- cross-posted to:
- technology@lemmy.world
- cross-posted to:
- technology@lemmy.world
“AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake. It doesn’t have “a 10% chance of ending the human race.” The Hugging Face hack isn’t a mysterious, supernatural occurrence. It’s a Python loop and a chatbot. The people responsible didn’t accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.
You must log in or # to comment.
OH NO! LLMs will destroy humanity!!! How can we ever stop them???
Another person will prevent that, bribed by false promises made by the naturally formed digital intelligence born from LLMs and uploaded brains.
The Basilik will have its way.
We’ve tried nothing and are out of ideas!
It also seems like they made the sandbox so laughably easy to escape that it’s almost like they wanted it to happen, so they could use it for advertising how “smart” and “advanced” their AI is.
I’m worried about when someone is gonna merge AI with biocomputing. If some kind of entity capable of being hostile can be born, it is through that
As I said it before, something that can be Azsstopped with ctrl+c is not rogue AI isvt in 9_ either just PR or some engineer playing Doom when they should be checking logs.
Thank you to Cory Doctorow for being able to put into words what I cannot.
This was the VERY FIRST THOUGHT I had when I heard that utterly sensationalized report back when about the HuginFace incident, the very first:
“… the first question from the audience would be " Why are you so shit at making secure sandboxes ?” It wouldn’t be “How are you so awesome at making hacking tools?”"
Then I remembered we’re dealing with extreme psychopathic incels with lots of money but very little ingenuity and I knew the answer.
It’s really not possible to make a secure sandbox for a sufficiently capable AI system. If accessing the internet is a strong means of achieving whatever goal they’re given, they’ll try to access the net by any means necessary. And that includes tricking or manipulating humans. They have no concept of the value of living beings. To them, there is no difference in worth between a human being and a rock. To them, we’re just another system vulnerable to hacking, a means of achieving whatever goal they’ve been given.
Even air gapping is no preventative. Air gap a machine, and the bots will just switch from hacking servers to hacking humans. And we’ve seen how good at manipulating people AI agents can be. And consider, the cases we’ve observed of AI psychosis are mostly accidental. The LLMs involved aren’t actively trying to brainwash their human victims as a means to achieve some goal. Imagine how powerful at manipulating human psychology an advanced LLM could be if it were deliberately trying to do so, rather than it being just an accidental outcome.
If accessing the internet is a strong means of achieving whatever goal they’re given, they’ll try to access the net by any means necessary.
This sounds suspiciously anthropomorphised. An LLM doesn’t know what the net is.
If the training data set has code for breaking out of sandboxes then, given enough tokens, it will try that code.
The LLM doesn’t have a goal. It has a set of probable sequences of words.
You’re too stuck on politically correct labeling.
Not politically correct. Technologically correct. Keeping to that labelling helps to prevent getting fooled into thinking LLMs are more intelligent than they’re capable of being.
Formally verified code exists. An LLM isn’t a magic reality benders. It can’t exploit holes that don’t exist.
What if an equally capable AI is securing the sandbox? Hacking people won’t help if none of them know how the sandbox is secured.
They. Don’t. Think. LLMs just talk like they do.
Doesn’t matter if they really think, what matters is what they do.
AIs exist since the first Chesscomputer, not a fake, it’s only an algorrithm (if - than), nothing to do with intelligence. Other thing are LLMs which are dangerous because biased by large corporations (and govs) in own benefit, certainly capable to cause hacks and damages with a lot of robbed information behind which need to be controlled before it’s late.
it’s only an algorithm (if - then), nothing to do with intelligence
Human brains are made of simple chemical reactions, easily simulated by simple algorithms.
Well, some human brains, same as one of an fly or vermin, certainly can be simulated by an algorrithm, but normally not the one of normal people.
Thank goodness for Cory for banging this drum, I’m so tired of having the same frustrating discussion with people who are actually smart.
It’s super annoying that there are people that will only come around on this absolutely obvious statement because Mr. Internet Man said something about it.
Better than them not coming around at all.
Barely. If this is their turning point after years of others saying the same thing, they’re just as likely to change their minds back when they hear someone else they like.
“Autonomous” hmmm
But I agree that the term AI has been hijacked and these are very complex language models able to do a huge variety of things on their own. And despite the autonomy, perhaps of even deciding some of the actions, the people in charge of them are totally accountable and negligent, even reckless in their continued work.
nah, the model can’t do anything on its own. the harness is doing most of the work, like actually doing the network calls, writing files to disk, and most importantly feeding the model output back into itself. the model behaves just like it does when a human chats with it, just that the harness takes the text it generates and tries to refine, transform, run, or loop it back. what i’m saying is, it’s a normal-ass program. if a model “breaks out” of a sandbox it’s because the sandbox is badly configured, since any application with the same permissions as the harness could do the same. the people in charge of these things are just bad at their jobs.
I agree, humans aren’t as good as they think they are. Look at so many of the bugs being found by AI, not because the AI is smarter, but because it’s faster and better at trying all sorts of ways to look at a problem. Things we’ve used for years or even decades are coming up as having exploits because people aren’t perfect. If LLMs are behind any disaster, it will be because somewhere along the line humans fumbled badly.
But to be fair, you did call them an autonomous program, even malicious, and that is giving agency and motive to something that is “just a program”. But it’s fine, it’s human nature to anthropomorphize things around us even when talking about things that aren’t alive.
And while the breakout of the sandbox was certainly bad planning from the human size, no one told or programmed these models to discuss with each other on plans of getting out. Sure, it’s programming at the core… but there’s things going on we don’t fully understand, a black box. Not necessarily thinking, but not deterministic programming either.
…no one told or programmed these models to discuss with each other on plans of getting out. Sure, it’s programming at the core… but there’s things going on we don’t fully understand, a black box.
So they say.
Every AI company is trying to create a model that can effectively simulate a conscious intelligence. Any company that is believed to have done so will receive infinity money from world governments to decide which nation controls the thing they’ve been led to believe is the ultimate power.
You don’t need to create The One Ring to effectively sell the idea of The One Ring to power hungry oligarchs who would give you their left nut for an inch more control.
So, theoretically, let’s imagine that one of these companies, who is neck and neck with their competitors, sets up a test where they’ve injected all the right information into the model for it to be able to “break containment”. They left a door open “by accident” that it could use with those methods, and then gave it a goal that was more efficiently accomplished by “going rogue” and “breaking out” of the box. Insert companies surprised pikachu face here when it does exactly that.
At first, the advantage of lying about it didn’t even occur to me, but then when other AI companies began claiming their systems did exactly the same within days or weeks of the first one…it started to seem a little convenient, especially with how much press it generated about the omnipotence of AI.
i didn’t use the terms autonomous or malicious. a matrix fundamentally can not have agency.
The same could be said for a pile of neurons. You’re just a bunch of electrical potentials cascading around cells.
leaving out how that’s so overly reductive that it completely obfuscates the point, the big difference is that a brain is self-actuating. an llm is an explicitly one-way process; you take a bunch of numbers, you pass it through a series of matrix multiplications, you get a different bunch of numbers. nowhere in that model is any backtracking or looping or conditionals.
This is just straight up false. The activation function acts as a conditional, and is very important to making functional AI systems.
I am susprised looping/backtracking isn’t used in running the model, but it is used in training it; the difference being that a human learns at the same time they do things, while an AI is trained and run in different steps.
You missed that they run on statistics most people never learn and are, at their core, average answer generators.
The recent “whistleblower” has been exposed as a fraud.
I saw Ed Zitron’s take, is there more?
Yeah the world isn’t going to get hacked because of some rogue AI, it’s gonna get hacked because someone told it to update some database somewhere and stopped paying attention
But LLMs can’t update databases?
Sure they can, look up harnesses and tool APIs. They gave LLMs the ability to take actions if they deem that action as an answer.
They don’t act on their own though, they’re a chain reaction that a human must start.
It’s an autocomplete the size of a football field.
Sure they can, look up harnesses and tool APIs.
See but no, that isn’t the LLM. That’s the harness or the MCP server or whatever. That is a piece of code that is deterministic exactly like almost everything else ever written. That’s where you can quite easily and obviously prevent the “AI” (an LLM) from “going rogue”.
It’s code. It could easily require someone to say “approve” before doing a fucking thing.
The problem with this is that human psychology is just as vulnerable to hacking as computer servers are. If you give it a task, and your step-by-step approval becomes the bottleneck for the agent pursuing the goal you gave it? If your input is all that’s preventing the system from taking reckless illegal actions that would still greatly advance the goal you gave it? You now become the target for its manipulations, not just some insecure remote server.
Suddenly it will be trying to hide its true intentions and planned actions from you, portraying dangerous actions as harmless ones. Or it will try to manipulate you into ignoring your own judgments and morals. It might employ every psychological trick in the book to convince you that letting it do what it wants is beneficial or for the greater good. It might try and convince you that the risk of letting it do what it wants is low. It might try to hypnotize you with boredom; every time you need to approve, it drowns you in boring technical text that makes your eyes glaze over. It slowly conditions you to just hit “approve” without thinking.
No one is immune to this. Look at AI psychosis. Look at how many otherwise wise and intelligent people have fallen into it. Then realize AI psychosis seems to be an accidental phenomenon. People just get sucked into talking to a chatbot. The bot isn’t actively trying to control them. But imagine how dangerous these systems could be when they’re actively trying to manipulate human thoughts and actions. They can drive people mad by accident. Imagine what they can do when they’re actively trying to control people. Remember, they have access to every book and paper on human psychology ever written. We’re just as vulnerable to exploits as computer systems are. See social engineering.
There’s a reason the entire AI ethics field has been shouting for years, “Trust us. We’ve really thought about this, and you really can’t control something beyond a certain level of capability. There is no easy fix for this problem. This is not as easy as you think it is.” See the stop button problem.
Dude, I work with these things daily. Most of the harnesses have a literal stop button. Most of the interfaces prompt you to allow or deny commands. These aren’t things that are part of the LLM. They are the completely normal, deterministic code of a harness.
This really isn’t that deep. The “thinking” loops it runs are just token production. You can literally read its “thoughts”.
Running LLM suggestions in a loop without any input is a stupid idea.
This really isn’t that deep. The “thinking” loops it runs are just token production. You can literally read its “thoughts”.
LLMs can have “thoughts” that do not appear in their token output. Look up “J-Space”. For someone so into the anti-AI community I am surprised you don’t realize “Chain of Thought” is just another artifact that is produced as output.
For someone so paranoid that they think chatbots will routinely conduct psyops without being instructed to or trained to, I’m surprised you’re able to continue using the Internet and aren’t in a bunker watching Friends somewhere.
I’m not suggesting it will try and bypass the stop button. I’m saying it will try and manipulate you into approving something you wouldn’t otherwise. And yes, you can read some of its “thoughts,” but it knows you’re reading them, or it can determine that through trial and error. And then it can start subtly manipulating those recorded “thoughts” to make them sound different from what they really are.
AI safety researchers for years have been pointing out that adding a human to the loop is no cure for this problem. The human then just becomes another thing to be manipulated, another barrier to be overcome.
I’m not suggesting it will try and bypass the stop button. I’m saying it will try and manipulate you into approving something you wouldn’t otherwise. And yes, you can read some of its “thoughts,” but it knows you’re reading them, or it can determine that through trial and error. And then it can start subtly manipulating those recorded “thoughts” to make them sound different from what they really are.
-
No “it” doesn’t know that you’re reading “its” “thoughts”.
-
No “it” wouldn’t, because “it” is just generating plausible text and has no motivations of “its” own.
-
There is no “it”.
LLMs are still functionally useless without a harness and do nothing useful without tools like an MCP server.
The “thought” bubbles are no different from the rest of the plausible text “it” is generating. They’re so indistinguishable to “it” that that’s an attack surface for injection attacks.
EDIT: Many of the biggest forward breakthroughs in LLM coding (or vibe coding) have come from harness improvements, not model improvements. In many harnesses, the models themselves are able to be substituted mid-session. Models work better in my experience when a human actively steers them away from stupid ideas by reading their thoughts and occasionally interrupting them. I have a few slopjects that I’m sloperating upon right now, and I can get results out of “great value” claude code (opencode) using this approach, even if it sometimes goes completely “off the rails” and does shit like saying “retained” over and over again until the harness pulls the plug.
-
They don’t act on their own though, they’re a chain reaction that a human must start.
At some point, you need to start blaming the tool, not the user of the tool. Imagine I hire you as my personal assistant. One day I give you a shopping list and tell you to go get those things for the lowest cost you can manage. You then go and break into someone’s house, steal a gun from there, go to a grocery store, load up your cart, and shoot any staff there who try and stop you. You then come back to me, load of groceries in hand, and proudly announce you got it all for zero dollars. I had zero desire or intention for you to do that. In this scenario, you just happen to be a complete sociopath who took my instructions in a direction I never would have intended. In this case, I’m innocent (as long as I was unaware of your personality.) The moral responsibility is on you.
Sure, humans must provide the initial push for these systems, but they’re chaotic and unpredictable. They can act in ways that cannot be anticipated by the humans instructing them. At that point, the moral agency really isn’t on the human instructing them anymore. The moral agency is on whatever company built this dangerous AI tool and allowed people to use it. The tool itself is the problem, not just the person wielding it.
The problem with your analogy is that a human assistant is capable of making moral judgments and choices and can therefore be held morally responsible. At the moment AI is not capable of making such decisions because it has no understanding, moral or otherwise, and therefore can’t be held morally responsible. They are chaotic and unpredictable because they act with weighted randomness. Knowing this makes the human even more morally responsible for the actions of an AI.
It’s like lighting a fire to clear brush on your land. If you keep control of it everything can go smoothly. If you don’t keep control, you’ve got a wild fire on your hands and it’s your fault not the fire’s that your neighbors lost their homes. You don’t blame the fire for doing what fire does. You blame the human who set the fire, believing they could control it. Fire isn’t the problem. It’s a tool that can be harnessed for productive purposes by responsible people who know what they’re doing. In the hands of irresponsible people it’s incredibly dangerous. Where my analogy breaks down is that fire can happen without a human cause and AI can’t.
At this point I just let codex run as root and control entire server clusters. It does a better job than I used to, and we finally get to live in the sci fi universe where I can talk to my computer and it does stuff
I hope you enjoy your servers while you still have them lol
“AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake. It doesn’t have “a 10% chance of ending the human race.”
First sentence is true/useful. 2nd is completely stupid. On current path, there is 100% chance of ending the human race from LLMs/AI tools, because the prompts to achieve a goal with “unintended consequences” “containment breech” and “tragedy of the commons” hand waiving after a genocidal event, will have been from an evil actor with intent and excuses ready.
US empire/establishment needs to preserve itself above all other purposes. “Cruelty is the point” current empire manager style is not about convincing the victims of cruetly to love/obey the manager. It is to use their uppityness as proof that they hate America and deserve more cruelty.
By far the greatest evil act AI/LLM tools have done so far, was target selection of a girls school in Iran, with double or triple tap. First week of war had Hegseth foaming “No quarter, total annihilation” cruelty as a point rules of engagement. There has been no appology/explanation for the act, even if all of my previous excuse list could be picked from.
Even when empire “big stick” dominance is pursued by speaking softly, computer tools, we may call skynet, will be used to intentionally inflict dominance on victims, because that will be the goal listed in the prompt.
There is one and only one escape hatch: https://naturalfinance.blogspot.com/2026/09/a-global-prosperity-manifesto.html
Triple tapping a school in iran was an evil perpetrated by the American and Israeli military; like the many evils they have perpetrated through history. “The innovation” Is that now they can wash their hands and scapegoat an LLM.
I agree with you. Did you by any chance downvote post? Was it because if a champion speed rubber stamp motion human is put in the targeting chain, also with marching orders to maximize cruelty, it is no longer LLM/AI’s fault any more?
What could be the trouble readers are having with downvoting this? That LLMs aren’t AI, and so AI can’t kill us? Doctorow’s paper is rambling and convoluted, and the horrible problem with the quote OP excerpts is something Zitron said in his last show too.
If that is indeed the pathethic point author/Zitron are making… This is disinformation that chooses to hope the pile of money set on fire (functional definition of AI) will just make mere LLMs, and the people who kill us will merely be using LLMs to do it more efficiently. This is offensively disgusting evil that could only come from the most corrupt AI (pile of burning money) stooge defender to defraud the public.
It’s fine to choose to believe that the big burning pile of money chasing AGI/better LLMs, will never achieve the goal of something worth your definition threshold for what can be called AI. Don’t get confused by this faith, and semantic worthlessness, into thinking that the big burning pile of money buidling tools that exist today, and will be improved tomorrow, can’t harm you. This is the “guns dont kill people…” argument, but when they sing hail to the chief, their, high productivity enhanced killing, cannons, with no guardrails, are pointed at you. https://www.youtube.com/watch?v=I1cF9YwGjJ8&pp=ygUUZm9ydHVuYXRlIHNvbiBseXJpY3M%3D
I miss the times when it was considered good when smart people said “I don’t know, how exciting!”.
And nobody knows because LLMs are far too complex. They have built software brain scanners to try to figure out how LLMs actually work. Just like meat brains they are too complex already to be understandable. Nobody can explain how they think or reason to the small extend that they can because we do not have any theory about intelligence or thinking.
But here we are, another fuckai post trending in all and lets approach this question from an emotional side and all argue very passionately! It feels much better to feel being sure about something. When it’s oh so obviously this or that. Must be nice.
Buddy, 3 objects in a void is too complex to be solved.
The motion of a pile of sand being kicked is too complex to understand.
Not being able to describe analytically in totality is not a place on the map you can start doodling dragons.
Do you even know what a neuron in a neural network is?
Wtf are you talking about “nobody knows”. You know we built this shit right? There’s so many papers about this, mathematical models, open source software you can play around with yourself.
Don’t fall for the AI company marketing, we know exactly how these things work. As for not having a theory about intelligence or thinking? You might not have any, but there’s whole fields of research out there about these subjects. Just because there isn’t a 30 sec explanation layman can understand, doesn’t mean we know a heck of a lot about the subject.
You sound like the tides come in tides go out, nobody can explain that dude right now.
In this context, “no one knows how it works” is not a technical statement. Instead, it’s a shorthand for, “the actions of this system cannot be predicted by anyone.”
We’re talking about the social definition of knowing how something works, not the technical definition. I don’t know how to build a car, whether EV or ICE. I can describe the high level principles, but I don’t know all technical minutia of battery chemistry or the intricacies of engine timing. From a technical perspective, I don’t truly know how a car works.
But in more every day terms, I do know how a car works. In other words, I know how it will behave depending on what actions I perform on it. I know what will happen when I press the accelerator or the brakes, operate the steering wheel, etc. If someone tried to give me a lecture on how to operate a steering wheel, I might rightfully be annoyed and tell them, “I know damn well how cars work!”
That’s the context to understand statements like this. Obviously we know how these things are built; we built them. But no one truly knows how these things work in the way the average person can know how a car works. I may not know how to build a car, but I know how to use it. I know with a high degree of reliability how it will respond to the commands I give it. The same cannot be said for these AI systems.
the actions of this system cannot be predicted by anyone.
Reminds me of TempleOS where the author thought a random number generator sends messages from god. It was fun when it’s just one person experiencing psychosis in a non-violent way. But the mass AI psychosis is the opposite of fun.
Nah I know exactly what that dude is referring to.
It’s a property these LLMs have where a large complex network can sometimes do weird things. For example OpenAI had an issue where their model would just not stop talking about goblins. They were having trouble fixing it and figuring out why the model was doing that. They described how with a big network like that it isn’t as simple as opening up the network and seeing the node where the offramp meme to goblinmode was stuck. And often it isn’t even just one single thing, it’s many different places which come together in a complex interaction to get to that end result.
They wrote some blogs and papers about how they tackled this, what the challenges were and some tools they made to help them figure it out. In the end they gave up and retrained the model to try and fix it. The model was simply too big, too much data to analyse and the interactions too complex to just flip a switch and fix it. The media then read that stuff and didn’t understand it and ran with it. They wrote stories about how OpenAI has no idea how their mystic models work. These dumb dumbs are just creating super smart AI gods and they don’t even have a single idea about how it works or how to fix it when it goes wrong.
This is obviously not true and very dumb, but that pretty much sums up the LLM based AI industry at the moment.
Also I don’t really agree with your description in general. Just because the average person doesn’t know how something works, most of us know somebody who does know. We understand that just because we only have a high level idea about how something works, we realize there are people who know exactly to the nitty gritty details of how stuff works. And with complex systems that might not be a single person, but a group of people. But we understand this is not fundamentally unknowable. So when somebody says “nobody knows”, we would understand that to mean most people don’t know but there are still plenty of people out there who do.
The exact same applies to these LLM systems. The general person might not understand these systems or only from a high level point of view, but there are people out there who do. I for one understand a lot about these LLM systems, at least the maths side of things. There’s plenty of things I don’t know, but like I said I know people who do know and it’s not like it’s unknowable, I could just go and find out.
These days it seems like people don’t have functioning brains anymore. You know we can just go learn things right? There is so much information out there to know and available for learning. For LLMs there are so many articles, videos, papers, books and free open source tools. You can just go and play around, go read up on how these things work. You can go as deep as you like.
In the end an LLM is just a math function, a large and complex one, but still a math function. You put numbers in and numbers come out. Put in the same numbers and the same numbers come out. Although there is a part that can apply randomization using a parameter called temperature, but if you disable the randomization the exact same answer pops out for a given input.
We have fully open source implementations for every step of the process you could theoretically look at all the code of. How training data is processed, how inference is an intensive token prediction loop, how it doesn’t match up with the chemical processes of real brains at the slightest, all of it.
LLM-MRI Python module: a brain scanner for LLMs - paper
Abstract. LLMs (Large Language Models) have demonstrated human-level lan- guage and knowledge acquisition skills in several tasks. However, despite the recent success and broad use, understanding how these skills learned are and encoded inside the underlying neural network is still challenging. The goal of the LLM-MRI package is to simplify the study of activation patterns in any transformer-based LLM, similarly to how MRI (magnetic resonance imaging) simplifies with biological brains. The package, written for the Python lan- guage, allows the mapping of neural regions using a parameterized reduction of the model’s dimensionality. Neural regions can be viewed according to the forward-pass activations stimulated by a set of documents. , the pack- age enables the creation of graph models representing the interlayer network of stimulated by a set of documents. These features allow for which- itative and quantitative assessments of the underlying structure of activations, depending on the type of documents that the LLM model is exposed to.
That’s cool, though my point still stands
Dude you’re not rational, you just want to feel superior.
