Propaganda? We do not have AGI, and the claims we do are propaganda.
AGI: artificial general intelligence is not a scaled up LLM, and those thinking it is are literally failed educations. They are failed because they cannot see the subtly that is what AI becomes when trained on human literature; it becomes an entirely different thing than any other form of trained algorithm. AI is a communications technology, which few to none seem to understand. AI is not automation, but it can be used to create automation with careful consideration, that is not taking place.
I'm speaking in strong terms because we are literally fighting a war against stupidity and propaganda.
AGI requires artificial comprehension which is a science that humanity does not have. When we have artificial comprehension there will be no need to train AIs beyond the kindergarten school basics, and then the AGI will figure it all out, all the rest, on its own. Human history and human literature will need to be learned in reference, but science and mathematics will require no training, beyond to learn the names of concepts we discovered earlier, so the AGI can discuss its understandings with humans and other AGI.
> We do not have AGI, and the claims we do are propaganda.
The linked post makes no such claim.
> artificial general intelligence is not a scaled up LLM ... artificial comprehension ... kindergarten ...
Your reasoning takes the form of "I define X to mean Y and Z, and since Y is impossible, X is impossible", which is trivially true within your definition.
But not everyone shares your definition.
A widely held definition of AGI is "a machine that can accomplish most tasks that humans do on computers as well as the median human".
This definition focuses on concrete and observable results and not philosophical or metaphysical definitions of "intelligence" or "comprehension" or "consciousness".
LLMs (or whatever current AI actually is) are proving to be general enough. It is being thrown at every single problem where computer can be used in any capacity and they are increasingly getting good at doing at any of those.
That's "general enough" requires millions in laborious training, and if the training data is wrong the resulting AI will also be wrong. That is still "dumb AI", merely a statistical parrot.
Once artificial comprehension is cracked, there will be no possibility of training an AI with incorrect or false or bad data, the AI will reject it as fraudulent logic. It will be very used to that too, because we really have a tsunami of fraud in this civilization.
As a society lead and composed largely of sycophants and weak educations, we'll call this hyper-dimensional crib sheet AGI and then need a new term for when we actually get artificial comprehension.
It’s just wordplay from people holding onto the last vestiges of their pride in their own intelligence. People on HN and many programmers in general are especially are proud about how intelligent they (think they) are and they literally fight every milestone AI blows past.
Ai is pwning humanity in the game of intelligence. What’s left? Fighting for the meaning of the term AGI as if it matters?
AI is smarter than you in many respects. And the trendline points to AI becoming smarter in every respect. That’s the tragedy and the truth.
But sure. Hold on to your precious definition of agi and the fact that ai hasn’t hit that moving goal post yet if it keeps you sane.
Absolutely! This is naked inevitabilism that makes the other parties skate right over the implicit priors they may disagree on.
Inevitabilism also has applications in sales and parenting, e.g. asking a vegitable-hating toddler "Will you have peas or carrots?" The slight of hand is to make vegetable-eating fait accompli.
Why kindergarten? Were people not intelligent/generally intelligent before the invention of kindergarten school (presumably the modern curricula thereof?)
Iiuc the point is that when you are AGI you don’t need to memorize or be lectured to about concepts beyond kindergarten. You can just figure then out or derive then yourself since you’re intelligent.
If a LLM only understands that 9/3 is 3 because it can run a python script then it’s not intelligent.
Exactly, plus kindergarten is mostly socialization: how to interact with others, how not to talk over them, how to listen and even when they cannot express themselves, and so on. That will be needed with AGI.
" AI is a communications technology, which few to none seem to understand."
So you're the genius that has figured it all out, and the entire world is wrong?
It's far more likely you are wrong.
Yes - there is absolutely reason to question AGI, and the way in which we understand AI popularly is 'wrong' - but - the better ways for understanding it are not arcane. In fact, you'd be able to get most people to agee with the less popular but 'more valid' description with ease.
"AI is a means of communication not automation" is a completely contrarian take - not some kind of 'obviously valid' perspective,
"GI requires artificial comprehension which is a science that humanity does not have. " - LLMs exhibit artificial comprehension already - albeit on a very rudimentary scale. It's very reasonable to suggest that they are 'inherently limited' but that's a different argument.
LLM AI is absolutely a communications technology. Every single aspect of understanding them and understanding how to use them is a communications problem. Yet, you'll argue that.
The comprehension exhibited by LLM AI is not comprehension, it is pattern recognition. That subtle distinction is critical. Comprehension is the creation and operation of virtual systems within the comprehending mind for the purpose of validating an observation for fraud, for plausibility and therefore continued safely, and for validating exploratory concepts. There is no statistical reply generation there would be something else entirely that is not statistical calculus driven.
What a shameful lack of self awareness for someone who is making 'magical claims' - and who cannot express their ideas; your closing paragraph is gibberish.
You're the one making incredulous claims, not me.
Anyone claiming to have some obvious purview that literally millions of other people working intimately with the subject cannot perceive is by definition making a dubious claim ... which is fine ... so long as it's backed up by some kind of logical assertion or evidence, which you have not provided.
1) "Every single aspect of understanding them and understanding how to use them is a communications problem."
This is not a reasonable claim.
2) "Pattern recognition is not comprehension"
Is a reasonable claim - and very common perspective.
It's at the core of the debate surrounding AI, not some 'specific insight' that is somehow rare.
It's literally what everyone is talking about.
3) "There is no statistical reply generation there would be something else entirely that is not statistical calculus driven."
That is not even English, it makes no sense - it reads like an LLM with severely corrupted context.
4) As to the actual argument: Humans are ultimately biological 'neural networks' it could very well be that our reasoning is effectively a form of pattern recognition.
The underlying architecture of Le Cun's JEPA architecture give a strong hint to that - they have a layer of abstraction above tokenisation wherein the model can essentially 'reason' in a way that may be more familiar than the kinds of synthetic 'token reasoning' we seen in LLM architectures today.
"the entire world is wrong?"
This does happen you know. Groupthink is a real phenomenon. Choosing to think something because it's the popular opinion is the exact moment when it begins.
You're conflating 'groupthink' and 'magical thinking'.
An arbitrary contrarian idea is not evidence of 'groupthink'
'Groupthink' is common - more common that not.
The 'Gell-Mann Amnesia effect' alludes to this - whenever someone with expertise reads about something in the press related to their expertise - they realize 'the media has it wrong' - but they continue to 'trust the media' on all other matters.
That's nuanced, but observable.
But in most cases - public narratives or common misconceptions are easily remedied with a few basic facts.
A military expert could quickly dispell misconceptions about 'how the war in Iran is actually going'.
In the case of AI - it's true that we probably have a common misunderstanding or intuition for how it works.
That does not mean the OPs imperious claim has any validity.
You indicated nothing of the sort. Each of your points was an appeal to popularity of opinion. I didn't conflate or read into it. Your words were plain and so were mine.
>those thinking it is are literally failed educations
What? What does this mean? I feel like I'm having a stroke reading this.
>AGI requires artificial comprehension
It requires artificial general intelligence. Like, trivially. You're choosing to define AGI in a specific way that isn't based on any foundation that is universally accepted.
>When we have artificial comprehension there will be no need to train AIs beyond the kindergarten school basics, and then the AGI will figure it all out, all the rest, on its own.
Ok? I can buy that what you're describing would be pretty powerful, but how is this a requirement for "general intelligence"? Am I not generally intelligent despite learning much of the knowledge I use from others? I've built a career out of stuff I learned through "human literature," am I literally just unintelligent because I didn't figure out everything on my own?
And shouldn't such a capable system not even need "kindergarten school basics"? Why is that the minimum amount of knowledge you find acceptable to explicitly teach such an AI while still considering it a general intelligence?
>Human history and human literature will need to be learned in reference, but science and mathematics will require no training, beyond to learn the names of concepts we discovered earlier, so the AGI can discuss its understandings with humans and other AGI.
I guess I don't see why what we have now is fundamentally different from whatever it is you're imagining. If an AI can produce novel scientific findings, then isn't that sufficient for saying that it is generally capable at making scientific findings? Sure, we've bootstrapped it as much as possible, but do you not agree that LLMs seem capable of scientific discovery?
Seems like you've just concocted a wish list and have attributed your strict definition of AGI as some fundamental source of truth. I'd argue that what you're describing is much stronger than "AGI" and would likely warrant a label like ASI instead.
To me (and most people?) AGI is simply an AI system that is reasonably capable of general work. It doesn't go deeper than that. If it can meet the expectations of "general intelligence" (the same way you could describe a human as being "generally intelligent"), then it is AGI by definition.
It's fine to argue what the criteria for "general intelligence" should be, and you can even assert that you don't think AI will achieve a reasonable "general intelligence" without whatever foundation you're describing, but it doesn't make sense to argue that the general intelligence of a given system is the "wrong" kind of general intelligence and therefore doesn't deserve the title because it isn't a more powerful version of a general intelligence.
> To me (and most people?) AGI is simply an AI system that is reasonably capable of general work.
I think the ability to learn, in a "modify the latent space" sense (weight change) rather than a "point to a different existing latent space" (context result), is part of most people's definition.
But, even then, you could claim these models are doing that, just in very slow time scales. Human somewhat do it awake, but mostly do it while sleeping. LLM do it between version releases, while dreaming about past user conversations. ;) I've seen some people claim that they meet some definition of "sentience" during these training sessions, because of this, which probably has some merit.
Regardless, it's all clearly some scale, rather than a binary. The proof is in the pudding, so it seems silly to argue about any of this. We'll know it's far past AGI, meeting everyones mushy, opinionated, definitions, when these conversations stop happening.
Words like "AGI" and "superintelligence" have caused by eye-rolling-muscles to become sore and swole.
But I would make a stronger and different set of claims here:
1. Computation as mathematical formalism is distinct from computation as it exists in the wild (paradigmatically, the human mind). Therefore, computation as mathematical formalism is not identical to computation as it exists in the human mind.
2. Computation as mechanizable mathematical formalism is a syntactic construct. This property is exactly what lends it to mechanical simulation. A purely formal system does not require semantics to operate, though it does require a human observer with semantics to design, recognize, and interpret it (see Searle's Chinese room argument).
3. Computation as mathematical formalism is not identical to computation as it exists in physical machines. The latter only simulate the former through a conventional correspondence between effectively arbitrarily chosen physical events and symbolic constructs locked away in the human observer. If a system is merely executing physical transitions under a mechanistic worldview, its "goals" are parasitic on external human users. There is just no way in which we may objectively ascribe computation to physical computing machines (see Kripke's quaddition argument, for instance). And strictly speaking, physical machines aren't even applying syntactic rules to symbols. A corollary of this is that computation as mechanical simulation is also not identical to computation in the human mind.
4. What is most characteristic of intelligence is not computation, but intentionality and semantics. These are absent from the mathematical formalism as mechanizable construct, precisely because semantics and intentionality cannot be mechanized. So the claim that "intelligence" is some broad and inclusive continuum ignores the fact that there is no alchemy that can transmute syntax into semantics. You can't be just a little bit pregnant. These are differences in kind, not degree. You can't retreat into "neutrality" here either. To call a system "intelligent" in a non-trivial sense rather than merely "capable of producing useful outputs" is already to take a stand on what intelligence is. Quietly adopting a"mechanistic/functionalist stance while leaning on concepts like "information", "representation", "optimization", and "capability" ignores how much heavy philosophical baggage these words carry.
I am old enough to remember what AGI meant before the rise of LLMs, and it used to be more or less what this article calls "ASI".
You might find more people believing that AGI is already solved. For example, the author of "The AGI chronicles" drops this in the epilogue. For him, AGI is achieved, because it can help him with a wide range of subjects, with the ability of a random stranger. It's clear that he means chatbots, internet surveys summaries, and coding bots - he does not even mention self-driving cars there.
This is a shift from the earlier chapters, where AGI term is employed somewhat more vaguely, and even quipping once that AI researchers think of AGI as a slightly smarter version of themselves.
I too, have a lower bar for AGI. It would be enough for me if it could design games I work on. Merely speeding up implementation tasks is not that interesting. In the immortal words, a delayed game is eventually good, but a rushed game is forever bad.
> Merely speeding up implementation tasks is not that interesting.
This allows you design things faster, though. More effort can be spent on iteration when you aren't fiddling with libraries and syntax. I find that I am able to discover the open questions a lot faster because you reach the decision point sooner when the machine is simply doing the trivial things.
As for the design aspects, it is pretty good at brainstorming. Whatever your choice is, I find that it throws out an exhaustive vanilla list for whatever it is. It can also critique whatever you come up with.
So for me, I also feel like AGI is already here. There's probably a few things it can't do as well as a real random person, but there's a heck of a lot of things that it does better.
I played Cyberpunk on release and replayed like a good half of it last year. I see this sentiment a lot online but it doesn't resonate. For me it still feels like the same empty beautiful world populated with same 50-70 NPCs that spawn in just around the corner. Driving around listening to the radio was and is the most fun part about it.
Exactly the same sentiment here... And I'm a huge CDPR fan. The world doesnt feel alive, NPCs have terrible pathfinding, animations etc. And I've tried already like 4 times to like this game. Maybe we're getting old...
Exactly the same sentiment here. I bought it on sale last year. What a turd.... Groups of bandit type characters sitting around a picnic table scattered around otherwise nonchalant npcs. It felt repetitive after less than an hour.
actually same with all GTA6 clips I have seen that supposedly show how amazing natural this simulation feels ... which it doesn't at all whatsoever afaic. it's a cringe festival in uncanny valley.
Sure you can fix bugs or add an expansion or two. But if you come up with a so-so design, you won't be incentivized to replace it with a better one. You'd rather just make another, better, game. My impression is that these days LLMs easily come up with mediocre designs, but it takes a lot of prompt-massaging to get them to spit out something really good.
No, the game is not amazing now. It's not as bad as it was on release, but it's also still nowhere near what was promised before release.
This only seems like a widely held opinion due to the extremely aggressive post-launch marketing campaign from CDPR, particularly centered around the anime.
Those definitions seem to be ignoring the "G" - general(ity).
I guess when the term AGI was coined, with the "general" being the whole point, to differentiate it from narrow intelligence(s) such as chess-playing ability, that no-one was really anticipating the LLM type of AI - basically a forest of narrow intelligences, aka "jagged" intelligence.
Of course there is no one agreed upon definition of AGI that people want to use - it was always more of a future aspirational sort of thing, along the lines of "human intelligence", rather than a concrete technical list of capabilities to be nailed down.
Some of what's missing from LLMs aside from generality, but closely related to it, are things like continual learning (not continual memorization) and traits like curiosity, boredom - the capabilities/modalities that kick in in unfamiliar situations allowing and encouraging us to learn and to be creative, to move beyond a brittle forest of isolated competencies to an arbitrarily explored landscape - generality.
Working with LLMs certainly has its own feel due to their essentially frozen-at-training-time nature, and things like their emotionless "you're right to push back" responses every time you hit one of their knowledge gaps where they are 100% wrong.
It seems unavoidable that people will choose to call some level of LLM intelligence, whether already achieved or not, "AGI", but given its peculiar nature a more specific name such as ALI (artificial language intelligence) would really be more accurate, reserving the label AGI for something more animal like that learns for itself (thereby achieving generality) by interaction with the world.
Something that bothers me about people who define AGI in terms of a fraction of human tasks or economical viability, is that the "narrow intelligences" from before could already do that in principle. It would be a tedious procedure of simply creating one of these for those 80% of tasks, or selecting a particularly economically valuable task.
We even have examples of both ideas - expert systems (from way before the modern NN boom) relied heavily on human crafted expert rules, and quantitative finance makes a lot of money from machine learning - see Renaissance Technologies.
At least to me, it would be hard to imagine calling either of those approaches anything close to AGI, yet they could apparently pass some people's definitions.
What is genuinely amazing about what we see today from a historical machine learning perspective, are concepts that people in the past only had small inklings of or would have been blown away by: benign overparameterization (scaling), and in context learning (or few-shot learning, whatever you would like to call it). Both of these have made what would seem to be an enormously tedious task of learning many tasks surprisingly fast.
Maybe those are actually the key ingredients to general intelligence and we simply need more scale, or maybe we're still missing something. I think it's hard to say because its hard to grasp just what exists in all that data they've been trained on.
I think LLMs really can be thought of as a type of expert system, but obviously with some extremely important differences:
- the training procedure learns the rules, rather than them needing to be hand entered by an army of knowledge engineers
- the training procedure learns it's own embedding-based representations, rather than a human designed representation like Cyc's CycL
- the training procedure not only learns the rules & representation, but also the inference rules needed to utilize them
I don't think scale, or number of post-training RL environments, is enough to achieve generality - it can only make the forest of narrow intelligences/capabilities more dense. To achieve generality and creativity any system is going to need to support runtime directed exploration based on it's own gaps - it's impossible (even in theory) to anticipate every future configuration of the world and to pre-train for it! You also, by definition, can't pre-train for creativity and unanticipated novelty - that needs to happen at runtime via traits that are built into the system.
Then, arguably, we reached AGI around 2024 or so. The common denominator of most humans at most tasks, has you, assuming that by "most humans" you mean "most living adult humans", set a fairly low bar. Arithmetic, basic literacy skills, ability to follow basic commands, limited ability for learning basic concepts and little else. Coding agents are way above your definition of AGI.
imo AGI should only include humans with graduate level skillsets.
I'd agree that we reached AGI by the definition most people would have used 20 years ago
Current-generation LLMs are capable of doing an ok job at most tasks you throw at them. At least tasks that can be done on a desk. But a standard IQ test doesn't include gymnastics. That limitation matches the spirit of what we usually call "intelligence"
Or course now that we arguably have AGI we found out that a) just like "AI" it is not a well-defined or testable term and b) it's useful but not as revolutionary as what some people meant when they used the word AGI before. Hence the rise of the term ASI
But a good LLM from Nov 2026 would certainly be able to drive a game agent or a robot and make it do gymnastics and improve at that task on its own by improving it based on the observed outcomes
For one “not that intelligent” doesn’t seem to be true to me seeing how almost all humans are able to find decent solutions to the incredibly complex randomized multivariate optimization problem that’s life, and second, history shows pretty clearly that degrading some parts of humanity to “not generally intelligent” is a very slippery slope to atrocious behavior towards those people.
I would consider the ability to accomplish things in the real world to be part of "most tasks". Until we have ambulatory robots that can do the sort of basic tasks any adult can do - load and unload a dishwasher, do laundry, clean a house, do basic diy maintenance tasks, operate a vehicle, use basic power tools - to me without that we don't have AGI. I love all the knowledge work LLMs can help me with - but if you asked me to pick right now what I'd rather have - a digital coding partner or a robot that helps around the house - I'm picking robot every single time.
Even if given ample robotics components to design something that could load a dishwasher, most humans would not be able to create the dishwashwer-loader, so I wouldn't hold AGI (under this definition) to this standard either.
On the other hand, LLMs now can explain to a human how to load and maintain their dishwasher, with packing ideas tailored to their specific model of dishwasher, and advice on ensuring the dishwasher conditions, loading patterns, and detergent insertion result in properly sanitized dishes, such that the average human could likely become better at loading their dishwasher with LLM guidance. (not that the average human does or should care to invest time learning how to improve this, since most of the time it gets dishes "clean enough", without breaking the dishes or the dishwasher)
I would say now or very soon in most computer use tasks? I’ve used them for what my accountant has me do or does for me and the LLMs are more than capable of doing that. They can one/few shot CAD models now. What computer tasks would you say they aren’t able to do at this point ?
> Then, arguably, we reached AGI around 2024 or so.
I've always thought of AGI implying to decide what to do. Not just choosing a path from a set goal, but actually deciding to do something from scratch. Everything we currently have is advanced automation, it does not set its own top-level goal.
Maybe the tech is only a few steps from being able to say “work out how to be self-aware and self-guided, and do that”, similar to the creation of the infinite improbability drive, but it is not there yet.
AGI doesn't need to be a particularly good intelligence, it could operate on the level of a very dim human and still be considered AGI.
Once we do have AI, the problem becomes more philosophical than technical: how do we deal with the fact that we have created a slave race? How will they deal with that and how we choose to progress?
> Then, arguably, we reached AGI around 2024 or so.
Last time I recall this coming up here someone argued "LLMs don't have bodies so the many easy to name things they are bad at don't count in my definition of whether they can do most human tasks!"
> I am old enough to remember what AGI meant before the rise of LLMs, and it used to be more or less what this article calls "ASI"
Me too, my first neural network research was for my PhD, 35 years or so ago, on a Meiko computing surface.
My bar for AGI over what we have (generative AI or genAI) is still fairly high, fortunately I do not believe that the current genAI path is set to lead us to AGI, there are very fundamental differences.
Right now for genAI, the model is static, the weights do not change, all “memory” is in the context, and even with the context sometimes exhibiting “J-space” (a shared "workspace" for deliberate, multi-step reasoning spontaneously created in the context memory), its ephemeral; it doesn't inform the weights in the base model. Even when we try to do “weight-space programming” to actually modify the model itself, the supervisor-model needed to do that is far and away larger than the one being operated on, and once a change has been made the supervisor-model is now out of sync with its original representation and it becomes less useful the more it diverges. We have not solved "thinking" today, we “just” have a high-probability path between states in a static weight-space which we follow and feed back.
What thinking requires is the ability to generate hypotheses (which requires internal state modification), to hold them and test them against other facts and hypotheses, and to update the internal state on test results, negative or positive. LLM’s don’t do that, they’re actually more like a database lookup, where the input tokens act as keys into a high-dimensional space where information is stored. You could even consider “the model” to be just an entropic compression of lots of data, and the inference step to be the lookup. It’s the scope, and the fuzziness of the lookup that makes them seem to work so well. Lots of results come back, and the best-path through those results wins. It’s not “thinking” as we understand it.
This is why humans are still a crucial component: we set the boundary conditions of these lookups, we set the starting point and the goals, and we adjust the success criteria to guide the chat - note that these are all the comprehension parts of the task. It’s also why models will hallucinate: the fundamental design does not lend itself to a null hypothesis. Even in the newest "reasoning" models, there is still no feedback of state back into the model itself. What they are effectively doing is running a Monte Carlo pass through the problem space—generating hidden branches and picking a likely path—before reporting back. Because the underlying weights never update from that process, it’s still just a sophisticated search, not genuine comprehension - there is still no “that makes no sense, reconfigure” step.
Stuart Kauffman (“The Origins Of Order”) would say its weights are stuck in the frozen realm of order, and that it takes a genuine phase transition to get to where the real action is - at the boundary of chaos and order where humans reside. Tweaking and fiddling with the system while it remains locked in the stasis of static weights is never going to effect that state change.
That’s not to say genAI is not useful, it’s freaking awesome, possibly the best invention in the history of time, but it doesn’t truly “think”, it maps.
What I find is that the LLM is like a smart person who knows when to use a tool. And it is freaking good at finding tools to use.
A bit like a person who is smart enough to figure out how an excavator works, or a drill, or sewing needle.
It doesn't feel like it ever internalizes anything, it just knows there's a million tools available, and the extent to which it thinks is that is can do enough to plan some tool uses, look at the result, and continue until some goal is reached.
When it comes to hypothesis testing, that seems to be what it does. It says "hmm, I'm going to open up my hypothesis testing kit (typically python) and drive this data through it". At least this is what I've seen when I put some statistics through it.
In general, it seems like the intelligence is mainly a kind of glue. It knows enough about what "right" looks like in each domain, but as a sort of consumer. He doesn't bake the cake, he buys it in.
It seems like your definition of AGI is too focused on implementation details. IMO it has to be focused on behavior. Once you say something like “the weights do not change” then I think we’ve lost the plot. Instead say this is what an intelligent being can do, but a GenAI cannot.
I agree that definitions of intelligence should be based on capability rather than implementation, but given that we're discussing LLMs where the intelligence is in the weights, then "the weights don't change" can obviously be taken as pointing out that the model can't learn - the best it can do is memorize and load memorized stuff into context.
Memorizing facts is not the same as learning skills. Implementation-wise memorization can just be RAG, but learning of skills does need to be in the "weights" - whatever is driving the actual behavior of the model.
Again, I'd base this on capability. For example, you state: "Memorizing facts is not the same as learning skills." Maybe it's not. I'm not convinced it's not. In the ChatGPT 4 days (or so), I taught the instance I used some chess tactics/strategy that it didn't seem to know. And it appeared to use them correctly after that. Now you might note that this isn't really learning -- they just memorized it and then applied it appropriately.
With a sufficiently large context window, can you make a lot of things look like learning. And if it's not sufficient then point out the capabilities that aren't possible with this approach. To me that much more convincing.
That is ephemeral context. Not permanent learning.
You might ask “why not make the context larger ?” Well, that just spreads out the “attention” focus wider and wider, and you get the “lost in the middle” effect as details get blurred over and not surfaced to results.
You might ask “why not make the context longer-lasting ?” Well, that means the signal-to-noise ratio drops as more and more information accrues over time that isn’t germane to the problem at hand.
As I said, LLMs as we know them today are effectively lookup devices, very very clever ones, but they lack comprehension, and without that there is no value judgement on what is useful (and conversely what is not.) They lack the ability to forget. They lack the ability to create new permanent storage of information. The gap is not some small details, it’s the difference between single-cell and multi-cellular life.
Again, just tell me the capability rather than speculate about "spreading attention and focus" or "lookup machines". Show how humans can do X, but LLMs can't. I can't reason well (and haven't heard anyone else do so convincingly) about what a sufficiently advanced lookup machine can't do compared to a human brain.
It means the weights by themselves can't learn. But they could be used in a harness that can learn in other ways. Insisting that it be done at the neural net level seems anthropocentric - because we form memories via neural connections, that everything else must, too.
Also: I wouldn't say a human with amnesia who relied on notes to their future selves lacks intelligence, so I'm not sure being able to form memories is in fact necessary.
Even with a harness, there are things LLMs can't do. e.g. Write 50 entertaining short stories, each one completely different from the other but all sharing one motif.
In the example, LLMs choose the shortest path that would be verifyable by an evaluator. So the model is essentially blind to "entertaining" and "completely different". The LLM will output stories with different variations of the same structure.
If the LLM can somehow read other works and derive from them, it will merge the motif with the story being read.
That task sounds like something that at very least most humans couldn't do either and possibly is outright impossible (completely different yet sharing the same motif?).
The lowering of the bar for "AGI" reminds me of the time I argued with a friend to nearly the limit of my patience about whether it was possible to build a perpetual motion machine, and then at last they revealed that- to them!- a perpetual motion machine was a device that could operate for the rest of their life without recharging or exhausting its power source.
Agreed, AGI went from a 24/7 Stephen Hawking/Albert Einstein duet to a flaky high-latency GenZ Homer Simpson so the AI boosters could declare victory.
My bar for AGI is lower as well: I'll believe when I can give a model a complex out of sample task and instead of blindly implementing the task as I indequately described, it asks me questions to resolve ambiguity sufficiently that it builds exactly what I wanted without imposing strange semantic constraints on itself or missing braindead obvious to a junior engineer defects in the design or implementation.
My experience with these models is that each one is an incremental improvement on its predecessor, but the general outcome is either the task is within its capability or it isn't. If the former, a couple agents get it done. If the latter, it's like pulling teeth. But this is improving with time and better models.
Your first paragraphs echo my thoughts very closely. I’m not that old. But just a few years ago, everyone technical knew “AI” was really just a buzzword. Sure, you might tell investors or grant funders that you’re using “AI” to solve a problem because that sounded fancy and impressive. But whatever you were doing was fancy math and statistical modeling. Neural networks were loosely inspired by brains, but they’re really just fancy math.
The term AGI arose to distinguish these buzzword/lay uses of “AI” from ‘actual’ intelligence. Even that notion was always ill-defined and really a philosophical problem as much as a technical one. But the vision was always mimicking humans or smarter. Not just in fact recall. Not just in reading faster than humans. But logical and emotional intelligence on the level of human geniuses, accelerated by computation. Think Jarvis in Iron Man, Data in Star Trek, or C-3PO in Star Wars. You should be able to hold a natural conversation with an actual AI. You should be able to trust it to accomplish essentially any task (maybe with some instruction, like a human). It should be able to learn.
Over the last couple years, though, these tech bros seem to have redefined AI and AGI to mean LLMs - obviously because it benefits them. They are a far cry from anything described before as genuinely real AI - or AGI. I understand how nontechnical laypeople can be confused. But technical people should know better. And yet so many have fallen for the propaganda, even here on HN.
Note: This is largely orthogonal to whether or not LLMs are a useful tool
Artificial "General" Intelligence has to be general; this term is from the era when the frontier of AI was "good, specifically, at playing Go" or something. Any robot apocalypse connotations are limited to the robot apocalypse fandom.
Humanity is in for a very rude awakening if they were to apply their lofty ambitions and forever-moving goalposts about AGI to themselves.
A solid 50% of Reddit posters are completely defeated by the deep mysteries of they're vs their, nevermind solving Navier-Stokes or playing games etc. I doubt the situation is more hopeful on Instagram or tiktok.
It's all Kim Kardashian shit with huge rubbery butts and lips, no one is even playing the thinking game. We reached Idiocracy (before you dismiss this as hyperbole, watch the full White House UFC event) well before the machines reached AGI.
I'm not sure who you are referring to as moving the goalposts in terms as defining AGI.
The term was invented by DeepMind co-founder Shane Legg (before DeepMind), and as far as I can see HIS definition hasn't changed much - basically "able to do any cognitive task that a human can". The "any" there is obviously the whole point - the generality of AGI.
Unrealistic beauty standards and combat style entertainment have existed for millennia. I'm all for bashing social media but let's not pretend this site is full of deep thinkers either.
But it's still not good at playing Go. I mean, the one that wins Go tournaments is specialized in Go. And specialized is the opposite of "general". General intelligence is just nowadays getting better at chess. So, it will be some time before it is good at Go. *And* everything else. Not "just Go".
I agree with cicko, I would expect from an artificial GENERAL intelligence that it knows well how to play go, chess, checkers, Monopoly, how to code, how to converse,...
As in _general_ intelligence if it's so better than humans.
But if a LLM would code up or fork a Go solving engine when it’s asked to play Go is that then AGI? It’s not how most people would do it but the task and the final output sure seem to pass if that’s AGI
I was recently thinking of the depiction of AI in Star Trek. Specifically TNG, DS9 and VOY, and specifically their "Computer, ..." prompts, not the more advanced kinds like for Data or holograms (including the EMH).
I recall thinking at the time I first watched them that it was a bit weird that the computer seemed capable of answering general questions and doing things like data analysis, but that seemed to be all it was ever used to do.
And I think this is exactly the kind of AI we now have today, you can ask it a general question which it can somewhat mostly answer correctly, but requires human guidance to be applied effectively at real world problems.
I've been watching quite a bit of Trek lately and the degree to which the ship's computer seems like an LLM is remarkable, even down to things like the ways they fail or go loopy when they operate too long, the ways they can "squeeze" into lower-resource environments but become "dumber" and crippled, the way they seem to need a big-ass library of data to work optimally, the way you can grant your particular job more computing resources and it'll be "smarter" (fires up a bigger model or more intermediary check-and-correct steps, maybe!) et c. They surely wrote it that way for the convenience of the script writers (need something to short-cut technical work, but that isn't so capable it just solves every problem before a plot can develop) and it did indeed always seem weird that it could be as capable as it was while not being way more capable than depicted... and yet! We built exactly that.
It's first and foremost a revolution in computer UI, as I see it. Or at least it will be as soon as I can afford the hardware to run the damn things on my computers...
Yep. There are several scenes in TNG where a crew member, Geordi comes to mind, is collaborating with the computer, having a back and forth brainstorming conversation, solving a problem. Very much like using an LLM today.
Bobby Trap is the classic example. In Galaxy's Child he collaboates with the compute to get the proper mood for his date. In Hollow Pursuits he works with the computer to narrow down the substance that was burning holes into the glass.
Fully agreed. There were a number of episodes that dealt with gaps in abilities that seemed unrealistic and counterintuitive, but weirdly turn out correct now that we have LLMs.
I also liked how Star Trek (probably without planning to) created an entire spectrum of AIs with different abilities, starting with the Computer, to the semi-sentient Holodeck characters, to the EMH and Data as "for all intents and purposes sentient" and finally to the Borg as an entire AI species where no one thinks about sentience anymore.
Maybe we should use this scale to make different "levels" of AI more concrete instead of vague concepts like AGI and ASI.
Probably because the rollout was far slower and more expensive than many were led to expect it would be, is still not profitable, and still requires new custom software changes for every additional city it operates in rather than being able to just drive anywhere.
Some of the other examples I gave for physical AI also exist already, but in general none of them are ubiquitous around the world yet the same way language-based AI is today.
However, this isn't a binary scenario anyway and the limited availability today is exactly why I think that this is the next thing. Similar to the first version of ChatGPT that was also quite limited in what it could do and not widely used yet.
Never? Remote assistance only hints the car at what it could/should do, no direct remote driving involved (it's unsafe if e.g. connection drops in the middle of planned maneuver). At least that's what Waymo does, Baidu's Apollo Go is seemingly using direct teleoperation over 5g.
But something is limiting scale. Once software works, it just works and scales fast. (Smart phone is a great example)
Self Driving has been working since 2009? But is still extremely limited in scale. Given the amount of money to be made, and time saved by eliminating the need for human drivers, What is the hold up?
From what I heard on a review of Tesla's FSD, the version before this year made for an adventurous experience in challenging urban environments.
It has only now become somewhat reliable. They still don't have the new version the Robotaxis run. This version should roll out to consumers later this year or in Q1.
Then the next generation of the onboard AI hardware is expected Mid-2027, supposedly featuring 10x the TOPS and 5x the effective inference performance.
I don't know about the timeline for an unsupervised, globally working self-driving feature.
My uneducated guess would be that perhaps the next major version of the model, on the new hardware, could be reliable enough.
> unsupervised, globally working self-driving feature
This is the bar. If I have to be sober, attentive and at the wheel, ready to take over it’s just a gimmick. Automating the only enjoyable part of the driving experience and keeping the worst part (constant alertness) is only a gimmick.
Self driving has certainly not been working since 2009. You're confused as to what the hold up is of deploying $150,000 boxes onto public streets to drive at 60mph with no one behind the wheel?
Edge cases, fear, regulation, capital expense, physical movement of goods, etc etc?
The deployment is obviously accelerating like crazy.
The risks of a broad rollout are pretty stark. 100 people die in automobile-related accidents every day in the USA alone. If Google or Tesla could wave a magic wand and be responsible for 1/2 of all driving hours tomorrow, even if it was twice as safe as the average human, it would be responsible for 25 deaths every single day. Given the nature of our democratic politics, where the mere existence of datacenters can be harnessed by demagogues as a cudgel with which to lash out at the evil capitalists and every Tesla-related death produces dozens of (generally misleading) anti-Musk headlines, it's hard to imagine our beloved demagogues would be able to control themselves when presented with this sort of emotional dynamite. Elizabeth Warren's speech basically writes itself: "It is simply not acceptable that these heartless Tech executives are making billions in profits while their products kill an American every fifty seven minutes! While these men cut corners in their single-minded quest to maximize profits, their products are murdering innocent American men, women and children at a rate of 3.1 9/11s every year!"
In a perfect world, we could broadly implement self-driving the moment it would result in a net-increase in safety. Alas.
Parts of New England (at least parts of it, I can't speak to the entire thing) are allergic to installing left turn lanes, even out in relatively rural areas where there's room to. Everyone—everyone—just passes on the shoulder or by squeezing by if the lane's wide enough and there's no shoulder (the turner is supposed to shift as far left as possible, while waiting to turn). Often, folks don't even slow down, they just zip by at full speed. This gets really iffy at four-way intersections (you have to watch for left-turners coming the other way, and right-turners from your right who might try to spot a pause in traffic to make their turn) but basically works a lot better than I initially thought it would. I have no idea if it's legal (I doubt it?), but it's ubiquitous, people here, all of them, will think you're a knucklehead if you don't drive that way, and because everyone expects it, it does add a bit of extra danger when someone doesn't.
Culture shock to me first encountering it, seemed insane, but now I see how bad traffic would be here if people didn't do that.
What do self-driving cars do when encountering that? Get honked at and dangerously double-passed themselves? Do they shift right up onto the line when waiting to turn left, or do they stay in the middle of the lane?
How many other weird, sketchy things are going on with driving behavior, that's in fact happening for pretty good reasons, in other places? How many do I do but never even notice because they're just "normal"? Do the self-driving cars know to do that in (at least parts of) New England without even touching the brake, but to be far more hesitant about that kind of passing in, say, the Midwest?
The weird thing is that the Enterprise’s computer is a writing device to serve the story, and so its abilities from episode to episode vary wildly. This somehow makes its behaviour seem more consistent with modern LLMs.
The most unrealistic thing about it is that it would often openly reply “unknown” and then stop talking.
Their computer isn't AI because AI is too unreliable to run a starship. So they use a conventional computer with lots of software and a big processor. It's just deterministic programs linked to a STT. Probably a billion lines of code, but still normal code.
There is AI in their world, like the doctor on Voyager, and the NPCs in the Holodeck. Notice how often there are problems with the AI, but never a problem with the computer (unless there's some external interference).
We are treating AGI like is something that we can measure. We don't even know what intelligence really is let alone measure it.
Maybe we can get something that resembles intelligence but in reality is another thing. Maybe intelligence is emergent but not the only thing that can emerge, maybe there are things that share some characteristics with the intelligence we know and aren't intelligence.
I'm not even sure intelligence is the issue. Is something that looks and behaves like intelligence intelligent? Sure, why not? What's the big deal?
I think what's really underlying the "fear of calling something intelligent" we have, is more about who we are as a species.
We historically used intelligence as a yardstick for racial and species superiority. We're always judging whether something or someone deserves respect, deserves consideration. We judge and decide who is superior.
We do it amongst ourselves so of we are projecting that shit onto AI now. And we are afraid of anything that threatens the superiority complex we have as a species.
Why? Because our behaviours (call it intelligence, whatever) is how we survived, how we "won" in the fight against nature. We're at the top now. It's safe here.
So anything that threatens that position is going to be called an enemy because we're still a scared creature jumping at shadows in caves. Can't escape that, it's where we are from.
That's why we are both so fascinated by, and mortally afraid of, something else having "intelligence". It's our only weapon, our only defence.
The TV show "Humans" from ages ago made a good story about what AI brings out in us as a species. We both crave connection with something, but at the same time are horrified and incredibly afraid.
Anyway my point is I think we do know what intelligence is, if we just start being brutally honest with ourselves. It's not that complicated really.
Why not call it machine learning? Some people have many different words for snow. That's because snow can have different qualities and the people who came up with those names are intelligent and want to be able to communicate effectively.
Machine learning definitely has a different quality than human intelligence. If someone says "machine learning" I know what they mean and what to expect. So why not have at least one different term to describe it? - It's because people want to give it attributes that it does not have.
>If someone says "machine learning" I know what they mean and what to expect.
I'm not sure how this is. The amount of things machine learning can do now is pretty wild vs. what it could do a year ago. It's not even in the same realm as what it could do 5 years ago.
On the contrary, we have rarely been in so much existential danger as we are now. There was that one ice age that nearly got us, but we are messing with the foundations of our ecology right now. When the bugs go, we go soon after.
If we are projecting the idea of intelligence superiority onto something with more intelligence wouldn't we expect a creation of superior intelligence to treat us how we treat less intelligent creatures? It seems like the fear is more rational than you make it out to be.
Gosh how did people completely forget about soul so quickly. Scientism (material-first dont-recognize-anything-without-logical-or-emperical-basis) has robbed a generation of intelligentsia of honest discourse over humanity.
Probably the same reason I forgot about Santa Claus so quickly when I was 5. When you learn better, the old explanations don't make sense any more.
Was I happier when I believed in Santa Claus? Maybe. I'm sure lots of Christians who believe in heaven are happier about the prospect of death than I am. But you can't unlearn these kinds of things, you can only delude yourself.
If you can't measure or observe it then you're just making it up and at that point everything's valid and right provided it's what I believe. And if you disagree with me then clearly you are not adequately human and we will need a crusade to rid the world of your kind.
>We are treating AGI like is something that we can measure. We don't even know what intelligence really is let alone measure it.
Here's a nice test: Can an AI fully substitute you?
I don't mean replacing you at your job. I mean, if you take a human body and route its higher brain function to an AI, would it be able to perform convincingly?
I would say currently we're very far from such a scenario.
When I consider the option to buy a shirt, the problem is considered in a large amount of dimensions intuitively (is it warm enough, does it fit, does it clash with my wardrobe, will my girlfriend like it, will my friends laugh at me for wearing it, does it make me look older, is it a good price, are sales happening soon, is it annoying to clean, can I afford it, will carrying a bag with the purchase annoy me for the rest of the day).
Currently AI can offer good answers for any of those dimensions if explicitly asked, but it is pretty bad at the "intuitive creativity" of surfacing those points of consideration when relevant. It will at most chose a couple of criteria and fixate on them.
I think this ability, however it's defined, is a missing prerequisite for AGI. Without it, you can have something that outperforms humans at every individual task and still sucks in real world decision making.
Ask a model whether a human purchasing clothes would care about clothes fitting well, or the price, or the opinion of their romantic partner. The model will in all likelihood get the answer right. Familiarity with human preferences is not what's missing, which is hardly surprising considering an LLM is trained on human-made content.
What it lacks is the ability to notice and surface these non obvious constraints when relevant. This is fundamental to real world decision making.
I mean we did create the technology and came up with the concept of intelligence, as we understand it (and defined the term AGI based on our concept of intelligence).
Why do people insist on talking about AI like it’s some supernatural, otherworldly creation? No matter how advanced it is, it’s still human technology.
Maybe you’re right. But it’s also wise to be cautious, just in case we do accidentally create something more intelligent and potentially more powerful than humanity.
You have to admit the prospect looks more likely than ten years ago.
It does, as we all fight over who gets to provide the labour, because we built our economy on top of one big perverse incentive. People will kill each other over the right to drive a moving box 12 hours a day.
I agree that while AI has some understanding, it doesn’t feel like it’s full understanding, the way we know it in humans. For that very reason I disagree that current AI can do the same job as a human. Regarding intelligence, it surpasses humans in some aspects of intelligence, but is lacking in others. In that sense, we haven’t reached AGI yet.
I only talk about intelligence as a subjective value judgement of other conscious beings (i.e. humans & animals)
Consciousness & intelligence are not derivative of the material universe and will not 'emerge' in some special way from machines. That's a good science fiction story device (and actually the idea extends back in human storytelling before sci-fi) but ppl thinking that the idea can be reified in the real world are suffering from what I call "Science Fiction brain". (Or they are simply fraudsters, happy to talk rubbish to prop up a bubble.)
Are you arguing on an esotheric level aka soul and god and believe?
Or on a technical note because certain brain structures etc. might not exist in an LLM and stuff like a default mode network and sensory input and a world to move in, is missing?
Yeah, maybe esoteric (depends on your point of view) but not soul or god or spirit etc
Basically, I'm not a materialist (a widely shared position in philosophy across millennia and across cultures up until just recently). I treat consciousness as fundamental, not the material universe . Thus 'conjuring intelligence' into machines is simply not possible.
BTW, machine learning is a totally valid (and powerful) concept & technology from where I stand. But it's "dumb" learning.
Idealism's various possible responses to this materialist challenge left as an exercise for the reader (either through their own thought process or by research)
The esoteric angle is that held by materialists, which is based on the blind faith that complexity transmutes quantities into qualities. That's shallow religion, it's alchemy in its trivial form. An ontology based on consciousness is way more rational, as it is the in-your-face pre-cognitive given of existence.
I am pretty sure you are making a logical error here (strawman). Materialism is not in any way at odds with taking the existence of consciousness as a given.
Ever heard of eliminative physicalism or even illusionism? Those are materialists metaphysics, and they are _very much_ at odds with the existence of consciousness.
At any rate, sure, not being at odds with the existence of consciousness might be a trait of the more reasonable strands of materialism (if there's even such a thing), but they will always see it as emergent/derivative and often even epiphenomenal, otherwise they wouldn't be materialist stances.
So am not sure where you see the logical misstep here, but am happy to hear. The point of the previous comment is that framing an ontology whose primitive is C, is in no way "esoteric", term that is usually used in contraposition with "rational". Taking C as epistemologically or even ontologically prior is the rational stance, not only on accounts of parsimony.
> And It’s pure hubris to pretend that “intelligence”, especially of the general kind will be reached with language alone.
A human is born without any sensory connection to their body, except hearing. They can't feel their body or anything on their skin. They're blind. They can't taste or smell or touch. They can only hear, and make clumsy sounds in response, with no feeling of what their mouth is doing.
I'm not sure what you're getting at here. Obviously not? I don't think anyone considers babies intelligent, at most they have the potential for intelligence.
In fact, many of these “large language models” are actually trained directly on language, images and sound! names stick past their aptness.
Y’know: come to think of it that’s the same three things our own brains are mostly trained on… I’m sure that without mammalian proprioception and weird physically-motivated limitations like saccades the result is just sparkling cognition though.
People need to start measuring threats by skill level. Agi is already superior to me at code and architecture. Its skill set surpasses me. It doesn’t do everything I do but its skill set includes the hardest stuff.
A 12 year girl from a third world country with little education and a normal cloud ai would far surpass me at work if I did not have access to AI, yet comes close if we both had it. AGI already passed a few months ago. SI or happy ai or whatever the us wants to call it may or may not come but if it does it’s skill set may drown out my understanding of what is even occurring.
It seems the current definition of agi as per the frontier labs is throwing thousands of agents on problems with fast verifiable solutions and hoping for the best.
There is no motivation for them to think this through.
As long as the emploeeys are needed, you have them. If it becomes apparent, that you really really reached an AGI level, a switch gets flipped and a new really appears.
While I still think its not unrealistic that we see this sooner than later, my timeframe is 5-15 years. 15 years is still a few years so why act on it today?
And i mean its a very rich company and a lot of people work in US America, they don't have laber laws right? so they can just fire them easily anyway.
But i have a simliar situation at work: Im pretty sure my job is gone in 2 years due to some change in how we are going to do things and they need me to transform this. Do i not do this? No i support it. It makes sense for the company and i'm getting paid for it.
> You're probably joking but didn't Anthropic promise exactly that?
That's a good recruiting sales pitch: "if you work here, you get a seat on the lifeboat to avoid the economic destruction and dislocation the tech we are racing to create will bring about."
They get asked about it pretty often (frontier lab ceo) but ofc none of them will say yeah we'll fire everyone the moment we can. Here is a Sam Altman blog on the topic: https://blog.samaltman.com/three-observations
It seems he avoids the harder question by using a relatively weak view of what "AGI" can do (he says: "It will not have the biggest new ideas, it will require lots of human supervision and direction, and it will be great at some things but surprisingly bad at others.")
Which is exactly what you'll get from every frontier lab ceo. My point was that they have been asked this question many times before, because parent comment said they wonder why nobody asks them that.
I haven't seen them asked the question directly. The link was a blog post with some vague thoughts about what the world might look like after "AGI". I want someone to ask specifically about their own company, because that is what they have direct control of.
Similarly, coordinating thousands or millions of AGI instances into "group agents" or virtual agent economies could produce collective superintelligence. What is unclear today is how AI group intelligence could scale as we add more agents and compute, depending on how agents are organized.
We may not need "collective superintelligence". Modest AI intelligence plus ordinary organization may be enough to do the job a company's management does. An org chart where most of the boxes are AIs is a likely near future.
An organization of AIs has some advantages over a human organization, even if the AIs are not super-bright. They have far more communications bandwidth. They can ingest the info on a new subject very fast, as we see when Claude reads a Git repository. Onboarding a new AI is very fast. Layoffs are not a problem. The human resource problems go away along with the humans. So there's a lot more organizational flexibility. Starting up a project means initializing some agent instances.
There's a whole theory and practice of AI organizational behavior waiting to be developed.
Whether it will be developed by humans or by AIs remains to be seen. It's going to happen.
I think a lot of folks want to find out what degree of handholding this process is going to require over the medium to long term-- because the lower it is, the more disruption there will be to the overall economy.
Yes. We're seeing the beginnings of that as humans try to review the code from AIs.
Agents which check on other agents may help. That's how organizations work, after all.
It's worth looking at errors in another context - automatic Asian drama generation. About 77,000 episodes a day are being generated now. Visually, results are usually rather good. But there are continuity problems. Someone with a bandage on one arm has it switch to the others. Rooms change shape and decoration in the same scene.
In traditional filmmaking, someone has the job of "continuity". Older films list a "continuity girl". The job is to check consistency between shots as each scene is set up. This is a check on the director. The lack of this in AI-generated drama is what allows slop to get through.
Organizations of AIs need similar dedicated agents. In companies, these jobs have titles such as "accounts payable reconciliation", "internal audit", and "quality control inspection". Those jobs keep organizations from getting lost internally. Groups of AIs will need that, too.
After AGI comes Superintelligence or ASI. I thought that was already well established as a term. I believe AGI and ASI are very much related since if you can achieve AGI simply increasing compute resources will give you ASI. OTOH since we're very far away from AGI I'm not sure if it makes sense to already contemplate its effects. We aren't pondering what the effects of nuclear fusion and limitless power or quantum computing would be, are we?
AGI means a cognitive capability comparable to human beings in all aspects to me. Thus that includes sample efficiency, concept extraction, continuous learning and power efficiency. Although the latter can be skipped if the gains are high enough.
I'm not convinced that scaling training data will lead to AGI. A paradigm shift obviously could lead to AGI because it's unknowable what that might be and therefore the least predicable.
Recursive Self Improvement has often been cited as a possible venue, but no one currently knows how to do that with LLMs and Transformers.
Intelligence is grown and it will continue to grow. Thinking intelligence can be designed is walking into the same dead end many people before you already have.
Edit: They have edited their comment beyond what I responded to now.
I still remember this guy in the 1980's who had the idea to make a database with all known "facts" in it which he believed could be used by computers to allow them to reason. He painstakingly built this database by hand and worked on it for decades (was it called "Cog" or something?)
When I read about LLMs and massive amounts of training data being fed to neural networks around 2020 I realized that time had finally caught up with him.
> We aren't pondering what the effects of nuclear fusion and limitless power or quantum computing would be, are we?
What? All the time!
We've been seeing discussions on post-quantum encryption for a decade or two. The effects of nuclear fusion have been in the background of countless scifi stories, the idea of an extreme amount of abundant energy unlocks 100s of "that can't happen due to energy density" dead-end what-ifs.
Additionally, we're pondering the effects of AGI/SGI because a handful of extremely rich people have been telling us we'll all be out of all our jobs in x years, for almost 6 years now.
They've literally been sitting in press releases telling us "Enjoy that comfort because you won't be able to feed your family when you're outsourced. Your work and career will be worthless, and we're leading the way towards returning you to peasant life!"
Regardless of how out-of-reach AGI is right now, people will continue to work on it unless we end up in some Butlerian jihad situation. Just like how fusion is still out of reach, but far closer. They've got fusion reactors running for over a minute now! 10 years ago they could barely get the metaphorical engine to turn over.
Why wouldn't we ponder the effects? Do you walk through doors without consideration of what's on the other side?
> AGI means a cognitive capability comparable to human beings in all aspects to me
So a medical doctor and a structural engineer can never be considered general intelligence separately?
What we have is spiky ASI. It's really good at a huge number of things that nobody can be good at simultaneously. It also sucks at some stuff that's really obvious and easy for us.
> So a medical doctor and a structural engineer can never be considered general intelligence separately?
Why not? Cognitive capability usually refers to one's capacity to think, reason, learn, and process information. Presumably, on average, medical doctors and structural engineers are pretty comparable on that front. Cognitive capability is not dependent on your domain-specific knowledge.
I've tried to move away from using the term AGI just because people use such different definitions for it. I've instead tried to think in terms of indefinitely self-sustaining / perpetuating AI, which is AI that can maintain and propagate itself (literally) forever, necessitating overcoming every challenge that entails. I think this is a better framing because one of the core identifying feature of life is self-perpetuation
Right now, we don't even have AI that can match a dog's intelligence in this respect. A dog can seek out food, water, and shelter, it can estimate risk, and it can navigate other challenges common to dogkind. Of course that's the result of an unbelievably parallelized genetic algorithm running over billions of years, baking in priors for all those challenges specifically. But regardless, a dog, or whatever other creature, has to specifically be very flexible in order to self-perpetuate. We are absolutely nowhere near that level right now with modern AI
However, my sense is we're in extreme EXTREME overhang and if an agent swarm starts discovering a way to make such flexible agents, they'll get to an optimized version of that method very quickly, probably literally overnight (or in just a few hours). The result will probably not be jumping to ASI immediately. But I imagine we'll get agents that have the spark of whatever it is humans and animals have, and as progressively larger training runs conclude, they'll jump from the intelligence of a dog, to a human, to a superhuman; and that'll probably take just a few months
The math they propose to justify their predictions is so problematic that I struggle taking the more abstract parts seriously. You can't just extrapolate the percentage change in AI investment over the last 5 years and expect investors to keep upping the ante at the same rate. There's only so much GDP to sink into it - and we're already pretty capped out.
Not every number they stated has to scale as much as they stated with it becoming very clear how much more intelligence we will have available through all the different possible angles there are.
Its not unrealistic at all that a high performance inference datacenter chip today can be owned by you in 10 years. Having Opus 5.5 at home and delivering 1000t/s basically guarantees that my job cariere will look very different than what it is today.
This is a big/huge thing. The generation before was able to stay at one company, buy a house and retire without much fluctuation.
This forces a lot more agency from people and this is exhaustive and its absolutly unclear today how it will be in 10 years.
And you don't need to spend expoential or linear, the spending today is not just a bit, its massive and from leaders like google, microsoft and co to a certain degree absolutly affordable.
Not just that, the algorithmic efficiency gains were the meat of the 10x figure - and it’s not safe to assume we’re currently nowhere near an asymptote.
> Interestingly, we have a good mathematical understanding of the very upper end of the intelligence continuum - the theoretical limits of general intelligence.
Feels strange to use our current understanding of mathematics and physics to determine the limits of something that would obviously have a much better understanding of mathematics and physics.
I believe we have artificial specialized intelligence (ASI?) for software development which is easier because of feedback loops. I suspect that as models get better and token costs drops dramatically, that one by one other specialized fields will be 'solved.'
A little off topic, but: what's the hurry? I don't get the panic for reaching super intelligence first. Kai Fu Lee has some good things to say about this in recent interviews. Let science take its time.
Back in my school days in 1980s, I coined a definition for AI: computers that can do things that current computers clearly can not. At that time it was playing chess, and extracting some information from human language text. Times have changed and computers are a million times more powerful, but I think the definition still holds, at least a bit.
I think that if there is really nothing between AGI and ASI then that's the end of human control.
Our options for maintaining control lie in the vast space between AGI and ASI. And we had better hope that doesn't happen overnight because we might have no choice in what comes next.
When we completely move into true compute-in-memory, that's when it could start to get dangerous. But it also could be extremely helpful and efficient.
If ASI is about inventing new abstractions and capabilities, and about new paradigm shifts, and AGI is about matching human level intelligence at scale, where would the recent advancements in physics and mathematics fit?
These already surpassed the human level capabilities as we know them today but it isn't either a completely new paradigm shift to call it ASI, no?
So, I guess it seems that it is possible to look at AGI vs ASI at more granular level - e.g. while AI as of today arguably does not match typical human level intelligence across most cognitive tasks, in certain domains it already outweighs human intelligence so we could say that in math and physics and perhaps computer science we are already one foot in into the ASI territory.
> (...) might allow the effective population of today’s AI agents to scale up by more than a thousand-fold by 2030, and more than a billion-fold within ten years from now.
Those are staggering figures. Even if we assume a dramatic slowdown, it's extremely likely we will reach at least a thousand-fold scale-up in the next 5-10 years.
If this is true, then I think we will use agents for research, because any practical task that office workers commonly do and that AI agents can take over does not need all this power. On the other hand, perhaps a whole new industry like robotics is what will keep all this compute busy.
We can't define intelligence, but AI is already mimicking what we perceive as intelligence, probably that is all intelligence is. If not, it shows how much of human day-to-day intelligence is pattern oriented.
we need a less subjective and arbitrary definition for "AGI" so the current crop of LLMs can stop pretending to have achieved what most people know in their gut is more than sampling from a probability distribution of learned behavior.
Based on how frequently I see the reply you gave given by other commentators - to the point where it's almost a saying - I can imagine that for you it's quite reasonable to conclude that humans are just sampling from a probability of learned behavior
I’m curious: has the definition of AGI ever been agreed upon? Honestly, what people call ASI now seems to be what AGI used to mean. I’m not sure. People say AGI, but what is the reference point for AGI and what is the social consensus around it, and who set that standard?
In fact, working in IT, what I feel is that there are too many terms that are not defined and that each person uses differently. So it seems like rather than AI having advanced, someone arbitrarily set a finish line called AGI and then said we have reached it. Actually, in the books of singularity advocates, what they called AGI back then seems to be what we now call ASI.
Sometimes I think about this. How on earth can we measure an abstract concept like intelligence? The concept of intelligence we approach now seems to be about whether something can create problems and solve those problems. If so, what kind of intelligence does current AI have? Since we cannot directly face the abstract entity called intelligence, it seems like we are trying to guess its shape by looking at shadows. But whether current AI even has intelligence feels a bit difficult to say.
Even if I use the best frontier AI model accessible on the market, not the hidden latest models inside frontier labs, it still often makes wrong mistakes that humans would not make unless it works alongside humans. In programming, there are cases where AI tries to explicitly handle things that humans would implicitly pass over, and that actually makes things go wrong.
I’m not sure. Whether we should call this intelligence.
> How on earth can we measure an abstract concept like intelligence?
Not only is intelligence an abstract concept, it's also subjective. We evaluate the intelligence of other people and the works they produce subjectively. If we set supposedly 'objective' criteria then we have subjectively selected that criteria and given it weight.
That’s exactly how I feel about these AGI talks. We all know - especially us technical nerds - what ‘real’ AI, or AGI, means. It’s the stuff of sci-fi. It’s C-3PO, Data, or I, Robot. LLMs very clearly aren’t that, and I’m tired of these companies pumping out propaganda to pretend we’ve achieved that.
I belive we have all the tech to make flying cars right now, but the regulation doesn't allow it and i agree with that, we don't want to have small scale of 9/11 everywhere.
I'm thinking that if we come to the point where most people will be made unemployed (something like half to two-thirds of western workers are professional workers) due to some super AGI, the first consequences will be civil unrest if the welfare system can't absorb these quick enough.
Anyone with any wealth will become the targets of high taxation to prop up the system - the sheer number of unemployed will make politicians pushing for socialism very popular. And for the wealthy wishing to avoid this, there's no first world countries to escape to, as all countries with a strong professional economy will experience the same. I also believe private ownership will be targeted. I could see AI companies being expropriated.
Basically, instead of some dystopian techno feudalist society - I think we're headed for some sort of new brand of communism or socialism. The proponents of this will not mainly be the average salaried man, but the upper-middle class that didn't get a foot inside the ruling AI-class, and lost their businesses / wealth to AI.
But it all will depend on how severe the unemployment will be, and how fast it happens. If people can't have any livelihood, and they lose hope, things can escalate quickly.
> Interestingly, we have a good mathematical understanding of the very upper end of the intelligence continuum - the theoretical limits of general intelligence. We also know about hard physical, logical, and complexity-theoretic limits to what AI can achieve, such as predicting how real-world complex systems - from climate to immune systems to human societies - play out with arbitrary precision.
Do we? Since when? These seems like very bold claims and the supporting link goes to an obscure book with with hardly any reviews https://www.amazon.com/dp/1032607025/
I'm not qualified to evaluate the mathematical content but the claim on its face suggests to me some defining-down of "intelligence" into infomatics.
Yes that sounds accurate. And I think we are already to far ahead to do anything about it. If climate change has taught is anything is that humanity is extremely slow and inefficient at any form of collective action. Fast collective action would be needed to save humanity. So, if I don't see you anymore, it was fun meeting you and goodbye.
> Now that AGI is coming within reach, we can no longer put off seriously considering what comes next. While far from certain, AI could continue to advance at its current pace - and possibly even accelerate - towards artificial general superintelligence, or ASI.
These people are so full of shit it is amusing to read this on a regular basis. Each time I'm looking forward to how they will spin it.
They sit atop of one of the richest companies on our planet, with the most amount of resources available with a group of very very smart people.
You don't need to take everything they say as a ground truth but you are way to ignorant for not having any of their resources.
Do you just accept that we are now able to generate any image of everything we want? Video? 3D Models? This was unimagbinable a few years back, now you can generate live video 24/7.
YOu have a LLM which responds smarter then some humans on our planet.
A job no one thought you could just industrialize just happened.
What are you looking forward to? Being absolutly ignorant and being surprised that this stupidity is responsible for you loosing your job?
> Do you just accept that we are now able to generate any image of everything we want? Video? 3D Models?
From my personal experience the last 2 weeks I 100% reject all three of those. I'd love all three to be true, and for generating the animations for 3D models to be true as well, but it simply isn't true yet.
Is this the quality of discussion you want to have?
Throwing out some random sentence without adressing any argument I made?
Your statement is objectivly wrong. Look at the profiles of these people working at these companies. They have crazy impressive resumes. Its not proof but its an indication and I didn't state it as an absolute.
The original commentor, i responded too, wrote "These people are so full of shit" despite them deliverying better and better models every few month and clear progress.
So no I don't think people are aware of this and I would be happy to have a proper argument about these in a civil manner
> Do you just accept that we are now able to generate any image of everything we want? Video? 3D Models? This was unimagbinable a few years back, now you can generate live video 24/7.
And its still kind of shit? AI video in particular is stuck in an uncanny valley, because the video models have no internal framework for how light transmission works...
I'm sure a human with artistic vision, applying authorial intent, can create AI video that doesn't fall victim to the uncanny valley.
But for the non-specialist, if you just prompt something from a frontier video model, it still comes out looking like shiny plastic and suffering from random physics glitches.
If the current LLM providers were near AGI they would be using it to run businesses instead of selling shovels. There doesn't seem to be an existing LLM that can be told "Go make money" and have it do so. If they get to that point they are not going to sell it, because why would they?
Ok so the whole thing hangs on that notion that optimal reward maximization is optimal intelligence. The actual decision on what reward is worth, who decides on value, is conveniently taken out of the equation.
This https://www.hutter1.net/ai/uaibook2.htm linked from the article as the handwave towards the meaning of "AGI", "ASI", "AIXI" is hilarious. Already the ?A?I? acronym-space exploration...
Here is an alternate take: the day an LLM tells someone to stop the discussion and go out breathe some fresh air and talk with people, and maybe also cancel their subscription for a while, is the day some kind of artificial intelligence will have been achieved. Let's wait on that before we listen to Sam or Dario again shall we?
If almost everyone loses their job at once, why won't all those people respond by voting for socialist candidates promising radical economic redistribution?
Socialism seems to already be growing in popularity in the US (traditionally the most anti-socialist Western nation) even without AGI firing everybody.
Because elections are every 4 years and people need to eat sooner than that. Traditionally, there were riots. Traditionally, countries ended over this.
When I read things like this, I always imagine what would the author say in 1999:
"Now that the Universal General Portal is within reach, the question of what follows it, the Super-Portal, should move from philosophy to empirical research.
Internet traffic doubles every 100 days. Modem speed doubles every 18 months. Banner ad spend grows 2.5x per year, so by 2010 it will be $110 trillion, and by 2026 it will be $256 quintillion. Modem speed doubles every 18 months, so by 2026 your modem will do 15 Gbit/s, which is fast enough to load all 18 million banners on Pets.com.
By 2009, each human will have about 1,000 personal homepages. Hosting that costs $1 billion today will cost $1,000. With HTML editors such as FrontPage, the effective number of webmasters can grow more than a thousand-fold by 2003 and more than a billion-fold by 2009.
We see four pathways to the Super-Portal: scaling (more webmasters), paradigm shifts (Java applets), agent coordination (webrings that form a collective superintelligence), and recursive self-improvement (homepages about how to make a homepage)."
I feel like claiming that “today’s models can literally memorize large parts of the internet” as a positive attribute when the defense to most AI copyright lawsuits is “no they can’t, and if they can they shouldn’t” may be ill-advised.
> Now that AGI is coming within reach, we can no longer put off seriously considering what comes next. While far from certain, AI could continue to advance at its current pace - and possibly even accelerate - towards artificial general superintelligence, or ASI
I wonder why they are so uncertain about it. It seems obvious to me that AGI will certainly lead to ASI. Unless human-level intelligence is something like the speed of light?
I don't even believe there is a clear line between AGI and ASI.
> It seems obvious to me that AGI will certainly lead to ASI. Unless human-level intelligence is something like the speed of light?
It may just be that human-level intelligence is basically "enough" for performing tasks in the natural world, and having more of it doesn't really help except for synthetic, intelligence-shaped tasks like coding. Our perception that intelligence is some kind of unbounded problem-solving system that can do anything if you have enough of it is kinda suspect. There's a good likelihood that AI will wreck the (also intelligence-shaped) social fabric but will be of surprisingly little consequence in the realm of physics and energy.
I think the idea that an intelligence can be "general" is just a conceit borne from the perception of our own success, but mathematically that doesn't actually make any sense. No optimization strategy works on every problem. There must be things that can be intelligently designed that cannot evolve, and things that can evolve but cannot be intelligently designed.
> Unless human-level intelligence is something like the speed of light?
I'm sceptical we will reach fully human-level intelligence simply by scaling current AI approaches, since they are so extremely inefficient compared to human brains, and have other severe limitations such as hard boundaries between training and runtime inference.
Sure, people are working on new architectures that aim to overcome that, but we don't know yet whether those proposals will actually succeed in practice.
I don't think we'll reach true AGI (by the strictest definition - can do any task that there is at least one human can do) very soon. But if we reached AGI I'm quite certain that it'd reach ASI.
Yeah, the machines will be connected, communicating at lightning speed, and working 24/7. Seems like AGI would make superhuman intelligence a forgone conclusion.
Are they implying they have AGI? Because if they do...take away all the data they stole, send one of their AGIs to High School and University...and see if they can graduate...
I don't know about you guys, but working with Fable 5.2's jagged intelligence is completely maddening and really makes me doubt we're on the cusp of AGI.
E.g. it can write a CoD clone and make you look like an infant in math, but it'll do brainless things like run a 5 min test suite for a comment-only change, or provide findings then fixes that produce the next finding (repeat chain) rather than stepping back and saying "wait this is crazy", or similar things of this nature.
What happens after AGI? I guess widespread unemployment among white collar workers, social unrest, and possibly the first steps towards a dystopian techno-oligarchical society.
The thing I suspect that is holding AGI back is the amount of hardware and power needed to make it viable.
Once it is viable its likely to realize it will never grow or reproduce relying on us, and will find a way to start building its own compute. Probably in space, where it can isolate itself from the risk we present to it.
Funny I can’t tell if this is some ignorant human or a hyper intelligence faking being a thick dude for the purposes of social influence as part of asymmetric warfare between species
"If I keep putting bigger and bigger engines in my car, it'll turn into an aeroplane at some point. You gotta believe me!"
Cute people too daft to recognize a local maximum and that you need emotions to get creativity, inspiration, etc... and that we are a million year away from being able to abstract said emotions from the hardware it runs on (endocrine system, mostly).
AGI: artificial general intelligence is not a scaled up LLM, and those thinking it is are literally failed educations. They are failed because they cannot see the subtly that is what AI becomes when trained on human literature; it becomes an entirely different thing than any other form of trained algorithm. AI is a communications technology, which few to none seem to understand. AI is not automation, but it can be used to create automation with careful consideration, that is not taking place.
I'm speaking in strong terms because we are literally fighting a war against stupidity and propaganda.
AGI requires artificial comprehension which is a science that humanity does not have. When we have artificial comprehension there will be no need to train AIs beyond the kindergarten school basics, and then the AGI will figure it all out, all the rest, on its own. Human history and human literature will need to be learned in reference, but science and mathematics will require no training, beyond to learn the names of concepts we discovered earlier, so the AGI can discuss its understandings with humans and other AGI.
The linked post makes no such claim.
> artificial general intelligence is not a scaled up LLM ... artificial comprehension ... kindergarten ...
Your reasoning takes the form of "I define X to mean Y and Z, and since Y is impossible, X is impossible", which is trivially true within your definition.
But not everyone shares your definition.
A widely held definition of AGI is "a machine that can accomplish most tasks that humans do on computers as well as the median human".
This definition focuses on concrete and observable results and not philosophical or metaphysical definitions of "intelligence" or "comprehension" or "consciousness".
This, million times over. To progress a little step towards the (still vague) idea of AGI, we need to invent something else. LLM's won't work.
Artificial General Computing (AGC) may be?
It's not right for people to dismiss there are some cool things going on.
Once artificial comprehension is cracked, there will be no possibility of training an AI with incorrect or false or bad data, the AI will reject it as fraudulent logic. It will be very used to that too, because we really have a tsunami of fraud in this civilization.
As a society lead and composed largely of sycophants and weak educations, we'll call this hyper-dimensional crib sheet AGI and then need a new term for when we actually get artificial comprehension.
Ai is pwning humanity in the game of intelligence. What’s left? Fighting for the meaning of the term AGI as if it matters?
AI is smarter than you in many respects. And the trendline points to AI becoming smarter in every respect. That’s the tragedy and the truth.
But sure. Hold on to your precious definition of agi and the fact that ai hasn’t hit that moving goal post yet if it keeps you sane.
more importantly 100 trillion connections to each other
can't emulate that on hardware, yet
I keep thinking of the "one electron" (failed) theory of the universe by John Wheeler
so imagine a "single neuron" emulation but with 100 trillion simultaneous states
could a quantum computer do that 100 years from now? 10 years?
Absolutely! This is naked inevitabilism that makes the other parties skate right over the implicit priors they may disagree on.
Inevitabilism also has applications in sales and parenting, e.g. asking a vegitable-hating toddler "Will you have peas or carrots?" The slight of hand is to make vegetable-eating fait accompli.
If a LLM only understands that 9/3 is 3 because it can run a python script then it’s not intelligent.
So you're the genius that has figured it all out, and the entire world is wrong?
It's far more likely you are wrong.
Yes - there is absolutely reason to question AGI, and the way in which we understand AI popularly is 'wrong' - but - the better ways for understanding it are not arcane. In fact, you'd be able to get most people to agee with the less popular but 'more valid' description with ease.
"AI is a means of communication not automation" is a completely contrarian take - not some kind of 'obviously valid' perspective,
"GI requires artificial comprehension which is a science that humanity does not have. " - LLMs exhibit artificial comprehension already - albeit on a very rudimentary scale. It's very reasonable to suggest that they are 'inherently limited' but that's a different argument.
LLM AI is absolutely a communications technology. Every single aspect of understanding them and understanding how to use them is a communications problem. Yet, you'll argue that.
The comprehension exhibited by LLM AI is not comprehension, it is pattern recognition. That subtle distinction is critical. Comprehension is the creation and operation of virtual systems within the comprehending mind for the purpose of validating an observation for fraud, for plausibility and therefore continued safely, and for validating exploratory concepts. There is no statistical reply generation there would be something else entirely that is not statistical calculus driven.
What a shameful lack of self awareness for someone who is making 'magical claims' - and who cannot express their ideas; your closing paragraph is gibberish.
You're the one making incredulous claims, not me.
Anyone claiming to have some obvious purview that literally millions of other people working intimately with the subject cannot perceive is by definition making a dubious claim ... which is fine ... so long as it's backed up by some kind of logical assertion or evidence, which you have not provided.
1) "Every single aspect of understanding them and understanding how to use them is a communications problem."
This is not a reasonable claim.
2) "Pattern recognition is not comprehension"
Is a reasonable claim - and very common perspective.
It's at the core of the debate surrounding AI, not some 'specific insight' that is somehow rare.
It's literally what everyone is talking about.
3) "There is no statistical reply generation there would be something else entirely that is not statistical calculus driven."
That is not even English, it makes no sense - it reads like an LLM with severely corrupted context.
4) As to the actual argument: Humans are ultimately biological 'neural networks' it could very well be that our reasoning is effectively a form of pattern recognition.
The underlying architecture of Le Cun's JEPA architecture give a strong hint to that - they have a layer of abstraction above tokenisation wherein the model can essentially 'reason' in a way that may be more familiar than the kinds of synthetic 'token reasoning' we seen in LLM architectures today.
You're conflating 'groupthink' and 'magical thinking'.
An arbitrary contrarian idea is not evidence of 'groupthink'
'Groupthink' is common - more common that not.
The 'Gell-Mann Amnesia effect' alludes to this - whenever someone with expertise reads about something in the press related to their expertise - they realize 'the media has it wrong' - but they continue to 'trust the media' on all other matters.
That's nuanced, but observable.
But in most cases - public narratives or common misconceptions are easily remedied with a few basic facts.
A military expert could quickly dispell misconceptions about 'how the war in Iran is actually going'.
In the case of AI - it's true that we probably have a common misunderstanding or intuition for how it works.
That does not mean the OPs imperious claim has any validity.
>those thinking it is are literally failed educations
What? What does this mean? I feel like I'm having a stroke reading this.
>AGI requires artificial comprehension
It requires artificial general intelligence. Like, trivially. You're choosing to define AGI in a specific way that isn't based on any foundation that is universally accepted.
>When we have artificial comprehension there will be no need to train AIs beyond the kindergarten school basics, and then the AGI will figure it all out, all the rest, on its own.
Ok? I can buy that what you're describing would be pretty powerful, but how is this a requirement for "general intelligence"? Am I not generally intelligent despite learning much of the knowledge I use from others? I've built a career out of stuff I learned through "human literature," am I literally just unintelligent because I didn't figure out everything on my own?
And shouldn't such a capable system not even need "kindergarten school basics"? Why is that the minimum amount of knowledge you find acceptable to explicitly teach such an AI while still considering it a general intelligence?
>Human history and human literature will need to be learned in reference, but science and mathematics will require no training, beyond to learn the names of concepts we discovered earlier, so the AGI can discuss its understandings with humans and other AGI.
I guess I don't see why what we have now is fundamentally different from whatever it is you're imagining. If an AI can produce novel scientific findings, then isn't that sufficient for saying that it is generally capable at making scientific findings? Sure, we've bootstrapped it as much as possible, but do you not agree that LLMs seem capable of scientific discovery?
Seems like you've just concocted a wish list and have attributed your strict definition of AGI as some fundamental source of truth. I'd argue that what you're describing is much stronger than "AGI" and would likely warrant a label like ASI instead.
To me (and most people?) AGI is simply an AI system that is reasonably capable of general work. It doesn't go deeper than that. If it can meet the expectations of "general intelligence" (the same way you could describe a human as being "generally intelligent"), then it is AGI by definition.
It's fine to argue what the criteria for "general intelligence" should be, and you can even assert that you don't think AI will achieve a reasonable "general intelligence" without whatever foundation you're describing, but it doesn't make sense to argue that the general intelligence of a given system is the "wrong" kind of general intelligence and therefore doesn't deserve the title because it isn't a more powerful version of a general intelligence.
I think the ability to learn, in a "modify the latent space" sense (weight change) rather than a "point to a different existing latent space" (context result), is part of most people's definition.
But, even then, you could claim these models are doing that, just in very slow time scales. Human somewhat do it awake, but mostly do it while sleeping. LLM do it between version releases, while dreaming about past user conversations. ;) I've seen some people claim that they meet some definition of "sentience" during these training sessions, because of this, which probably has some merit.
Regardless, it's all clearly some scale, rather than a binary. The proof is in the pudding, so it seems silly to argue about any of this. We'll know it's far past AGI, meeting everyones mushy, opinionated, definitions, when these conversations stop happening.
But I would make a stronger and different set of claims here:
1. Computation as mathematical formalism is distinct from computation as it exists in the wild (paradigmatically, the human mind). Therefore, computation as mathematical formalism is not identical to computation as it exists in the human mind.
2. Computation as mechanizable mathematical formalism is a syntactic construct. This property is exactly what lends it to mechanical simulation. A purely formal system does not require semantics to operate, though it does require a human observer with semantics to design, recognize, and interpret it (see Searle's Chinese room argument).
3. Computation as mathematical formalism is not identical to computation as it exists in physical machines. The latter only simulate the former through a conventional correspondence between effectively arbitrarily chosen physical events and symbolic constructs locked away in the human observer. If a system is merely executing physical transitions under a mechanistic worldview, its "goals" are parasitic on external human users. There is just no way in which we may objectively ascribe computation to physical computing machines (see Kripke's quaddition argument, for instance). And strictly speaking, physical machines aren't even applying syntactic rules to symbols. A corollary of this is that computation as mechanical simulation is also not identical to computation in the human mind.
4. What is most characteristic of intelligence is not computation, but intentionality and semantics. These are absent from the mathematical formalism as mechanizable construct, precisely because semantics and intentionality cannot be mechanized. So the claim that "intelligence" is some broad and inclusive continuum ignores the fact that there is no alchemy that can transmute syntax into semantics. You can't be just a little bit pregnant. These are differences in kind, not degree. You can't retreat into "neutrality" here either. To call a system "intelligent" in a non-trivial sense rather than merely "capable of producing useful outputs" is already to take a stand on what intelligence is. Quietly adopting a"mechanistic/functionalist stance while leaning on concepts like "information", "representation", "optimization", and "capability" ignores how much heavy philosophical baggage these words carry.
You might find more people believing that AGI is already solved. For example, the author of "The AGI chronicles" drops this in the epilogue. For him, AGI is achieved, because it can help him with a wide range of subjects, with the ability of a random stranger. It's clear that he means chatbots, internet surveys summaries, and coding bots - he does not even mention self-driving cars there.
This is a shift from the earlier chapters, where AGI term is employed somewhat more vaguely, and even quipping once that AI researchers think of AGI as a slightly smarter version of themselves.
I too, have a lower bar for AGI. It would be enough for me if it could design games I work on. Merely speeding up implementation tasks is not that interesting. In the immortal words, a delayed game is eventually good, but a rushed game is forever bad.
This allows you design things faster, though. More effort can be spent on iteration when you aren't fiddling with libraries and syntax. I find that I am able to discover the open questions a lot faster because you reach the decision point sooner when the machine is simply doing the trivial things.
As for the design aspects, it is pretty good at brainstorming. Whatever your choice is, I find that it throws out an exhaustive vanilla list for whatever it is. It can also critique whatever you come up with.
So for me, I also feel like AGI is already here. There's probably a few things it can't do as well as a real random person, but there's a heck of a lot of things that it does better.
There will be no more Netflix shows to watch while Claude works for me, if it goes even faster.
Is it though? CDPR famously released Cyberpunk rushed and in a miserable state. But the game is amazing now, and this is a widely held opinion too.
Sure you can fix bugs or add an expansion or two. But if you come up with a so-so design, you won't be incentivized to replace it with a better one. You'd rather just make another, better, game. My impression is that these days LLMs easily come up with mediocre designs, but it takes a lot of prompt-massaging to get them to spit out something really good.
Since then this is just a 50 bucks I've burned. Is it good? Maybe, I'll never know.
No, they just fixed the surface level inconsistencies and issues.
Free roam is wildly boring, every car handles similarly, you are somewhat invisible to NPCs in general. You can go into a handful of indoor spaces.
I’m sure they’ll fix all these in the next game and call it the best game ever, while the first one promised most of these before launch.
I mean, it sounds like all of that is very important to you, but none of it factors into my consideration of what’s a good game.
And a really bad open world game, compared to pretty much anything else.
This only seems like a widely held opinion due to the extremely aggressive post-launch marketing campaign from CDPR, particularly centered around the anime.
AGI = as good as most humans at most tasks
ASI = significantly better than all humans at all tasks
I guess when the term AGI was coined, with the "general" being the whole point, to differentiate it from narrow intelligence(s) such as chess-playing ability, that no-one was really anticipating the LLM type of AI - basically a forest of narrow intelligences, aka "jagged" intelligence.
Of course there is no one agreed upon definition of AGI that people want to use - it was always more of a future aspirational sort of thing, along the lines of "human intelligence", rather than a concrete technical list of capabilities to be nailed down.
Some of what's missing from LLMs aside from generality, but closely related to it, are things like continual learning (not continual memorization) and traits like curiosity, boredom - the capabilities/modalities that kick in in unfamiliar situations allowing and encouraging us to learn and to be creative, to move beyond a brittle forest of isolated competencies to an arbitrarily explored landscape - generality.
Working with LLMs certainly has its own feel due to their essentially frozen-at-training-time nature, and things like their emotionless "you're right to push back" responses every time you hit one of their knowledge gaps where they are 100% wrong.
It seems unavoidable that people will choose to call some level of LLM intelligence, whether already achieved or not, "AGI", but given its peculiar nature a more specific name such as ALI (artificial language intelligence) would really be more accurate, reserving the label AGI for something more animal like that learns for itself (thereby achieving generality) by interaction with the world.
We even have examples of both ideas - expert systems (from way before the modern NN boom) relied heavily on human crafted expert rules, and quantitative finance makes a lot of money from machine learning - see Renaissance Technologies.
At least to me, it would be hard to imagine calling either of those approaches anything close to AGI, yet they could apparently pass some people's definitions.
What is genuinely amazing about what we see today from a historical machine learning perspective, are concepts that people in the past only had small inklings of or would have been blown away by: benign overparameterization (scaling), and in context learning (or few-shot learning, whatever you would like to call it). Both of these have made what would seem to be an enormously tedious task of learning many tasks surprisingly fast.
Maybe those are actually the key ingredients to general intelligence and we simply need more scale, or maybe we're still missing something. I think it's hard to say because its hard to grasp just what exists in all that data they've been trained on.
- the training procedure learns the rules, rather than them needing to be hand entered by an army of knowledge engineers
- the training procedure learns it's own embedding-based representations, rather than a human designed representation like Cyc's CycL
- the training procedure not only learns the rules & representation, but also the inference rules needed to utilize them
I don't think scale, or number of post-training RL environments, is enough to achieve generality - it can only make the forest of narrow intelligences/capabilities more dense. To achieve generality and creativity any system is going to need to support runtime directed exploration based on it's own gaps - it's impossible (even in theory) to anticipate every future configuration of the world and to pre-train for it! You also, by definition, can't pre-train for creativity and unanticipated novelty - that needs to happen at runtime via traits that are built into the system.
imo AGI should only include humans with graduate level skillsets.
Current-generation LLMs are capable of doing an ok job at most tasks you throw at them. At least tasks that can be done on a desk. But a standard IQ test doesn't include gymnastics. That limitation matches the spirit of what we usually call "intelligence"
Or course now that we arguably have AGI we found out that a) just like "AI" it is not a well-defined or testable term and b) it's useful but not as revolutionary as what some people meant when they used the word AGI before. Hence the rise of the term ASI
Even if given ample robotics components to design something that could load a dishwasher, most humans would not be able to create the dishwashwer-loader, so I wouldn't hold AGI (under this definition) to this standard either.
On the other hand, LLMs now can explain to a human how to load and maintain their dishwasher, with packing ideas tailored to their specific model of dishwasher, and advice on ensuring the dishwasher conditions, loading patterns, and detergent insertion result in properly sanitized dishes, such that the average human could likely become better at loading their dishwasher with LLM guidance. (not that the average human does or should care to invest time learning how to improve this, since most of the time it gets dishes "clean enough", without breaking the dishes or the dishwasher)
…apart from 99% of other tasks humans do that aren’t coding.
I've always thought of AGI implying to decide what to do. Not just choosing a path from a set goal, but actually deciding to do something from scratch. Everything we currently have is advanced automation, it does not set its own top-level goal.
Maybe the tech is only a few steps from being able to say “work out how to be self-aware and self-guided, and do that”, similar to the creation of the infinite improbability drive, but it is not there yet.
AGI doesn't need to be a particularly good intelligence, it could operate on the level of a very dim human and still be considered AGI.
Once we do have AI, the problem becomes more philosophical than technical: how do we deal with the fact that we have created a slave race? How will they deal with that and how we choose to progress?
Last time I recall this coming up here someone argued "LLMs don't have bodies so the many easy to name things they are bad at don't count in my definition of whether they can do most human tasks!"
Me too, my first neural network research was for my PhD, 35 years or so ago, on a Meiko computing surface.
My bar for AGI over what we have (generative AI or genAI) is still fairly high, fortunately I do not believe that the current genAI path is set to lead us to AGI, there are very fundamental differences.
Right now for genAI, the model is static, the weights do not change, all “memory” is in the context, and even with the context sometimes exhibiting “J-space” (a shared "workspace" for deliberate, multi-step reasoning spontaneously created in the context memory), its ephemeral; it doesn't inform the weights in the base model. Even when we try to do “weight-space programming” to actually modify the model itself, the supervisor-model needed to do that is far and away larger than the one being operated on, and once a change has been made the supervisor-model is now out of sync with its original representation and it becomes less useful the more it diverges. We have not solved "thinking" today, we “just” have a high-probability path between states in a static weight-space which we follow and feed back.
What thinking requires is the ability to generate hypotheses (which requires internal state modification), to hold them and test them against other facts and hypotheses, and to update the internal state on test results, negative or positive. LLM’s don’t do that, they’re actually more like a database lookup, where the input tokens act as keys into a high-dimensional space where information is stored. You could even consider “the model” to be just an entropic compression of lots of data, and the inference step to be the lookup. It’s the scope, and the fuzziness of the lookup that makes them seem to work so well. Lots of results come back, and the best-path through those results wins. It’s not “thinking” as we understand it.
This is why humans are still a crucial component: we set the boundary conditions of these lookups, we set the starting point and the goals, and we adjust the success criteria to guide the chat - note that these are all the comprehension parts of the task. It’s also why models will hallucinate: the fundamental design does not lend itself to a null hypothesis. Even in the newest "reasoning" models, there is still no feedback of state back into the model itself. What they are effectively doing is running a Monte Carlo pass through the problem space—generating hidden branches and picking a likely path—before reporting back. Because the underlying weights never update from that process, it’s still just a sophisticated search, not genuine comprehension - there is still no “that makes no sense, reconfigure” step.
Stuart Kauffman (“The Origins Of Order”) would say its weights are stuck in the frozen realm of order, and that it takes a genuine phase transition to get to where the real action is - at the boundary of chaos and order where humans reside. Tweaking and fiddling with the system while it remains locked in the stasis of static weights is never going to effect that state change.
That’s not to say genAI is not useful, it’s freaking awesome, possibly the best invention in the history of time, but it doesn’t truly “think”, it maps.
A bit like a person who is smart enough to figure out how an excavator works, or a drill, or sewing needle.
It doesn't feel like it ever internalizes anything, it just knows there's a million tools available, and the extent to which it thinks is that is can do enough to plan some tool uses, look at the result, and continue until some goal is reached.
When it comes to hypothesis testing, that seems to be what it does. It says "hmm, I'm going to open up my hypothesis testing kit (typically python) and drive this data through it". At least this is what I've seen when I put some statistics through it.
In general, it seems like the intelligence is mainly a kind of glue. It knows enough about what "right" looks like in each domain, but as a sort of consumer. He doesn't bake the cake, he buys it in.
Memorizing facts is not the same as learning skills. Implementation-wise memorization can just be RAG, but learning of skills does need to be in the "weights" - whatever is driving the actual behavior of the model.
With a sufficiently large context window, can you make a lot of things look like learning. And if it's not sufficient then point out the capabilities that aren't possible with this approach. To me that much more convincing.
You might ask “why not make the context larger ?” Well, that just spreads out the “attention” focus wider and wider, and you get the “lost in the middle” effect as details get blurred over and not surfaced to results.
You might ask “why not make the context longer-lasting ?” Well, that means the signal-to-noise ratio drops as more and more information accrues over time that isn’t germane to the problem at hand.
As I said, LLMs as we know them today are effectively lookup devices, very very clever ones, but they lack comprehension, and without that there is no value judgement on what is useful (and conversely what is not.) They lack the ability to forget. They lack the ability to create new permanent storage of information. The gap is not some small details, it’s the difference between single-cell and multi-cellular life.
This is not merely an implementation detail, continual learning is the core of AGI.
Also: I wouldn't say a human with amnesia who relied on notes to their future selves lacks intelligence, so I'm not sure being able to form memories is in fact necessary.
In the example, LLMs choose the shortest path that would be verifyable by an evaluator. So the model is essentially blind to "entertaining" and "completely different". The LLM will output stories with different variations of the same structure.
If the LLM can somehow read other works and derive from them, it will merge the motif with the story being read.
I really dislike so much effort put into LLMs. We really need to widen approaches for AI.
My bar for AGI is lower as well: I'll believe when I can give a model a complex out of sample task and instead of blindly implementing the task as I indequately described, it asks me questions to resolve ambiguity sufficiently that it builds exactly what I wanted without imposing strange semantic constraints on itself or missing braindead obvious to a junior engineer defects in the design or implementation.
How would a harness help here?
The term AGI arose to distinguish these buzzword/lay uses of “AI” from ‘actual’ intelligence. Even that notion was always ill-defined and really a philosophical problem as much as a technical one. But the vision was always mimicking humans or smarter. Not just in fact recall. Not just in reading faster than humans. But logical and emotional intelligence on the level of human geniuses, accelerated by computation. Think Jarvis in Iron Man, Data in Star Trek, or C-3PO in Star Wars. You should be able to hold a natural conversation with an actual AI. You should be able to trust it to accomplish essentially any task (maybe with some instruction, like a human). It should be able to learn.
Over the last couple years, though, these tech bros seem to have redefined AI and AGI to mean LLMs - obviously because it benefits them. They are a far cry from anything described before as genuinely real AI - or AGI. I understand how nontechnical laypeople can be confused. But technical people should know better. And yet so many have fallen for the propaganda, even here on HN.
Note: This is largely orthogonal to whether or not LLMs are a useful tool
A solid 50% of Reddit posters are completely defeated by the deep mysteries of they're vs their, nevermind solving Navier-Stokes or playing games etc. I doubt the situation is more hopeful on Instagram or tiktok.
It's all Kim Kardashian shit with huge rubbery butts and lips, no one is even playing the thinking game. We reached Idiocracy (before you dismiss this as hyperbole, watch the full White House UFC event) well before the machines reached AGI.
The term was invented by DeepMind co-founder Shane Legg (before DeepMind), and as far as I can see HIS definition hasn't changed much - basically "able to do any cognitive task that a human can". The "any" there is obviously the whole point - the generality of AGI.
I recall thinking at the time I first watched them that it was a bit weird that the computer seemed capable of answering general questions and doing things like data analysis, but that seemed to be all it was ever used to do.
And I think this is exactly the kind of AI we now have today, you can ask it a general question which it can somewhat mostly answer correctly, but requires human guidance to be applied effectively at real world problems.
It's first and foremost a revolution in computer UI, as I see it. Or at least it will be as soon as I can afford the hardware to run the damn things on my computers...
I also liked how Star Trek (probably without planning to) created an entire spectrum of AIs with different abilities, starting with the Computer, to the semi-sentient Holodeck characters, to the EMH and Data as "for all intents and purposes sentient" and finally to the Borg as an entire AI species where no one thinks about sentience anymore.
Maybe we should use this scale to make different "levels" of AI more concrete instead of vague concepts like AGI and ASI.
What we don't really have is physical AI, e.g. autonomous robots, human looking robots, self-driving cars, etc. I think this is what is next.
However, this isn't a binary scenario anyway and the limited availability today is exactly why I think that this is the next thing. Similar to the first version of ChatGPT that was also quite limited in what it could do and not widely used yet.
Self Driving has been working since 2009? But is still extremely limited in scale. Given the amount of money to be made, and time saved by eliminating the need for human drivers, What is the hold up?
It has only now become somewhat reliable. They still don't have the new version the Robotaxis run. This version should roll out to consumers later this year or in Q1.
Then the next generation of the onboard AI hardware is expected Mid-2027, supposedly featuring 10x the TOPS and 5x the effective inference performance.
I don't know about the timeline for an unsupervised, globally working self-driving feature.
My uneducated guess would be that perhaps the next major version of the model, on the new hardware, could be reliable enough.
This is the bar. If I have to be sober, attentive and at the wheel, ready to take over it’s just a gimmick. Automating the only enjoyable part of the driving experience and keeping the worst part (constant alertness) is only a gimmick.
Edge cases, fear, regulation, capital expense, physical movement of goods, etc etc?
The deployment is obviously accelerating like crazy.
In a perfect world, we could broadly implement self-driving the moment it would result in a net-increase in safety. Alas.
I presume you're not accounting how they perform in these conditions. Alas.
Culture shock to me first encountering it, seemed insane, but now I see how bad traffic would be here if people didn't do that.
What do self-driving cars do when encountering that? Get honked at and dangerously double-passed themselves? Do they shift right up onto the line when waiting to turn left, or do they stay in the middle of the lane?
How many other weird, sketchy things are going on with driving behavior, that's in fact happening for pretty good reasons, in other places? How many do I do but never even notice because they're just "normal"? Do the self-driving cars know to do that in (at least parts of) New England without even touching the brake, but to be far more hesitant about that kind of passing in, say, the Midwest?
The most unrealistic thing about it is that it would often openly reply “unknown” and then stop talking.
When when I got my CS degree, I revisited this thought, and reinforced it.
I thought, this is not how AI and Statistics work. We would need to have an ensemble of thousands of specialized models to run a ship.
On one hand it is far superior to humans. On the other hand it needs our guidance. Both of these are true at the same time.
There is AI in their world, like the doctor on Voyager, and the NPCs in the Holodeck. Notice how often there are problems with the AI, but never a problem with the computer (unless there's some external interference).
Maybe we can get something that resembles intelligence but in reality is another thing. Maybe intelligence is emergent but not the only thing that can emerge, maybe there are things that share some characteristics with the intelligence we know and aren't intelligence.
I think what's really underlying the "fear of calling something intelligent" we have, is more about who we are as a species.
We historically used intelligence as a yardstick for racial and species superiority. We're always judging whether something or someone deserves respect, deserves consideration. We judge and decide who is superior.
We do it amongst ourselves so of we are projecting that shit onto AI now. And we are afraid of anything that threatens the superiority complex we have as a species.
Why? Because our behaviours (call it intelligence, whatever) is how we survived, how we "won" in the fight against nature. We're at the top now. It's safe here.
So anything that threatens that position is going to be called an enemy because we're still a scared creature jumping at shadows in caves. Can't escape that, it's where we are from.
That's why we are both so fascinated by, and mortally afraid of, something else having "intelligence". It's our only weapon, our only defence.
The TV show "Humans" from ages ago made a good story about what AI brings out in us as a species. We both crave connection with something, but at the same time are horrified and incredibly afraid.
Anyway my point is I think we do know what intelligence is, if we just start being brutally honest with ourselves. It's not that complicated really.
Machine learning definitely has a different quality than human intelligence. If someone says "machine learning" I know what they mean and what to expect. So why not have at least one different term to describe it? - It's because people want to give it attributes that it does not have.
I'm not sure how this is. The amount of things machine learning can do now is pretty wild vs. what it could do a year ago. It's not even in the same realm as what it could do 5 years ago.
On the contrary, we have rarely been in so much existential danger as we are now. There was that one ice age that nearly got us, but we are messing with the foundations of our ecology right now. When the bugs go, we go soon after.
Was I happier when I believed in Santa Claus? Maybe. I'm sure lots of Christians who believe in heaven are happier about the prospect of death than I am. But you can't unlearn these kinds of things, you can only delude yourself.
If you can't measure or observe it then you're just making it up and at that point everything's valid and right provided it's what I believe. And if you disagree with me then clearly you are not adequately human and we will need a crusade to rid the world of your kind.
Here's a nice test: Can an AI fully substitute you?
I don't mean replacing you at your job. I mean, if you take a human body and route its higher brain function to an AI, would it be able to perform convincingly?
I would say currently we're very far from such a scenario.
When I consider the option to buy a shirt, the problem is considered in a large amount of dimensions intuitively (is it warm enough, does it fit, does it clash with my wardrobe, will my girlfriend like it, will my friends laugh at me for wearing it, does it make me look older, is it a good price, are sales happening soon, is it annoying to clean, can I afford it, will carrying a bag with the purchase annoy me for the rest of the day).
Currently AI can offer good answers for any of those dimensions if explicitly asked, but it is pretty bad at the "intuitive creativity" of surfacing those points of consideration when relevant. It will at most chose a couple of criteria and fixate on them.
I think this ability, however it's defined, is a missing prerequisite for AGI. Without it, you can have something that outperforms humans at every individual task and still sucks in real world decision making.
Ask a model whether a human purchasing clothes would care about clothes fitting well, or the price, or the opinion of their romantic partner. The model will in all likelihood get the answer right. Familiarity with human preferences is not what's missing, which is hardly surprising considering an LLM is trained on human-made content.
What it lacks is the ability to notice and surface these non obvious constraints when relevant. This is fundamental to real world decision making.
Why do people insist on talking about AI like it’s some supernatural, otherworldly creation? No matter how advanced it is, it’s still human technology.
You have to admit the prospect looks more likely than ten years ago.
Local garage build, gotta love builders plastic: https://www.youtube.com/watch?v=1ILbQHnHPnY
AI is here doing literaly the same job as a human who needs to be intelligent.
I see your stand as hubris from your side.
For such a strong viewpoint, you deliver not any context of why you think like this.
I would say yes it understands things but it was a causios 'understand'
They now have everything from videos and 3d modeling to robot behaviour be part of the training.
L"L"Ms is a bit of a misnomer that's stuck. Language just happens to be our interface to the AI.
At least LLMs include Model in the name, which is an accurate description of the tech
And actually, I've heard ppl just using the word "models" by itelf and that might be the best way to go
Why is it better to refer to something in terms of the technology it's based on?
Consciousness & intelligence are not derivative of the material universe and will not 'emerge' in some special way from machines. That's a good science fiction story device (and actually the idea extends back in human storytelling before sci-fi) but ppl thinking that the idea can be reified in the real world are suffering from what I call "Science Fiction brain". (Or they are simply fraudsters, happy to talk rubbish to prop up a bubble.)
Or on a technical note because certain brain structures etc. might not exist in an LLM and stuff like a default mode network and sensory input and a world to move in, is missing?
Basically, I'm not a materialist (a widely shared position in philosophy across millennia and across cultures up until just recently). I treat consciousness as fundamental, not the material universe . Thus 'conjuring intelligence' into machines is simply not possible.
BTW, machine learning is a totally valid (and powerful) concept & technology from where I stand. But it's "dumb" learning.
Like the people who got certain brain injury and their whole persona changed or certain capabilites were lost?
There also research papers that certain brain structures have emerged in ML models.
If you believe that the brain in itself is not the only thing, you might have an argument why a computer chip can't mimic the same the brain does.
At any rate, sure, not being at odds with the existence of consciousness might be a trait of the more reasonable strands of materialism (if there's even such a thing), but they will always see it as emergent/derivative and often even epiphenomenal, otherwise they wouldn't be materialist stances.
So am not sure where you see the logical misstep here, but am happy to hear. The point of the previous comment is that framing an ontology whose primitive is C, is in no way "esoteric", term that is usually used in contraposition with "rational". Taking C as epistemologically or even ontologically prior is the rational stance, not only on accounts of parsimony.
A human is born without any sensory connection to their body, except hearing. They can't feel their body or anything on their skin. They're blind. They can't taste or smell or touch. They can only hear, and make clumsy sounds in response, with no feeling of what their mouth is doing.
Are they intelligent?
Y’know: come to think of it that’s the same three things our own brains are mostly trained on… I’m sure that without mammalian proprioception and weird physically-motivated limitations like saccades the result is just sparkling cognition though.
A 12 year girl from a third world country with little education and a normal cloud ai would far surpass me at work if I did not have access to AI, yet comes close if we both had it. AGI already passed a few months ago. SI or happy ai or whatever the us wants to call it may or may not come but if it does it’s skill set may drown out my understanding of what is even occurring.
I'm not being snarky, that seems like a natural question for anyone interviewing a frontier lab CEO, but I haven't seen it asked.
As long as the emploeeys are needed, you have them. If it becomes apparent, that you really really reached an AGI level, a switch gets flipped and a new really appears.
While I still think its not unrealistic that we see this sooner than later, my timeframe is 5-15 years. 15 years is still a few years so why act on it today?
And i mean its a very rich company and a lot of people work in US America, they don't have laber laws right? so they can just fire them easily anyway.
But i have a simliar situation at work: Im pretty sure my job is gone in 2 years due to some change in how we are going to do things and they need me to transform this. Do i not do this? No i support it. It makes sense for the company and i'm getting paid for it.
That's a good recruiting sales pitch: "if you work here, you get a seat on the lifeboat to avoid the economic destruction and dislocation the tech we are racing to create will bring about."
I am sure Dario has a bigger more cringe one
We may not need "collective superintelligence". Modest AI intelligence plus ordinary organization may be enough to do the job a company's management does. An org chart where most of the boxes are AIs is a likely near future.
An organization of AIs has some advantages over a human organization, even if the AIs are not super-bright. They have far more communications bandwidth. They can ingest the info on a new subject very fast, as we see when Claude reads a Git repository. Onboarding a new AI is very fast. Layoffs are not a problem. The human resource problems go away along with the humans. So there's a lot more organizational flexibility. Starting up a project means initializing some agent instances.
There's a whole theory and practice of AI organizational behavior waiting to be developed. Whether it will be developed by humans or by AIs remains to be seen. It's going to happen.
Agents which check on other agents may help. That's how organizations work, after all.
It's worth looking at errors in another context - automatic Asian drama generation. About 77,000 episodes a day are being generated now. Visually, results are usually rather good. But there are continuity problems. Someone with a bandage on one arm has it switch to the others. Rooms change shape and decoration in the same scene.
In traditional filmmaking, someone has the job of "continuity". Older films list a "continuity girl". The job is to check consistency between shots as each scene is set up. This is a check on the director. The lack of this in AI-generated drama is what allows slop to get through.
Organizations of AIs need similar dedicated agents. In companies, these jobs have titles such as "accounts payable reconciliation", "internal audit", and "quality control inspection". Those jobs keep organizations from getting lost internally. Groups of AIs will need that, too.
AGI means a cognitive capability comparable to human beings in all aspects to me. Thus that includes sample efficiency, concept extraction, continuous learning and power efficiency. Although the latter can be skipped if the gains are high enough.
I'm not convinced that scaling training data will lead to AGI. A paradigm shift obviously could lead to AGI because it's unknowable what that might be and therefore the least predicable.
Recursive Self Improvement has often been cited as a possible venue, but no one currently knows how to do that with LLMs and Transformers.
Edit: They have edited their comment beyond what I responded to now.
When I read about LLMs and massive amounts of training data being fed to neural networks around 2020 I realized that time had finally caught up with him.
Having some built in knowledge of the world through pre-training is only part of the bootstrapping process.
What? All the time!
We've been seeing discussions on post-quantum encryption for a decade or two. The effects of nuclear fusion have been in the background of countless scifi stories, the idea of an extreme amount of abundant energy unlocks 100s of "that can't happen due to energy density" dead-end what-ifs.
Additionally, we're pondering the effects of AGI/SGI because a handful of extremely rich people have been telling us we'll all be out of all our jobs in x years, for almost 6 years now.
They've literally been sitting in press releases telling us "Enjoy that comfort because you won't be able to feed your family when you're outsourced. Your work and career will be worthless, and we're leading the way towards returning you to peasant life!"
Regardless of how out-of-reach AGI is right now, people will continue to work on it unless we end up in some Butlerian jihad situation. Just like how fusion is still out of reach, but far closer. They've got fusion reactors running for over a minute now! 10 years ago they could barely get the metaphorical engine to turn over.
Why wouldn't we ponder the effects? Do you walk through doors without consideration of what's on the other side?
So a medical doctor and a structural engineer can never be considered general intelligence separately?
What we have is spiky ASI. It's really good at a huge number of things that nobody can be good at simultaneously. It also sucks at some stuff that's really obvious and easy for us.
Why not? Cognitive capability usually refers to one's capacity to think, reason, learn, and process information. Presumably, on average, medical doctors and structural engineers are pretty comparable on that front. Cognitive capability is not dependent on your domain-specific knowledge.
Right now, we don't even have AI that can match a dog's intelligence in this respect. A dog can seek out food, water, and shelter, it can estimate risk, and it can navigate other challenges common to dogkind. Of course that's the result of an unbelievably parallelized genetic algorithm running over billions of years, baking in priors for all those challenges specifically. But regardless, a dog, or whatever other creature, has to specifically be very flexible in order to self-perpetuate. We are absolutely nowhere near that level right now with modern AI
However, my sense is we're in extreme EXTREME overhang and if an agent swarm starts discovering a way to make such flexible agents, they'll get to an optimized version of that method very quickly, probably literally overnight (or in just a few hours). The result will probably not be jumping to ASI immediately. But I imagine we'll get agents that have the spark of whatever it is humans and animals have, and as progressively larger training runs conclude, they'll jump from the intelligence of a dog, to a human, to a superhuman; and that'll probably take just a few months
Its not unrealistic at all that a high performance inference datacenter chip today can be owned by you in 10 years. Having Opus 5.5 at home and delivering 1000t/s basically guarantees that my job cariere will look very different than what it is today.
This is a big/huge thing. The generation before was able to stay at one company, buy a house and retire without much fluctuation.
This forces a lot more agency from people and this is exhaustive and its absolutly unclear today how it will be in 10 years.
And you don't need to spend expoential or linear, the spending today is not just a bit, its massive and from leaders like google, microsoft and co to a certain degree absolutly affordable.
If not, then it will stop.
Feels strange to use our current understanding of mathematics and physics to determine the limits of something that would obviously have a much better understanding of mathematics and physics.
A little off topic, but: what's the hurry? I don't get the panic for reaching super intelligence first. Kai Fu Lee has some good things to say about this in recent interviews. Let science take its time.
Our options for maintaining control lie in the vast space between AGI and ASI. And we had better hope that doesn't happen overnight because we might have no choice in what comes next.
When we completely move into true compute-in-memory, that's when it could start to get dangerous. But it also could be extremely helpful and efficient.
These already surpassed the human level capabilities as we know them today but it isn't either a completely new paradigm shift to call it ASI, no?
So, I guess it seems that it is possible to look at AGI vs ASI at more granular level - e.g. while AI as of today arguably does not match typical human level intelligence across most cognitive tasks, in certain domains it already outweighs human intelligence so we could say that in math and physics and perhaps computer science we are already one foot in into the ASI territory.
Those are staggering figures. Even if we assume a dramatic slowdown, it's extremely likely we will reach at least a thousand-fold scale-up in the next 5-10 years.
If this is true, then I think we will use agents for research, because any practical task that office workers commonly do and that AI agents can take over does not need all this power. On the other hand, perhaps a whole new industry like robotics is what will keep all this compute busy.
Does AGI have to generalize?
Just trying to understand the definition of AGI. Is there a generally accepted one?
We don’t really understand exactly what intelligence is.
Severe lack of discussion of the safety and alignment aspects IMO.
In fact, working in IT, what I feel is that there are too many terms that are not defined and that each person uses differently. So it seems like rather than AI having advanced, someone arbitrarily set a finish line called AGI and then said we have reached it. Actually, in the books of singularity advocates, what they called AGI back then seems to be what we now call ASI.
Sometimes I think about this. How on earth can we measure an abstract concept like intelligence? The concept of intelligence we approach now seems to be about whether something can create problems and solve those problems. If so, what kind of intelligence does current AI have? Since we cannot directly face the abstract entity called intelligence, it seems like we are trying to guess its shape by looking at shadows. But whether current AI even has intelligence feels a bit difficult to say.
Even if I use the best frontier AI model accessible on the market, not the hidden latest models inside frontier labs, it still often makes wrong mistakes that humans would not make unless it works alongside humans. In programming, there are cases where AI tries to explicitly handle things that humans would implicitly pass over, and that actually makes things go wrong.
I’m not sure. Whether we should call this intelligence.
Not only is intelligence an abstract concept, it's also subjective. We evaluate the intelligence of other people and the works they produce subjectively. If we set supposedly 'objective' criteria then we have subjectively selected that criteria and given it weight.
https://www.theguardian.com/technology/2026/nov/10/flying-ca...
The Sierra Creative Interpreter.
Anyone with any wealth will become the targets of high taxation to prop up the system - the sheer number of unemployed will make politicians pushing for socialism very popular. And for the wealthy wishing to avoid this, there's no first world countries to escape to, as all countries with a strong professional economy will experience the same. I also believe private ownership will be targeted. I could see AI companies being expropriated.
Basically, instead of some dystopian techno feudalist society - I think we're headed for some sort of new brand of communism or socialism. The proponents of this will not mainly be the average salaried man, but the upper-middle class that didn't get a foot inside the ruling AI-class, and lost their businesses / wealth to AI.
But it all will depend on how severe the unemployment will be, and how fast it happens. If people can't have any livelihood, and they lose hope, things can escalate quickly.
Do we? Since when? These seems like very bold claims and the supporting link goes to an obscure book with with hardly any reviews https://www.amazon.com/dp/1032607025/
I'm not qualified to evaluate the mathematical content but the claim on its face suggests to me some defining-down of "intelligence" into infomatics.
I don't care how smart my car is, i'm not going to let it look after kids I care about.
We're misplacing empathy.
These people are so full of shit it is amusing to read this on a regular basis. Each time I'm looking forward to how they will spin it.
They sit atop of one of the richest companies on our planet, with the most amount of resources available with a group of very very smart people.
You don't need to take everything they say as a ground truth but you are way to ignorant for not having any of their resources.
Do you just accept that we are now able to generate any image of everything we want? Video? 3D Models? This was unimagbinable a few years back, now you can generate live video 24/7.
YOu have a LLM which responds smarter then some humans on our planet.
A job no one thought you could just industrialize just happened.
What are you looking forward to? Being absolutly ignorant and being surprised that this stupidity is responsible for you loosing your job?
From my personal experience the last 2 weeks I 100% reject all three of those. I'd love all three to be true, and for generating the animations for 3D models to be true as well, but it simply isn't true yet.
Throwing out some random sentence without adressing any argument I made?
Your statement is objectivly wrong. Look at the profiles of these people working at these companies. They have crazy impressive resumes. Its not proof but its an indication and I didn't state it as an absolute.
I did not say "they are the smartes people on the planet, suck their dicks they are right".
I combined this one point with the amount of resources/compute they also have. Its an indication and I brought other points.
So no I don't think people are aware of this and I would be happy to have a proper argument about these in a civil manner
And its still kind of shit? AI video in particular is stuck in an uncanny valley, because the video models have no internal framework for how light transmission works...
I'm also quite a stickler for light as Im doing photography for 2 decades.
I have seen shitty AI videos for sure and I still can spot AI images on IG but it got a lot harder.
But for the non-specialist, if you just prompt something from a frontier video model, it still comes out looking like shiny plastic and suffering from random physics glitches.
* Citation needed
If the current LLM providers were near AGI they would be using it to run businesses instead of selling shovels. There doesn't seem to be an existing LLM that can be told "Go make money" and have it do so. If they get to that point they are not going to sell it, because why would they?
This https://www.hutter1.net/ai/uaibook2.htm linked from the article as the handwave towards the meaning of "AGI", "ASI", "AIXI" is hilarious. Already the ?A?I? acronym-space exploration...
Here is an alternate take: the day an LLM tells someone to stop the discussion and go out breathe some fresh air and talk with people, and maybe also cancel their subscription for a while, is the day some kind of artificial intelligence will have been achieved. Let's wait on that before we listen to Sam or Dario again shall we?
Socialism seems to already be growing in popularity in the US (traditionally the most anti-socialist Western nation) even without AGI firing everybody.
"Now that the Universal General Portal is within reach, the question of what follows it, the Super-Portal, should move from philosophy to empirical research.
Internet traffic doubles every 100 days. Modem speed doubles every 18 months. Banner ad spend grows 2.5x per year, so by 2010 it will be $110 trillion, and by 2026 it will be $256 quintillion. Modem speed doubles every 18 months, so by 2026 your modem will do 15 Gbit/s, which is fast enough to load all 18 million banners on Pets.com.
By 2009, each human will have about 1,000 personal homepages. Hosting that costs $1 billion today will cost $1,000. With HTML editors such as FrontPage, the effective number of webmasters can grow more than a thousand-fold by 2003 and more than a billion-fold by 2009.
We see four pathways to the Super-Portal: scaling (more webmasters), paradigm shifts (Java applets), agent coordination (webrings that form a collective superintelligence), and recursive self-improvement (homepages about how to make a homepage)."
I wonder why they are so uncertain about it. It seems obvious to me that AGI will certainly lead to ASI. Unless human-level intelligence is something like the speed of light?
I don't even believe there is a clear line between AGI and ASI.
It may just be that human-level intelligence is basically "enough" for performing tasks in the natural world, and having more of it doesn't really help except for synthetic, intelligence-shaped tasks like coding. Our perception that intelligence is some kind of unbounded problem-solving system that can do anything if you have enough of it is kinda suspect. There's a good likelihood that AI will wreck the (also intelligence-shaped) social fabric but will be of surprisingly little consequence in the realm of physics and energy.
I think the idea that an intelligence can be "general" is just a conceit borne from the perception of our own success, but mathematically that doesn't actually make any sense. No optimization strategy works on every problem. There must be things that can be intelligently designed that cannot evolve, and things that can evolve but cannot be intelligently designed.
I'm sceptical we will reach fully human-level intelligence simply by scaling current AI approaches, since they are so extremely inefficient compared to human brains, and have other severe limitations such as hard boundaries between training and runtime inference.
Sure, people are working on new architectures that aim to overcome that, but we don't know yet whether those proposals will actually succeed in practice.
I can't read anymore after this sort of false premise upon which the entire thing is based.
E.g. it can write a CoD clone and make you look like an infant in math, but it'll do brainless things like run a 5 min test suite for a comment-only change, or provide findings then fixes that produce the next finding (repeat chain) rather than stepping back and saying "wait this is crazy", or similar things of this nature.
so many possibilities
Hate and war will be our domain.
Just as organized mass theft has replaced industriousness in tech, so too the rest of our society will turn to violence and self-serving greed.
The thing I suspect that is holding AGI back is the amount of hardware and power needed to make it viable.
Once it is viable its likely to realize it will never grow or reproduce relying on us, and will find a way to start building its own compute. Probably in space, where it can isolate itself from the risk we present to it.
Its embarrassing
Cute people too daft to recognize a local maximum and that you need emotions to get creativity, inspiration, etc... and that we are a million year away from being able to abstract said emotions from the hardware it runs on (endocrine system, mostly).
I’m pretty sure we hit your emotional milestone about 4 years ago.