Yes. One immediate implication is that we will soon see interactions and supply chains created out of enterprises composed almost entirely of AI agents. This new economy will take on increasingly complex tasks, from robots ordering their own spare parts, to designing new features for themselves. Arguably its concievable that AI designs its own next generation of chips and convinces ASML/TMC to build them...or builds fabs that can do it. In practice it will be the interaction between this AI economy and the human economy that will shape where it goes.
Justin, this is a rich and ambitious piece. What I particularly value in it is the attempt to create a common language across AI, organisms, and organisations without falling back into the tired binary of “intelligent or not”. The emphasis on coordination, feedback, credit assignment, and the structure of interaction is especially strong. That shift in focus matters.
I also think the article lands an important point about organisations: fluency, confidence, and operational competence can persist long after the underlying conditions for real learning have begun to erode. That is a useful insight, and the parallel with AI failure modes is productive.
Where I would add a note of caution is in the move from resemblance to identity. There are clearly shared dynamics across these different systems, but the differences in how they maintain coherence, handle novelty, and remain answerable to reality still matter a great deal. The parallels are illuminating. The substrates are not interchangeable.
I am also not fully persuaded by the treatment of hallucination as mainly a function of being outside the training distribution. That is part of the story, but only part. Models can fail inside familiar territory as well, especially where the prompt structure, truth conditions, or constraints are underdetermined. The issue seems deeper than novelty alone. It concerns the relation between fluent generation and reliable world-tracking.
Even so, the article does something valuable. It pushes the discussion away from surface judgements about AI and toward the conditions under which any collective, artificial or human, can actually learn. That is a worthwhile move.
The question it left me with is slightly prior to the one you pose: not only what makes a collective intelligent, but what allows a collective to remain coherent enough for that intelligence to hold under pressure when its environment changes. That, to me, is where the next layer of the discussion begins.
Thanks Abol - thoughtful and wise as always. I take both of your major points. I am taking a rather radical pragmatist position here and not a functionalist one. Language is not representational and there is no need to answer the question of substrate dependence in order to agree that inferential claims can be made of both LLM and human statements. See Brandom on this for similar. The same goes for the identity argument - I certainly don’t argue for identity. I merely argue that by ignoring substrate and looking at both as language machines and language as a mechanism for evaluating knowledge claims they both resemble each other close enough to make comparisons which at this level are illuminating for both.
Justin, that helps and the Brandom reference clarifies the move.
I agree that treating language inferentially allows us to evaluate outputs across systems without settling questions of inner states or substrate.
My hesitation is that Brandom’s account still depends on a practice of commitment and accountability carried by agents over time. LLM outputs can be evaluated within that space, but the system itself does not yet participate in it in the same way.
So the comparison works at the level you describe. The open question for me is what allows a system to sustain that kind of normative participation under changing conditions.
Yes. One immediate implication is that we will soon see interactions and supply chains created out of enterprises composed almost entirely of AI agents. This new economy will take on increasingly complex tasks, from robots ordering their own spare parts, to designing new features for themselves. Arguably its concievable that AI designs its own next generation of chips and convinces ASML/TMC to build them...or builds fabs that can do it. In practice it will be the interaction between this AI economy and the human economy that will shape where it goes.
Justin, this is a rich and ambitious piece. What I particularly value in it is the attempt to create a common language across AI, organisms, and organisations without falling back into the tired binary of “intelligent or not”. The emphasis on coordination, feedback, credit assignment, and the structure of interaction is especially strong. That shift in focus matters.
I also think the article lands an important point about organisations: fluency, confidence, and operational competence can persist long after the underlying conditions for real learning have begun to erode. That is a useful insight, and the parallel with AI failure modes is productive.
Where I would add a note of caution is in the move from resemblance to identity. There are clearly shared dynamics across these different systems, but the differences in how they maintain coherence, handle novelty, and remain answerable to reality still matter a great deal. The parallels are illuminating. The substrates are not interchangeable.
I am also not fully persuaded by the treatment of hallucination as mainly a function of being outside the training distribution. That is part of the story, but only part. Models can fail inside familiar territory as well, especially where the prompt structure, truth conditions, or constraints are underdetermined. The issue seems deeper than novelty alone. It concerns the relation between fluent generation and reliable world-tracking.
Even so, the article does something valuable. It pushes the discussion away from surface judgements about AI and toward the conditions under which any collective, artificial or human, can actually learn. That is a worthwhile move.
The question it left me with is slightly prior to the one you pose: not only what makes a collective intelligent, but what allows a collective to remain coherent enough for that intelligence to hold under pressure when its environment changes. That, to me, is where the next layer of the discussion begins.
Thanks Abol - thoughtful and wise as always. I take both of your major points. I am taking a rather radical pragmatist position here and not a functionalist one. Language is not representational and there is no need to answer the question of substrate dependence in order to agree that inferential claims can be made of both LLM and human statements. See Brandom on this for similar. The same goes for the identity argument - I certainly don’t argue for identity. I merely argue that by ignoring substrate and looking at both as language machines and language as a mechanism for evaluating knowledge claims they both resemble each other close enough to make comparisons which at this level are illuminating for both.
Justin, that helps and the Brandom reference clarifies the move.
I agree that treating language inferentially allows us to evaluate outputs across systems without settling questions of inner states or substrate.
My hesitation is that Brandom’s account still depends on a practice of commitment and accountability carried by agents over time. LLM outputs can be evaluated within that space, but the system itself does not yet participate in it in the same way.
So the comparison works at the level you describe. The open question for me is what allows a system to sustain that kind of normative participation under changing conditions.