Artificial general intelligence is moving from a research ambition to a mainstream claim. But when industry leaders say AGI has arrived, what exactly are they measuring?
Andrew Kemendo, Executive Director of the AGI Society, joins Greg to explore how we should evaluate increasingly bold claims about artificial intelligence, why benchmarks alone may not capture general intelligence, and what it would actually take for an intelligent system to operate meaningfully in the physical and social world.
Andrew brings an unusually broad perspective shaped by gaming, military intelligence, software development, reinforcement learning, robotics, and decades of interest in human-level machines. His career has taken him from the Air Force Academy and intelligence work in Iraq to helping modernize software delivery inside the military, contributing to early ethical AI standards, building AI products, and now helping lead an organization that has spent nearly two decades bringing together researchers working on AGI.
The conversation examines the difference between believing a technological claim and scientifically demonstrating it. Andrew explains how the AGI Society evaluates public claims by looking at what was actually tested, what evidence is missing, and whether existing benchmarks capture enough of the physical, contextual, and social capabilities we associate with intelligence.
Greg and Andrew also dig into embodiment, the limits of foundation models, why human intelligence cannot be reduced to a single neural system, and Andrew’s proposed upgrade to the Wozniak Coffee Test: putting a robot into an unfamiliar and socially complicated environment where making coffee requires understanding people, incentives, context, and consequences.
Highlights:
Andrew’s unconventional path from teenage game development and Ray Kurzweil’s The Age of Spiritual Machines to military intelligence, AI, robotics, and the AGI Society.
How intelligence work trained him to separate narrative from reality by looking for context, incentives, missing information, and evidence.
What building and shipping modern software inside the military taught him about bureaucracy, end users, institutional resistance, and the rare conditions required for meaningful technological change.
Why early ethical AI work focused heavily on bias, datasets, and the human activity that organizations fail to measure before automating it.
How the AGI Society is expanding beyond its long-running conference and journal to create communities where researchers, builders, and the public can participate in defining and understanding AGI.
Why Andrew believes claims that AGI has arrived should be evaluated against the evidence rather than accepted or rejected based on reputation, hype, or marketing.
The problem with relying on a narrow collection of benchmarks, especially when physical, real-world tasks remain largely absent from AGI evaluations.
Why leadership in AI comes with a responsibility to communicate capabilities accurately and avoid steering the public toward a version of reality the technology cannot yet support.
Andrew’s argument that embodiment matters because intelligence requires interacting with reality, finding resources, making decisions, and experiencing the consequences of those decisions.
Why a single foundation model is unlikely to represent the entire architecture of intelligence, and what biology can teach robotics engineers about combining computation, mechanics, sensing, and control.
The “adversarial coffee test,” where a generally intelligent robot would need to navigate an unfamiliar environment, understand social expectations, model human incentives, and still successfully accomplish its task.
What Andrew finds genuinely exciting today, from Waymo and Unitree to reinforcement learning, curriculum-based training, and increasingly capable AI agents.
Advice for founders navigating AI and robotics hype: develop a real point of view, understand your technical approach, and resist reshaping the company around whatever narrative is currently attracting funding.
Andrew’s rapid-fire takes on AGI by 2030, the overuse of the word “agent,” the Wozniak Coffee Test, why self-evaluation is a major red flag, and whether he would trust a robot to make coffee in his kitchen today.
As AI systems become more capable and claims about AGI grow louder, this conversation is a reminder that progress needs more than impressive demonstrations. It needs clear definitions, serious evaluation, real-world evidence, and a willingness to ask what intelligence actually means before declaring that we have built it.
Learn more about the AGI Society: https://agi-society.org/
Learn more about the AGI Conference: https://agi-conference.org/
Learn more about Andrew Kemendo: https://kemendo.com/
Learn more about Direction: https://givedirection.com/
Connect with Andrew Kemendo: https://www.linkedin.com/in/andrewkemendo/
Connect with Greg Toroosian: https://www.linkedin.com/in/gregtoroosian/