The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks.
ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels.
Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations.
You can try the tasks yourself here: https://arcprize.org/arc-agi/3
Here is the current leaderboard for ARC-AGI 3, using state of the art models
- OpenAI GPT-5.4 High - 0.3% success rate at $5.2K
- Google Gemini 3.1 Pro - 0.2% success rate at $2.2K
- Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K
- xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K.
(Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.)
https://arcprize.org/leaderboard
Technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf
In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training
LLMs might suck at this game but I’m pretty sure Deepmind’s deep reinforcement learning AI could solve these easily.
EDIT: I know you guys hate AI around here, but you need to at least be aware of what the technology is capable of.
From 11 years ago:
https://youtu.be/V1eYniJ0Rnk
if only it would exist
Wdym? It’s existed for at least a decade. Plenty of papers about it. It mastered Atari and Mario. It became the best Go player.
Yeah RL exist since 80s (or in some form earlier) but have it solve the benchmark?
I’m talking specifically about the variant of RL used by systems like AlphaGo.
Yes – this game has some fixed, relative small set of rules so the RL could learn to play by playing millions of games at random but following the rules of the game. Confront this with dounting (infinite) number of situations which may approach one in a daily life.
Yeah, for a fixed ruleset that can be provided up front the Alpha-Zero approach seems to work great.
These tasks strike me as a bit different. I’m sure the ruleset is fixed somewhere, but it’s not disclosed to the participants. In the task I walked myself through, there was a new wrinkle in each part – a new interactable, a (more) hidden goal, or an information limit. And, of course, part of the task is “discovering” all that from the bitmap frame(s) provided.
I’m unconvinced of the hype around “AI”, but this does seem like a legitimate research target that might stymie the Alpha{Go,Zero,Fold} series at least a bit.
If only…
How Alpha Fold Solved the Protein Folding Problem and Changed Science Forever
Edit:
Source
OK, RL exists end results like the protein design or Go are impressive, but does exist a RL solving the benchmark problem?
This is as concrete as Sam Altman saying “AI will actually discover new science”
They won the Nobel prize for it.
Then say they won the same prize that was awarded to the inventor of the lobotomy, don’t link to a puff piece with an indefensibly bad title
No because it’s designed with all the things AI can’t do. Breakout is a quick repetitive loop of pass/fail linear progression. AI melts down when it has to backtrack and keep track of multiple pieces of context and figure out how to do something but not do it yet.
I guess the idea is that yes, machine learning algorithms could be used to solve these, but that’s essentially brute forcing. You can make a simple algorithm learn how to complete Super Mario Bros or how to make a virtual robot walk, it just takes millions of iterations. The promised actual artificial intelligence wouldn’t need that.
The founder of ARC worked at Google until 2024 and wrote 2.5+ books in Deep Learning. So, I expect some of these benchmarks are based on limitations seen in Deepmind.
That said, it would be interesting to see how well Deepmind does at these tasks. My understanding is that the private tasks would still be dynamic enough to require “on the job training” so an Alpha-Go / Alpha-Zero / Alpha-Fold approach is unlikely to do well on ARC-AGI-3.
Still, I think commentary around models (including, but not limited to something from Deepmind) attempting these tasks would be much more interesting than most of the discourse around generative AI, whether text, image, video, or code generation.