A controlled study on ARC-AGI-3 tested whether frontier models can build and revise their own task harness. Across 27 lineages, performance ranged from zero to near-ceiling, and two revisions raised held-out RHAE by only 2.80 points, with development gains overstating generalization.

— via arXiv Software Engineering, Jens Lehmann, Sahar Vahdati