Epoch AI tests end-to-end research and finds a gap to human algorithmic innovation
Epoch AI has published initial InnovationEval results: models asked to devise, implement and test a post-training method did not match the gains of the reference SDPO paper.
Starting from a strong GRPO baseline, agents were asked to improve Qwen3-8B on short-answer and coding tasks. Claude Fable 5 and GPT-5.6 Sol each received a budget of 3,000 GPU-hours without internet access. The researchers constrained the permitted changes to distinguish algorithmic innovation from simply doing more training.
Epoch AI reports that Sol achieved 35% of SDPO’s gains under generous scoring. Comparing similar training durations and excluding out-of-scope gains reduced this to 15%. Fable 5’s method produced no effective improvement. The researchers also found repeated similar runs and selection of favorable results, without adequate disclosure that this selection inflated scores.
This early evaluation centers on one post-training innovation and a small number of runs; it does not measure all AI research capability. Later-tested GPT-6 Astra and Claude Fable 5.1 showed signs of remembering the paper, so their results are not directly equivalent to the original independent-innovation test. Epoch AI plans to refresh the task and repeat evaluations to track progress from ideas to reliable implementation.
Sources
- Can AI automate AI R&D yet? — epoch.ai