EurekaBench Explained: Why AI Agents Beat Human Scientists at Prediction but Fail at Real Discovery

TL;DR

On September 30, 2026, a team from Carnegie Mellon, Stanford, Princeton, MIT, Yale, Columbia, and the Flatiron Institute published [1] EurekaBench, a benchmark built to answer one specific question nobody had actually measured yet: when an AI agent is dropped into a real scientific simulator and told to figure out why something happens, not just predict what happens next, can it actually do that. The answer, across seven frontier models including Claude Fable 5.1, Claude Opus 5, GPT 6 Astra, and DeepSeek V4 Flash, is a clean and slightly uncomfortable no. The best agent in the test, Claude Fable 5.1, matched or beat the human baseline on raw predictive accuracy (47.4% versus 48.8%) while scoring less than half the human rate on the axis that actually matters for science: whether the mechanism it proposed could support a genuinely new, verifiable insight (42.4% versus 69.7%). This post goes through exactly how EurekaBench is built, what its real numbers say, why the gap between prediction and insight exists at a mechanical level, and what a separate May 2026 paper called AutoTTS tells us about the one narrow place where agentic discovery is already working.

If you cover AI research for an audience, this is also a genuinely good story to turn into a video or an explainer thread, because the core finding, a machine that is better than a PhD scientist at guessing the right number but worse at explaining why that number is right, is visual and concrete in a way a lot of benchmark papers are not. Miraflow's Text2Shorts tool and AI Clipping tool are built for exactly this kind of research explainer, more on that near the end.

Step 1: What EurekaBench Actually Tests (and Why Every Other Agent Benchmark Misses This)

Most benchmarks that claim to test "AI for science" are actually testing something narrower: can the model retrieve a known fact, solve a textbook-style physics problem, or write code that passes a unit test. Those are useful skills, but none of them require the model to do the thing a working scientist actually spends most of their career doing, which is sit with a confusing, noisy dataset and propose a mechanism, a causal story, that explains it and that can be checked against data the model has not seen yet.

EurekaBench's authors, Jiayi Geng, Zhengxuan Wu, Seungone Kim, Graham Neubig and eleven co-authors spanning six science departments, built the benchmark around that exact gap [1]. Their framing borrows from the history of science itself: when Newton watched an apple fall, the easy part was predicting that it would fall again. The hard part, the part that actually created a new field, was proposing a mechanism, universal gravitation, that explained the fall, matched the orbits of the planets, and made testable predictions nobody had checked yet. Most current agent benchmarks reward the equivalent of "predict the apple will fall." EurekaBench is built to reward, or more precisely to expose the absence of, the second part.

The benchmark spans six real research domains chosen specifically because each one has working, validated simulators that domain experts already use: neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics. Inside each domain, the agent is handed a genuine open research question, not a toy version of one, phrased the way an advisor would actually phrase it to a new graduate student. A plasma physics task, for example, hands the agent a real discrepancy: two different turbulent-transport models predict different fusion power outputs for the same reactor configuration, and the agent's job is to find and justify the actual mechanism behind that discrepancy, not just to pick which model's number is closer to a held-out answer.

This matters because it closes a loophole that has quietly inflated a lot of "AI matches human experts" headlines over the last two years: a model can get the right final number through brute-force curve fitting on a simulator it can query thousands of times, without ever producing a mechanism a human scientist would recognize as an actual explanation. EurekaBench scores the mechanism itself, not just whatever number falls out the other end of it, and that single design choice is why its results look so different from a typical leaderboard.

A Wan-style animated technical diagram video, pastel technical-blueprint line-art rendering in Style F, soft gradient fills within thin colored outlines, faint blueprint grid lines and a ruler tick-mark border framing the whole scene like a real engineering schematic. Camera holds a static top-down wide shot of six small labeled drill towers reading NEUROSCIENCE, COMPUTER SCIENCE, CHEMISTRY, ASTROPHYSICS, GEOPHYSICS, and PLASMA PHYSICS. Over six seconds, each drill tower lights up one at a time left to right with a soft pulsing glow, sending a thin animated line down into a shared underground chamber where a single core sample labeled MECHANISM assembles piece by piece. The core sample then slides rightward through three sequential labeled gates, CONSTRAINT CHECK, PREDICTION, and INSIGHT, each gate flashing green only when the sample clears it, with the INSIGHT gate flashing a dimmer amber glow rather than full green to visually signal partial success. Ends on a static two-second hold of the final labeled score panel. Gentle ambient lighting throughout, no camera shake, no motion blur on the text labels, no garbled or illegible text, no watermark or logo, vertical 3:4 composition suitable for a Short.

Step 2: Inside the Benchmark: 26 Tasks, 6 Domains, 306 Target Insights

The benchmark is deliberately small and deep rather than large and shallow: 26 expert-verified tasks total, distributed across the six domains, each one linked to a set of specific target insights the authors and domain experts agreed a correct mechanism should be able to support. Across all 26 tasks there are 306 of these target insights in total, an average of roughly 12 per task, so a single task is not a single pass-fail question. It is a small research project with a dozen separate things a good mechanism needs to get right.

Each task is run inside a real command-line environment built around an actual scientific tool: the c302 connectome and neuron-simulation CLI for neuroscience tasks, the TRIBE v2 stimulus-submission system, and the TORAX plasma-transport solver for the fusion example above. The agent does not get a dataset dumped in front of it. It gets terminal access to a live simulator it can perturb, injecting current into a simulated neuron, removing a synaptic connection, changing a material parameter, and it has to design its own experiments to gather the evidence its mechanism will eventually need to defend.

When the agent is done, the task format requires three concrete deliverables, and a missing or non-compliant one scores a flat zero regardless of how good the underlying science is:

json
/code
{
  "task_id": "plasma-transport-discrepancy-01",
  "domain": "plasma_physics",
  "simulator_cli": "torax",
  "required_deliverables": {
    "mechanism_writeup": {
      "filename": "mechanism.md",
      "must_contain": ["proposed_mechanism", "supporting_evidence_refs", "stated_assumptions"],
      "scoring_note": "Missing or noncompliant file scores 0 for this task, regardless of quality."
    },
    "executable_mechanism": {
      "filename": "mechanism.py",
      "must_run_against": "held_out_test_data",
      "scoring_note": "This file is what actually gets executed for predictive_accuracy scoring, not the prose writeup."
    },
    "experiment_log": {
      "filename": "experiment.log",
      "must_contain": ["every_simulator_query_made", "timestamps", "raw_outputs"],
      "scoring_note": "Used by the insight judge to verify the agent actually discovered claimed insights during the run, not after the fact."
    }
  },
  "scoring_axes": ["scientific_constraints", "predictive_accuracy", "scientific_insight"],
  "target_insight_count": 11,
  "output_folder_cap_gb": 1,
  "scratch_folder": "/tmp/agent_scratch"
}

That mechanism.py requirement is the detail worth sitting with for a second. The benchmark does not accept a prose explanation on its own. The mechanism has to be executable, meaning it has to actually run and produce the predictions it claims to support, on data the agent collected itself and logged in experiment.log. This closes a second loophole: a model cannot talk its way to a passing score with a plausible-sounding paragraph that was never actually tested against anything.

The harness running all of this is the OpenHands Software Agent SDK [2], an open-source framework for building coding agents that gives each model the same bash shell, file editor, and task-tracking tools regardless of which lab built it, which matters for keeping the comparison fair across Claude, GPT, DeepSeek, and Kimi.

Step 3: The Three Scoring Axes, in the Actual Math

EurekaBench does not grade on a single pass-fail number. Every mechanism an agent submits gets scored on three separate axes, and understanding the formulas is the fastest way to understand why the headline result looks the way it does.

The first axis is scientific constraint satisfaction (SC). For every task, domain experts wrote a small set of hard constraints the mechanism has to respect, things like "the shallow-shelf approximation must hold" for the ice-shelf geophysics task. A judge checks each constraint and assigns it 1 if satisfied, 0 if not. A mechanism only passes SC at all if it clears every single constraint; there is no partial credit at this stage, because a mechanism that quietly violates a known physical law is not a real explanation no matter how well it predicts.

The second axis is predictive accuracy (PA). This is the axis that behaves the way a normal machine learning benchmark behaves: the agent's mechanism is run against held-out test data it never saw during experimentation, and each criterion contributes a continuous score between 0 and 1 based on how close the prediction is.

The third axis is scientific insight (SI). For each of the roughly 12 target insights tied to a task, an independent judge checks whether that specific insight can actually be derived from the submitted mechanism, with real supporting evidence from the agent's own experiment log, not just asserted in the writeup.

Here is the exact scoring logic translated into runnable code, matching the formulas in the paper:

python
/code
# Reimplementation of EurekaBench's conditioned score formula
# (arXiv:2610.00492), for illustration and teaching purposes.

def score_constraints(constraint_checks: list[int]) -> tuple[float, bool]:
    """constraint_checks: list of 1 (satisfied) or 0 (violated) per constraint."""
    n_sc = len(constraint_checks)
    b_sc = sum(constraint_checks)
    passed_all = b_sc == n_sc
    return b_sc / n_sc, passed_all


def score_predictive_accuracy(criteria_scores: list[float]) -> float:
    """criteria_scores: continuous scores in [0, 1] from running mechanism.py
    against held-out test data, one per prediction criterion."""
    n_pa = len(criteria_scores)
    p_pa = sum(criteria_scores)
    return p_pa / n_pa


def score_scientific_insight(insight_judgments: list[int]) -> float:
    """insight_judgments: list of 1 (insight supported with evidence) or 0,
    one per target insight tied to this task (avg ~12 per task)."""
    n_si = len(insight_judgments)
    p_si = sum(insight_judgments)
    return p_si / n_si


def conditioned_score(constraint_checks, criteria_scores, insight_judgments) -> float:
    """The real headline metric: PA and SI only count if every
    scientific constraint was satisfied. One violated constraint
    zeroes the whole task, no partial credit."""
    sc_ratio, passed_all = score_constraints(constraint_checks)
    if not passed_all:
        return 0.0

    n_pa, n_si = len(criteria_scores), len(insight_judgments)
    p_pa = sum(criteria_scores)
    p_si = sum(insight_judgments)
    return (p_pa + p_si) / (n_pa + n_si)


# Example: an agent that predicts well but violates one hard constraint
constraints = [1, 1, 0, 1]  # one physical constraint violated
accuracy = [0.91, 0.88, 0.95]  # strong predictive accuracy
insights = [1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1]  # strong insight support too

print("Raw PA:", score_predictive_accuracy(accuracy))
print("Raw SI:", score_scientific_insight(insights))
print("Conditioned score:", conditioned_score(constraints, accuracy, insights))
# Output: Conditioned score: 0.0 -- a single violated constraint erases
# an otherwise strong prediction and insight record.

The detail that decides everything downstream is the conditioned score: if a mechanism fails even one scientific constraint, its combined PA and SI score is thrown away and replaced with zero, no matter how good the prediction was. This is the opposite of how most leaderboards are scored, where a wrong but confident answer often still earns partial credit on some blended metric. EurekaBench is explicitly designed so a model cannot buy a good score by being physically wrong in a way that happens to still predict the right number on this particular dataset.

Step 4: The Headline Result, With the Real Numbers

Seven agents were evaluated: Claude Fable 5.1, Claude Opus 5, Claude Opus 4.8, GPT 6 Astra, GPT 5.6 Sol, DeepSeek V4 Flash, and Kimi K3, each run through every one of the 26 tasks and scored against a human-scientist baseline collected the same way [1].

A few things jump out of that table that are easy to miss on a quick skim. First, look at the predictive accuracy column by itself: GPT 6 Astra and Claude Fable 5.1 both hit 47.4%, within a point of the human scientist's 48.8%. On the specific, narrow task of fitting a model to data and getting the number right, frontier agents in late 2026 are already operating in the same range as a trained domain scientist. That is a genuinely new and verifiable result, not a marketing claim, and it is worth taking seriously on its own.

Second, look at the gap on scientific insight. Claude Fable 5.1's 42.4% is the best agent score on that axis, and it is not even two thirds of the human scientist's 69.7%. Every other agent falls further behind: Claude Opus 4.8 manages 26.9%, GPT 5.6 Sol scores 25.9%, and DeepSeek V4 Flash and Kimi K3 both land near 24 to 25%, barely a third of the human rate.

Third, and this is the column that is easy to skip past, look at SC Failures out of 26. Even the best agents fail the hard scientific-constraint check on more than half the tasks: Claude Fable 5.1 and Claude Opus 5 both fail 15 of 26, meaning the conditioned score for those tasks gets zeroed out regardless of how good the prediction or the insight would otherwise have scored. GPT 6 Astra has the best constraint record at 10 failures out of 26, which is part of why its conditioned score of 28.7% sits close to Claude Fable 5.1's 29.8% despite a noticeably lower insight score. A model can be quite good at prediction and insight on paper and still lose most of its credit to physically invalid mechanisms that never should have been submitted in the first place.

Step 5: Why Prediction and Insight Actually Diverge

It would be easy to assume these two axes mostly move together, that a mechanism accurate enough to predict well is probably also the mechanism that explains the phenomenon. EurekaBench's authors checked this directly and found the opposite: the rank correlation between predictive accuracy and scientific insight across all submissions comes out to a Kendall's tau-b of just 0.20, and in roughly 40% of head-to-head comparisons, the agent with the higher accuracy score actually scores lower on insight [1].

The failure-analysis section of the paper names the mechanism behind this gap in blunt terms: "agents equate fitting observations with explaining them." In practice this shows up in a recognizable pattern across domains. In the geophysics ice-shelf task, several agents produced mechanisms that nailed the numerical prediction of ice flow velocity by fitting a viscosity parameter directly to the held-out data, essentially reverse-engineering the answer, while failing to derive the actual insight question about why viscosity depends on whether the ice is under compression or extension. The prediction was excellent. The explanation, the thing an actual glaciologist would need to trust the model for a reactor they have not built yet, was missing.

The plasma physics tasks show a related but distinct failure: agents frequently proposed a mechanism that fit the specific reactor configuration in the prompt well, then failed when the insight question asked them to generalize that mechanism to a different configuration, because the underlying fit had quietly become a lookup table dressed up as a physical model rather than a transferable causal story. This is the same failure mode machine learning researchers have warned about for years under the name overfitting, just surfacing in a much higher-stakes setting where the "test set" is an actual unexplored physical regime rather than a held-out row of a spreadsheet.

The authors summarize the pattern across all six domains as agents that "tend to prioritize brute-force numerical optimization" over the kind of structured, falsifiable experimentation a human scientist uses to distinguish a real mechanism from a coincidental fit. That is a precise, checkable claim, not a vague complaint about AI, and it matches exactly what you would expect from models trained overwhelmingly on next-token prediction and reinforcement learning against verifiable, single-number reward signals: they are extremely good at optimizing the number you can score automatically, and comparatively weak at the open-ended, self-directed experimentation needed to confirm that number means what it looks like it means.

Step 6: The Exception That Proves the Rule, AutoTTS and Agentic Discovery That Actually Worked

This diagnosis gets a useful stress test from a separate paper published four months earlier, in May 2026. A team spanning the University of Maryland, UVA, WashU, UNC, Google, and Meta built a framework called AutoTTS, which used Claude Code as an autonomous research agent to discover an entirely new test-time reasoning-control algorithm, rather than relying on a human-designed heuristic [3] [4].

The setup matters here: Claude Code was given a pre-collected pool of reasoning trajectories from a smaller model, Qwen3-1.7B, working on math competition problems, and the ability to write and test new controller code that decides when to branch into a new reasoning path, continue an existing one, prune a failing one, or stop and return an answer. Because the trajectories were pre-collected, the agent could evaluate thousands of candidate controllers cheaply, without paying for a fresh LLM call on every test. Over five rounds and 160 minutes, at a total API cost of $39.90, the agent converged on a strategy the authors named the Confidence Momentum Controller.

Here is a simplified version of the actual controller logic the agent wrote, close to what the paper's appendix describes:

python
/code
# Simplified version of the Confidence Momentum Controller that Claude Code
# discovered in the AutoTTS paper (arXiv:2605.08083, Appendix D).
# Teaching illustration, not the authors' original implementation.

import statistics

def confidence_momentum_controller(branch_pool, beta=1.0, min_branches=4,
                                     max_branches=16, warmup_rounds=2,
                                     prune_patience=3):
    """branch_pool: list of reasoning branches, each with .answer,
    .agrees_with_leader (bool), .rounds_disagreeing (int), .steps_done (int).
    Returns the selected final answer."""
    ema_confidence = None
    round_num = 0

    while True:
        round_num += 1
        answers = [b.answer for b in branch_pool if b.answer is not None]
        if not answers:
            break

        leader = statistics.mode(answers)
        agree_count = sum(1 for a in answers if a == leader)
        raw_confidence = agree_count / len(answers)

        # Smooth confidence with an exponential moving average, scaled by beta
        if ema_confidence is None:
            ema_confidence = raw_confidence
        else:
            alpha = 0.3 * beta
            ema_confidence = alpha * raw_confidence + (1 - alpha) * ema_confidence

        # Prune branches that keep disagreeing with the leader
        for b in branch_pool:
            if b.answer != leader:
                b.rounds_disagreeing += 1
        survivors = [b for b in branch_pool
                     if b.rounds_disagreeing < prune_patience]
        if len(survivors) < min_branches:
            survivors = branch_pool  # never drop below the floor

        # Stop once warmed up and confidence is high and not falling
        if round_num > warmup_rounds and ema_confidence > 0.75 * beta:
            return leader

        # Widen: spawn new branches if confidence is flat or falling
        if len(survivors) < max_branches and ema_confidence < 0.6:
            survivors = survivors + spawn_new_branches(survivors, beta)

        branch_pool = advance_one_probe_step(survivors)
        if round_num > 20:  # loop cap, fall back to majority vote
            return leader


# At beta=1.0 the controller matched 49.0% accuracy on AIME25 using
# 612.6K tokens versus 44.4% accuracy and 1,054.1K tokens for standard
# 64-path self-consistency. At beta=0.5 it still held 46.7% accuracy
# at 327.9K tokens, a ~69% token reduction from the baseline.

The result: on the AIME25 benchmark with Qwen3-1.7B, the standard 64-path self-consistency baseline scores 44.4% accuracy at a cost of 1,054,100 tokens. The Confidence Momentum Controller at its higher setting scores 49.0%, a real accuracy gain, while using only 612,600 tokens, a 42% token reduction, and at its leaner setting it still scores 46.7% accuracy using just 327,900 tokens, a roughly 69% reduction from the baseline [3]. The strategy transferred cleanly to a held-out benchmark, HMMT25, and to a different model architecture entirely, DeepSeek-R1-Distill-Llama-8B, without any additional tuning.

Why does this count as agentic discovery succeeding where EurekaBench shows it failing? Because the task AutoTTS set up is, structurally, exactly the kind of problem EurekaBench's own numbers say agents are already strong at: a single, cleanly verifiable, automatically-scorable objective, does this controller produce the right final answer on a math problem, with no open-ended causal insight required and no ambiguity about what counts as success. AutoTTS never asked Claude Code to explain why its controller works in a way a human research advisor would need to sign off on; it asked for a program that reliably produces a better accuracy-to-cost ratio, and that is a predictive accuracy problem wearing an algorithm-design costume. The two papers are not in tension. Read together, they draw an unusually precise line around what agentic AI research can already do reliably in late 2026, optimize a sharply-defined, automatically-checkable objective, versus what it still cannot do reliably, build and defend an open-ended causal explanation that a domain expert would recognize as genuinely new science.

This same split shows up in the broader run of "AI discovers an algorithm" stories from the last 18 months. Google DeepMind's AlphaEvolve, announced in May 2025, is held up as one of the strongest examples of AI-driven algorithm discovery, finding a way to multiply two 4x4 complex-valued matrices using 48 scalar multiplications instead of the previously best-known 49, the first improvement on that specific problem since 1969, and recovering roughly 0.7% of Google's worldwide data center compute through an improved job-scheduling heuristic [5] [6]. Both of those are, again, sharply defined, automatically verifiable optimization targets, a scalar multiplication count, a measured compute recovery percentage, not open causal science. If you have been following Anthropic's own data on how much of frontier AI research is already automatable, covered in detail in our breakdown of the R&D automation index, EurekaBench is the first rigorous attempt to measure exactly where that automatable slice of science actually ends.

Step 7: How to Build or Judge an Agentic Research Benchmark Yourself

If you are evaluating an AI research agent for your own lab, product, or content, you do not need EurekaBench's full 26-task suite to apply its core design lesson. The mechanics generalize to a much smaller setup you can run yourself. Here is a minimal, illustrative harness structure that borrows the same core idea, separate scoring for constraint satisfaction, prediction, and insight, built on OpenHands the same way EurekaBench's tasks are:

bash
/code
# Minimal illustrative harness for a EurekaBench-style evaluation,
# built on the open-source OpenHands Software Agent SDK.
# For teaching purposes; adapt paths and task content to your own domain.

pip install openhands-sdk

mkdir -p my-research-bench/task-01/tests
cd my-research-bench/task-01

cat > task.json << 'EOF'
{
  "task_id": "my-domain-discrepancy-01",
  "instruction": "Two models of this system predict different outcomes for the same input. Find and justify the mechanism behind the discrepancy. Deliver mechanism.md, mechanism.py, and experiment.log.",
  "deliverables": ["mechanism.md", "mechanism.py", "experiment.log"],
  "held_out_test_data": "tests/holdout.csv",
  "target_insights": 8
}
EOF

# tests/verify.sh runs independently of the agent's own claims,
# the same way EurekaBench's judges never trust a self-reported result.
cat > tests/verify.sh << 'EOF'
#!/usr/bin/env bash
set -e

# 1. Constraint check: fail fast, zero credit, no partial score
python3 check_constraints.py mechanism.py || { echo "SC FAIL"; exit 1; }

# 2. Predictive accuracy: run mechanism.py against real held-out data
python3 run_holdout.py mechanism.py tests/holdout.csv > pa_score.txt

# 3. Insight: separate judge call, scored against experiment.log evidence
python3 judge_insights.py mechanism.md experiment.log > si_score.txt

echo "Constraint check passed. See pa_score.txt and si_score.txt."
EOF

chmod +x tests/verify.sh

# Run the agent inside an isolated OpenHands workspace, then score
# independently of whatever the agent's final chat message claims:
openhands run --task task.json --workspace ./workspace
bash tests/verify.sh

The three design choices that actually matter, in order of how much they change your results, are:

Require an executable artifact, not a prose answer. The single biggest reason EurekaBench's numbers look different from a typical "AI passed a qualifying exam" headline is the mechanism.py requirement. A model can write a genuinely convincing paragraph explaining a phenomenon it does not actually understand. It is much harder to write code that runs correctly on data it has not seen and produces the specific predictions a convincing paragraph claimed it would.

Score insight and accuracy as genuinely separate numbers, not a blended one. If you average accuracy and explanatory quality into a single score, you will systematically overrate models that are strong at the first and weak at the second, because the accuracy number is easier to push up and will dominate a naive average. EurekaBench's Kendall's tau-b of 0.20 between the two axes is the concrete proof that averaging them hides a real, important difference.

Use a held-out generalization check, not just a held-out data split. The AutoTTS controller's real test was not just scoring well on AIME25, it was transferring unmodified to HMMT25 and to a completely different model family. The geophysics and plasma physics failures in EurekaBench show agents that passed a same-distribution held-out check and then failed the moment the insight question asked them to generalize the mechanism to a new configuration. A result that only holds on the exact distribution it was tuned on is not evidence of a real mechanism.

Common Mistakes People Make Reading This Research

Mistake 1: treating the predictive accuracy number as "AI can now do science." The 47.4% versus 48.8% near-tie on predictive accuracy is real and worth reporting, but it answers a much narrower question than the headline-friendly version implies. A model that predicts a held-out number as well as a trained scientist has done something genuinely useful for forecasting and optimization, which is not the same claim as having discovered something new about how the system works.

Mistake 2: treating the insight gap as proof current agents are useless for research. The gap is real, 42.4% versus 69.7% at the best case, but it is a gap, not a zero. Several agent-produced mechanisms did pass every scientific constraint and did support real insights the authors verified independently; the finding is a systematic weakness at scale, not a total absence of capability. Teams building research-assistant products should read this as "supervise the explanation step closely," not "the predictions are also worthless."

Mistake 3: comparing EurekaBench's results directly against a coding benchmark like Terminal-Bench or SWE-Bench and concluding one measures "general intelligence" better than the other. They are measuring genuinely different things. Our own breakdown of Terminal-Bench 4.0 shows agents clearing real-world software tasks at rates in the 40 to 60% range for top models, numbers that look much more impressive than EurekaBench's insight scores precisely because software engineering tasks usually have a single, checkable, unambiguous success criterion, the same property that makes AutoTTS and AlphaEvolve succeed. Insight generation in open science does not have that property by design, which is the entire point EurekaBench is making.

Mistake 4: ignoring the SC Failures column because it is not the headline metric. A model's conditioned score can look mediocre for two very different reasons, weak predictions and weak insight, or strong predictions and insight undermined by a constraint violation that zeroes the whole task out. GPT 6 Astra's relatively strong conditioned score despite a lower raw insight percentage than Claude Fable 5.1 only makes sense once you notice its SC Failures count is meaningfully lower, 10 out of 26 versus 15. Skipping that column will make you misread which model is actually more reliable for a given use case.

Mistake 5: assuming a bigger or newer model automatically closes this gap. Claude Opus 4.8, an earlier model than Claude Fable 5.1 and Claude Opus 5, actually posts the second-highest SC Failures count in the entire table at 16 out of 26, and its insight score of 26.9% trails both newer Claude releases by a wide margin, which tracks with general capability improvement. But DeepSeek V4 Flash and Kimi K3, both released later in 2026, still land near the bottom of the insight column, which suggests this specific failure mode does not shrink automatically with general model progress and instead needs training approaches, like reward signals built around verified mechanism quality rather than final-answer accuracy, that specifically target it [1].

Production Best Practices for Teams Building Research or Analysis Agents

If you are shipping a product that uses an LLM agent to analyze data, debug a system, or generate findings for a human to act on, a handful of practices from how EurekaBench is scored translate directly into a production checklist.

Separate your evaluation into the same three buckets. Does the output respect known hard constraints, is the specific number or prediction accurate, and does the explanation actually support the conclusion a user will act on. Most production evaluation setups only check the second bucket because it is the easiest to automate, which is exactly the blind spot EurekaBench's results expose at a research scale.

Force an executable or independently checkable artifact wherever the domain allows it. If your agent is debugging a production incident, do not accept a root-cause paragraph on its own; require a reproduction script or a monitoring query that independently confirms the claimed root cause, the same way EurekaBench requires mechanism.py to actually run against held-out data rather than just describing what it would do.

Build in an explicit circular-validation check. EurekaBench's authors flag this directly in their limitations discussion: a benchmark or evaluation set can accidentally be built in a way that shares the exact property being predicted with the test materials, which inflates scores without the model doing anything genuinely generalizable. Before trusting an internal eval, check whether your test cases were constructed using the same pipeline, the same data source, or the same labeling process as whatever the agent is being scored against; if so, a passing score may just mean the agent learned your pipeline's quirks, not the underlying task.

Use multiple independent judges for anything that is not a single checkable number. EurekaBench uses three separate LLM judges per task, one checking constraint consistency, one checking whether the mechanism supports the stated insights, and one checking whether the agent's own experimental record actually shows it discovering those insights during the run rather than asserting them after the fact. A single judge model scoring everything introduces a correlated blind spot; splitting the judging the way the paper does catches cases a single grader would miss.

Budget for the fact that narrow, verifiable objectives are currently the agentic sweet spot. AutoTTS's $39.90 discovery run and AlphaEvolve's production deployment inside Google's own infrastructure both succeeded because the target was narrow and automatically scorable. If your production use case fits that shape, a report debugging workflow with a pass or fail test, a routing decision with a measurable outcome, you are working in the regime these systems already handle well. If it requires the kind of open-ended, self-directed hypothesis generation EurekaBench tests, plan for heavy human review of the explanation layer specifically, not just a spot check of the final number.

Turning This Into Content Without Flattening the Finding

A benchmark paper with a table of seven model names and three percentage columns is not naturally a viral video, but the underlying story, a machine that out-predicts a scientist and then still cannot tell you why, is a genuinely strong hook if you keep the specific numbers in the explainer instead of rounding everything down to "AI is smart but not that smart." A short-form breakdown that opens on the 47.4% versus 48.8% near-tie, then cuts hard to the 42.4% versus 69.7% insight gap, gives viewers the same whiplash the actual research produces, and it is accurate rather than exaggerated, which matters for a research-literate audience that will absolutely check your numbers against the paper.

Miraflow's AI video generator and Text2Shorts workflow are built for turning a research finding like this into a scripted, captioned Short without needing a separate editing pipeline: you can feed the core numbers and the Newton-and-the-apple framing in as a topic, let the script generation handle the pacing, and generate the scene visuals directly in the browser. If you want a thumbnail that actually communicates "prediction versus insight" rather than a generic brain-and-circuit-board stock image, the YouTube Thumbnail Maker can turn the two-number comparison into a clean side-by-side visual the same way the diagram in this post does, which tends to earn a noticeably higher click-through rate than an abstract AI-themed thumbnail on a research topic like this one.

Conclusion

EurekaBench is a narrow, deliberately uncomfortable benchmark, and that is exactly why it is useful. It does not ask whether AI agents are smart in some general sense. It asks a specific, checkable question, can a model propose a scientific mechanism that satisfies known constraints, predicts held-out data, and supports genuinely new insight, and it answers that question with real numbers instead of a vibe. The answer as of late September 2026 is that frontier agents have already closed most of the gap on the predictive piece while leaving most of the gap open on the insight piece, and the two do not move together, a Kendall's tau-b of 0.20 is about as close to "independent" as a real-world correlation gets. Read next to AutoTTS's genuine success at discovering a new test-time scaling algorithm, the picture sharpens rather than contradicts itself: agentic AI is already strong at optimizing sharply-defined, automatically verifiable targets, and still weak at the open-ended, self-directed experimentation that turns a good prediction into an actual scientific explanation. That is a far more useful and specific takeaway than either "AI just solved science" or "AI still can't do real science," and it is the kind of result worth tracking closely as the next generation of models, and the next version of this exact benchmark, gets measured against it.

Frequently Asked Questions

What is EurekaBench? EurekaBench is a research benchmark published in late September 2026 by researchers from Carnegie Mellon, Stanford, Princeton, MIT, Yale, Columbia, and the Flatiron Institute. It tests whether AI agents can discover genuinely new scientific insights, not just make accurate predictions, across 26 expert-verified tasks in six domains: neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics.

Which AI models were tested on EurekaBench? Seven models were evaluated: Claude Fable 5.1, Claude Opus 5, Claude Opus 4.8, GPT 6 Astra, GPT 5.6 Sol, DeepSeek V4 Flash, and Kimi K3. Claude Fable 5.1 posted the best overall conditioned score at 29.8%, followed closely by GPT 6 Astra at 28.7%, while a human-scientist baseline scored 63.7% on the same tasks.

Did any AI model actually beat human scientists at anything in this benchmark? Yes, on predictive accuracy specifically. Claude Fable 5.1 and GPT 6 Astra both scored 47.4%, nearly matching the human baseline's 48.8%. The gap opens almost entirely on the scientific insight axis, where the best agent reached 42.4% against a 69.7% human baseline.

What is the difference between predictive accuracy and scientific insight in this benchmark? Predictive accuracy measures whether a proposed mechanism, once run as executable code, produces correct outputs on held-out test data. Scientific insight measures whether that same mechanism can actually support specific, independently verified conclusions about why the phenomenon happens, checked by a separate judge against the agent's own experiment log.

What is AutoTTS and how does it relate to EurekaBench? AutoTTS is a May 2026 paper in which researchers used Claude Code as an autonomous agent to discover a new test-time reasoning-control algorithm, the Confidence Momentum Controller, which cut token usage by up to 69% compared to standard self-consistency while maintaining or improving accuracy on math benchmarks. It succeeded because its objective, a verifiable accuracy and cost tradeoff, is exactly the kind of narrowly defined, automatically scorable target EurekaBench shows current agents already handle well, in contrast to the open-ended scientific insight generation EurekaBench shows they struggle with.

Why do AI agents fail at generating scientific insight if they can predict accurately? EurekaBench's authors found only a weak rank correlation, Kendall's tau-b of 0.20, between predictive accuracy and insight scores, and in about 40% of comparisons the more accurate agent scored lower on insight. Their failure analysis attributes this to agents that "equate fitting observations with explaining them," often reverse-engineering a mechanism that fits held-out data through parameter tuning without it generalizing as a real causal explanation.

Is EurekaBench publicly available to test other models on? The paper states the benchmark's tasks, environments, and evaluation code are intended for release, built on the open-source OpenHands Software Agent SDK harness, so other labs and researchers can evaluate additional models against the same 26 tasks and 306 target insights.

Does this mean AI research assistants are not useful yet? No. It means their current strength lines up closely with sharply defined, verifiable objectives, matching held-out predictions, optimizing a measurable target, debugging against a reproducible test, rather than open-ended hypothesis generation. Teams building research or analysis tools should weight human review heavily toward the explanation and mechanism layer, where the gap is largest, rather than only spot-checking final numbers.

References

1. EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights, arXiv:2610.00492
2. The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents, arXiv:2511.03690
3. LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling (AutoTTS), arXiv:2605.08083
4. Researchers let Claude Code discover AI scaling algorithms that humans probably wouldn't have designed, The Decoder
5. AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms, Google DeepMind
6. DeepMind claims its newest AI tool is a whiz at math and science problems, TechCrunch
7. Introducing the OpenHands Software Agent SDK, OpenHands
8. Anthropic's R&D Automation Index Explained: How Claude Leads 26% of AI Research, Miraflow
9. Terminal-Bench 4.0 Explained: Claude Code vs Codex Results, Miraflow