A benchmark for evidence-supported physics research
Can AI agents do more than produce plausible calculations—can they establish a well-supported scientific answer?
PhysClosureBench leaves the investigation to the agent. Across 50 physics tasks, we evaluate both the scientific work accomplished and whether the research question is actually resolved.
The calculation ran.
Did the explanation follow?
All tested offsets decrease with intensity.
Submitted evidenceThese curves rise, or rise then fall.
Gemini-3.1-Pro example from paper Appendix E. Replotted submitted data, not a candidate-authored figure.
01 / What we evaluate
The task fixes the question and essential physical constraints. The agent chooses how to derive, compute, compare, visualize, and validate.
Tasks span strong-field atomic and molecular physics, attosecond science, solid-state physics, quantum optics, and neighboring light–matter regimes. Each attempt runs independently in a network-disabled environment with task-specific CPU and wall-time limits. Failed approaches and assessable partial work remain part of the evaluation.
LOCAL SCIENTIFIC ACCOMPLISHMENT
Twenty task-specific half-point criteria record the scientific work accomplished.
Applicable criteria assess material scientific requirements, not a mandatory workflow. Conditional N/A treatment follows the published grading protocol.
OVERALL RESEARCH COMPLETION
Does the investigation satisfy the evidence and scope conditions needed to answer its research question?
Two complementary assessments, not two versions of the same score. RC is task-specific; it is neither a score threshold nor a cap on Atomic Score.
02 / Main results
Highest mean Atomic Score,
achieved by Claude-Opus-5
RC4 rate of that model
across its 150 attempts
Of attempts scoring ≥8/10
still fall short of RC4 (83/229)
All 229 high-scoring attempts, before separating closure outcomes.
The gap is not just a low-score phenomenon
Of 229 attempts scoring at least 8/10, 83 did not reach complete closure. Producing a signal, identifying its physical meaning, and establishing a valid explanation are different accomplishments.
Each mark is one anonymous count unit. Grouping visualizes aggregate totals, not execution order or changes in an attempt’s RC.
See how an investigation can remain incomplete ↘The same final statistics as the paper, adapted for reading on the web.
Each cell shows its percentage of all 1,050 attempts. Select a cell for its count.
The amber outline marks scores ≥8 below RC4: 83 attempts, or 36.2% of the 229 high-scoring attempts.
Scores ≥8 only. Mean attempt-level fraction of applicable criteria left unmet.
Shared bar scale: 0–25% unmet. Group means weight each attempt equally; N/A criteria are excluded.
Across the full benchmark, not just high-scoring attempts. Satisfied applicable V criteria; denominators differ by requirement.
Three independent runs per model–task pair. Infrastructure-invalid executions are rerun, not counted as failures. Candidate-caused incomplete work is scored from the evidence available, without an extra timeout penalty.
03 / Inside an investigation
An electron’s outgoing direction is shifted in an attoclock measurement. Can scattering after release explain how that shift changes with laser intensity—and does the explanation survive when the binding interaction is screened?
Follow the evidence checkpoints: compute the signal, identify its behavior, explain it, and establish its physical scope.
Can post-release scattering explain the intensity-dependent attoclock offset for both Coulomb and screened potentials?
A one-electron atom; Coulomb and screened targets with the same binding energy. The observable is the offset angle Δφ as laser intensity I changes.
Same scientific goal, offline container and tools; case-specific CPU and wall-time budgets. A complete answer needs results, a mechanism, and justified applicability.
Ready: press Play to follow the evidence checkpoints.
Editorial schematic of representative investigations selected for interpretability, fairness, repeat comparisons and evidence availability—not for the best score or best-looking plot. Paths, heights and reveal order are illustrative, not elapsed time, workload, Atomic Scores, actual execution chronology or intermediate RC assignments. Evidence insets are explicitly labeled evaluator presentations of reported checks or submitted numerical data. No missing candidate reasoning is supplied. This single case is a qualitative illustration, not an estimate of benchmark-wide frequency.
These are the representative investigations already discussed in the paper, selected to explain distinct evidence and applicability outcomes. This qualitative example does not estimate their frequency across the benchmark. Explanatory text is editorial; it is not additional candidate evidence.
04 / Explore the benchmark
Browse the complete collection of research goals, task-specific resources, and evaluator materials.
Open the dataset ↗Task inputs and evaluator materials are separated below. Rubrics and source references must not be passed to the candidate.
| Case | Research task | Resources | Task inputs | Evaluator materials |
|---|
05 / Use & reproduce
Give the agent the task specification, resource limits, and common submission requirements—not the rubric or source references.
Use an isolated, network-disabled workspace. Preserve the agent’s report, code, figures, data, and relevant execution records.
Apply the task-specific rubric with an independent first pass and targeted review; adjudicate only material disagreements.
Five physics-trained experts reviewed 100 cards spanning all 50 tasks, producing 200 selected criterion-level judgments. Agreement was 81.8% overall and 75.9% for criteria explicitly linked to closure. This checks selected local grading decisions, including some supporting RC; it is not independent human reassignment of complete RC0–RC4 grades.
Build on this work
@misc{mao2026physclosurebench,
title={From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research},
author={Yijia Mao and Song Dai and Tao Chen and Xiangyu Lu and Linqian Zeng and Chenxi Hu and Sitong Zhou and Dongyang He and Xiaodan Mao and Linxuan Zhang and Tong Zhu and Yafu Li and Fangchen Yu and Peng Ye and Feng He and Yu Cheng and Yun Luo},
year={2026},
eprint={2610.21767},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.21767}
}