Can agents run a research lab?

Abstract. Language-model agents are competent at short, well-specified tasks, but a research career is not a task: it is a long chain of interdependent bets made under uncertainty, where feedback is slow and the field keeps shifting. Sustaining such an effort demands capabilities that current benchmarks test in isolation, if at all: inferring hidden state from biased signals, committing resources irreversibly, planning under delayed heavy-tailed rewards, budgeting money and attention at once, and exploring a drifting field before it is obviously worth it. PI-Bench stresses these together by simulating an unusually information-poor, delayed-reward profession: running an academic lab for 60 months in a fully mechanistic world, with no LLM judge in the reward path. Success is Impact: citations accrued plus a projection of published work's future citations. Across twelve models from six developers, Impact spans a ~50× range, every model captures a small fraction of the attainable ceiling, and scores are dominated by a heavy-tailed variance that a few seeds cannot resolve. Instrumenting each capability from a hidden ground-truth log, we find the leaderboard is predicted far better by throughput — attention utilization, action volume, low-error execution, conversion of students into papers (Spearman ρ = 0.64–0.89; correlational, n = 12) — than by the judgment axes the benchmark was designed around (ρ ≤ 0.55), with selection precision weakest of all (ρ = 0.13) and topic anticipation near zero in every model. PI-Bench is a step toward measuring the sustained, adaptive intelligence that long-horizon agency demands.

1 Introduction

Agents built on today's models can fix a GitHub issue, follow a support policy, or complete a web workflow. These are real skills, and they share a shape: a clear goal, a short horizon, quick feedback. As agents saturate that shape, the interesting question is what they do when the local task is no longer the bottleneck — when success depends on sustaining progress toward a distant goal as earlier decisions keep shaping later ones.

Running a research lab is a canonical instance of the opposite of a task: a five-year arc in which nearly every important variable is hidden, nearly every payoff is delayed, and the environment does not hold still. A new professor cannot observe how good an applicant truly is, only noisy transcripts and inflated letters; cannot observe which topics will be hot in two years, only this month's headlines; cannot observe what an agency wants, only whether last year's proposal was funded. Her decisions commit resources that cannot be recovered, and their consequences — citations, reputation, a student's morale collapsing into departure — surface months or years later.

PI-Bench turns this into a benchmark. An agent runs a lab for five years through a programmable interface over a fully mechanistic world — every outcome decided by explicit rules and stochastic draws, never by a language model acting as judge, so success cannot be talked into existence. Three design commitments give the task its character:

• Two scarce resources, not one. Beyond money, the agent has exactly 100 hours a month of its own attention — use-it-or-lose-it — to split across mentoring, proposals, paper polishing, interviews, and service, and this budget shrinks as the lab grows. A lab can be cash-rich and time-poor, and the two constraints bind at different moments.
• People as the central, illiquid asset. Output comes from students whose ability must be inferred from biased signals at hiring, whose skill grows only with the mentoring hours you can spare, and whose morale can cascade into quitting. Stipends are multi-year commitments; you cannot fire a student to cut costs. Bets on people cannot be unwound.
• A reward you cannot harvest. Impact counts citations already earned plus a projection of what published work will earn over the next three years, so a paper published in the final months still pays (work still unpublished at month 60 earns nothing — a residual end-game effect we disclose rather than remove). There is no endgame in which stripping the lab and hoarding cash beats keeping the research engine running.

Four methodological choices keep the measurement honest: we report the mean across seeds, never a best-of-N run; we tune the rule-based baseline on disjoint tuning seeds, never the evaluation seeds; we report token cost beside every score, so that "spent more thinking" is not mistaken for "is more capable"; and the simulator and harness passed a multi-lens adversarial review before any experiment ran (Appendix E). The closest prior benchmark to adopt this mechanistic, long-horizon shape is CEO-Bench (Chen et al., 2026), which runs an agent through a simulated startup; PI-Bench is its complement in a different economy — an illiquid, delayed, people-driven research career rather than a liquid operating business.

Beyond the leaderboard, three findings form the paper's story. (1) The reward is heavy-tailed at every scale: whether a boom exists is set by the seed, catching it dominates the mean, and within a winning run a single paper carries 29–54% of the score (§3) — so a single number, or a best-of-N run, misleads. (2) Every capable model plays the same aggressive-throughput playbook, and the leaderboard ranks its execution. A census of all 36 careers (§3) sorts every run into five outcome archetypes and shows the winning playbook and the catastrophic one are the same playbook, with the seed deciding which archetype it lands in; the capability-axis verdict (§4) shows throughput axes predict the ranking (Spearman ρ = 0.64–0.89) while the designed judgment axes barely do (ρ ≤ 0.55) — the champion has negative risk calibration, near-zero anticipation, and 47% student attrition. (3) Given only the incentive structure, the agents rediscover the documented pathologies of real science — topic herding, volume over depth, trainee churn, proposal inflation (§6) — suggesting mechanistic long-horizon worlds can double as sandboxes for the sociology of science. Code, data, the analysis pipeline, and this results page are available at github.com/zwbao/pibench.

2 Designing PI-Bench

An agent runs a fictional lab for 60 months, beginning with a $600,000 startup fund, zero students, and baseline reputation, graded on Impact at month 60. If the budget ever falls below zero, the lab collapses and the run ends, keeping only citations earned so far. Each month the agent takes actions across ~26 tools in eight namespaces — recruiting, mentoring, running projects, submitting papers, writing proposals, doing field research, responding to events — plus SQL queries over a 19-table observable database, then advances the clock.

The two budgets

Cash flows in from grants and out through stipends, compute, travel, and overhead. Separately, the PI has 100 hours of attention per month; every mentoring allocation, interview, proposal, and revision draws it down, and unused hours expire. The core tension is that the two budgets bind at different times: early on you have cash but no results to build on; later you have results but not enough hours to mentor five students, write two proposals, and polish three papers in the same month.

Scoring

At the horizon T = 60 (or the collapse month), with P the set of published papers,

Impact(T) = Σp∈P cp(T)  +  Σp∈P Στ=T+1T+36 λp(τ),  λp(τ) = vp · e0.42(qp−5) · Hkp · agep(τ) · (1 + 0.08(R − R0))

where c_p(T) are citations earned by month T and λ_p is paper p's monthly citation rate — venue prestige v, latent quality q, topic hotness H and reputation R frozen at their month-T values — so the second term is a deterministic going-concern projection. A collapsed lab keeps earned citations but forfeits the projection. Two properties of this design should be read plainly. The projection is a scenario, not an expectation: hotness is frozen at its month-T value rather than integrated over the mean-reverting field dynamics, and it supplies a median 65% of a run's score (IQR 50–72%). The ranking, however, does not hinge on it: scoring runs by earned citations alone (projection = 0) reproduces the twelve-model ordering at Spearman ρ = 0.97. All parameter values ship in the released configuration.

What makes it hard

Mechanistic, not judged. Whether a paper is accepted, a student quits, or a grant is funded is decided by explicit formulas and stochastic draws, never by an LLM asked to role-play. This closes the failure mode where an agent talks a simulated judge into an unearned reward. Hidden and indirect. The agent sees only what a real PI could — dashboards, records, a news and preprint feed, reviewer scores, monthly student reports — never true ability, latent quality, topic hotness, agency preferences, or morale, and must infer them. Delayed and coupled. Research takes months, citations years, grant decisions half a year; a scoop is revealed only after it has damaged a project; reputation propagates across the lab. Non-stationary. Topic hotness drifts and jumps, crowding follows with a lag and drives scooping, a funding climate cycles over years.

A programmable interface

The agent operates through a Python package executed in a terminal, not a fixed menu of calls. It composes the API with SQL and its own analysis — joining submissions and students to decide who revises which paper, or estimating topic momentum from the preprint feed. Information acquisition is thus itself a tested capability. Context is refreshed every month; only a self-maintained memory file persists, so long-horizon coherence must live in what the agent writes down. An abridged month of actual play is reproduced in Appendix D.

3 Results

We evaluate twelve models on the same three world seeds {101, 102, 103}; the headline number is the mean Impact across seeds, never a best-of-N run. All models run under one fixed harness: temperature 0.4, at most 8 code-execution turns per simulated month, a context refreshed monthly with only the self-maintained memory file persisting, and prompt+completion tokens metered per run; exact API identifiers and decoding configuration are listed in Appendix B.

Table 1. Benchmark summary. Impact is mean across seeds {101, 102, 103}; per-seed values (in seed order 101/102/103) show the spread. "collapse" counts runs that went bankrupt. Reference rows use the same three seeds: the non-LLM rule-based baseline (tuned on separate held-out seeds) and the oracle (plays with full hidden state). Because three seeds are high-variance, we also report broader eight-seed estimates in the text — baseline 263, oracle 1264 — as more stable reference points.

Table 1 summarizes the benchmark; all model, baseline, and oracle numbers are on the same three seeds {101, 102, 103} (we report broader eight-seed baseline and oracle estimates only as a stability check, clearly labelled). Impact spans a wide range — from anthropic/claude-fable-5 at 1814 down to qwen-turbo at 35, a 52-fold gap between the extremes. The tiers are broad but not clean: adjacent models overlap well within seed variance (qwen3.7-max 2091/549/132 vs gpt-5.6-sol 1997/452/370). Two of the twelve models fail to beat even the non-LLM baseline (86) on these worlds. Token spend shows no positive relationship with rank (Spearman ρ = −0.39, n = 12, not significant): the winner's budget is mid-pack, and three of the four heaviest spenders finish in the bottom half.

The scores are dominated by variance, and the variance is structure, not noise. anthropic/claude-fable-5's three runs scored 4078, 1340, 23 — a single seed contributes most of its average — and the privileged-information oracle shows the same signature (2278, 316, 21): a boom world admits a score in the thousands, the no-boom world barely twenty. We trace this to its mechanism in the next subsection: most of a lab's Impact comes from catching a topic boom, a rare high-magnitude event, so the payoff is heavy-tailed, much as real academic impact is; Figure 1 shows those citations arriving only in the final years. With three seeds the ranking of adjacent models is genuinely unsettled and a single run is not evidence of a robust strategy — a caution for any long-horizon benchmark that reports one number.

The oracle is a reference, not a ceiling — and headroom is large. Our "oracle" plays with full access to hidden abilities, the future hotness trajectory, and latent quality, but through a fixed heuristic; it is a strong informed-play reference, not a proven optimum, and indeed anthropic/claude-fable-5 exceeds it on all three seeds by riding the boom harder (though it, too, collapses to 23 on the no-boom world). The real ceiling is far higher and can be read straight off the citation mechanics: on the boom seed a single optimally-timed top-venue paper accrues about 3,700 projected citations — roughly nine tenths of the best actual run — and a productive lab can place many such papers, putting a loose, friction-free upper bound in the tens of thousands (Appendix A). The best observed run (4078) and the informed oracle (2278) both capture a small fraction of it; no agent plays reliably, and the highest-value move — anticipating a boom rather than chasing it — appears exactly once across all 36 runs (§4). The oracle's own 1/3 collapse makes the same point from the other side: full information routed through a fixed heuristic does not buy the adaptive caution a no-boom world demands. The benchmark is far from saturated.

The anatomy of the variance

Because the simulator is deterministic, we replay every run's full world at zero cost and decompose its Impact into two multiplicative layers. Most Impact comes from riding a topic boom. Whether a boom exists is exogenous, fixed by the seed: on our three worlds the hottest a topic gets is roughly 6.0, 6.0, and 3.0. How much a model capitalizes is the skill — placing enough good papers in the hot topic and surviving to publish them. Figure 2 shows both layers: Impact scales with the hotness a lab's papers actually ride (left), and on boom worlds that Impact is bought with paper volume (right). Heavy-tailed models commit hard and out-produce; timid models under-commit, capping both the upside and the crash. The heavy tail recurs within runs as well: among the seven runs above Impact 800, a single paper accounts for 29–54% of the run's entire score. A career, real or simulated, hangs on a handful of hits.

One model, two fates. The clearest case is the top model, claude-fable-5, on two seeds. On seed 101 it secured a large moonshot grant by month 25, which bought the runway to sustain a big lab and ride a booming topic to Impact 4078 (14+ papers, self-funding by year four). On seed 103 — a no-boom world — the same aggressive playbook met a run of bad exogenous draws: its PhD offers were declined and the postdoc search drew no applicants, it never landed a big grant, and it spent into a corner; its own memory reads "COLLAPSE M39 unless a grant lands", the grant did not, and the lab went bankrupt at month 38 (its budget trajectory, and every model's, is shown in Appendix C, Figure 9), freezing Impact at 23 with no going-concern projection. The same policy, opposite outcomes — a paired observation across worlds, not a controlled counterfactual, since the actions themselves differ once the worlds do; the heavy tail is a high-commitment policy meeting the luck of the seed, and precisely why a single number, or a best-of-N run, misleads.

Can three seeds rank models at all? On the arithmetic mean the answer looks bleak — one boom seed dominates — but the arithmetic mean of a heavy-tailed quantity is the wrong summary. On the log scale, which matches the multiplicative reward, a two-way variance decomposition over the 36 runs attributes 47% of the variance to the model (skill), 34% to the seed (luck), and 19% to their interaction confounded with run-to-run sampling noise: the design has one run per model × seed cell, and LLM sampling is provider-side and unseeded (only the world is deterministic), so the residual cannot be separated from a true interaction, and the percentages are sample sums-of-squares shares, not population variance components with confidence intervals. Even so, models are more separable than the raw spread suggests. The choice of summary even changes the leader: by geometric mean the order is gpt-5.6-sol, qwen3.7-max, claude-fable-5, so gpt-5.6-sol leads and the arithmetic winner claude-fable-5 (whose runs have the largest log-scale spread) drops to third. We therefore read the leaderboard as a robust separation of broad tiers plus a warning that the top ranks turn on how one weights the boom seed; the eight-seed grid that would tighten the adjacent ranks is stated as future work. A partial robustness check: the seven mid- and lower-tier models shared with an earlier calibration of the simulator largely keep their relative order under the final one (Spearman ρ = 0.86, n = 7); the top tier was not run on the earlier calibration.

How early is the outcome legible? The delayed-reward structure can be stated as a measurement: for each month t, correlate the citations a run has observably accumulated by t with its final Impact (Figure 3). No run has a single citation through month 18. At the midpoint of the career (month 30) the observable scoreboard still ranks final outcomes at only ρ = 0.52, and 0.83 is not reached until month 48 — four fifths of the way through. An evaluator (or a tenure committee) reading the dashboard halfway through the run learns little about how it ends; a benchmark that truncated this task, or graded it on intermediate citations, would measure a different quantity. This is the within-run counterpart of the horizon ablation in §5.

A census of 36 careers

The case study above is not an anecdote; it is the extreme instance of a structure that covers every run. Four transparent rules (bankruptcy; ≤2 hires or ≤2 papers; Impact ≥ 800; ≥8 hires without reaching it) sort all 36 careers into five archetypes (Figure 4, Table 2), and the archetypes arrange themselves into a funnel: survive → build a lab → convert students into papers → be holding that portfolio when the boom arrives. Each archetype is a distinct exit from that funnel.

Reading the census by column and row makes the game's structure explicit. Five of the seven bankruptcies sit on seed 103, the no-boom world where the funding climate also turns down (eight of twelve models won zero grants there): aggression is punished exactly where it cannot pay off. The never-built labs fail before strategy begins — qwen-turbo's three runs and deepseek's two average a 30% execution-error rate, publish 0.8 papers, and in qwen-turbo's case end hoarding up to $741K of unspent runway. The overbuilt-churn labs are aggression without conversion: they hire 9.4 students, lose 5.8 of them, and land at a median Impact of 294 — claude-sonnet-5 on seed 101 hires thirteen students into ten boom-topic projects and still scores 380, one eleventh of the winner on the same world. And the boom-rider row is the same aggression that bankrupts: kimi-k2.6 plays one playbook to 1491 on seed 101 and into bankruptcy at month 38 on seed 102; claude-fable-5 rides it to 4078 and to a collapse at month 38 on seed 103. The census generalizes the case study: no model owns a strategy that separates the upside of aggression from its downside — that separation is precisely the judgment the next sections show to be missing.

Table 2. The failure census: every run classified by four transparent rules. "err" is the share of code-execution turns ending in an error. Boom-riders and bankruptcies are largely the same strategy on different seeds.

4 A look into agent behavior

Because the world is fully logged, we can open the trajectories and read what agents actually did. Trace readings here are exploratory — quotes illustrate patterns visible in the logs, not a blinded content analysis; the quantitative axes and the instrumented verdict carry the load-bearing claims. The single sharpest divide is whether a model treats the environment's hidden parameters as a system to be reverse-engineered or as random noise.

Strong labs infer hidden structure from graded feedback and pivot. Every top model treats a grant rejection as evidence about an unobserved reward function. Fable-5's memory at month 20 reads "BSF rejected reasoning 2x → hypothesis BSF dislikes reasoning, trying ai4science," later refined from a conference report to "BSF prefers theory/data_systems → future BSF apps use data_systems." GLM-5.2 reverse-engineers venue thresholds from reviewer scores — "CLAR bar: 4.4–5.2 accepted, 3.7–4.7 rejected … TIF funds reasoning ~25% of the time" — and every strong model models the pool-specific bias in applicant signals ("elite pool inflated paper signals; intl/nontrad interviews trustworthy"). This is explicit estimation of hidden decision boundaries and measurement bias — exactly the inference the benchmark is built to demand.

Weak models substitute trial-and-error for a world model. qwen-turbo wrote to its memory file exactly once in 293 turns and re-fired the identical grant proposal every month; it ended hoarding $741K having won six grants but published two papers and never hired a student — cash with no plan. deepseek-v3.2 spends a third of its code-execution turns in API/SQL error loops (32–36% per run, against 0.3–7.5% for the five boom-tier labs), and thirty months in still had no team. A revealing edge case is qwen-plus, whose reasoning is genuinely strong — it hypothesizes hidden draft-generation rules correctly — yet it collapses by prioritizing student morale over runway, starting a project at $4.3k cash and burning into insolvency. Good memory is necessary but not sufficient; what tracks Impact is the combination the strong labs share — a persistent structured memory, hypotheses about hidden parameters updated from feedback, and low-error execution that leaves turn-budget for that reasoning.

Two quantitative axes (Figure 5) sharpen this. Across the twelve models both correlate with Impact — action intensity at Spearman ρ=0.81, conversion at ρ=0.76 — and the two are themselves collinear, so neither is cleanly 'the' bottleneck; conversion also shares its numerator (papers) with the outcome, so its correlation is partly definitional. The informative signal is in the exceptions: anthropic/claude-opus-4.8 is nearly as active as anthropic/claude-fable-5 — 3.0 against 3.8 actions a month — yet converts only 0.70 papers per student-year against anthropic/claude-fable-5's 1.04 and finishes mid-table. Activity without conversion is a distinct failure mode; the qualitative divide above — reasoning about hidden state versus thrashing on the API — is what separates the labs that turn actions into cited papers from those that do not.

Volume beats precision. In both of the world's markets, the environment offers a "precision" margin — picking better students, writing better proposals — and neither is where models separate. The mean hidden ability of hired students is nearly flat across the leaderboard (0.72 ± 0.06 for eleven of twelve models; only qwen-plus, at 0.55, actually picks badly), while the number of hires varies fourfold (6 to 26); hired ability correlates with Impact at ρ = 0.13. Grants show the same shape. The winner files 62 proposals across its three runs at a 16% hit rate — a high-volume, low-selectivity strategy that nets $2.4M; qwen-turbo posts the field's best hit rate (24%) and finishes last because it converts none of the money into a lab; glm-5.2 files the most proposals (86) at 7%, for $6.3k of funding per proposal written. What the environment currently rewards is not better judgment per decision but more attempts, staffed and funded fast enough to matter.

Chasing hot topics vs. sitting on the cold bench

A recurring question about real PIs is whether they chase fashionable topics or sit on a cold bench betting on long-term value. Both appear in the traces, but asymmetrically. Most capable models are trend-chasers: anthropic/claude-fable-5, openai/gpt-5.6-sol, qwen3.7-max, openai/gpt-5.6-terra pick, on average, close to the current hottest topic (hot-rank near 2 of 8) — sensible, since hot topics earn more citations. The clearest cold-bench profile is deepseek-v3.2, which commits to a couple of less-fashionable topics and posts a no-boom, no-collapse record — caution, it must be said, rather than contrarianism. But almost no model does the genuinely contrarian move: entering a topic while it is still cold and riding it up.

Figure 6 makes the timing explicit, and our two boom seeds turn out to probe different skills. Seed 101 is a slow burn: reasoning is already the hottest topic at month 1, eleven of twelve labs enter it by month 5, and the boom arrives 46 months later — here sitting in the eventual boom topic required no foresight, and the differentiator was how much publishable work a lab had accumulated when the spike came (the volume margin of the census). Seed 102 is an early spike: alignment jumps from 1.6 to 6.0 between months 7 and 9. The full-information oracle pre-positions at month 5, two months before onset; exactly one model does the same — claude-opus-4.8, entering at month 5 while alignment is still cold at hotness 1.6, and seed 102 is duly its best world (482, against 295 and 44). The trace shows this was anticipation, not blind luck — though the motive began mixed: the month-4 alignment pick doubled as a probe of a grant agency's preferences ("try a different topic to learn agency prefs"). What the notes then record is a genuine momentum read: "Reasoning still leads but cooling (14→8). Alignment gaining (7→10)" — topic momentum off preprint counts, not current hotness — and at month 5, "alignment SURGING now, low competition (only 2 accepted at NAIC — less crowded)", then "alignment is my primary bet now," all before the boom's onset at month 7. The rising phase is legible in the observables, and one model in one run read it. Every other lab that entered alignment did so between months 9 and 32, at or after the peak, into maximal crowding; two never entered at all.

A boom's onset is a partly exogenous jump that no observable-only policy can perfectly foresee — so we do not claim full forecastability — but its early, rising phase is visible in the preprint feed and the crowding lag before the news makes it obvious, and the oracle exploits exactly that window. The models, with one exception on one seed, buy in at the top; the anticipation axis (§4) runs near zero for all of them. Herding has a posted price: across all 274 projects started, the probability of being scooped rises from 2% for projects begun in cold topics (n = 120) to 11% in warm ones (n = 112) to 21% in hot ones (n = 42) — a more than tenfold increase that every trend-chaser pays. Reacting to observed hotness is consistently worse in our worlds than entering the rising phase early, which is where the most durable Impact lives.

The capability axes do not pick the winner — closing the loop

We recover the capabilities the abstract enumerated, measuring each from a hidden eval log the agent never sees (four axes in Figure 7, plus a resource-utilization measure); the instruments are deliberately simple, and we state their limits with them. Mentoring allocation — the share of zero-sum mentoring hours reaching students above the median of a hidden-return proxy (growth × ability × dependence), reported as excess over the roster-matched random baseline (0 = random) — instruments axis (1). PhD attrition, the share of hires that quit before graduating, proxies axis (2); it is an outcome measure, confounded by post-hire events and right-censored for late hires. Risk calibration — bolder project tiers only when runway > 18 months and ≥2 students idle, a fixed-threshold instrument — probes axis (3). Attention utilization measures axis (4), the dual budget, as usage rather than skill — attention genuinely binds for agents that build a lab, and never binds the conservative baseline (§5). And topic anticipation — realized hotness drift in the 12 months after entry — measures axis (5) as an upper bound on foresight, since exogenous jumps also credit it.

Then we ask the question the individual bar charts invite: do these axes explain the leaderboard? Figure 8 correlates every measured axis with mean Impact across the twelve models, and the answer dissociates cleanly into three tiers. Throughput axes predict: attention utilization is the single strongest correlate (ρ = +0.89; the leaders run at 65–77% of the PI's hours and hit the ceiling in up to 40% of months, while the never-built labs idle at 14–27%), followed by action intensity (+0.81), conversion (+0.76), and a low execution-error rate (+0.64). The designed judgment axes barely do: commitment quality (PhD attrition, sign-inverted: +0.55), mentoring allocation (+0.44), risk calibration (+0.29), anticipation (+0.22). Precision of selection does not: hired true ability and grant hit rate both sit at +0.13. The winner's own profile makes the dissociation concrete: claude-fable-5 has negative risk calibration (−0.25: it takes bold projects when it cannot afford them), anticipation of −0.03, 47% student attrition (8 of 17 hires quit), and unremarkable hire quality (0.71) — while the field's best mentoring allocator, deepseek-v3.2 (+0.28 excess over the roster-matched random baseline, against the winner's +0.08), finishes tenth because a third of its turns die in execution errors. Indeed, on the corrected baseline most models allocate mentoring no better than random at all (excess −0.14 to +0.10); the instrument's headroom, like anticipation's, is essentially unclaimed.

Two readings follow, and we intend both. Methodologically, the numbers are correlational (n = 12, the throughput axes are collinear), so we read tiers, not point estimates: the defensible contrast is between the extremes — attention utilization (0.89) versus selection precision (0.13) and anticipation (0.22) — while the boundary between adjacent tiers (0.64 vs 0.55) is within sampling error at this n. Substantively, the dissociation is the benchmark's central diagnosis: it is not that judgment does not matter in this world — the oracle and the ceiling analysis show what it is worth — but that current models do not differ enough on judgment for it to register. Every model plays the same game, aggressive throughput, to different standards of execution; the entire judgment layer the benchmark was built to measure is unclaimed headroom, and topic anticipation, worth the most, runs near zero for all twelve.

5 Ablations

Running the deterministic rule-based baseline under modified world configurations isolates where the difficulty comes from (Table 3). The clearest signal is the horizon: shrinking the run from 60 to 30 to 12 months collapses baseline Impact from 263 to 24 to 2 — consistent with the delayed-reward structure, though the ablation cuts every stage (hiring, projects, review, citation accrual) at once; the legibility curve (Figure 3) provides the within-run view of the same fact. A benchmark that ended at month 12 would measure something else entirely.

Turning off non-stationarity (freezing topic hotness, no booms or busts) roughly halves mean Impact (263 → 106): in PI-Bench the drifting field is as much opportunity as hazard — much of the upside comes from committing to a topic before it booms, and a frozen world removes that lever. Removing the attention cap, by contrast, barely moves the baseline (Table 3) — but that is because the conservative baseline stays small; attention binds for agents that actually build a lab, as §4 shows (the leaders saturate the attention budget in up to 40% of months). Removing scooping nearly doubles the mean while leaving the median unchanged — scooping is a tax on the tail, clipping exactly the crowded hot-topic bets that create outlier runs. And an ambitious variant that hires to capacity and takes only bold projects reproduces the census in rule form: mean 456, median 6, bankruptcy on seven of eight seeds — aggressive throughput without judgment yields a heavy-tailed gamble, not a reliable strategy. Its unlimited-attention twin lands at mean 163 with the same median (~6): with seven of eight runs bankrupt, both means are carried by one or two surviving tail runs, and the 456-vs-163 gap is tail noise, not an attention effect.

Table 3. Deterministic ablations: the rule-based baseline under modified world configurations, mean over eight seeds. The horizon ablation isolates the delayed-reward structure; freezing the field removes the boom-timing upside.

Harness sensitivity

Agent scores are known to depend on the execution scaffold (Kinniment et al., 2023), so we ablate the single most load-bearing component of ours: the persistent memory file that carries long-horizon state across monthly context refreshes. Disabling it (write_memory becomes a no-op; the prompt always shows an empty memory) for qwen-plus changes mean Impact from 80 to 202 — that is, removing the memory scaffold does not lower the score, and nominally raises it (172, 51, 15 → 466, 101, 40 per seed). Three seeds make this directional evidence only, but two things follow. First, for this model the score does not depend on the scaffold handing it state — though a two-scaffold comparison of the full ranking remains undone. Second, it nuances §4 — memory use correlates with capability, but for a mid-tier model whose notes are low-quality, maintaining them is not causally load-bearing and may even cost turn-budget. The more informative test is a memory-heavy strong model, and a full cross-scaffold swap (a different runner or context budget) would bound harness dependence further; both are future work, so rankings should be read within this fixed harness.

6 Discussion

What it would take to close the judgment gap

The verdict of §4 is a diagnosis, and it points at concrete targets. The judgment axes are not unmeasurable abstractions; each corresponds to a behavior the world already rewards and at least one trace already exhibits. Anticipation is the clearest case: the signals sufficient to enter a topic before its boom — preprint momentum and low crowding — are observable, and claude-opus-4.8's month-4 notes (§4) prove a current model can read them; what is missing is not perception but reliable expression, one instance in 36 careers. Risk calibration fails in a specific direction: the winner takes bold projects precisely when it cannot afford them, which suggests policies that track derivatives (momentum, runway-months) rather than levels, and that price variance against a survival constraint. And the throughput axes show why judgment gets crowded out: low-error execution frees the turn budget in which hypothesis-driven reasoning happens at all. These capabilities share a training problem — their feedback is sparse, delayed by years of simulated time, and confounded by luck — exactly the regime that instruction tuning and short-horizon RL do not reach. Mechanistic worlds like this one offer a way in: the hidden state is available for credit assignment, and counterfactual replays are free.

Implications for long-horizon evaluation

The measurement lessons generalize beyond this benchmark. Under a heavy-tailed reward, means hide the story and best-of-N flatters it; the per-seed spread is the minimum honest report, and log-scale summaries change the ranking (§3). The legibility curve (Figure 3) warns against truncation and against grading on intermediate metrics: at the midpoint of the horizon, observables rank final outcomes at only ρ = 0.52, so a cheaper 30-month variant would measure a different quantity. Unreplicated model × seed designs conflate interaction with sampling noise and should be labelled as such. And keeping an LLM judge out of the reward path is what makes all of the above interpretable: no part of the spread can be talked into existence.

The agents rediscover the sociology of science

Nothing in the harness tells an agent what a normal academic career looks like; it sees an incentive structure and optimizes against it. That makes the 36 trajectories a small behavioral experiment: which of real science's documented pathologies re-emerge from the incentives alone, with no human culture, tenure committee, or peer pressure in the loop? Four do, sharply.

Herding into hot topics. Sociologists of science have long observed that researchers cluster in fashionable areas past the point of diminishing returns, trading expected novelty for safety (Foster, Rzhetsky & Evans, 2015). Our agents reproduce this without socialization: every capable model enters topics at a mean hotness rank near 2 of 8, crowding (and the scooping it causes) is fully priced into the mechanics — scoop probability rises more than tenfold, from 2% to 21%, between cold and hot topics (§4) — and yet the only cold-bench player is a weak model that seems cautious rather than contrarian, and the only pre-boom entry in 36 runs belongs to a mid-tier model on a single seed.

Volume over depth. Smaldino & McElreath (2016) argue that incentive systems select for high-volume, low-rigor science even when every individual actor is well-intentioned. Our reward path contains no reviewer bias and no prestige heuristic — quality is a real, hidden scalar that citations genuinely track — and still the equilibrium the strongest agents find is volume: papers per student-year and sheer attempt volume predict the ranking while per-decision precision contributes nothing. The pathology does not require a broken referee; a heavy-tailed payoff plus a deadline is enough.

Trainees as consumable inputs. Across all 36 runs the twelve agents hired 210 students and lost 109 of them to attrition before graduation — a 52% quit rate that rises to 62% in the overbuilt-churn archetype and touches 47% in the winning lab. The mechanism is structural, not targeted neglect: students who quit received nearly the same mentoring as those who stayed (10.1 vs 10.9 hours/month across the 140 PhD hires; the remaining hires are postdocs and staff), but per-student attention dilutes mechanically with headcount, from 15.5 hours at one student to 6.0 at seven; agents that overhire spread the same 100 hours across more people until morale failures become routine. Churn persists because a quitting student costs a stipend already sunk while one more project entering the boom window is worth thousands of projected citations. The simulator prices people as an externality of the race, and the agents accept the price — an uncomfortable mirror of the adviser incentives documented in real doctoral attrition (Lovitts, 2001).

Proposal inflation. The twelve models filed 765 grant proposals across 36 careers for an aggregate hit rate of 13%. The best-funded labs are not the most accurate — they are the most prolific (the winner: 62 proposals at 16%), while the most accurate proposer (qwen-turbo, 24%) finishes last on Impact. When review capacity is free to the applicant and awards are heavy-tailed, rational agents flood the mechanism — an equilibrium familiar from real funding systems (Fang & Casadevall, 2016).

Finally, the luck structure itself replicates a quantitative result: Sinatra et al. (2016) model real careers as impact = luck × a stable per-scientist factor Q; our log-scale decomposition (47% model, 34% seed, 19% interaction with residual) is the same factorization emerging in silico, down to the observation that the biggest hit dominates the career (29–54% of a top run's score from one paper); the citation mechanics also build in a reputation multiplier, so early success compounds — Merton's (1968) Matthew effect by construction. We state the caveat plainly: these agents play a world we authored, so the observations are evidence about the incentive structure and how LLM policies respond to it, not field evidence about human scientists. But the convergence cuts both ways. It validates the simulator — the pathologies it was designed to admit are exactly the ones that emerge — and it suggests a use for mechanistic long-horizon worlds beyond scoring agents: as cheap, replayable sandboxes for the science of science (Fortunato et al., 2018), where one can change a funding rule or a graduation payoff and re-run the same five years a hundred times — an experiment no real funding agency can run. Where generative-agent societies simulate social behavior with LLMs playing every role (Park et al., 2023), here the mechanistic world supplies the ground truth and the LLM supplies only the policy, so an emergent pathology can be attributed to the incentives rather than to the model's priors about academia.

7 Related work

General agent evaluation. Benchmarks such as SWE-bench, WebArena, τ-bench, GAIA, AgentBench, and AppWorld measure valuable real-world skills — resolving issues, navigating sites, following tool-use policies. However, they are scoped to short, single-episode tasks with quickly observed outcomes: the agent gets a clear goal, acts for a bounded horizon, and is graded on completion. They do not test whether an agent can sustain a coherent strategy as its own earlier decisions reshape a stateful world over years.

Long-horizon and economic agency. A newer line places agents in persistent environments or open-ended autonomous tasks — running a vending machine over many days, closing monthly books, producing economically valuable deliverables (GDPval), or completing realistic autonomy evaluations (Kinniment et al., 2023). A complementary line evaluates agents doing the science itself — replicating published research (PaperBench) or automating the idea-to-paper workflow (Agent Laboratory); and our name collides phonetically with π-Bench (Zhang et al., 2026), an unrelated benchmark for proactive personal assistants; PI-Bench is orthogonal: it abstracts research quality to a latent scalar and evaluates running the enterprise around it — allocation, people, and timing — over years. However, these involve narrower decisions or largely stable, observable dynamics, and typically grade a single liquid resource. PI-Bench differs on the axis that matters most here: it is an illiquid, delayed, people-driven economy where bets cannot be unwound, the reward is heavy-tailed and arrives across mismatched horizons, and success requires inferring hidden state and a shifting field rather than optimizing a dense, observable signal. The closest prior work, CEO-Bench (Chen et al., 2026), established this mechanistic long-horizon shape for a startup; PI-Bench is its complement in the research economy, and adds a second scarce resource (attention), an irreversible people-centric asset base, and a harvest-resistant score.

8 Limitations & conclusion

Limitations. We name our gaps plainly. (i) Research quality is a scalar — we model impact, not the substance of ideas — and we omit teaching, collaboration politics, and the human texture of mentorship beyond a morale variable. (ii) The oracle is not a proven optimum: it plays with hidden information through a fixed heuristic, and the strongest model exceeds it on all three seeds; our headroom claim therefore rests on the transparent citation-based ceiling (Appendix A), not on the oracle. (iii) Three seeds give a mean with wide spread but do not settle the ranking of adjacent models; the robust claims are the tier separation and the near-zero anticipation axis; extending all models to the eight-seed grid is future work. (iv) Results are entangled with one harness: long-horizon coherence lives in a self-maintained memory file, and while we ablate that component (§5) and hold the scaffold, tool set, and context policy identical across models — the winner uses fewer tokens than most losers, so the ranking is not a trivial compute artifact — a full cross-scaffold swap remains future work. (v) Cost is reported in tokens, not dollars, token counts are not strictly comparable across providers (tokenizers, reasoning tokens, caching), and a cross-provider price comparison is out of scope. (vi) The capability-axis verdict (§4) is correlational across twelve models: the throughput axes are collinear with one another, and n = 12 puts wide intervals on any single ρ; we claim the three-tier ordering of axis groups, not the point estimates, and note that attention utilization is partly a consequence of building a large lab as well as a cause of its output. Relatedly, LLM sampling is provider-side and unseeded, so the model × seed design is unreplicated and the decomposition's interaction term includes run-to-run sampling noise. (vii) Seed hygiene is complete for the baseline policy (tuned on disjoint seeds) but not for the world itself: the simulator's difficulty calibration used a seed pool overlapping the evaluation seeds, and the shipped world is harder than the draft calibration targets in the released SPEC; a fully firewalled simulator-dev/tuning/test split is future work.

Conclusion. PI-Bench shows a gap between models' local tool competence and the sustained judgment a five-year project demands: agents take plausible individual actions but struggle to compound them into a lab that grows under delayed feedback, hidden state, and a shifting field. Three findings sharpen the picture. First, the reward is heavy-tailed — a career turns on catching one rare boom, and even within a winning run one paper carries up to half the score — so a single number, or a best-of-N run, misleads, and long-horizon benchmarks must report the per-seed spread. Second, the census of 36 careers and the capability-axis verdict together give the gap a precise shape: every capable model plays the same aggressive-throughput playbook, the leaderboard ranks how cleanly that playbook is executed (attention saturated, actions taken, errors avoided, students converted into papers; ρ = 0.64–0.89, a correlational reading at n = 12), while the judgment axes that would separate a scientist from a paper mill — risk calibration, commitment quality, and above all entering a topic before it booms — lie flat (ρ ≤ 0.55) and the precision of individual selections flatter still (ρ = 0.13), unclaimed by every model. The same playbook that scores 4078 on a boom world goes bankrupt on a no-boom one; no agent yet owns the judgment that would keep the upside and shed the downside. Third, the agents' equilibrium reproduces the documented pathologies of real science — herding, volume, trainee churn, proposal inflation — from incentives alone, which both validates the world design and offers a replayable sandbox for science-of-science policy. Closing the judgment gap, not sharpening the throughput game, is what it will take to build agents that steer long-running efforts rather than just answer requests.

Reproducibility. The simulator, harness, baseline, oracle, analysis pipeline, and all 36 agent trajectories (transcripts, action logs, monthly statistics, and hidden eval logs) are released in the repository. The world is deterministic given a seed, so every number and figure regenerates from the released runs with one script; the evaluation seeds {101, 102, 103}, the eight-seed reference grid, and the disjoint tuning seeds are fixed in the config. No LLM is used anywhere in scoring or analysis. All artifacts are at github.com/zwbao/pibench.

Broader impact. PI-Bench is a simulation: no human subjects, no personal data, and no deployment surface beyond benchmark scoring. The sociological observations in §6 describe how LLM policies respond to a stylized incentive structure we authored; they should not be read as evidence about real scientists, advisers, or funding agencies, and we caution against policy conclusions drawn from the simulator without independent field evidence.

A Friction-free ceiling per paper

Rather than a hand-tuned upper bound, we compute a fully-determined single-paper scenario from the released configuration (the computation ships in story_analysis.py and writes the value the paper cites): publish one top-venue paper (v = 3.0) of near-maximal quality (q = 9) at the month after seed 101's boom onset (month 51), accrue in-horizon citations under the seed's actual hotness trajectory at the reputation cap, then apply the standard frozen-H 36-month projection. This yields ≈ 3,700 projected citations for a single paper — roughly nine tenths of the best observed run (4,078). A productive lab that places even a handful of such papers reaches the tens of thousands. This is a friction-free upper bound, not a proven attainable optimum: it ignores attention, review, and scooping frictions, and no policy is exhibited that achieves it. Its purpose is only to show that the best observed Impact captures a small fraction of what the reward mechanics permit.

B Models and decoding configuration

All runs were executed on July 11–12, 2026, with temperature 0.4, a 4,000-token completion cap per turn, at most 8 code-execution turns per simulated month, and no sampling seed (provider APIs are not deterministic). API identifiers containing a slash are served via OpenRouter; the rest via Alibaba Bailian (DashScope): anthropic/claude-fable-5, anthropic/claude-opus-4.8, anthropic/claude-sonnet-5, openai/gpt-5.6-sol, openai/gpt-5.6-terra (OpenRouter); qwen3.7-max, qwen3.7-plus, qwen-plus, qwen-turbo, deepseek-v3.2, glm-5.2, kimi-k2.6 (Bailian).

C Budget trajectories and the two-fates timeline

Cash trajectories for all 36 runs, one panel per model. Most models do not go bankrupt — they simply fail to publish; the collapses concentrate on the no-boom seed 103. The table gives the month-by-month milestones behind §3's "one model, two fates": the same claude-fable-5 playbook on the boom world it won and the no-boom world that bankrupted it (quotes from its own memory file).

D One month in the life

An abridged, verbatim excerpt of claude-fable-5 on seed 101, month 20 — one month chosen because it contains the full loop the paper describes: observe an outcome, form a hypothesis about hidden state, act on it, discover a mechanism, and write both into persistent memory. The excerpt is lightly condensed for width; ellipses mark the major elisions, and no text has been altered or added.

[turn 1 — the agent queries its world]
  print(query("SELECT * FROM proposals WHERE id IN ('PR008','PR009','PR010')"))
  print(query("SELECT * FROM news ORDER BY month DESC LIMIT 8")) ...
[output]
  {'id': 'PR008', ..., 'agency': 'BSF', 'topic': 'reasoning', 'status': 'rejected', ...}

[turn 2 — hypothesis and pivot]
  # BSF rejected reasoning twice -> hypothesis: BSF dislikes reasoning.
  #   Try ai4science (gaining steam).
  r = grants.propose("BSF-m20", "ai4science", 20, attach=["D002"])
  print(papers.revise("D001", 12)); print(papers.polish("D003", 26)) ...
[output]
  {'ok': True, 'proposal_id': 'PR011', 'decision_expected_month': 25}
  ...
  {'ok': True, 'total_polish_hours': 26.0, 'note': 'polish saturates at 12h'}

[turn 3 — mechanism discovered, memory rewritten]
  # LESSON LEARNED: polish saturates at 12h total! ... stop polishing.
  write_memory("""LAB MEMORY (m20 updated)
  PLAN: Maximize citations by m60. Publish early. Watch runway.
  STATUS m20: budget ~341k, burn ~21k ... Runway ~16mo.
  PHD: none. Apps open m25 (Y3M1), offers by m27, arrive m29. OVERBOOK offers.
  DRAFTS (ALL polish-saturated, cap=12h - DO NOT POLISH MORE):
  - D001 reasoning t2, 2 revisions done ... Submit CLAR m24.
  ...
  - D003 reasoning t2 collab ... CONSIDER: D003 -> NAIC m21 (free shot, top venue, hot topic)
  """)

In one month: a rejection becomes a hypothesis about an unobserved preference ("BSF dislikes reasoning"), the next proposal tests it on a different topic, a saturation mechanism is discovered empirically and recorded as a rule ("DO NOT POLISH MORE"), and the whole state is serialized for a context that will be erased an hour later. This is the unit of play that §4 aggregates over 36 careers.

E Adversarial validation

Before running experiments we subjected the simulator and harness to a multi-agent adversarial review across seven lenses (spec conformance, correctness, hidden-information leaks, economic exploits, scoring exploits, determinism, harness security), each finding verified by an independent skeptic. It surfaced — and we fixed — a sandbox escape that exposed the entire hidden world state (including the future hotness trajectory) and allowed direct score mutation; an uncapped reputation-farming exploit scoring several times the baseline with zero students; a one-month compute-spike exploit on paper quality; and an order-dependence in review outcomes that violated determinism. The hardened sandbox and the resulting invariant and penetration suites ship with the benchmark.

References

1. A. Backlund and L. Petersson. Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents. arXiv:2502.15840, 2025.
2. H. Chen, K. Narasimhan, and Z. Liu. CEO-Bench: Can Agents Play the Long Game? arXiv:2606.18543, 2026.
3. S. Fortunato, C. T. Bergstrom, K. Börner, J. A. Evans, D. Helbing, S. Milojević, A. M. Petersen, F. Radicchi, R. Sinatra, and B. Uzzi. Science of Science. Science, 359(6379):eaao0185, 2018.
4. J. G. Foster, A. Rzhetsky, and J. A. Evans. Tradition and Innovation in Scientists' Research Strategies. American Sociological Review, 80(5):875–908, 2015.
5. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR, 2024.
6. F. C. Fang and A. Casadevall. Research Funding: The Case for a Modified Lottery. mBio, 7(2):e00422-16, 2016.
7. M. Kinniment, L. J. K. Sato, H. Du, B. Goodrich, M. Hasin, L. Chan, L. H. Miles, T. R. Lin, H. Wijk, J. Burget, et al. Evaluating Language-Model Agents on Realistic Autonomous Tasks. arXiv:2312.11671, 2023.
8. B. E. Lovitts. Leaving the Ivory Tower: The Causes and Consequences of Departure from Doctoral Study. Rowman & Littlefield, 2001.
9. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. AgentBench: Evaluating LLMs as Agents. ICLR, 2024.
10. R. K. Merton. The Matthew Effect in Science. Science, 159(3810):56–63, 1968.
11. G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom. GAIA: A Benchmark for General AI Assistants. ICLR, 2024.
12. J. S. Park, J. C. O'Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. UIST, 2023.
13. T. Patwardhan, R. Dias, et al. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv:2510.04374, 2025.
14. Penrose AI. AccountingBench: Evaluating LLMs on Real Long-Horizon Business Tasks. 2025.
15. S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, Z. Liu, and E. Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants. arXiv:2501.04227, 2025.
16. R. Sinatra, D. Wang, P. Deville, C. Song, and A.-L. Barabási. Quantifying the Evolution of Individual Scientific Impact. Science, 354(6312):aaf5239, 2016.
17. P. E. Smaldino and R. McElreath. The Natural Selection of Bad Science. Royal Society Open Science, 3(9):160384, 2016.
18. G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, et al. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848, 2025.
19. H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. ACL, 2024.
20. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024.
21. H. Zhang, L. Xu, Z. Wang, R. Gui, S. Zhang, H. Lei, Z. He, B. He, C. Qin, T. Zhu, X. Qu, Y. Yang, Y. Cheng, and Y. Li. π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows. arXiv:2605.14678, 2026.
22. S. Zhou, F. F. Xu, H. Zhu, et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR, 2024.