The most useful number in this paper is a gap between two arms of the same system: 0.96 Sharpe when the agent re-runs its own discovery loop every twelve months, 0.71 when it starts from the identical factor library and never re-discovers. Both gross of costs, both mean plus or minus SE over three replicates on the JKP US panel, out-of-sample from 2020. If that gap holds up, the thing worth owning in an agentic factor pipeline is the loop rather than the factors already in the drawer.
We did not reproduce this. The JKP US panel, the 87-variable CRSP/Compustat panel and the Bryzgalova reference panel are not among our available datasets, and the prompts and sandbox configuration behind SEADS are described in the paper but not released with the preprint, so we could not rebuild them. Every number below is the paper's own.
What Pan, Ding and Giesecke actually built
The setup is a factor mill with a language model in the generator seat. Pan, Ding and Giesecke call the general category Agentic Empirical Asset Pricing, meaning a system where LLM agents run hypothesis, formalization and evaluation without a human doing any of those three steps. Their instance is SEADS, their own system. A Planner emits one seed per candidate slot, split 70% explore and 30% exploit by default. Each seed goes to its own Proposer, which writes a Python formula over a characteristic panel. The paper describes a deterministic AST walk enforcing seven rule classes plus a taxonomy whitelist, which rejects look-ahead constructions before any code executes. Then a nine-check gate decides admission.
The gate is the strict part. Nine checks, in order. Two structural ones come first: a duplicate fingerprint, and a self-collapse check rejecting anything correlating 0.95 or above with a plain rank of one of its own declared inputs. Three statistical checks follow: coverage at least 0.5, a Rank-IC t-stat of at least max(3.0, a Bonferroni-dynamic bar that rises with cumulative trials), and out-of-decade regime sign agreement. A robustness trio closes it: novelty below 0.7 max correlation, partial-IC t at least 1.0 against the top five correlated controls, Fama-MacBeth t at least 1.0 against the same, with sign agreement required across Rank-IC, partial-IC and the Fama-MacBeth beta. Survivors go into a library. A Ridge regression with alpha=1 on the library's rank-normalized values ranks stocks monthly into a long-short portfolio.
Two panels, five rivals, 300 candidates each
Panel A is the US subset of the Jensen, Kelly and Pedersen global characteristics database, ex-microcap: 1,936,944 stock-months, 21,052 unique stocks, roughly 400 characteristics. Novelty there is checked against that same already-mined zoo. Panel B hands the system 87 primitive CRSP/Compustat variables instead, with novelty checked against the 45-characteristic Bryzgalova et al. panel. Both panels use an in-sample window of 2010 to 2019. Out-of-sample runs to December 2025 on A (about 71 months) and November 2024 on B (about 59). Five prior systems are re-implemented and run at a matched budget of 300 initial candidates each, same GPT-5.2 backbone, scored out-of-sample by identical code.
SEADS's static headline: mean per-factor OOS Sharpe 0.25 on Panel A and 0.16 on Panel B, with 14.0 and 13.8 admissions out of 300. Novelty comes in at 0.65 and 0.46, and the spanning-test alpha t-stat at 1.21 and 1.28. Means across five replicates, all gross of trading costs.
The evaluation move is the contribution
Every system in their audit table, five re-implementations plus SEADS itself, does the same thing: discover once, hold out a window, backtest the output. Pan, Ding and Giesecke call the missing piece a point-in-time process, and write that it is "the property no existing AEAP system we survey satisfies." Their protocol re-executes the entire loop at a sequence of historical decision dates. Repropose every 12 months, refit the combinator every 4, on a fixed-length 10-year sliding in-sample window. The library, the novelty reference set and the accumulated lessons carry forward.
This is the right question to ask of a research process rather than a research output. A gate that admits 14 of 300 candidates might be well calibrated, might be leaking, or might have caught a friendly window. One run's Sharpe cannot tell you which. Re-running under sliding information constraints at least starts to.
The authors themselves say the protocol is the same sliding-window re-evaluation discipline standard in trading-strategy validation, applied to the discovery process instead of a fixed model's parameters. The machinery is standard; the target is new.
Does the rolling comparison isolate persistence?
On Panel A, all five discovery systems beat the no-discovery floor, where the same raw 400-characteristic panel is refit with no admitted factors: mean Sharpe 0.82 to 0.96 against 0.62. The floor is fully deterministic, with no LLM step and no replicate variance. Every system clears it by a wide margin.
Among the five, SEADS leads at 0.96 plus or minus 0.04. Beyond Prompting, one of the five re-implemented rivals, is at 0.91 plus or minus 0.02, and QuantaAlpha at 0.90 plus or minus 0.02. The authors say plainly that SEADS is not reliably distinguishable from either. RD-Agent(Q) and AlphaAgent, both at 0.82 plus or minus 0.03, are the ones they call more clearly separated. And they flag that this compares each system's best rolling configuration, so gate design and reconstruction fidelity are in the comparison alongside persistence.
Which leaves the within-system contrast doing the real work. Same starting library, same refit schedule, only re-discovery removed: 0.71 against 0.96.
The static-versus-rolling contrast is the cleanest evidence in the paper, and the authors draw the same conclusion in their own words: "most of SEADS's rolling advantage is repropose itself, not the initial library." It argues for refreshing the search. The static arm also loses the ability to add any new factor at all, so the specific carry-forward mechanisms are not what is being tested here. What the authors do not claim is that the gap survives trading costs or more than three replicates, and neither claim is on the page.
Does the gate certify anything the portfolio wants?
The authors put this tension in their own limitations. Across 15 replicates, three per system on Panel A, 14 fall below the diagonal comparing mean per-factor OOS Sharpe against combined-portfolio OOS Sharpe. Combining independently-admitted factors typically does not preserve their average individual quality. Nine sequential checks certify a factor's marginal in-sample signal. None certifies what it contributes next to the factors admitted after it.
The productivity axis makes the same point from the other side. Beyond Prompting admits 96.4 on Panel A against SEADS's 14.0, and leads Panel A on mean per-factor performance too (0.41 against SEADS's 0.25, with AlphaForge second at 0.34 on 5.0 admissions). It loses Panel B, 0.11 against SEADS's 0.16. AlphaForge is worth pausing on: a non-agentic random search that beats SEADS on Panel A performance out of five total admissions, and the worst novelty of all six there at 0.96. But Beyond Prompting's 94 nominal Panel A admits have mean pairwise absolute correlation of 0.281 and an effective independent count of 3.46. SEADS's five Panel A replicates run 10 to 18 nominal admits with effective counts of 3.51 to 4.49. The larger set is one idea. The ablation is blunter still: turn off the Planner and admissions nearly triple to 37.2 while Sharpe collapses to 0.101 from 0.254. Admission count moves opposite to quality. Novelty barely responds anywhere in that battery, running 0.627 to 0.676 against a full-pipeline 0.653, which is its own argument against reading any single axis.
Structurally the systems find different objects. Pooling SEADS's flagship replicates across both panels, 83.8% of its admits are multiplicative or gate-style formulas and 9.0% additive spreads; Beyond Prompting's own natively-admitted set is 14.5% and 78.7%. A flat difference of two ranked characteristics is already spanned by a linear model holding both inputs, so a large additive admit set mostly adds arithmetic.
One finding deserves separate mention because it is about the tooling everyone is building on. In one matched 300-candidate run, RD-Agent(Q)'s native evaluator admitted 57 candidates and the authors' external firewall flagged 18 of them, 31.6%, for temporal look-ahead or cross-stock rank pooling, leaving the 39 they report. Their own Alignment Checker, an LLM asked to confirm each formula does what its stated rationale says, had to be grounded after it rejected a formula by citing a.shift(1) call that appeared nowhere in the code.
What stops this being a trading decision
Every Sharpe here is gross. The paper says so: "we model neither transaction costs nor turnover." A cost-adjusted return at 50bps per side, round-trip, scaled by each factor's own realized monthly rank turnover, is computed as a side diagnostic and does not enter any reported number. The spanning t-stats, 1.21 and 1.28, are below conventional bars, and the authors read them as suggestive rather than confirmatory evidence of value added beyond the reference set.
Sample size is thin in both directions. Panel B's 59 OOS months give a Lo (2002) standard error on Sharpe of 0.138 at a Sharpe of 0.5; Panel A's 71 months give 0.126. Replicates are 5 static and 3 rolling, set by API budget. Compute is not matched even though candidate counts are, and self-evolution, the source of the retries that break the match, is the single most load-bearing mechanism: removing it takes admissions from 14.0 to 7.8 and mean per-factor Sharpe from 0.254 to 0.080.
The backbone problem is the one I would push hardest on. GPT-5.2 answers at every historical decision date, and Panel A's out-of-sample window runs to December 2025. The authors call the residual leakage soft, on the argument that proposals come from feature descriptions rather than realized returns. They note the alternative, a release-dated model such as GPT-4o-mini from October 2023, leaves only about 25 clean OOS months, and that those months are confounded with a weaker, older model's reasoning capacity. Their fix is worse than the problem, which I accept. But the rolling protocol queries that same backbone at several dates, so whatever leakage exists is applied repeatedly to exactly the arm carrying the headline result.
What we can and cannot check
We cannot reproduce this. The three panels are not in our data, so the feature universe, the novelty reference and every cross-system comparison are off the table as stated. Anything we build substitutes a different discovery engine over our own point-in-time fundamentals and prices, and tests that substitute. Panel construction and cleaning conventions will differ materially. A reader should not come away thinking we ran their method.
What is portable is the protocol. Repropose on a fixed cadence, refit the combinator on a finer one, carry the library and reference set forward, and always run a static arm from an identical starting library so the refresh effect is separable from the initial draw. That comparison costs one extra arm and is the cleanest separation in the paper.
They built the release-dated-backbone path and disclose that no result uses it, because it would leave about 25 clean OOS months. Running it anyway, on more than three replicates, with turnover-scaled costs, is what would move the headline for me. Absent that, the paper has established that re-running an agentic search adds something over freezing it, on a metric that models neither transaction costs nor turnover, and the decomposition it offers, repropose against static, is worth more than the six-system ranking table it sits behind.