Random forests still own the one-quarter statement forecast. I expected Forma to win there too, and its failure explains why the rest of the paper matters.
Johnson, Jiang, Chaudhuri, Chen, Falvey and O'Cofaigh introduce a benchmark and model for a task that, by their account, prior work has not covered: jointly forecast the entire financial statement, comprising all 78 Compustat quarterly line items, from one to twenty quarters ahead. The use case is a discounted cash flow. Most enterprise value lies beyond the one-year window covered by the earnings-prediction literature, and a terminal-year estimate needs margins, reinvestment and financing that fit together. The forecast object is the complete pro forma.
ProForma-20Q gives the model twelve quarters of reported history, a Fama-French 48 industry code and a scale deflator, then requests the next twenty quarters. Firm identity, returns and analyst data are excluded. The Compustat quarterly panel covers non-financials, with 32,851 firms and 1,173,598 firm-quarters. Time determines the splits: train 1971-2001, validate 2002-2009, test 2010-2024 (352,962 firm-quarters). Evaluation uses out-of-sample R-squared on changes in standardized space, preventing a model from earning credit by carrying levels forward.
Forma treats statement history as an unordered set of (account, quarter, value) tuples. Missing disclosures produce no token. That design matters in a sparse panel. The average firm-quarter contains 64.1 of the 78 items, rising to 73.0 in the test period, while only 4.67% of test firm-quarters contain all 78. Listwise deletion would discard 95.3% of the test panel. Mask tokens represent future cells. The model has four layers, d_model 128, four heads and 942,210 parameters. It is trained against masked values with a Gaussian beta-NLL, and five seeds are combined as a mixture. Each seed takes sixteen and a half hours on one A100. Cloud compute for the set costs about $120.
The full test result is 0.289 change-space R-squared. Learned baselines reach 0.272 for a per-(item, horizon) random forest, 0.258 for elastic net, and 0.253 and 0.247 for two feed-forward mixtures. Simpler anchors fall farther behind: 0.183 for a pooled fade/AR(1) using only the current level, 0.155 for zero-shot Chronos-2, and minus 0.041 for a seasonal random walk.
Why the smaller specialist wins
Parameter count does not rescue the larger networks. The 4.2M-parameter feed-forward network scores 24.7%, and its 3.1M-parameter linear sibling reaches 25.3%. Both finish 3.6 to 4.2 points behind the 0.9M-parameter transformer. The authors attribute the difference to architecture rather than scale.
The baseline progression gives that interpretation some support. A pooled AR(1) already captures 18.3%, more than half Forma's total. The remaining opportunity lies in conditional behavior, especially reversion speeds that vary with the rest of the statement. Set attention can inspect every reported account at once, an option unavailable to a per-item regression. The paper does not test this mechanism directly, so it remains my reading. Its shared-parameter feed-forward nets lose as well.
The frontier-LLM comparison is entertaining and carries the thinnest evidence. It covers 2,103 origins across 133 firms, with 2,152,314 of a possible 3,280,680 cells. Forma scores 0.299 on that sample. Claude Opus 4.8 reaches 0.186, GPT-5.5 records 0.174, and Claude Sonnet 5 gets 0.158. The gap expands from 5.5 points at h=1 to 15.9 at h=20. Opus drops from 32.5% to 7.8% over the same span.
MAE leaves the LLMs almost tied, at 0.362 to 0.368 against 0.348. The authors interpret this pattern as implicit conditional-median forecasting, which cannot support a DCF because medians do not add. The narrow sample has one offsetting feature. According to the authors, the comparison is conservatively biased toward the generalists because statement realizations from 2010-2024 plausibly appeared in their pretraining data. That bias cuts against the reported result.
The lead changes at h=3
Forma loses the first quarter to the random forest, 39.0% versus 39.8%, with a DM t of +12.3 favouring the forest. They draw at h=2. Forma moves ahead from h=3 and ends at 0.225 versus 0.193 at h=20.
The re-implemented chained GBM of Geertsema et al. follows the same curve on its 25-item footprint. It wins at h=1, scoring 32.6% against 32.1%, with t=+6.9. By h=10 it loses 25.1% to 17.2%, followed by 19.7% to 13.3% at h=20. Tree ensembles lead only in the one-quarter cell. The authors interpret that result as evidence that RF handles short-term earnings timing better. Forma controls quarters three through twenty, a sequence spanning five years and reaching the point where terminal-year estimates are set.
The long end carries a caveat that the authors state plainly. A cell at horizon h can be scored only when the firm still reports at t+h, so absolute skill at h=20 applies to survivors. They also found that allowing the future query grid to depend on cells the firm later reports introduced survivorship leakage worth about 0.7pp of test R-squared. They closed it.
Coherence costs 3.8 points
Forma's raw dollar statements come close to adding up. Across 124.7M enforced instances, median absolute identity violation is 3.7% of identity gross scale, despite the absence of an imposed constraint. Projection onto the coherent subspace, weighted by each item's predictive variance, reduces violations to zero.
The authors report the accuracy charge. R-squared declines from 0.289 to 0.251, a 3.8 point drop, with a DM t of minus 1.4. MAE improves from 0.408 to 0.407. Their abstract calls this no statistically significant accuracy cost. Equal-weighted projection is a disaster, producing R-squared minus 5.51, because spreading a dollar plug uniformly makes it enormous relative to minority interest.
Statistical insignificance comes from an effective sample of 60 calendar quarters and Newey-West bandwidth 19. The 3.8-point decline exceeds twice Forma's 1.7-point lead over the random forest. Failure to reject a zero cost does not establish a zero cost. Disclosure is handled properly, since Table 3 shows the decline. A footnote also identifies the comparison problem: reconciled Forma should not be compared with unreconciled full-sample entries. Coherent-to-coherent evaluation would impose each model's reconciliation charge, while competitors lacking predictive variances cannot assign that charge observation by observation.
Fair.
Anyone requiring balanced statements owns the 0.251 result, with the reconciled forest result still unknown.
The scenario interface is the feature I would use. Future assumptions enter as more tuples, allowing an analyst to pin a revenue path and re-forecast everything else without retraining. Supplying the true realized twenty-quarter revenue path raises R-squared on all remaining cells from 30.5% to 34.8%. MAE declines from 0.409 to 0.383. The improvement grows from 0.9pp at h=1 to 7.4pp at h=20. Balance-sheet items gain +8.9pp, compared with +5.6pp for income-statement items and +0.7pp for cash flow.
The authors correctly describe this exercise as an oracle and upper bound on scenario value. An analyst's revenue forecast will differ from the realized path. The paper does not measure how much of the 4.3-point improvement remains when that view is wrong.
Our separate implementation
We could not reproduce the benchmark. Doing so requires a Compustat/WRDS panel from 1971 to 2024 and the paper's exact 78-item schema, while our usable fundamentals and price history start around 2010. Everything below concerns an analogous set-transformer trained on the filings available to us. It is no test of the paper's claim. The account mapping follows a Sharadar-style structure instead of the paper's 78-item footprint.
We added a DCF and portfolio layer that appears nowhere in the paper. It sums expected free cash flow at h=17-20, then calculates five-year FCF growth, operating-margin improvement and DCF upside using a 10% discount rate and 2.5% terminal growth. Those three signals are z-scored within sector. The portfolio holds the top quintile long and the bottom quintile short, runs at 200% gross, remains sector dollar-neutral, and rebalances quarterly within the top-500 liquid universe.
From 2020-01-01 through 2025-12-31, the strategy returned 17.46% cumulatively. Its Sharpe was 0.57 on 5.11% realized volatility, with a maximum drawdown of -10.54%. Beta to SPY was -0.01. The sector dollar-neutral construction therefore performed its intended job, leaving the portfolio with essentially no market exposure.
The paper reports no Sharpe, no return and no drawdown. Its performance statistic is 0.289 change-space R-squared across 327.2M test cells from 2010-2024. The figures answer different questions on different samples, so their distance does not represent a replication failure. Five per cent volatility on 200% gross fits a composite with very little cross-sectional separation after winsorization, sector neutralization, a 10-point rank buffer and 50% partial rebalancing.
Ours is one automated pass.
Several differences matter. The accounting corpus changes, and our six-year window is dominated by COVID, the 2022 rate move and 2023-2025 mega-cap dispersion. The paper instead pools 60 test quarters. Our training block most likely omits the beta-NLL objective and the five specified seeds. It also most likely omits the horizon curriculum, which increases the maximum queried horizon from 4 quarters to 20 (4, 4, 8, 12, 16, 20).
Two assumptions flatter our performance. We charged four tenths of a cent a share and zero slippage, while the design called for 10 bps per side and 50 bps annual borrow. The signal also accepts any filing dated at or before the cutoff, without a T+1 availability lag. We cannot fully explain the distance between our 0.57 and the paper's forecast skill from the evidence available. The honest interpretation is that we measured our own pipeline, rather than reached a verdict on the authors' work.
Has anyone priced the forecasts?
The authors explicitly evaluate forecast quality alone. Valuation still requires assumptions about discount rates and terminal values. Their scope is correct, but it leaves the main trading question open. Change-space R-squared of 29% leaves most variation in statement changes unexplained. The paper anticipates that objection: in change space, 20-40% at multi-year horizons is strong, while level-space intuitions around 90% are inapplicable.
I accept the metric. I still want to know what 29% buys when a position is put on. The reported results contain no analyst-consensus or practitioner pro forma benchmark. Analyst data is excluded as a feature by design so that Forma remains applicable to private firms.
Much of the test I want already exists. The LLM protocol samples consecutive origins to create overlapping-target panels for a companion analysis of forecast revisions. Revisions to a joint statement forecast supply the natural event study. Do firms whose twenty-quarter cash-flow paths are marked up subsequently outperform after costs? Does the result survive controls for the profitability and investment factors that already price fundamentals?
Until that test is run, this remains a very good forecasting paper with a genuinely new task and no evidence about money. Show me the revision sort with a t-stat, and I will change my view on whether the advantage from h=3 through h=20 is worth paying for.