A free standardized log count of matched headlines reduced SPY's next-day variance loss by 0.001720 across 856 evaluation dates, with a familywise interval of [0.000719, 0.002830]. The paid semantic score improved no endpoint. That comparison is Mohanty's result that matters to an implementer. It is also the only result we can rebuild in part.
We cannot reproduce the study. Its selected headline packets come from GDELT, a public archive of crawled news items that we do not have, while the scoring model, gpt-5.6-luna at high reasoning effort, is unavailable to us as a fixed artifact. Our version can test the headline-count control and a text-derived importance proxy. The proxy stands in for the paper's LLM feature; it is not the same feature.
What was tested?
An LLM reads each day's macro and market headlines, then rates their importance for broad U.S. stocks on a 0 to 100 scale. Any information about tomorrow's risk beyond option prices should improve a next-day variance forecast based on VIX. Mohanty scored one packet for each trading-day bucket, running from the previous NYSE close to the current close. Each packet contained up to 25 whitelisted headlines, capped at five per domain. Scoring used gpt-5.6-luna through the Batch Responses API at the provider's "high" reasoning setting.
The score became z = (importance - 50)/25. A single coefficient then added it to an option-implied one-day variance baseline, using VIX for SPY and VXN for QQQ as a same-date transfer check.
Each asset had two endpoints. The first was a binary indicator for whether tomorrow's absolute return exceeded the trailing five-year 90th percentile, fitted by log loss. The second was squared percent return, fitted by Gaussian QLIKE. Mohanty fitted everything on 250 usable 2022 dates, locked the models, and applied them without refitting from 2023-01-03 through 2026-06-30. Each asset had 856 strict-valid dates from 865 eligible dates, for 98.96% coverage. Prespecified inference used 95% intervals from 20,000 moving-block bootstrap draws with 20-day blocks. The study contains no returns and no cost model anywhere. Its outputs are forecast-loss differences.
Calibration assigned all four importance coefficients exactly zero. Every augmented forecast therefore matched its baseline on every date, producing an improvement of 0 with a [0, 0] interval on every endpoint. Mohanty calls this calibration-induced degeneracy and bases the paper's prescription on it. The proposed calibration-viability checkpoint fits the mapping, perturbs the candidate feature over a frozen support grid, and buys holdout feature values only when forecast response D exceeds a threshold δ.
The paper also establishes the limit of that result. Signed weights reactivated all four mappings, yet none improved forecasts after familywise correction. Mohanty concedes that the prespecified comparison had zero ability to detect a score effect. The supported conclusion stays narrow: the procedure did not evaluate incremental score value, rather than showing every mapping is uninformative.
Calibration killed the score
The score varied substantially. During 2022 it ranged from 4 to 98, with a median of 73, an SD of 17.43, and 60 distinct values. The later sample ranged from 5 to 97 across 76 values. The bound created the zero. Mohanty restricted the news coefficient to [0, 5], imposing the directional prior that more important news should not lower next-day risk. The optimizer reached the lower bound and remained there.
This feature never entered the forecast.
A failed out-of-sample feature and one blocked during calibration are different objects. Mohanty's feature was the second. The entire diagnostic reduces to one perturbation: moving the score from 0 to 100 left the forecast unchanged.
Did signed weights help?
For the post-hoc repair, Mohanty widened the bounds to [-5, 5] and refitted the same 250 rows. All four optima moved inside the bounds and turned negative: SPY binary -0.00536, SPY variance -0.15878, QQQ binary -0.09386, and QQQ variance -0.17830. The sign constraint was binding.
The repaired mappings delivered four nonpositive improvements in the later period. SPY variance registered -0.007452, familywise [-0.018435, 0.000266]. QQQ variance registered -0.008548, familywise [-0.022584, 0.000948]. Nothing improved after correction. The four-endpoint Bonferroni familywise intervals apply to the two post-hoc families, signed repair and headline count, at 98.75% per endpoint. Reopening the route from feature to forecast was necessary and nowhere near sufficient.
The conditional-rank results show why. Raw later-period Spearman between importance and next-day absolute return measured 0.1413 for SPY and 0.1284 for QQQ. Conditioning only on implied volatility lowered those figures to 0.0557 and 0.0546, declines of 60.6% and 57.5%. Conditioning on packet mechanics barely changed them, at 0.1393 and 0.1269. Most of the score behaved like VIX translated into words.
An importance-only forecast did improve SPY binary loss by 0.00440 at ordinary 95% coverage, with [0.00094, 0.00789]. Its absolute binary loss, however, was 0.328, compared with 0.178 for GARCH and 0.179 for VIX.
Enough detail, two missing pieces
The paper prints the forecast equations, including a variance floor of 1e-8 and a probability clip of 1e-6. It lists all 26 whitelisted domains and all 38 query phrases. Deduplication follows exact rules: containment matches of at least 25 characters, SequenceMatcher similarity of at least 0.85, and retention of the earliest crawl observation. The first pass had a token ceiling of 8,192. The single permitted retry allowed 16,384. Buckets ended at 16:00, or at 13:00 on prespecified early closes.
No number is supplied for δ, even though the prescription turns on that parameter. The paper says it should be fixed from the forecast scale, scoring rule, and minimum change worth paying to evaluate. The omission makes little difference here because D was exactly zero, so any positive δ stops the run. When a fitted response is small and nonzero, the gate becomes a free parameter.
The paper's GARCH grid illustrates the issue. It selected the nested zero-weight model whenever the training-loss improvement was no more than 1e-12. The signed repair's SPY binary training gain was 1.12e-6. The tolerance determines whether the feature counts as "active".
Two further implementation choices matter. The large-move threshold yielded only 40 SPY events and 46 QQQ events during 856 days. The paper calls large moves rare, in those words. Changing the percentile creates a different test. It also generated one score per day and checked order robustness only during the 27-date prompt bakeoff. An implementer therefore has no measured score noise against which to judge a substitute.
Mohanty describes the archive plainly. It permits recalculation from released derived rows, while end-to-end recreation of headline collection or stochastic model scoring remains unavailable. A fresh API run would constitute a new experiment. He also discloses prior exposure to SPY outcomes through June 2026 through a separate project.
A coverage correction changed a recorded prompt decision from STOP to GO. P1 passed validity at 53/54 = 0.9815 against a 0.98 threshold. Exactly 16 truncated 2022 outputs were retried after the first-pass market result was known. That retry reduced the screening rank correlation from 0.1133 to 0.0980, while the screen remained recorded as passing.
The $16.17 lesson
Full-history paid scoring cost $16.1749105 from a $25.0817470 program, and the spending occurred before the calibration fit. A perturbation test requiring no holdout outcomes would have halted it. Mohanty treats the amount as an operational illustration rather than a loss estimate. He also says the gate authorizes an evaluation without promising a favorable result.
As protection against purchasing data that a model cannot use, the rule is cheap and correct. The paper makes the same limited claim for its gate: it checks a necessary condition for interpretation rather than predictive accuracy. It does nothing about overfitting. Nor does it solve the problem of a score whose 0.1413 rank correlation with next-day absolute return falls to 0.0557 after conditioning on VIX.
What we could build
Our packets come from timestamped financial-news tables instead of GDELT selections. We compute the importance proxy in code from headline text. This build tests the headline-count control and that in-code proxy, which remains a stand-in for the paper's feature.
The count is the only feature with a familywise-corrected forecast-loss improvement. Mohanty says plainly that it is not a generally superior forecast. It worsened SPY binary loss (-0.007913, familywise [-0.013017, -0.003549]) and QQQ variance (-0.007883, familywise [-0.014661, -0.002344]). QQQ binary remained unresolved at +0.000150, familywise [-0.000020, 0.000332]. The result is one winning endpoint out of four, from a family designed after the null was known.
We have previously examined a risk signal that works in one episode and fails in others in correlation geometry. The selection arithmetic is the same here. A free packet-volume statistic beat the expensive semantic feature on the sole endpoint where either worked, making it a fair and demanding hurdle for paid scores. Evidence for tradable variance information beyond VIX is much thinner. We would not size on a 0.001720 QLIKE improvement without a cost model.
A date-masked rescoring would change our view if it removed the sign restriction, included headline count in the same fit, used an independently built corpus, and showed the semantic score beating count on the variance endpoint out of sample. Until then, the buildable contribution is the paper's five-step gate, fit then perturb then acquire. It uses no holdout outcomes and would have saved $16.17.