Rank the stocks with news by average daily headline polarity, buy the top 15%, and short the bottom 15% at the next open. Luo's 2019 sample gives that trade a best one-day cross-sectional rank IC of 0.0143. By day two, every tested model has changed sign.
The paper benchmarks financial sentiment classifiers under two scorecards. Its seven models range from TF-IDF Multinomial Naive Bayes and the off-the-shelf FinBERT and Financial-RoBERTa encoders to 7B and 8B large language models adapted with QLoRA. QLoRA places small low-rank adapters over a frozen 4-bit quantized backbone, allowing a 7B model to be fine-tuned on one GPU rather than a cluster. The adapters enter the attention and feed-forward projections at rank 16 and alpha 32. The backbones are Qwen2.5-7B, LLaMA3-8B and Mistral-7B.
Experiment 1 tests language classification on 33,549 labeled texts, drawn from five sources and reconciled into three classes. Experiment 2 tests returns. Seven frozen classifiers score 10,637 unlabeled Benzinga headlines from 2019 against a fixed S&P 100 universe, then the scores are matched to later returns.
The headline classification result belongs to Mistral-7B with QLoRA: 0.8840 accuracy and 0.8771 macro-F1. QLoRA moves Qwen2.5-7B from 0.7274 macro-F1 zero-shot to 0.8615. The economic results are much thinner. Across the seven models, the highest one-day rank IC is 0.0143, and all 28 model-horizon tests fail false-discovery-rate correction.
The trading signal needs little machinery. For each headline, a classifier supplies three probabilities: negative, neutral and positive. Subtracting the negative probability from the positive probability produces the score. Its range is [-1, 1], with neutral at zero. All headlines for the same stock and calendar date are averaged, preventing a name covered four times from receiving a larger signal simply because it drew more stories.
Fresh news determines the cross-section. Stock-dates without it are omitted instead of being assigned zero, so coverage changes the daily universe. Probability outputs also differ in calibration across model families. Each model is therefore ranked on its own scores.
The intended source of profit, as I read it, is short-horizon post-news drift following analyst-rating headlines. An upgrade or downgrade may contain information about revisions that liquid large caps have yet to absorb fully within one session. The paper says only that its downstream sample comes from the Benzinga analyst-ratings data. Trades enter at the adjusted open of the first trading session strictly after the signal date, making the one-day result an open-to-close return. This conservative timing choice carries much of the economic argument.
The controlled half of the paper
Separating the language and money tests is the right design. For the first experiment, Luo combines five financial text sources: Financial PhraseBank, an FOMC monetary-policy corpus, SEntFiN 1.0, Twitter Financial News Sentiment and NASDAQ news. The resulting three-class benchmark contains 33,549 observations and uses a fixed-seed split of 25,664 / 2,852 / 5,033. FiQA is left out because converting its continuous scores into three classes would require researcher-chosen cutoffs.
Mistral-7B with QLoRA leads the unified Experiment 1 test set of 5,033 held-out texts, reaching 0.8840 accuracy and 0.8771 macro-F1. LLaMA3-8B comes next at 0.8814 / 0.8753. For Qwen, inverse-frequency class weighting slightly reduces macro-F1 from 0.8615 to 0.8595.
The cleanest comparison keeps the backbone fixed. Zero-shot Qwen2.5-7B records 0.7274 macro-F1, while its QLoRA-adapted counterpart reaches 0.8615, a gain of 13.41 points. That 13.41-point improvement captures the value of task-specific adaptation in one configuration: r=16, alpha=32, lr 2e-5 and three epochs. It is the paper's firmest contribution.
Rows spanning different architectures answer another question, as Luo explicitly acknowledges. FinBERT posts 0.6753 macro-F1 and Financial-RoBERTa 0.6679. Both are off-the-shelf checkpoints, and neither received fine-tuning on the merged training split. Their distance from the QLoRA models combines the effects of adaptation and transfer. Treating it as an encoder-versus-decoder verdict would exceed the paper's claim.
Can the return test carry the result?
Experiment 2 applies the seven probability-producing models to Benzinga analyst-ratings headlines from calendar 2019. The universe is the S&P 100 membership fixed on 1 January 2019. Luo has 10,637 unique headlines and URLs, 13,115 headline-stock observations and 253 calendar dates. Only 72 of the 100 names have usable headlines.
Each unique headline receives inference once, after which its score is mapped to every associated security. Multi-stock articles therefore avoid duplicate scoring. The return horizons are 1, 2, 3 and 5 sessions. Overlapping cohorts receive 1/h of capital each. Newey-West standard errors are used for mean IC tests, while Benjamini-Hochberg correction is imposed jointly on all 28 model-horizon tests.
This inference setup makes the section useful.
The source of the one-day result
Every model has a positive one-day IC. The range begins at 0.0013 for LLaMA3 QLoRA and ends at 0.0143 for FinBERT. Financial-RoBERTa and TF-IDF Naive Bayes both reach 0.0141, while Mistral QLoRA reaches 0.0085. At two days, all seven ICs are negative. Every five-day IC is negative as well.
Correction removes the apparent evidence. Among the 28 tests, the minimum adjusted q-value is 0.9622. Even the unadjusted Newey-West p of 0.0397 on Financial-RoBERTa's two-day result disappears after correction. Financial-RoBERTa has the largest one-day annualized ICIR at 1.015, paired with a long-short return of 3.17%.
FinBERT's main portfolio row is worth giving in full. Its one-day long-short gross return is 12.96%, with Sharpe 1.11 and max drawdown -6.37%. The long leg returns 4.05% at Sharpe 0.31. The short leg returns 19.73% at Sharpe 0.96. Luo reports a cohort win rate below 50% for that short leg, making a few large observations the source of the return rather than persistent daily directional accuracy.
At three and five days, FinBERT's long-only book makes 39.64% and 33.89%, with Sharpe ratios of 2.29 and 2.25. The short-only book loses 22.15% and 20.15%. In 2019, those rows amount to market exposure filtered through news.
Two other results deserve attention. Naive Bayes assigns a neutral label to 80.4% of the 2019 headlines, yet matches the transformer encoders on one-day IC, at 0.0141 against 0.0143. Financial-RoBERTa, meanwhile, has mean out-of-domain prediction confidence of 97.6% while posting the study's lowest benchmark accuracy, 0.6622.
Confidence carries no information by itself.
The abstract accepts the negative economic result while retaining the positive classification finding: QLoRA works for financial sentiment adaptation, and classification accuracy does not translate cleanly into tradable cross-sectional signals. Luo's conclusion is sharper. He says the study does not find statistically reliable evidence that the resulting daily sentiment signals predict later stock returns. The gap, he writes, "may be amplified by limited ticker coverage, uneven news frequency, short sample duration, daily aggregation, and a mismatch between news-arrival timing and the evaluation horizon".
The timing objection is already addressed there. Still, the downstream test has little power by Luo's own account. Coverage reaches 72 constituents out of 100, and some daily cross-sections include only a few stocks with fresh news. Across 253 dates and 28 tests, a minimum q of 0.9622 leaves the absence of predictability unproven.
The sample does establish one narrower point: macro-F1 is a poor basis for selecting a trading signal. Mistral, the best classifier at 0.8771 macro-F1, generates a one-day IC of 0.0085 and a long-short return of 2.69%. FinBERT sits below all four QLoRA models on the label task at 0.6753 macro-F1, yet produces 0.0143 and 12.96%. Both ICs fail correction, so this reversal remains one ordering from one 253-date sample.
We ran the trade elsewhere
Our figures require several disclosures first. We could not use either the paper's data or its models. Luo's 2019 Benzinga analyst-ratings headlines are unavailable to us, so we applied the same portfolio design to our own news feeds, consisting of general stock news plus company press releases, from 2020 onward. We also cannot fine-tune 7B to 8B models here. An off-the-shelf FinBERT-style classifier supplied the traded score. A deterministic substitute score was used if that model failed to load, and we cannot rule out that this occurred. The run is one automated pass. It tests our implementation before it tests anything in the paper.
Our sample runs from 2020-01-02 through 2025-10-08. It contains 1,450 daily rows and 70,022 trades on an annually re-ranked top-100 US large-cap universe. Cumulative gross return is -61.38%, with a Sharpe of -1.16. Luo's one-day FinBERT long-short reports +12.96% gross and Sharpe 1.11 in 2019. Ours records -61.38% and -1.16 over 2020 to 2025. Different signals, corpora and non-overlapping periods produced those figures, so the gap does not constitute a failed replication of the paper's 12.96%.
The losses arrived steadily. Over 5.7 years, the trade win rate is 49.47%, profit factor is 0.95, annualized volatility is 9.18%, and max drawdown reaches -64.15%. Those figures show how an uninformative cross-sectional rank behaves without any cost of carry. Luo's FinBERT one-day long-short has a -6.37% max drawdown during 2019. Both results are gross.
Four differences lean toward the outcome we observed. The time windows have no overlap. FinBERT's 2019 one-day long-short depends largely on a short leg returning +19.73%, during a year when large caps with negative news happened to fall. Our 2020 to 2025 period includes two mega-cap melt-ups. The result is consistent with repeated losses from shorting negative-news mega-caps through those moves, although we could not quantify that explanation.
The source text changes too. Company press releases are firm-issued and, we suspect, promotional. A broad newsfeed also contains syndicated stories and non-event items. If those features pushed low-information names into our tails instead of analyst actions, we could not measure the effect.
Universe construction differs as well. We select again each year using capitalization, which plausibly favors recent winners and makes the short side adverse. This effect is also unquantified. Luo instead fixes membership on 1 January 2019. Both tests exclude costs, so costs explain none of the gap. With 70,022 trades, our net result would be worse.
We cannot fully account for the divergence from the evidence available to us. None of the four differences has been quantified. Some of the gap may simply reflect a one-year estimate resting on a short leg with a sub-50% win rate. FinBERT's one-day IC also fails FDR correction, and the lowest adjusted q across all 28 tests is 0.9622. A setup change need not be large to reverse an estimate this thin.
What evidence would change the trade verdict?
The mechanism that matters lies beyond what this sample can resolve. Luo lacks reliable intraday publication times across the full sample, so a signal attached to a calendar date enters at the following open. The paper observes that public information in liquid large-cap equities "may be incorporated into prices within minutes or hours of publication". Entry can therefore occur after most of the reaction.
We raised the same timing concern about another sentiment signal in our note on FinSMART. With all seven models reversing sign at two days, the pattern resembles what remains after price discovery rather than an edge fading gradually.
I would accept this paper as evidence that QLoRA offers an inexpensive route to a better financial text classifier. The classification leaderboard still fails as a ranking of signal quality. A macro-F1 of 0.8771 corresponds to 0.0085 IC and 2.69%, while 0.6753 corresponds to 0.0143 and 12.96%. None of these return results is significant.
Timestamped headlines aligned with the price immediately before publication would change my view of the trade. The relevant abnormal returns should be tested over minutes to hours. The universe also needs to expand. Usable Benzinga headlines cover only 72 of the 100 constituents, leaving some daily cross-sections with a handful of names and weakening the stability and statistical power of daily rank IC. Luo identifies this as the next step and supplies its exact specification. Until that test is run, 0.0143 remains a measure of the residual after price discovery.