Calibration ranges chosen a year early decide 4-bit forecast skill

Compressing a volatility forecaster's activations to 4 bits can destroy more ranking skill than the network added over HAR. With the default abs-max statistic, static W4A4 costs TSMixer 0.096 mean daily IC and the vanilla Transformer 0.123. Their full-precision advantage over pooled HAR is about 0.08 IC. Compression can therefore reverse the model choice already made upstream.

The experiment Ye and Wanjiku ran

The target is cross-sectional. On each date and for each stock, the model predicts the within-date z-score of log realized volatility over the next five days. Inputs cover a 64-day history of eleven features: lagged return means, 20-day volatility, RSI, MACD and its signal line, plus the 20-day max, min and skewness of returns. Performance is mean daily cross-sectional Spearman IC over test dates.

Ye and Wanjiku train seven architectures: DLinear, TSMixer, TimeMixer, a vanilla Transformer, PatchTST, iTransformer and SegRNN. Their panel contains 501 current S&P 500 constituents from June 2008 to December 2025, or roughly 2.1 million asset-days. Each walk-forward fold has seven training years, one validation year and one test year. The eight test years run from 2018 to 2025, with ten seeds apiece and 560 trained checkpoints in total.

Post-training quantization is simulated in FP32 through quantize then dequantize. Trained parameters supply the weight ranges directly. Activation ranges require calibration. For each test year, the authors draw 1,024 sequence windows from the preceding validation year, estimate a range, and hold it fixed throughout the test year. They use either abs-max, the greatest calibration magnitude, or a percentile of activation magnitudes swept from p90 to p99.99. Damage is the paired daily IC difference from the same FP32 checkpoint, evaluated on identical stocks and dates.

Bit width separates the results sharply. At full precision, TSMixer scores 0.490, Transformer 0.490, SegRNN 0.458, TimeMixer 0.303, iTransformer 0.197, PatchTST 0.181 and DLinear 0.026. HAR reaches 0.408, while trailing-5-day persistence reaches 0.315.

Static W8A8 costs six of seven architectures at most 0.0007 IC. Weight-only 4-bit never gives up more than 2.5% of FP32 IC. The largest absolute change is SegRNN at +0.0116, with TimeMixer at +0.0075 and also near 2.5%.

Abs-max W4A4 is another matter.

Across the affected architectures, it removes 11% to 62% of full-precision IC: PatchTST +0.019, iTransformer +0.023, TSMixer +0.096, Transformer +0.123, SegRNN +0.271 and TimeMixer +0.188. DLinear falls outside that comparison because a 0.026 FP32 IC makes percentages uninformative. Using the best percentile cuts Transformer damage to +0.007 and TSMixer damage to +0.019. SegRNN still loses +0.072, and TimeMixer +0.089. Architecture family offers little guidance. Attention models span 11% to 25% damage, while mixers span 20% to 62%.

Where implementation becomes judgment

The quantizer itself is explicit. At 4 bits it has 15 representable levels and step size s = A/7. Quantization coverage is also specified operation by operation. Every learned matrix multiplication is quantized together with its input activation, including convolutional token embeddings. For SegRNN, both the step input and hidden-state operands are quantized at every unrolled time step. Biases, normalization layers, elementwise nonlinearities and activation-only attention products remain in FP32.

The split rule is reproducible: a sample enters the split containing t+5, while its 64-day input window may extend backward across the boundary. Hyperparameters are selected once on a 2009-2015 / 2016 development fold, then frozen. The search covers d_model in {16,32,64,128,256}, dropout in {0.1,0.3,0.5}, depth 2 and three seeds per configuration.

More discretion enters through the hindsight-free selection rule. The paper's initial percentile sweep chooses the best range from test-year outcomes, an upper bound Ye and Wanjiku identify themselves. The range-recoverable share R and residual damage L use the minimising percentile on the test year, and the authors say so before presenting those figures.

Their deployable rule reruns the sweep on the validation year. It chooses the percentile with the smallest validation damage and carries that choice into the next test year. Across the four swept architectures, mean regret against the oracle percentile is 0.007 IC and median regret is 0.002. The rule matches or beats a fixed p99 in 27 of the 32 architecture-fold cells.

The setup appears to draw both calibration windows and the selection sweep from the validation year. If so, the observations used to set the range also score the percentile choice. Narrow ranges receive favorable treatment because the calibration envelope necessarily covers that selection period. The chosen percentile then moves into the following test year, leaving the 0.007 IC regret genuinely out of sample.

An implementer could calibrate on the first half of the validation year and select on the second. The paper's worst case supports that separation. SegRNN chooses its percentile during the 2020 stress validation year, applies it to calm 2021, and incurs 0.057 IC regret, eight times the mean.

When the envelope breaks

The envelope analysis belongs on a monitoring screen. The calibration envelope is defined as the validation year's largest daily cross-sectional return dispersion. Under p99, test days beyond that boundary suffer 1.5x to 3.6x the damage seen inside it: Transformer 3.6x, SegRNN 2.5x, TSMixer 2.2x and TimeMixer 1.5x.

Abs-max behaves differently. Its outside-to-inside ratios are 0.8 for Transformer, 0.9 for TSMixer, 1.1 for SegRNN and 1.4 for TimeMixer, with no comparable penalty from the wider range. Dispersion in 2020 reached 2.03x the envelope. The percentile fix removes 94% of Transformer damage, 80% of TSMixer damage, 73% of SegRNN damage and 53% of TimeMixer damage, yet concentrates what remains on the days when a volatility ranking is most likely to matter.

The paper states the problem directly in its abstract: "Narrow ranges improve resolution under typical market conditions but lose part of their advantage when test-period market dispersion exceeds the calibration history". Its deployment advice follows immediately. Where degradation remains substantial, the authors prefer 8-bit activations or weight-only 4-bit quantization. They also propose out-of-envelope days as a trigger for recalibration, a more conservative p99.9, or fallback to 8-bit activations.

The missing piece is the value of that fallback. The study never measures its memory or latency benefit.

A matched experiment recalibrates the 2020 fold on 2020 itself. Under p99, TSMixer damage drops from +0.077 to +0.043, while SegRNN falls from +0.164 to +0.119. Calibration mismatch accounts for roughly a quarter to a half of the 2020 excess. The remainder is clipping that survives every percentile.

The regime evidence is thinner than the framing suggests. The authors identify only two major stress episodes across eight test years. SegRNN's greatest abs-max damage in any fold also occurs during calm 2021 rather than COVID. Excessively wide calibration can fail as well.

This leaves half the trade-off unmeasured, as the authors acknowledge. Quantization is simulated, so the study reports no backend memory or latency savings. W4A4's IC cost is carried to three decimals. Its compensating benefit has no measurement.

Safer choices already work

W8A8 costs at most 0.0007 IC for six of seven architectures, while weight-only W4 remains below 2.5% of FP32 IC. A single-layer repair makes the strongest case for selective precision. The Transformer's convolutional token embedding alone reproduces 72% of full-model W4A4 damage. Keeping that layer at 8 bits reduces damage from +0.123 to +0.015. On the same architecture, the best tested percentile does slightly better, reaching +0.007 from the same +0.123.

TSMixer offers no similar bottleneck. Protecting any single layer leaves damage near +0.086. The paper recommends percentile W4A4 where range tuning recovers most of the loss, then 8-bit activations or weight-only W4 for architectures with large residual damage. In its results, those are SegRNN at +0.072 and TimeMixer at +0.089.

My threshold is stricter. Full 4-bit activations deserve deployment only after direct measurement, and in most cases I would decline them.

Our run omitted the central treatment

We could not reproduce the paper's panel. Point-in-time S&P 500 membership was unavailable to us, so we substituted our platform's annual top-500-by-capitalization US equity universe. The paper's training scale was also too costly for one pass: seven architectures, eight folds, ten seeds and the hyperparameter grid.

We ran a reduced design. Compact sequence summaries with random projections and ridge regression served as architecture proxies. The test covered six folds from 2020 rather than eight from 2018, used 14 features rather than 11, and comprised 420 stated runs. Our executable quantization included weights only. We did not implement static per-tensor activation ranges, the 1,024-window calibration sample or fixed test-year ranges. The treatment at the center of the paper was therefore absent from our run.

Our extension converts the forecast target into a portfolio: long-only, lowest predicted-volatility quintile, daily market-on-close rebalancing, a 10% position cap, four tenths of a cent a share and zero modelled slippage. From 2020-01-02 to 2025-10-08, it produced 41.70% total return, 6.24% CAGR, 17.47% volatility, a 0.41 Sharpe, a 37.20% maximum drawdown and beta 0.68.

The start date and drawdown belong together. Our window begins in 2020, the fold the paper highlights for calibration-test mismatch, and excludes the calm 2018-2019 folds. A long-only, low-beta equity sleeve entering January 2020 acquires that drawdown mechanically. It establishes no skill in either direction. Across 148,425 trades, we charged no spread and no market impact. With a profit factor of 1.16, the return estimate is optimistic.

The paper reports forecast performance only: 0.490 FP32 mean daily IC for TSMixer and Transformer and 0.408 for HAR. It gives no strategy returns, Sharpe or cost assumptions. Our run supplies the opposite set of outputs, a 0.41 Sharpe and no IC. These figures measure different objects. Comparing 0.41 against 0.490 creates no shortfall.

We cannot measure a gap on any shared metric, and the causes cannot be assigned cleanly. We expect ridge proxies to rank forward volatility less sharply than tuned sequence models, although our run contains no forecast-skill measurement. Our point-in-time universe also removes the survivorship tilt the authors flag in their own IC levels. And none of the W4A4 results could appear because activation calibration was missing.

A real backend result could change my conclusion: actual memory and latency won by 4-bit activations under the same inference load, large enough to justify 0.019 IC on TSMixer. For now, the Transformer's layer exception is the cleanest result. The validation-year percentile rule, with 0.007 IC mean regret across the four swept architectures, is the part I would deploy.

For each test year Y:
  1. Select the point-in-time top 500 non-ADR US stocks by capitalization;
     hold membership fixed for calendar year Y.
  2. Assign Y-8 through Y-2 to training, Y-1 to validation, and Y to test.
     The 2025 test fold ends on 2025-10-08.
  3. For every eligible stock-date, construct a [64, 14] sequence from returns,
     rolling technical indicators, OHLC range, and dollar-volume features.
     Drop samples containing unavailable or non-finite required values.
  4. Fit feature z-scores on training data only and apply the same statistics
     to validation and test data.
  5. Define the target as the within-date cross-sectional z-score of log
     volatility over returns t+1 through t+5. Assign the complete target window
     to a split according to t+5.
  6. Fit the configured architecture representation and regression model using
     training data; use validation loss for model selection where implemented.
  7. At each test close, forecast every eligible stock's standardized relative
     volatility score and rank forecasts from lowest to highest.
  8. Buy the lowest 20% of forecasts. Either equal-weight them or assign raw
     weights exp(-forecast_score), normalize long-only weights to 100%, and
     apply the 10% position cap.
  9. Submit market-on-close orders using observed close prices. Skip an order
     when its execution price is missing; never synthesize a fill.
 10. Repeat daily, subject to the 4.0 maximum-leverage constraint.