A simulator can pass twelve distributional checks even after its levels 2 to 10 have been shuffled. Bacalum, Wang, Olby, Garaj and Stillman construct an adversarial Level-2 sequence that preserves prices, timestamps, messages and every Level-1 quantity. Only the size and order-count columns at levels 2 to 10 move. Across 18 stock-days, the sequence scores 95% (plus or minus 1%) below LOB-Bench's real-real noise floor on the twelve statistics tested. A score of -100% means the metric detects no difference whatsoever. By that measure, the fake book appears more real than another genuine sample from the same trading day.

That should worry anyone tuning an execution schedule inside a synthetic order book.

We ran nothing here, so there is no result of ours to compare with theirs. The paper studies synthetic Level-2 order-book data for five HKEX equities. Our nearest tradable substitute is US equities, for which we have OHLCV bars. Those bars contain neither depth beyond the touch nor event timestamps. LOB-ID consumes windows of 100 event-indexed Level-2 book states, and no resampling frequency can recover the multi-level bid-ask book or its event-time dynamics from bars. The paper gives no P&L, Sharpe or cost figure of any kind. A trading comparison therefore cannot be made.

Inside the metric

This is an evaluation method for other people's simulators. Transfer to a backtest requires somebody to build the simulator first.

The design comes from image-generation evaluation. DeepLOB, comprising convolutions, an Inception module and an LSTM head, is trained to classify three-way mid-price movement over the next 100 messages. Each input contains 100 events by 40 features: price and size at the ten best levels on both sides. The held-out test set produces 75.44% accuracy and 75.29% macro-F1.

The authors then discard the LSTM and classification head. Averaging the Inception output through time yields a 96-dimensional embedding. Real and synthetic windows pass through the same network, producing two clouds that are compared with FID and MIND. FID imposes Gaussian fits, then compares their means and covariances. MIND takes a scaled average of squared 2-Wasserstein distances across 256 random one-dimensional projections. Following Berthet and co-authors, alpha is set to 3d = 288. MIND is the paper's main score.

The source data comprises four months of message-level HKEX feed, September to December 2025, covering Tencent, Kuaishou, NetEase, Meituan and Baidu. Each book is rebuilt to ten levels per side. The chronological split uses 1 September to 12 December for training, 13 to 20 December for validation, and 21 to 31 December for testing, giving six test dates. Year-end participation is materially lower, and the authors deliberately use that window to create a liquidity shift. Five generators are ranked: Zero Intelligence, Compound Hawkes, LOBGAN, LOBS5 and DiffLOB. Tests cover three stocks across six dates, or 18 stock-days in total.

Why synthetic books need this test

Synthetic market data supports backtesting, execution analysis, risk measurement and microstructure research. A synthetic LOB model must generate message flow because price-time priority evolves the book through matched order streams.

Evaluators have generally relied on stylised facts, including heavy tails, volatility clustering and persistent order-flow dependence, alongside distributions for selected microstructure statistics. The authors target both families of checks.

Their five generators cover very different mechanisms. Zero Intelligence independently samples side, type, timing, volume and depth, preventing any relationship between order flow and book state from developing. Compound Hawkes uses an eight-dimensional self-exciting process, conditioned on 1024 prior real events before generating the next 1024. LOBGAN conditions every new order on an LSTM summary of the preceding 30 nine-dimensional market states. LOBS5 autoregressively creates 22-token messages with structured state-space layers. DiffLOB applies a regime-conditioned diffusion model to Level-2 trajectories.

The distance makes the difference

The actionable contribution is the choice of distance. Earlier research has wrapped embedding networks around market simulators, including calibration work cited from the paper's own group.

For the moment-matching attack, the authors optimise 1024 real LOB windows directly in input space. The objective maximises projection-shape loss while keeping the embedding mean and covariance discrepancy within tau, defined as the moment discrepancy from a genuine real batch of equal size. The optimiser reaches a moment loss of 0.0033. FID increases from a real-real floor of 0.17 to 0.59, giving a ratio of 3.5. MIND jumps from 0.54 to 40.3, a ratio of 74.5 and roughly 21 times the FID ratio. Both calculations use the same 96-dimensional vectors.

Distance choice explains the separation. FID and MIND see identical 96-dimensional inputs, yet FID captures less structure. Replacing it with 256 one-dimensional projections, calculated by sorting, eliminates the 96 by 96 covariance estimate. Every projected distance is exact for empirical samples of equal size. The authors describe MIND as more resistant to this attack class while remaining vulnerable to it.

An attack designed to register

The deep-book result comes from the authors' own construction, as their abstract states. They devise a moment-matching attack on FID and a deep-book perturbation that escapes statistic-based evaluation. The permutation preserves precisely the information used by the competing benchmark while disrupting the relationships MIND measures. Nine of the twelve statistics stay fixed by definition. The impact response function also remains unchanged. Total bid volume, total ask volume and volume per minute are the three measures that move, yet all remain below their real-real baselines.

LOB-ID scores the same data 75% (plus or minus 17%) above the floor, with positive readings for all three names: +40% for NetEase, +144% for Kuaishou and +42% for Tencent. The method detects the attack on conditions selected by its authors.

The paper does not dismiss LOB-Bench. Its discussion says statistic-based benchmarks "provide clear and interpretable diagnostics, while LOB-ID captures temporal and cross-level relationships that individual statistics may miss", adding that this "supports using the two approaches together". Fair enough. The ranking claim remains untested outside a self-built attack. Constructing the failure mode proves that MIND can react while LOB-Bench remains quiet. It cannot establish that MIND ranks actual generators better when nobody engineered the distortion.

Two limits are disclosed by the authors. They use only 12 of the 21 LOB-Bench measures, selecting those that do not require message-level data. They also omit LOB-Bench's trained discriminator, which they describe as serving a different evaluation objective by measuring empirical separability between real and generated sequences. A direct empirical comparison of inception distances with trained discriminator scores is identified as an important direction for future work. I would want that gap closed before treating LOB-ID as the headline score, since a discriminator is the obvious competitor for detecting an incoherent deep book.

Can a mid-price classifier represent realism?

The embedding reflects whichever features helped its directional classifier. During sensitivity tests, random row deletion shifts MIND about as much as a persistent random-walk price distortion. The deletion experiment uses a smaller epsilon range because deletion changes the event index instead of feature values. Event spacing seems to dominate the representation. That emphasis is defensible for event-time books, though it limits the score as a general realism statistic.

The paper acknowledges that LOB-ID does not directly assess the complete message-level order-flow process because each input is a Level-2 book state. Several evaluated generators work at message level. The embedding network is also trained on the same five names later used for scoring. Its training period runs from 1 September to 12 December, before the six test dates from 21 to 31 December. We did not find a held-out instrument test for the embedding.

Validation across markets, instruments and regimes remains future work, the authors concede, "although the stable ranking across checkpoints suggests limited sensitivity to checkpoint selection within the chosen representation". Stability across checkpoints shows that the training stage stopped changing the result. It offers no evidence about a name or exchange absent from network training. The paper further warns that absolute LOB-ID values can be compared across studies only when the network, preprocessing, sample size and projection protocol are identical.

Rankings, with the oracle apart

The MIND ranking starts with DiffLOB under null conditioning at 6.0251 (sd 3.7417), followed by LOBS5 at 9.1201 (4.2673), LOBGAN at 17.9711 (8.6250), Compound Hawkes at 24.4254 (9.8464) and Zero Intelligence at 26.5514 (11.3785). Future-conditioned DiffLOB reaches 2.8204 (0.9424). The authors exclude that configuration from the ranking because its trend, volatility, liquidity and imbalance inputs come from the realised future. As they do, treat it as an upper reference.

Disagreements between metrics tell us more. Compound Hawkes reproduces 6.06 (0.87) of Cont's 11 stylised facts, the highest count among generators other than DiffLOB, yet has the worst Wasserstein-1 across all six configurations at 2.0498 (0.5278). LOBS5 reproduces just 5.28 (1.78) stylised facts and records the best Wasserstein-1 among the five non-oracle generators at 0.7515 (0.4162). The oracle row is lower again at 0.2124 (0.0683).

The L1 column makes the disagreement plain. Compound Hawkes records 0.5170 (0.0433) under L1, ahead of LOBGAN at 0.5642 (0.0903). LOB-ID reverses them, with 24.4254 against 17.9711. Historical data reproduces 7.67 (0.59) of the 11 stylised facts, compared with 7.33 (0.69) for null-conditioned DiffLOB. Counting stylised facts cannot distinguish those two. Dispersion here is calculated across three names and six holiday-period dates.

Two stability findings justify using the score. In the pairwise stock-day matrix, the same stock remains close across nearby dates while different stocks separate clearly, apart from a handful of larger Christmas distances. Across five DeepLOB training checkpoints, MIND values settle into plateaus. Changes are small between roughly 40,000 and 50,000 training steps, and generator order stays fixed at every checkpoint.

Our earlier piece on an LLM tilting TWAP inside a simulated Shenzhen book, link, explains why this matters to a trading desk. That edge existed inside a simulated book. MIND put shuffled levels 2 to 10 at +40%, +144% and +42% above the per-stock sampling floor, while the twelve LOB-Bench statistics came in 95% below it. Such a gap earns MIND a place in the acceptance suite for any simulator used to fill orders.

One result would persuade me to adopt it as the headline measure rather than one score among several: a MIND ranking based on an embedding trained on instruments and a market excluded from the scored set. For now, it is a stress test carrying a home-field advantage.