TOP1 loses its edge when strategic flow enters the book. Its mean 30-second markout at 20 lots drops to -0.06 ticks, while the same size posted one tick behind retains +0.59. Sustained adverse selection turns queue priority into a liability. This comparison has broader force than most results in the paper because the fixed benchmark rules place orders identically across the three environments. Each markout cell contains 10,000 limit order fills.
One unit governs every other result. Normalized cash flow is total fill cash flow plus the terminal mark, divided by the lot budget M, across one 600-second episode. The authors average 10,000 out-of-sample test episodes for each environment and lot size, using seeds separate from training. The paper gives no Sharpe ratio, no t-statistic and no standard errors anywhere.
The agent's choices
Cheridito and Weiss let their market maker trade for a 600-second episode. It acts every 30 seconds, for 20 decisions in all. At each decision, the agent spreads a fixed budget of M lots among nine slots: hold nothing, market buy, the best three bid levels, market sell and the best three ask levels. Its allocation is a simplex point sampled from a multivariate logistic-normal distribution, which the authors call LN, then rounded to whole lots through Hamilton apportionment.
Three of the RL papers they cite ask an agent to choose a distance from the mid or a reservation price before posting one unit on each side. Five more reduce the problem to a discrete action set. Here, the policy distributes size across levels and sides.
A deep-set encoder handles the agent's resting orders. Each order enters as a (queue position, size) pair, with embeddings averaged at each price level. The network therefore sees priority regardless of storage order. Reward consists of realised cash flow plus a mid-price mark-to-market term, using potential-based shaping that leaves the optimal policy unchanged, minus gamma times absolute inventory. The total is divided by M. At the horizon, a market order force-flattens inventory and walks the book.
Spread capture supplies the cash flow, with inventory held near zero. The authors state the mechanism directly: the algorithm "does not generate its cash flows by leaning to one side of the market, but by quoting on both sides while keeping its inventory centered at zero."
The paper uses no historical data. Its order flow combines Poisson noise traders, with intensities taken from Abergel and Jedidi (2013), and tactical traders responding to instantaneous weighted volume imbalance at sensitivity 4. Strategic traders follow a smoothed form of the same signal, with smoothing 0.1 and sensitivity 2.
The headline table is strong inside that setup. In the noise-only book, LN earns 6.03 (sd 2.81) at M=2 and 4.66 (1.41) at M=20. The best heuristic earns 4.83 and 4.26. With tactical traders added, LN reaches 9.11 and 5.68, against TOP1 at 5.47 and 2.88. Once strategic traders arrive, LN keeps 8.55 and 4.99. TOP1 falls to 0.73, while the inventory-skew rule reaches -1.44.
Too large for sampling error
Only point estimates are reported. Still, sd 1.07 for LN and 2.73 for TOP1 across 10,000 episodes imply sampling errors of about 0.011 and 0.027. The 4.26 gap in normalized cash flow, 4.99 versus 0.73, cannot plausibly be a sampling artefact.
Costs offer no rescue.
The model includes no exchange fees, rebates or latency. Crossing the spread and walking the book with market orders are the only charged costs. The initial book is 1000/1001, and the spread is usually one tick, making a tick a tenth of a percent of the price. The 4.99 figure is mean normalized cash flow under the definition above. That metric excludes the running inventory penalty, keeping results comparable across values of gamma.
One hundred times the cancellation intensities
The authors multiply the borrowed noise-trader cancellation intensities by 100. Their reason is computational: shorter queues make the simulation more efficient. Yet queue length determines the value of priority. Queue positions are normalised by 100 because average queue sizes remain below that level.
The paper does not test whether this book's level-1 to level-2 fill-rate ratio resembles any traded instrument. The level-2-over-level-1 finding depends on that ratio. Under the paper's own explanation, orders at the second-best prices avoid fills tied to adverse price moves. With strategic traders present, TOP2 beats TOP1 at 20 lots, 2.05 versus 0.73.
Other traders observe the agent's orders only through book volumes, and those traders respond to imbalance. At M=20, the agent places 20 lots against price levels whose average queues, according to the paper, remain below 100. Its quotes could form a visible share of the signal followed by its counterparties. Some of the M=20 gain could therefore come from the policy shaping the imbalance that tactical and strategic agents subsequently chase. I did not find a decomposition excluding that channel.
LN's own markouts would be the most useful missing figure. The paper withholds them deliberately, for a reasonable stated cause: LN changes its placement across environments, so its markouts would combine market properties with the policy's adaptation. As a result, only the fixed quoting rules quantify the adverse-selection channel.
Behaviour worth carrying forward
Informed flow pushes the learned policy deeper into the book. At M=20, filled volume originally inserted at the third-best price rises from 2.10% in the noise-and-tactical book to 8.30% when all three trader types are present. Fill rates at M=20 decline from 59.65% in the noise book to 43.99% with all three trader types, and the cancellation share in that hardest book reaches 56.01%.
The policy rarely chooses aggression. Market orders account for 1.90% to 2.74% of placed volume across all markets and lot sizes, with almost all of that volume coming from mandatory terminal liquidation. Average inventory stays flat at zero. At M=2, it remains within roughly one lot. At M=20, the one-standard-deviation band extends to about six lots. The behaviour resembles a market maker.
The output distribution matters more than expected. A Dirichlet head attached to the same architecture loses 6.18 in the hardest book at M=20, where the logistic-normal earns 4.99, and its training is visibly unstable. That result is useful before anyone spends a week implementing the obvious simplex distribution.
Two disclosed hand-set knobs
The reported LN figures use gamma=0.01 with forced terminal flattening. When gamma is set to zero, every mean increases and dispersion rises as well. In the noise book at M=2, the result moves from 6.03 (2.81) to 7.95 (8.20).
Allowing terminal flattening to use half the budget raises means again in all six cells. For all-traders M=20, the mean goes from 5.16 to 5.73 with gamma=0. The authors reach the appropriate conclusion: a static inventory penalty is ineffective when trade intensities depend on state. They propose a state-dependent risk parameter for future work. The headline table uses the lower-mean configuration.
The benchmarks deserve more pressure. TOP1 and TOP2 place M/2 on each side every 30 seconds, without a signal or skew. INV applies a single linear skew at one tested aggressiveness setting, alpha=1.0. These rules establish a floor, although the conclusion calls them "competitive benchmarks." I did not find a comparison with an Avellaneda-Stoikov style control, cited in the introduction, or with another RL market maker.
The authors make no claim about real-data performance. According to the abstract, they "illustrate the performance of the method in three simulated market environments". The conclusion acknowledges the simulation limitation, then answers that the algorithm "can be trained on any market simulator." True. I would not carry any of these six cells over with the method. The policy is fitted to one generative model, and its noise-trader cancellation intensities are 100 times the borrowed values. The sharper question concerns what a simulator built this way can establish about level-1 versus level-2 fill economics.
We could not test this. The test requires resting-order queue positions and order-by-order fills. We have one-minute bars, and synthetic fills would recreate the exact problem the paper gives for avoiding a backtest.
Evidence that would change my view would use the same architecture and the same nine-slot allocation, trained on a queue-reactive book calibrated to real cancellation intensities. The level-2-over-level-1 ordering at 20 lots under strategic flow would also need to survive.