Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

Guanqun Zhao Note: Work done during an internship at Baidu. Zijun Xie Note: Work done during an internship at Baidu. Binbin Zheng Note: Work done during an internship at Baidu. Enlei Gong\corresponding Jiafeng Lu Yehan Yang Aoqi Hu Zeyu Chen\corresponding

Abstract

Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse. Existing methods typically retain or discard tokens based solely on the magnitude of their importance ratios, applying the same threshold uniformly across token positions. In this work, we reveal that the natural scale of the importance ratio varies systematically with token entropy. Under asynchronous dynamics, this entropy-ratio scaling dictates two distinct phenomena: at low entropy, the inherent train-inference discrepancy is drastically amplified into substantial sampling noise; at high entropy, in-flight weight updates naturally induce pronounced, legitimate exploratory deviations. Consequently, magnitude-only correction inadvertently admits the amplified noise while strictly masking out the essential exploration triggered by in-flight updates. To address this, we propose the Entropy-Scaled Trust Region (ESTR), which scales each token’s off-policy deviation by its local entropy, requiring no auxiliary forward passes or explicit version-switch detection. Across long-horizon agentic tasks and mathematical reasoning benchmarks, ESTR consistently outperforms existing asynchronous methods and achieves the best train-inference consistency. Compared with synchronous GRPO, ESTR attains comparable accuracy while improving training speed by $2.6\times$ .

¹Beijing University of Posts and Telecommunications, ²Peking University, ³University of Science and Technology of China

⁴Institute of Computing Technology, Chinese Academy of Sciences, ⁵Baidu Inc.

1 Introduction

Refer to caption — Figure 1: Entropy and ratio scale. The per-bin mean $|\delta_{t}|$ (purple) grows with $H_{t}$ . A fixed threshold (dashed) admits low-entropy noise (red) and clips high-entropy exploration (blue), whereas our entropy-scaled boundary (solid) separates them.

Reinforcement learning (RL) has become the dominant paradigm for aligning large language models (DeepSeek-AI 2026) and for eliciting long-horizon reasoning (Chen et al. 2025; Yu et al. 2025; Zheng et al. 2025a) and agentic behavior (GLM-5-Team et al. 2026). To scale RL to long, multi-turn rollouts, modern systems increasingly adopt asynchronous training (Sheng et al. 2025; Fu et al. 2026a; Lu et al. 2025; Piché et al. 2025; Hu et al. 2026; Wu et al. 2025; Bai et al. 2026), where trajectory generation and policy optimization are decoupled and run concurrently. This dramatically improves throughput (Li et al. 2026a) but breaks the on-policy assumption (Li et al. 2026b): each update consumes rollouts sampled from behavior policies that lag behind and diverge from the target policy being optimized. In long multi-turn rollouts, a single trajectory may even span multiple weight versions. Optimizing against such stale, ill-defined behavior policies injects high-variance off-policy gradients that, in the worst case, drive training into irreversible policy collapse (Zheng et al. 2025b). Mitigating this instability without sacrificing the throughput benefits of asynchrony is thus a central challenge for scaling RL on LLMs.

Existing stabilization methods fall into three families: importance sampling correction, i.e., interval clipping (Li et al. 2026c; Çağatan et al. 2026; Fu et al. 2026b; Luo et al. 2026; Xie et al. 2026) and hard masking (Yao et al. 2025; Team et al. 2025; Guo et al. 2026; Zheng et al. 2025b); staleness-mismatch decoupling (Fu et al. 2026a; Guan et al. 2026); and off-policy objective design (Ritter et al. 2026; Yuan et al. 2025). Despite their differences, these methods rest on two implicit premises that both fail in asynchronous agentic RL. One is that a token’s trustworthiness can be read from the magnitude of its importance ratio. We find instead that this magnitude is intrinsically governed by token entropy, its natural scale growing with entropy. At confident, low-entropy tokens, a near-zero probability amplifies a small train-inference gap into a large ratio that is mere noise rather than genuine drift, whereas at high entropy asynchronous in-flight weight updates inject large ratios that carry genuine exploration, which a magnitude threshold wrongly suppresses. The other is that each trajectory comes from a single, well-defined behavior policy. Yet a long multi-turn rollout is produced by a mixture of weight versions diverging from the target to varying degrees, so no single reference behavior policy is even well defined.

Motivated by these empirical findings, we propose the Entropy-Scaled Trust Region (ESTR). Specifically, our main contributions are:

•

We establish a fundamental entropy-ratio relation, proving that the importance ratio’s natural scale is governed by token entropy. This reveals that conventional magnitude-only masking fundamentally fails at low entropy, where it erroneously admits amplified sampling noise.
•

We decompose asynchronous staleness to identify an overlooked intra-trajectory version switching mechanism. We demonstrate that these in-flight updates trigger concurrent surges in token entropy and importance ratios, which constitute genuine exploration that conventional methods indiscriminately suppress.
•

We propose the Entropy-Scaled Trust Region (ESTR), a token-level trust region for asynchronous RL whose boundary expands and contracts with each token’s entropy, requiring no magnitude bias or behavior-policy reconstruction. Across long-horizon agentic tasks and complex reasoning benchmarks, ESTR consistently surpasses the strongest asynchronous baseline, reaching 37.3 avg@1 on BrowseComp-Plus versus 34.9 for the best prior method, and matches synchronous GRPO while training $2.6\times$ faster.

2 Related Work

To mitigate the instability of asynchronous LLM RL, prior works have explored various stabilization strategies. While these methods perform strongly in standard settings, their effectiveness rests on assumptions that break down under the highly dynamic nature of asynchronous agentic training.

A prominent line of work controls variance through importance-sampling corrections. One family discards out-of-bounds tokens directly, either by interval clipping (Çağatan et al. 2026; Fu et al. 2026b; Shen et al. 2026; Li et al. 2026c), which constrains each ratio to a fixed interval, or by hard masking (Yao et al. 2025; Team et al. 2025; Guo et al. 2026; Zheng et al. 2025b), which zeroes out the contribution of any token whose ratio exceeds a preset bound. Both rest on the same premise, reading a token’s untrustworthiness off the magnitude of its importance ratio and letting a single global threshold, shared across all positions irrespective of their local uncertainty, decide which tokens are kept. A second family modulates updates through finer-grained stability signals: ESPO (Sheng et al. 2026) regroups sequences according to their entropy before correction, but such regrouping presumes a relatively stable entropy structure across a batch, whereas under asynchronous rollouts staleness and in-flight version switches perturb per-token entropy and make the grouping itself noisy; AEPO (Dong et al. 2025) balances entropy across the trajectory, yet requires explicitly locating specific high-entropy positions and the turn-switching points of agentic rollouts, introducing additional overhead; and VCPO (Huang et al. 2026) uses the effective sample size as an aggregated variance signal to dynamically scale updates. Although these methods move beyond raw magnitude, entropy or variance still enters as an external reweighting or grouping heuristic layered on top of a trust region that is treated as position-invariant, leaving the boundary of a valid deviation uniform rather than redefined according to each token’s local state.

Another direction is staleness–mismatch decoupling (Fu et al. 2026a; Guan et al. 2026), which approximates the behavior policy that actually generated each rollout in order to separate genuine policy drift from benign staleness, and correct only the former. This isolation, however, hinges on the existence of a single, well-defined behavior policy for the trajectory—an assumption that does not hold in the asynchronous agentic setting. In long multi-turn rollouts, in-flight weight updates synchronize the inference engine to newer parameters mid-generation, so a single trajectory is progressively produced by a mixture of weight versions rather than one fixed policy. There is thus no complete, uniform behavior policy to recover for the whole trajectory, and the version that generated any given token must itself be inferred, making the clean separation of drift from staleness difficult to realize.

A related direction is off-policy objective design (Ritter et al. 2026; Shen et al. 2026; Luo et al. 2026; Yuan et al. 2025), which redesigns the objective to reuse stale data without hard cutoffs, either by softening clipping into a distributional variance penalty or by regressing against a reference policy without importance ratios. Yet the former imposes a batch-level, position-invariant budget that still cannot separate low-entropy noise from high-entropy exploration, while the latter forgoes importance correction altogether, treating every token uniformly regardless of its off-policy deviation.

Across these lines, the common limitation is a boundary of trust that is fixed either by ratio magnitude or at the sequence level, decoupled from the local state of each token. Our work departs from this view. Rather than tuning a global cutoff or reconstructing a behavior policy, ESTR scales the trust region to each token’s local entropy, standardizing each deviation by its own local entropy. This design is grounded in an empirical observation that the natural scale of the importance ratio is tightly coupled with token entropy, the relationship that motivates and underpins the method.

3 Motivation

To understand why magnitude-only suppression struggles in asynchronous agentic RL, we analyze the joint distribution of token entropy $H_{t}$ and the log-importance-ratio $|\delta_{t}|$ on real rollouts using Qwen3-30B-A3B on BrowseComp-Plus (Chen et al. 2025). We find that $|\delta_{t}|$ is inherently coupled with token entropy, suggesting that a single global cutoff may not fully accommodate this state-dependent variance.

Figure 3: Token entropy

H

and ratio magnitude

|\delta|

aligned at an intra-trajectory version switch where

t=0

. Both surge synchronously.

3.1 Ratio Magnitude Relates to Entropy

Empirically, the magnitude of the ratio is not uniform but grows with token entropy. As shown by the per-bin mean in Figure 1 (purple curve), the average $|\delta_{t}|$ increases monotonically with $H_{t}$ . We summarize this core relation as follows:

\mathbb{E}[\delta_{t}^{2}]\;\propto\;H_{t}.

(1)

Rather than a constant horizontal line, the bound of valid $|\delta_{t}|$ expands as entropy increases. Formal mathematical proofs for this relation are provided in the Appendix. Given that the natural scale of $|\delta_{t}|$ is governed by entropy, we must examine how the massive deviations observed in asynchronous RL align with this expected scale across different entropy regimes.

3.2 Dual Phenomena under Asynchronous Dynamics

As illustrated in Figure 2, unlike synchronous RL, asynchronous RL maximizes throughput by eliminating pipeline bubbles. However, this causes in-flight weight updates to sync mid-generation. Let $v_{\mathrm{tgt}}$ be the target policy version at the current update, and let $v_{\mathrm{first}}$ and $v_{\mathrm{last}}$ be the weight versions that generated the first and last tokens of the rollout, respectively. We formalize this by decomposing the total staleness $v_{\mathrm{tgt}}-v_{\mathrm{first}}$ into two components:

\Delta^{\mathrm{intra}}\triangleq v_{\mathrm{last}}-v_{\mathrm{first}},\qquad\Delta^{\mathrm{inter}}\triangleq v_{\mathrm{tgt}}-v_{\mathrm{last}}.

(2)

The term $\Delta^{\mathrm{intra}}$ captures the intra-trajectory version switch, meaning a single long trajectory spans multiple weight versions.

When a version switch occurs mid-trajectory, the newly updated weights evaluate a prefix generated by older weights, causing both $H_{t}$ and $|\delta_{t}|$ to spike concurrently at the switch point ( $t=0$ ; Figure 3). By Eq. (1), this joint surge factorizes as

|\delta_{t}|=\sigma_{t}\sqrt{H_{t}},\qquad z_{t}\triangleq\frac{\delta_{t}}{\sqrt{H_{t}}},\qquad\mathbb{E}[z_{t}^{2}]\approx\sigma_{t}^{2},

(3)

so the spike is carried entirely by the entropy factor $\sqrt{H_{t}}$ , while the standardized deviation $z_{t}$ stays at scale $\sigma_{t}=O(1)$ , invariant to it (Appendix). Hence at a switch

|\delta_{t}|=\sigma_{t}\sqrt{H_{t}}>c\quad\text{yet}\quad z_{t}^{2}\approx\sigma_{t}^{2}\ \text{bounded}:

(4)

a fixed threshold $|\delta_{t}|\leq c$ discards these ratios as drift, whereas they are in fact valid exploration within the natural high-entropy envelope.

Conversely, the low-entropy region in Figure 1 exhibits massive $|\delta_{t}|$ values tracing concave arcs. At a confident position the model reduces to a binary choice between a dominant token ( $1-q$ ) and a sampled off-mode alternative ( $q\approx 0$ ), so the entropy hugs the binary bound

H_{t}\approx H_{\mathrm{bin}}(q)=-q\log q-(1-q)\log(1-q),

(5)

independent of vocabulary size. When the off-mode token is sampled, the train-inference mismatch $\xi$ is amplified by the near-zero denominator, and the delta method gives

\mathbb{E}[\delta_{t}^{2}\mid q]\approx\frac{\mathrm{Var}(\xi)}{q^{2}}\propto\frac{1-q}{q}\xrightarrow[q\to 0]{}\infty.

(6)

Since $H_{\mathrm{bin}}(q)\to 0$ as $q\to 0$ while $\tfrac{1-q}{q}$ diverges, Eq. (5)–(6) jointly produce the concave, entropy-opening arcs (derivation in the Appendix). These outliers are amplified sampling noise, not policy drift: fixed-magnitude thresholds admit them, whereas our entropy-scaled rule discards them.

3.3 An Entropy-Scaled Design Principle

These observations establish a trust criterion, where the entropy-ratio relation itself dictates the boundary between valid updates and harmful drift. Fluctuations within the natural envelope of their local entropy are genuine exploration, whereas outliers exceeding this scale are amplified sampling noise. A fixed-magnitude threshold applies one uniform boundary across the joint distribution, suppressing valid exploration at high entropy while admitting inflated noise at low entropy. This motivates a principled paradigm: the trust region should be dynamically scaled by local entropy. Enforcing such an entropy-scaled threshold preserves deviations that match their natural scale and better fits the dynamics of asynchronous RL, which we formalize next.

4 Method

The preceding analysis established that the natural scale of the importance ratio is dictated by local entropy, and that a trustworthy update criterion must be relative to this scale. We now formalize this principle as the Entropy-Scaled Trust Region (ESTR).

At token position $t$ with state $s_{t}$ , let $\mu_{t}(\cdot\mid s_{t})$ denote the behavior policy that generated token $y_{t}$ , and $\pi_{\theta}(\cdot\mid s_{t})$ the target policy being optimized. We define the log-importance-ratio $\delta_{t}$ and the behavior-policy entropy $H_{t}$ as:

	$\displaystyle\delta_{t}$	$\displaystyle\triangleq\log\frac{\pi_{\theta}(y_{t}\mid s_{t})}{\mu_{t}(y_{t}\mid s_{t})},$		(7)
	$\displaystyle H_{t}$	$\displaystyle\triangleq-\sum_{w\in V}\mu_{t}(w\mid s_{t})\log\mu_{t}(w\mid s_{t}).$		(8)

4.1 A Second-Order Trust Region

To construct an entropy-scaled threshold, we must first examine how deviations are constrained. Asynchronous RL estimates the policy gradient for $\pi_{\theta}$ using data sampled by the behavior policy $\mu_{t}$ . When the two are locally close, the per-token KL divergence admits a second-order expansion (derivation in the Appendix):

	$\displaystyle D_{\mathrm{KL}}(\mu_{t}\,\\|\,\pi_{\theta})$	$\displaystyle=\mathbb{E}_{v\sim\mu_{t}}\!\left[\frac{\pi_{\theta}(v)}{\mu_{t}(v)}-1-\log\frac{\pi_{\theta}(v)}{\mu_{t}(v)}\right]$
		$\displaystyle\approx\tfrac{1}{2}\,\mathbb{E}_{v\sim\mu_{t}}\!\left[\delta_{t}(v)^{2}\right].$		(9)

A trust region therefore bounds the second moment of the log-ratio. The conventional fixed budget $\delta_{t}^{2}\leq c$ implicitly assumes this moment is position-invariant (homoscedastic). The scaling relation of Eq. (1), however, shows that it grows with local entropy: the data are intrinsically heteroscedastic, and a constant budget is mis-scaled at every entropy level.

4.2 An Entropy-Standardized Budget

The remedy is to constrain not the raw deviation but the deviation measured in units of its own local scale. Let $\nu_{t}$ denote the position-dependent scale of $\delta_{t}^{2}$ ; by Eq. (1), $\nu_{t}\propto H_{t}$ . We define the standardized deviation and impose a uniform budget on it:

z_{t}\;\triangleq\;\frac{\delta_{t}}{\sqrt{\nu_{t}}},\qquad z_{t}^{2}\leq\tau,

(10)

which restores a single, position-invariant trust region across all entropy regimes. Concretely, we instantiate the scale as

\nu_{t}\;\triangleq\;H_{t}+\epsilon,\qquad\epsilon>0,

(11)

where $\epsilon$ guards against degenerate near-zero entropy and $H_{t}$ is read off the inference-side logits at no extra cost. Modeling $\pi_{\theta}$ as a small logit perturbation of $\mu_{t}$ with per-logit variance $\sigma_{t}^{2}$ , a first-order expansion gives (Appendix)

\mathbb{E}[\delta_{t}^{2}]=\sigma_{t}^{2}\sum_{w\in V}\mu_{t}(w)\big(1-\mu_{t}(w)\big)\approx\sigma_{t}^{2}\,H_{t},

(12)

so the standardized deviation carries a position-invariant scale,

\mathbb{E}[z_{t}^{2}]=\mathbb{E}\!\left[\frac{\delta_{t}^{2}}{\nu_{t}}\right]\approx\sigma_{t}^{2},

(13)

independent of entropy. The same scale governs the low-entropy regime, whose amplified-noise envelope $(1-q)/q$ (Eq. (6)) diverges as $q\to 0$ yet stays monotonically tied to $H_{t}$ (Appendix). Normalizing by $\nu_{t}=H_{t}+\epsilon$ thus tightens the budget exactly where this noise dominates, while the bounded $H_{t}$ keeps the criterion numerically stable.

4.3 Entropy-Scaled Keep Mask

Substituting the entropy scale (11) into the standardized trust region (10) yields the entropy-scaled keep rule. Defining the entropy-scaled score

S_{t}\;\triangleq\;\frac{\delta_{t}^{2}}{H_{t}+\epsilon},

(14)

a token is retained iff it lies within the entropy-dependent boundary:

S_{t}\leq\tau\quad\Longleftrightarrow\quad|\delta_{t}|\leq\sqrt{\tau\,(H_{t}+\epsilon)}.

(15)

The score $S_{t}$ measures each token’s deviation relative to the natural scale permitted by its local entropy, while $\tau$ controls the acceptance budget. The criterion therefore tightens the boundary where low-entropy noise dominates and widens it where high-entropy deviations reflect exploration, filtering amplified sampling noise without the magnitude bias of a fixed threshold.

Proposition 1 (Strict Generalization)

The keep rule (15) accepts a token inside the entropy-dependent boundary

b_{\mathrm{ESTR}}(H_{t})\triangleq\sqrt{\tau\,(H_{t}+\epsilon)}.

(16)

Forcing the scale to be constant, $H_{t}+\epsilon\equiv C$ , collapses this to the fixed threshold $|\delta_{t}|\leq\sqrt{\tau C}$ of prior methods, which the two boundaries share at the single crossover $H^{\star}=C-\epsilon$ . Away from it, $b_{\mathrm{ESTR}}$ is strictly tighter at low entropy, where large ratios are amplified noise, and strictly wider at high entropy, where they are exploration, contracting to a floor $\sqrt{\tau\epsilon}$ as $H_{t}\to 0$ and growing like $\sqrt{\tau H_{t}}$ as entropy increases. Proof in the Appendix.

Method	Qwen3-30B-A3B	Qwen2.5-7B
	BrowseComp-Plus	GSM8K	AIME24		AIME25		AIME26		AIME Avg.
	avg@1	avg@4	avg@4	pass@4	avg@4	pass@4	avg@4	pass@4	avg@4	pass@4
GRPO(Sync)	38.55	96.07	19.84	27.73	16.67	27.20	16.12	28.12	17.54	27.68
GRPO(Async)	28.91	60.72	14.17	25.69	14.16	22.57	12.50	20.75	13.61	23.00
ICEPop	32.53	65.01	17.52	27.42	15.78	25.45	14.16	22.01	15.82	24.96
KPop	34.94	70.51	18.74	26.61	15.12	25.08	14.79	24.73	16.22	25.47
ESTR (Ours)	37.34	95.69	20.03	31.78	15.64	26.23	15.46	27.14	17.04	28.38

Table 1: Main results. ESTR consistently outperforms all asynchronous baselines across agentic search, multi-turn math, and AIME, matching the synchronous GRPO.

4.4 The ESTR Objective

ESTR gates the per-token policy gradient with the entropy-scaled keep mask $M_{i,t}$ . With group-normalized advantage $A_{i,t}$ , the per-token quantities are

r_{i,t}=\frac{\pi_{\theta}(o_{i,t}\mid s_{i,t})}{\mu_{i,t}(o_{i,t})},\qquad\delta_{i,t}=\log r_{i,t},

(17)

M_{i,t}=\mathbf{1}\!\left[S_{i,t}\leq\tau\right],\qquad S_{i,t}=\frac{\delta_{i,t}^{2}}{H_{i,t}+\epsilon}.

(18)

Though both act on the same importance ratio, the mask and the PPO clip play orthogonal roles. The clip bounds the step size of trusted tokens, whereas $M_{i,t}$ enforces the trust region by zeroing out any token that violates the entropy-scaled boundary, preventing severe off-policy deviations from corrupting the update. The full objective is

\mathcal{L}_{\mathrm{ESTR}}(\theta)=-\,\mathbb{E}_{x\sim\mathcal{D},\ \{o_{i}\}_{i=1}^{G}\sim\mu(\cdot\mid x)}\\ \Bigg[\,\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}M_{i,t}\cdot\\ \min\!\big(r_{i,t}\,A_{i,t},\ \mathrm{clip}(r_{i,t},\,1-\epsilon_{\mathrm{low}},\,1+\epsilon_{\mathrm{high}})\,A_{i,t}\big)\Bigg].

(19)

Since $\mu_{i,t}$ is fixed, $\nabla_{\theta}r_{i,t}=r_{i,t}\nabla_{\theta}\log\pi_{\theta}$ and the hard mask is a stop-gradient gate, yielding

\nabla_{\theta}\mathcal{L}_{\mathrm{ESTR}}=-\,\mathbb{E}\Bigg[\frac{1}{\sum_{i}|o_{i}|}\sum_{i,t}M_{i,t}\,C_{i,t}\,r_{i,t}\,A_{i,t}\,\\ \times\,\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid s_{i,t})\Bigg],

(20)

where $C_{i,t}\in\{0,1\}$ is the clip indicator (Appendix). A masked token ( $M_{i,t}=0$ ) is dropped from the loss entirely, whereas a clipped token ( $C_{i,t}=0$ ) still enters the loss but with zero gradient. As a diagnostic of how aggressively the trust region discards tokens, we track the empirical masked fraction

\rho_{\mathrm{mask}}=1-\frac{1}{\sum_{i}|o_{i}|}\sum_{i,t}M_{i,t}.

(21)

In summary, ESTR is an entropy-scaled trust region for asynchronous RL. By scaling off-policy deviations against the local entropy scale, it filters amplified sampling noise at low entropy while preserving the genuine exploration injected by intra-trajectory version switches, precisely the signal that fixed-magnitude thresholds discard, securing training stability at the full throughput of asynchronous optimization.

5 Experiment

5.1 Setup

Tasks.

We evaluate ESTR on two task families. Long-horizon multi-turn tool use produces long rollouts that interleave reasoning with external tool calls: (i) agentic deep search on BrowseComp-Plus (Chen et al. 2025) with Qwen3-30B-A3B (Team 2025); and (ii) tool-integrated math on multi-turn GSM8K (Cobbe et al. 2021) under a binary exact-match reward. Mathematical reasoning tests generalization beyond the long-horizon regime: we train on DAPO-Math (Yu et al. 2025) with Qwen2.5-7B (Team 2024) and evaluate on AIME 2024 (Zhang and Math-AI 2024), 2025 (Zhang and Math-AI 2025), and 2026 (Zhang and Math-AI 2026).

Baselines.

All methods share the same experimental configuration, differing only in the token-level acceptance rule; all masking baselines act on the same off-policy ratio $\pi_{\theta}/\mu$ as ESTR. Sync is synchronous GRPO (Guo et al. 2025); Async is vanilla asynchronous training with no correction; IcePop (Team et al. 2025) keeps a token when its ratio lies within a fixed interval; and KPop (Guo et al. 2026) keeps a token when the bidirectional binary KL between behavior and target token probabilities stays below a threshold.

Implementation.

All experiments are implemented in the verl framework (Sheng et al. 2025) on H800 GPUs (8 per node), using disaggregated rollout and training pools in the asynchronous setting. The synchronous baseline colocates generation and training on the same total hardware, so all efficiency comparisons are made under equal resource budgets. Full hyperparameters, per-task resource splits, training configurations, and hyperparameter sensitivity studies are provided in the appendix.

Figure 4: Validation accuracy (avg@1) on BrowseComp-Plus across asynchronous methods.

5.2 Long-Horizon Multi-Turn Tool Use

Using the staleness definition from Eq. (2), we evaluate two long-horizon settings: agentic search on BrowseComp-Plus ( $\Delta^{\mathrm{intra}}{=}5$ , $\Delta^{\mathrm{inter}}{=}1$ ), and tool-integrated math on GSM8K ( $\Delta^{\mathrm{intra}}{=}5$ , $\Delta^{\mathrm{inter}}{=}13$ ). In both settings, ESTR significantly outperforms all other asynchronous baselines. By scaling the trust region to local entropy, ESTR successfully stabilizes training while preserving the legitimate exploration naturally induced by intra-trajectory version switches.

On BrowseComp-Plus, ESTR reaches 37.3 avg@1, surpassing all asynchronous baselines by a clear margin, as detailed in Table 1 (Async 28.9, IcePop 32.5, KPop 34.9). The underlying training dynamics provide further insights. Figure 5(a) illustrates that ESTR maintains a steady increase in policy entropy, ensuring progressive exploration. Concurrently, Figure 5(b) demonstrates that ESTR achieves the best train-inference consistency by effectively controlling the rollout–target KL divergence. As a result, its validation accuracy consistently outperforms the other asynchronous methods throughout the training process, as depicted in Figure 4. Additional training dynamics for all tasks are provided in the Appendix.

(a) Policy entropy

(b) Rollout–target KL

Figure 5: BCP training dynamics (Qwen3-30B-A3B): ESTR sustains a steady, stable policy entropy and achieves the best train-inference consistency (lowest rollout–target KL) among all asynchronous methods.

Under the more aggressive GSM8K configuration, Figure 6 shows that IcePop and KPop destabilize, with their training scores dropping irreversibly within a few hundred steps, whereas ESTR trains stably and achieves the highest performance. As reported in Table 1, ESTR reaches 95.7 on GSM8K, essentially matching synchronous GRPO, while the asynchronous baselines collapse to 60–71. This shared failure stems from the baselines’ fundamental design. Magnitude-only correction discards the high- $|\delta_{t}|$ tokens that carry essential exploration at high entropy, so these methods degrade and eventually fail as the off-policy gap expands. ESTR instead evaluates each token relative to its entropy scale, remaining well-calibrated across both settings.

Figure 6: Training score on multi-turn GSM8K (Qwen2.5-7B). The asynchronous baselines collapse within a few hundred steps, whereas ESTR trains stably.

5.3 Mathematical Reasoning

Figure 7 shows training rewards on DAPO-Math with Qwen2.5-7B ( $\Delta^{\mathrm{intra}}{=}8$ , $\Delta^{\mathrm{inter}}{=}10$ ). Naïve asynchronous training collapses irreversibly early in training; IcePop and KPop avoid collapse but plateau at a lower reward; ESTR improves throughout training and attains the highest final reward. Table 1 reports downstream accuracy on AIME 2024–2026: ESTR reaches an avg@4 of $17.04$ , closely approaching the synchronous baseline of $17.54$ despite the aggressive staleness. Furthermore, it achieves the highest pass@4 of $28.38$ compared to $27.68$ for the synchronous baseline, indicating that the preserved high-entropy exploration translates into broader solution coverage rather than mere stability.

Mask behavior and variance control.

Figure 8 examines how each keep rule attains its stability. As shown in the left panel of Figure 8, ESTR attains the lowest masked fraction $\rho_{\mathrm{mask}}$ (Eq. (21)), consistently discarding the fewest tokens—roughly an order of magnitude below the fixed-threshold baselines. Despite sacrificing this minimal amount of tokens, it successfully maintains the lowest and most stable IS-ratio deviation throughout training, as illustrated in the right panel of Figure 8. In contrast, IcePop’s deviation drifts steadily upward in late training even though its $\rho_{\mathrm{mask}}$ is substantially higher. By achieving superior training stability at the lowest token cost, ESTR demonstrates that its entropy-scaled boundary removes precisely the harmful, noise-driven outliers rather than indiscriminately truncating high-ratio exploration. This directly instantiates the trust-region argument of Eq. (9): bounding the standardized deviation is sufficient to keep the second moment of the log-ratio in check.

Robustness to each staleness component.

Using the decomposition of Eq. (2), we stress each staleness source in isolation. The left panel of Figure 9 presents a sweep over the intra-trajectory staleness $\Delta^{\mathrm{intra}}\in\{1,5,7,9\}$ with $\Delta^{\mathrm{inter}}{=}1$ , while the right panel displays the inter-trajectory staleness $\Delta^{\mathrm{inter}}\in\{1,5,15,20,30\}$ with $\Delta^{\mathrm{intra}}{=}1$ . Under these stress tests, no configuration collapses. Instead, rewards degrade gracefully and monotonically as staleness grows, confirming that the entropy-scaled trust region absorbs both mid-trajectory version switches and stale off-policy batches without requiring any staleness-specific tuning.

Figure 7: Training rewards on DAPO-Math (Qwen2.5-7B,

\Delta^{\mathrm{intra}}{=}8

\Delta^{\mathrm{inter}}{=}10

). Vanilla GRPO collapses irreversibly; IcePop and KPop plateau; ESTR improves throughout and attains the highest reward.

Figure 8: Mask behavior on DAPO-Math. Left: masked fraction

\rho_{\mathrm{mask}}

; right: IS-ratio standard deviation. ESTR attains the lowest

\rho_{\mathrm{mask}}

yet keeps the lowest, most stable IS-ratio deviation, targeting harmful noise rather than exploration.

Figure 9: Staleness sweeps on DAPO-Math. Left: intra-trajectory staleness

\Delta^{\mathrm{intra}}\in\{1,5,7,9\}

(

\Delta^{\mathrm{inter}}{=}1

). Right: inter-trajectory staleness

\Delta^{\mathrm{inter}}\in\{1,5,15,20,30\}

(

\Delta^{\mathrm{intra}}{=}1

Method	Throughput	s/step	Speedup
GRPO(Sync)	82.56	1356.47	$1.0\times$
ESTR	214.38	514.84	$\mathbf{2.6\times}$

Table 2: Training efficiency on DAPO-Math. Throughput is in tokens/s per GPU; s/step is per-step wall-clock time; speedup is relative to synchronous GRPO.

5.4 Training Efficiency

Table 2 quantifies the throughput benefit that motivates asynchronous training in the first place, comparing ESTR against the synchronous baseline on identical hardware and parallelism. By decoupling generation from parameter updates, ESTR lifts rollout throughput by $2.6\times$ and shortens the per-step wall-clock time by $62\%$ . Crucially, this speedup comes at no accuracy cost: as shown in Table 1, ESTR matches the synchronous run on avg@4 and surpasses it on pass@4. The keep rule itself adds no measurable overhead—one entropy read per token—so the asynchronous speedup is realized essentially for free.

6 Conclusion

We introduce the Entropy-Scaled Trust Region (ESTR) to address the instability of asynchronous LLM reinforcement learning. By demonstrating that off-policy deviations scale intrinsically with token entropy, ESTR replaces conventional fixed-magnitude thresholds with an entropy-scaled trust region. This formulation systematically filters low-entropy noise while preserving high-entropy exploration, naturally absorbing the perturbations of intra-trajectory version switches without auxiliary overhead. Consequently, ESTR achieves synchronous-level accuracy while retaining the full throughput advantages of asynchronous optimization. Future work will explore extending this framework to tackle problem-solving in even longer-horizon agentic tasks under extreme asynchronous regimes.

References

Bai et al. (2026) Y. Bai, X. Tong, J. Wang, H. Liu, L. Pan, J. Li, Z. Wang, J. HAO, M. Yuan, and F. Wu D-arl: a distribution-matched asynchronous reinforcement learning framework for language reasoning. Cited by: §1.
Çağatan et al. (2026) Ö. V. Çağatan, B. Akgün, G. G. Şahin, and X. Zhao Clipping-free policy optimization for large language models. External Links: 2601.22801, Link Cited by: §1, §2.
Chen et al. (2025) Z. Chen, X. Ma, S. Zhuang, P. Nie, K. Zou, A. Liu, J. Green, K. Patel, R. Meng, M. Su, S. Sharifymoghaddam, Y. Li, H. Hong, X. Shi, X. Liu, N. Thakur, C. Zhang, L. Gao, W. Chen, and J. Lin BrowseComp-plus: a more fair and transparent evaluation benchmark of deep-research agent. External Links: 2508.06600, Link Cited by: §1, §3, §5.1.
Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §5.1.
DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4: towards highly efficient million-token context intelligence. Cited by: §1.
Dong et al. (2025) G. Dong, L. Bao, Z. Wang, K. Zhao, X. Li, J. Jin, J. Yang, H. Mao, F. Zhang, K. Gai, G. Zhou, Y. Zhu, J. Wen, and Z. Dou Agentic entropy-balanced policy optimization. External Links: 2510.14545, Link Cited by: §2.
Fu et al. (2026a) W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, T. Yang, B. Yuan, and Y. Wu AReaL: a large-scale asynchronous reinforcement learning system for language reasoning. External Links: 2505.24298, Link Cited by: §1, §1, §2.
Fu et al. (2026b) X. Fu, J. Lin, Y. Fang, C. Hu, C. Qin, Z. Shao, B. Zheng, L. Pan, and K. Zeng From $\log\pi$ to $\pi$ : taming divergence in soft clipping via bilateral decoupled decay of probability gradient weight. External Links: 2603.14389, Link Cited by: §1, §2.
GLM-5-Team et al. (2026) GLM-5-Team, :, A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, C. Zhu, C. Yin, C. Wang, G. Pan, H. Zeng, H. Zhang, H. Wang, H. Chen, J. Zhang, J. Jiao, J. Guo, J. Wang, J. Du, J. Wu, K. Wang, L. Li, L. Fan, L. Zhong, M. Liu, M. Zhao, P. Du, Q. Dong, R. Lu, Shuang-Li, S. Cao, S. Liu, T. Jiang, X. Chen, X. Zhang, X. Huang, X. Dong, Y. Xu, Y. Wei, Y. An, Y. Niu, Y. Zhu, Y. Wen, Y. Cen, Y. Bai, Z. Qiao, Z. Wang, Z. Wang, Z. Zhu, Z. Liu, Z. Li, B. Wang, B. Wen, C. Huang, C. Cai, C. Yu, C. Li, C. Hu, C. Zhang, D. Zhang, D. Lin, D. Yang, D. Wang, D. Ai, E. Zhu, F. Yi, F. Chen, G. Wen, H. Sun, H. Zhao, H. Hu, H. Zhang, H. Liu, H. Zhang, H. Peng, H. Tai, H. Zhang, H. Liu, H. Wang, H. Yan, H. Ge, H. Liu, H. Chu, J. Zhao, J. Wang, J. Zhao, J. Ren, J. Wang, J. Zhang, J. Gui, J. Zhao, J. Li, J. An, J. Li, J. Yuan, J. Du, J. Liu, J. Zhi, J. Duan, K. Zhou, K. Wei, K. Wang, K. Luo, L. Zhang, L. Sha, L. Xu, L. Wu, L. Ding, L. Chen, M. Li, N. Lin, P. Ta, Q. Zou, R. Song, R. Yang, S. Tu, S. Yang, S. Wu, S. Zhang, S. Li, S. Li, S. Fan, W. Qin, W. Tian, W. Zhang, W. Yu, W. Liang, X. Kuang, X. Cheng, X. Li, X. Yan, X. Hu, X. Ling, X. Fan, X. Xia, X. Zhang, X. Zhang, X. Pan, X. Zou, X. Zhang, Y. Liu, Y. Wu, Y. Li, Y. Wang, Y. Zhu, Y. Tan, Y. Zhou, Y. Pan, Y. Zhang, Y. Su, Y. Geng, Y. Yan, Y. Tan, Y. Bi, Y. Shen, Y. Yang, Y. Li, Y. Liu, Y. Wang, Y. Li, Y. Wu, Y. Zhang, Y. Duan, Y. Zhang, Z. Liu, Z. Jiang, Z. Yan, Z. Zhang, Z. Wei, Z. Chen, Z. Feng, Z. Yao, Z. Chai, Z. Wang, Z. Zhang, B. Xu, M. Huang, H. Wang, J. Li, Y. Dong, and J. Tang GLM-5: from vibe coding to agentic engineering. External Links: 2602.15763, Link Cited by: §1.
Guan et al. (2026) Z. Guan, Y. Guo, H. Sun, W. Huang, S. Di, L. Wu, X. J. Wu, and H. Zhao Missing old logits in asynchronous agentic rl: semantic mismatch and repair methods for off-policy correction. External Links: 2605.12070, Link Cited by: §1, §2.
Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Chen, J. Yuan, J. Tu, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. You, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Zhou, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), pp. 633–638. External Links: ISSN 1476-4687, Link, Document Cited by: §5.1.
Guo et al. (2026) J. Guo, Y. Sun, Z. Huang, Z. Wang, Z. Wen, Z. Zhang, J. Zhou, and S. Kok KPop: taming training–inference mismatch in reinforcement learning with adaptive masking regions. External Links: Link Cited by: §1, §2, §5.1.
Hu et al. (2026) T. Hu, X. Liu, Y. Xiao, Y. Zheng, X. Huang, J. Ding, Y. Zhang, T. Liang, H. Zang, Q. Chen, Y. Sun, W. Shi, C. Zhang, W. Wang, Q. Gu, Y. Sun, Y. Xie, and X. Cai DORA: a scalable asynchronous reinforcement learning system for language model training. External Links: 2604.26256, Link Cited by: §1.
Huang et al. (2026) L. J. Huang, Z. Zhang, Q. Hu, S. Yang, and S. Han Stable asynchrony: variance-controlled off-policy rl for llms. External Links: 2602.17616, Link Cited by: §2.
Li et al. (2026a) H. Li, S. Lin, F. Fu, Y. Zhou, X. Ji, Y. Zhao, L. Wang, J. Jiang, and B. Cui Unleashing efficient asynchronous rl post-training via staleness-constrained rollout coordination. External Links: 2601.12784, Link Cited by: §1.
Li et al. (2026b) Y. Li, J. Liu, J. Xu, Y. Tong, Z. Li, Q. Liu, and B. Wang Trust region masking for long-horizon llm reinforcement learning. External Links: 2512.23075, Link Cited by: §1.
Li et al. (2026c) Y. Li, B. Wang, Y. Gao, Y. Yao, X. Wang, Z. Yin, and X. Qiu BandPO: bridging trust regions and ratio clipping via probability-aware bounds for llm reinforcement learning. External Links: 2603.04918, Link Cited by: §1, §2.
Lu et al. (2025) H. Lu, Z. Liu, S. Xiong, Y. He, W. Gao, Y. Wu, W. Wang, J. Liu, Y. Li, H. Zhao, J. Huang, S. Yang, X. Li, Y. Luo, Z. Liu, L. Pan, J. Yan, W. Wang, W. Su, J. Wang, L. Qu, and B. Zheng Part ii: roll flash – accelerating rlvr and agentic training with asynchrony. External Links: 2510.11345, Link Cited by: §1.
Luo et al. (2026) Y. Luo, S. Han, Y. Hu, D. Li, and J. Hao Ratio-variance regularized policy optimization for efficient llm fine-tuning. External Links: 2601.03320, Link Cited by: §1, §2.
Piché et al. (2025) A. Piché, E. Kamalloo, R. Pardinas, X. Chen, and D. Bahdanau PipelineRL: faster on-policy reinforcement learning for long sequence generation. External Links: 2509.19128, Link Cited by: §1.
Ritter et al. (2026) D. Ritter, O. Oertell, B. Guo, J. Chang, K. Brantley, and W. Sun LLMs can learn to reason via off-policy rl. External Links: 2602.19362, Link Cited by: §1, §2.
Shen et al. (2026) G. Shen, C. Zhao, X. Cheng, L. Huang, and X. Yu VESPO: variational sequence-level soft policy optimization for stable off-policy llm training. External Links: 2602.10693, Link Cited by: §2, §2.
Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, pp. 1279–1297. External Links: Link, Document Cited by: §1, §5.1.
Sheng et al. (2026) Y. Sheng, Y. Huang, S. Liu, A. Zeng, and H. Zhang ESPO: entropy importance sampling policy optimization. External Links: 2512.00499, Link Cited by: §2.
Team et al. (2025) L. Team, A. Shen, B. Li, B. Hu, B. Jing, C. Chen, C. Huang, C. Zhang, C. Yang, C. Lin, C. Wen, C. Li, D. Zhao, D. Yuan, D. You, F. Mao, F. Meng, F. Xu, G. Li, G. Wang, H. Dai, H. Zheng, H. Liu, J. Guo, J. Liu, J. Liu, J. Fu, J. Shi, J. Wang, J. Lai, J. Yang, J. Mei, J. Zhou, J. Zhao, J. Zhao, K. Xu, L. Su, L. Chen, L. Tang, L. Jiang, L. Fu, L. Xu, L. Shi, L. Liao, L. Zheng, M. Li, M. Chen, Q. Zuo, Q. Cheng, Q. Cao, Q. Shi, Q. Guo, S. Zhu, S. Wang, S. Zheng, S. Li, S. Gu, S. Chen, T. Wu, T. Zhang, T. Zhang, T. Zhou, T. Bie, T. Yang, W. Hong, W. Ren, W. Chen, W. Yu, W. Zheng, X. Wang, X. Yan, X. Wan, X. Zhao, X. Kong, X. Tang, X. Han, X. Wang, X. Yang, X. Hu, Y. Zhang, Y. Sun, Y. Shan, Y. Wang, Y. Xu, Y. Liu, Y. Guo, Y. Wang, Y. Yan, Y. Wang, Y. Guo, Z. Li, Z. Xu, Z. Li, Z. Zhang, Z. Gui, Z. Pan, Z. Huang, Z. Lan, Z. Ding, Z. Zhang, Z. Li, Z. Liu, Z. Wang, and Z. Wen Every step evolves: scaling reinforcement learning for trillion-scale thinking model. External Links: 2510.18855, Link Cited by: §1, §2, §5.1.
Team (2024) Q. Team Qwen2.5: a party of foundation models. External Links: Link Cited by: §5.1.
Team (2025) Q. Team Qwen3 technical report. External Links: 2505.09388, Link Cited by: §5.1.
Wu et al. (2025) B. Wu, S. Wang, Y. Tang, J. Ding, E. Helenowski, L. Tan, T. Xu, T. Gowda, Z. Chen, C. Zhu, X. Tang, Y. Qian, B. Zhu, and R. Hou LlamaRL: a distributed asynchronous reinforcement learning framework for efficient large-scale llm training. External Links: 2505.24034, Link Cited by: §1.
Xie et al. (2026) Z. Xie, Y. You, Y. Li, E. Gong, Z. Chen, Q. Chen, Y. Cheng, P. Jiang, and Y. Mu ACPO: adaptive credit policy optimization via fine-grained surrogate entropy. External Links: 2607.03126, Link Cited by: §1.
Yao et al. (2025) F. Yao, L. Liu, D. Zhang, C. Dong, J. Shang, and J. Gao Your efficient rl framework secretly brings you off-policy rl training. External Links: Link Cited by: §1, §2.
Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §1, §5.1.
Yuan et al. (2025) Y. Yuan, F. Chen, Z. Jia, A. Rakhlin, and T. Xie Trajectory bellman residual minimization: a simple value-based method for llm reasoning. External Links: 2505.15311, Link Cited by: §1, §2.
Zhang and Math-AI (2024) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2024. Cited by: §5.1.
Zhang and Math-AI (2025) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2025. Cited by: §5.1.
Zhang and Math-AI (2026) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2026. Cited by: §5.1.
Zheng et al. (2025a) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin Group sequence policy optimization. External Links: 2507.18071, Link Cited by: §1.
Zheng et al. (2025b) H. Zheng, J. Zhao, and B. Chen Prosperity before collapse: how far can off-policy rl reach with stale data on llms?. External Links: 2510.01161, Link Cited by: §1, §1, §2.

Appendix A Proof of the Entropy–Ratio Scaling Law

The Motivation section of the main text establishes the empirical scaling law $\mathbb{E}[\delta_{t}^{2}]\propto H_{t}$ (Eq. (1) of the main text). We now derive it analytically by modeling asynchronous off-policy deviation as a logit perturbation.

Setup and notation.

At step $t$ with state $s_{t}$ , let $V$ be the vocabulary. Let $\rho=(\rho_{w})_{w\in V}$ be the inference-side (behavior) distribution used during rollout, with logits $z$ ; that is,

\rho_{w}=\frac{e^{z_{w}}}{\sum_{v\in V}e^{z_{v}}},\qquad H_{t}\triangleq-\sum_{w\in V}\rho_{w}\log\rho_{w},

(22)

where $H_{t}$ is the Shannon entropy. The token $y$ is sampled from $\rho$ . We model the training-side (target) policy as a logit perturbation of the inference distribution:

\pi_{\theta}(w)=\frac{e^{z_{w}+\eta_{w}}}{\sum_{v\in V}e^{z_{v}+\eta_{v}}},\quad\mathbb{E}[\eta_{w}]=0,\quad\mathrm{Var}(\eta_{w})=\sigma_{t}^{2},

(23)

where the $\eta_{w}$ are mutually independent across the vocabulary and $\sigma_{t}^{2}$ measures the local off-policy perturbation strength induced by asynchronous staleness. Define the log-importance-ratio

\delta(y)\triangleq\log\frac{\pi_{\theta}(y)}{\rho(y)},

(24)

let $\mathbf{1}[\cdot]$ denote the indicator function, and write

\mathrm{LSE}(x)\triangleq\log\sum_{w\in V}e^{x_{w}}.

(25)

Proposition 2 (Entropy–ratio scaling)

Under the first-order (small- $\eta$ ) approximation of (23),

\mathbb{E}_{y\sim\rho}\big[\delta(y)^{2}\big]=\sigma_{t}^{2}\sum_{w\in V}\rho_{w}(1-\rho_{w}),

(26)

and, applying the pointwise approximation $1-\rho_{w}\approx-\log\rho_{w}$ , which is accurate for concentrated distributions,

\mathbb{E}_{y\sim\rho}\big[\delta(y)^{2}\big]\approx\sigma_{t}^{2}\,H_{t}.

(27)

Proof.

Taking logarithms of

\rho(y)=\frac{e^{z_{y}}}{\sum_{v}e^{z_{v}}},\qquad\pi_{\theta}(y)=\frac{e^{z_{y}+\eta_{y}}}{\sum_{v}e^{z_{v}+\eta_{v}}},

(28)

and subtracting (the $z_{y}$ terms cancel) gives

\delta(y)=\eta_{y}-\big[\mathrm{LSE}(z+\eta)-\mathrm{LSE}(z)\big].

(29)

Since $\partial\,\mathrm{LSE}(z)/\partial z_{w}=\rho_{w}$ , a first-order expansion at $\eta=0$ yields

\mathrm{LSE}(z+\eta)\approx\mathrm{LSE}(z)+\sum_{w}\rho_{w}\eta_{w}.

(30)

Substituting into (29),

\delta(y)\approx\eta_{y}-\sum_{w}\rho_{w}\eta_{w}=\sum_{w}\big(\mathbf{1}[w=y]-\rho_{w}\big)\eta_{w}.

(31)

Because $\mathbb{E}[\eta_{w}]=0$ , we have $\mathbb{E}[\delta(y)]=0$ ; in particular, $\mathbb{E}[\delta^{2}]$ coincides with $\mathrm{Var}(\delta)$ to this order. By the independence of the $\eta_{w}$ and $\mathrm{Var}(\eta_{w})=\sigma_{t}^{2}$ ,

	$\displaystyle\mathbb{E}\big[\delta(y)^{2}\mid y\big]$	$\displaystyle=\sigma_{t}^{2}\sum_{w}\big(\mathbf{1}[w=y]-\rho_{w}\big)^{2}$
		$\displaystyle=\sigma_{t}^{2}\Big(1-2\rho_{y}+\sum_{w}\rho_{w}^{2}\Big).$		(32)

Averaging over the sampled token $y\sim\rho$ , and using $\sum_{y}\rho_{y}=1$ and $\sum_{y}\rho_{y}\cdot\rho_{y}=\sum_{w}\rho_{w}^{2}$ ,

	$\displaystyle\mathbb{E}_{y\sim\rho}\big[\delta(y)^{2}\big]$	$\displaystyle=\sigma_{t}^{2}\Big(1-2\sum_{w}\rho_{w}^{2}+\sum_{w}\rho_{w}^{2}\Big)$
		$\displaystyle=\sigma_{t}^{2}\Big(1-\sum_{w}\rho_{w}^{2}\Big)=\sigma_{t}^{2}\sum_{w}\rho_{w}(1-\rho_{w}),$		(33)

which proves (26). Applying $1-\rho_{w}\approx-\log\rho_{w}$ term by term gives

\sum_{w}\rho_{w}(1-\rho_{w})\approx-\sum_{w}\rho_{w}\log\rho_{w}=H_{t},

(34)

establishing (27).

Switch-point invariance of the standardized deviation.

The scaling law of Proposition 2 also explains why an intra-trajectory version switch, which inflates both $H_{t}$ and $|\delta_{t}|$ , does not destabilize the entropy-scaled criterion. At a switch the behavior policy $\mu_{t}$ (old weights) and the target $\pi_{\theta}$ (new weights) differ by a logit perturbation of per-logit variance $\sigma_{t}^{2}$ , as in (23).

Proposition 3 (Switch-point invariance)

Under the perturbation model of (23), the standardized deviation $z_{t}\triangleq\delta_{t}/\sqrt{H_{t}+\epsilon}$ satisfies

\mathbb{E}[z_{t}^{2}]=\frac{\mathbb{E}[\delta_{t}^{2}]}{H_{t}+\epsilon}\approx\frac{\sigma_{t}^{2}\,H_{t}}{H_{t}+\epsilon}\leq\sigma_{t}^{2},

(35)

with equality as $\epsilon\to 0$ . Hence, although a switch inflates both the entropy $H_{t}$ and the raw deviation $|\delta_{t}|=\sigma_{t}\sqrt{H_{t}}$ , the standardized deviation $z_{t}$ is governed by the perturbation strength $\sigma_{t}$ alone and is invariant to the entropy spike.

Proof.

By Proposition 2, $\mathbb{E}[\delta_{t}^{2}]\approx\sigma_{t}^{2}H_{t}$ . Dividing by the deterministic scale $H_{t}+\epsilon$ gives the equality in (35); since $H_{t}/(H_{t}+\epsilon)\leq 1$ for $\epsilon\geq 0$ , the bound $\mathbb{E}[z_{t}^{2}]\leq\sigma_{t}^{2}$ follows, and $H_{t}/(H_{t}+\epsilon)\to 1$ as $\epsilon\to 0$ . The raw deviation $|\delta_{t}|=\sqrt{\mathbb{E}[\delta_{t}^{2}]}=\sigma_{t}\sqrt{H_{t}}$ scales with the entropy factor $\sqrt{H_{t}}$ , whereas dividing by $\sqrt{H_{t}+\epsilon}$ removes it, leaving only $\sigma_{t}$ . A fixed-magnitude rule $|\delta_{t}|\leq c$ is therefore triggered by the $\sqrt{H_{t}}$ spike alone, while the standardized criterion is not. $\blacksquare$

Empirical symmetry of the signed deviation.

Eq. (31) implies $\mathbb{E}[\delta(y)]=0$ to first order, so the off-policy deviation is symmetric about zero and its natural scale is fully captured by the second moment. Figure 10 confirms this on real BrowseComp-Plus rollouts: the signed $\delta_{t}$ distributes symmetrically around zero at every entropy level, with the per-bin mean envelope ( $\pm$ ) expanding as $\sqrt{H_{t}}$ , in line with Proposition 2. This symmetry justifies gating tokens by the sign-free score $\delta_{t}^{2}/(H_{t}+\epsilon)$ without treating positive and negative deviations separately. The signed view also visualizes the two failure modes of a fixed threshold: it newly masks legitimate high-entropy exploration (blue) while admitting amplified low-entropy noise (red), both of which the entropy-scaled boundary handles correctly.

Appendix B Low-Entropy Regime: Amplified Off-Mode Noise and the Concave Arc

The preceding section characterizes the expected scale of $\delta_{t}^{2}$ . We now explain the extreme outliers observed at low entropy in the joint distribution (Figure 2 of the main text), which manifest as distinct concave arcs, and derive the variance law $\mathbb{E}[\delta_{t}^{2}]\approx\frac{1-q}{q}$ invoked in the Method section.

Variance law via the delta method.

At position $t$ , let $q\triangleq\pi_{\theta}(y_{t}\mid s_{t})$ be the realized target probability of the sampled token. Model the behavior probability as a sampling-side perturbation, $\mu_{t}(y_{t})=q\,(1+\eta)$ with $\eta=\Delta/q$ and $\Delta\triangleq\mu_{t}(y_{t})-q$ . Treating the realization of the token as a Bernoulli event gives $\mathrm{Var}(\Delta)\propto q(1-q)$ . Since $\delta_{t}=-\log(1+\eta)\approx-\eta$ to first order, the delta method yields

\mathrm{Var}(\delta_{t}\mid q)\;\approx\;\frac{\mathrm{Var}(\Delta)}{q^{2}}\;\propto\;\frac{q(1-q)}{q^{2}}=\frac{1-q}{q}.

(36)

As $q\to 0$ —an off-mode token drawn at a highly confident position—the variance diverges purely because the near-zero denominator inflates a marginal absolute probability gap between the training and inference engines. Such outliers are amplified sampling noise, not policy drift.

The concave binary-entropy envelope.

To see why these outliers trace concave arcs in the $(H,|\delta|)$ plane, consider the minimum entropy attainable at a given confidence level. For a position whose mode carries probability $1-u$ (with $u\triangleq 1-q$ the off-mode mass), entropy is minimized when the residual mass $u$ concentrates on a single alternative token, yielding the binary entropy

H_{\mathrm{bin}}=-(1-u)\log(1-u)-u\log u,

(37)

independently of vocabulary size; empirically the low-entropy population hugs this lower bound. On the high-confidence branch $u\to 0^{+}$ , a Taylor expansion gives

H_{\mathrm{bin}}(u)=u\,(1-\log u)+O(u^{2}),

(38)

while by (36) the standard-deviation scale of the log-ratio is

\bar{\delta}\propto\sqrt{\tfrac{u}{1-u}}\approx\sqrt{u}.

(39)

Eliminating $u$ between the two expressions,

\bar{\delta}\;\approx\;\sqrt{\frac{H_{t}}{\,1-\log u\,}}\;\xrightarrow[H_{t}\to 0]{}\;0.

(40)

Since $-\log u\to+\infty$ , the envelope vanishes strictly more slowly than $\sqrt{H_{t}}$ , forcing the bounding envelope into the concave, entropy-opening arc observed in Figure 2 of the main text rather than a horizontal line.

Appendix C Second-Order Expansion of the KL Divergence

The Method section (“A Second-Order Trust Region”) states that

D_{\mathrm{KL}}(\mu_{t}\,\|\,\pi_{\theta})\approx\tfrac{1}{2}\,\mathbb{E}_{v\sim\mu_{t}}\big[\delta_{t}(v)^{2}\big].

(41)

Writing $\delta_{t}(v)=\log\big(\pi_{\theta}(v)/\mu_{t}(v)\big)$ , note first the exact identity

\mathbb{E}_{v\sim\mu_{t}}\!\left[\frac{\pi_{\theta}(v)}{\mu_{t}(v)}-1\right]=\sum_{v}\pi_{\theta}(v)-\sum_{v}\mu_{t}(v)=0,

(42)

so the KL divergence can be written without approximation as

D_{\mathrm{KL}}(\mu_{t}\,\|\,\pi_{\theta})=\mathbb{E}_{v\sim\mu_{t}}\!\left[\frac{\pi_{\theta}(v)}{\mu_{t}(v)}-1-\log\frac{\pi_{\theta}(v)}{\mu_{t}(v)}\right].

(43)

The integrand equals $e^{\delta_{t}(v)}-1-\delta_{t}(v)$ , whose second-order Taylor expansion is $\tfrac{1}{2}\,\delta_{t}(v)^{2}+O(\delta_{t}^{3})$ . Hence, when $\mu_{t}$ and $\pi_{\theta}$ are locally close,

D_{\mathrm{KL}}(\mu_{t}\,\|\,\pi_{\theta})\approx\tfrac{1}{2}\,\mathbb{E}_{v\sim\mu_{t}}\big[\delta_{t}(v)^{2}\big].

(44)

Moreover, since $\mathbb{E}[\delta_{t}]\approx 0$ under the perturbation model of the entropy–ratio scaling law derived above, the second moment coincides with the variance to this order, recovering the trust-region-on-variance reading used in the main text.

Appendix D Joint Monotonicity of $H_{\mathrm{bin}}(q)$ and $\frac{1-q}{q}$

The Method section replaces the singular variance scale $\frac{1-q}{q}$ with the entropy $H_{t}$ . Here $q\in(0,\tfrac{1}{2}]$ denotes the vanishing off-mode mass at a confident position. We show that this substitution is order-preserving: the two quantities are strictly monotone in $q$ on $(0,\tfrac{1}{2}]$ , so one is a strictly monotone reparameterization of the other and thresholding them is equivalent up to a monotone change of scale.

Monotonicity of the variance scale.

Define

f(q)\;\triangleq\;\frac{1-q}{q}\;=\;\frac{1}{q}-1.

(45)

Differentiating,

f^{\prime}(q)=-\frac{1}{q^{2}}<0,\qquad f^{\prime\prime}(q)=\frac{2}{q^{3}}>0,

(46)

so $f$ is strictly decreasing and convex on $(0,\tfrac{1}{2}]$ . Its range is

\lim_{q\to 0^{+}}f(q)=+\infty,\qquad f\!\left(\tfrac{1}{2}\right)=1,

(47)

i.e., the variance scale grows without bound as the off-mode mass vanishes.

Monotonicity of the binary entropy.

For

H_{\mathrm{bin}}(q)=-q\log q-(1-q)\log(1-q),

(48)

term-by-term differentiation gives

	$\displaystyle\frac{d}{dq}\big[-q\log q\big]$	$\displaystyle=-(\log q+1),$		(49)
	$\displaystyle\frac{d}{dq}\big[-(1-q)\log(1-q)\big]$	$\displaystyle=\log(1-q)+1,$		(50)

and summing the two yields

H_{\mathrm{bin}}^{\prime}(q)=\log\frac{1-q}{q}.

(51)

For $q\in(0,\tfrac{1}{2})$ we have $\frac{1-q}{q}>1$ , hence $H_{\mathrm{bin}}^{\prime}(q)>0$ : $H_{\mathrm{bin}}$ is strictly increasing on $(0,\tfrac{1}{2}]$ . Differentiating (51) once more,

H_{\mathrm{bin}}^{\prime\prime}(q)=-\frac{1}{q}-\frac{1}{1-q}<0,

(52)

so $H_{\mathrm{bin}}$ is strictly concave, and it vanishes at the confident limit,

\lim_{q\to 0^{+}}H_{\mathrm{bin}}(q)=0.

(53)

Monotone reparameterization.

Since $f$ is a strictly decreasing bijection from $(0,\tfrac{1}{2}]$ onto $[1,\infty)$ , it is invertible with

q=f^{-1}(y)=\frac{1}{1+y},\qquad y\in[1,\infty).

(54)

Composing, define $\varphi\triangleq H_{\mathrm{bin}}\circ f^{-1}$ , i.e.,

\varphi(y)=H_{\mathrm{bin}}\!\left(\frac{1}{1+y}\right).

(55)

By the chain rule, using $H_{\mathrm{bin}}^{\prime}>0$ and $(f^{-1})^{\prime}(y)=-\tfrac{1}{(1+y)^{2}}<0$ ,

\varphi^{\prime}(y)=H_{\mathrm{bin}}^{\prime}\!\big(f^{-1}(y)\big)\cdot(f^{-1})^{\prime}(y)<0,

(56)

so $\varphi$ is strictly decreasing. Hence $H_{\mathrm{bin}}$ is a strictly monotone (order-reversing) reparameterization of $f=\frac{1-q}{q}$ , and for any budget $c$ ,

\frac{1-q}{q}\leq c\quad\Longleftrightarrow\quad H_{\mathrm{bin}}(q)\geq\varphi(c),

(57)

so a bound on the variance scale is equivalent to an entropy threshold.

	BrowseComp-Plus	DAPO-Math	GSM8K
Policy model	Qwen3-30B-A3B (MoE)	Qwen2.5-7B	Qwen2.5-7B
Train backend	Megatron	Megatron	Megatron
Rollout engine	SGLang	SGLang	SGLang
Task type	multi-turn agentic	single-turn	multi-turn tool
Reward	LLM judge	rule-based	rule-based
Total nodes ( $\times$ 8 H800)	4	2	2
Train nodes	2	1	1
Rollout nodes	2	1	1
Train GPUs / Rollout GPUs	16 / 16	8 / 8	8 / 8
Actor TP/PP/CP/EP	2/1/8/8	2/1/1/1	2/1/1/1
Ref TP/PP/CP/EP	2/1/4/8	2/1/1/1	2/1/1/1
Rollout TP/EP	8/1	2/1	2/1
Rollout samples $n$	8	8	8
Global batch (prompts)	16	128	256
Prompt length	4096	2048	2048
Response length	32768	12288	8192
Max model length	40960	15376	12288
Max turns	100	100	100
ESTR $\tau$ (default)	0.7	1.6	1.6
Entropy floor $\epsilon$	0.01	0.01	0.01
Staleness threshold	5	20	10
Param-sync interval	1	2	4
Partial rollout	yes	yes	yes
Rollout mem. util.	0.50	0.70	0.70
Epochs	5	12	12

Table 3: Per-task training configuration. All runs are fully asynchronous (disaggregated train/rollout pools).

Why entropy is the preferred scale.

Although $\frac{1-q}{q}$ and $H_{\mathrm{bin}}$ are threshold-equivalent, they differ sharply in numerical behavior. As $q\to 0^{+}$ the raw scale $\frac{1-q}{q}$ diverges, whereas $H_{\mathrm{bin}}$ stays bounded in $[0,\log 2]$ . Moreover the empirically used $H_{t}$ is an aggregate statistic of the full next-token distribution, rather than a function of a single realized probability, so it is robust to single-token sampling noise. These properties justify using $H_{t}$ as the local scale in the keep rule.

Probability-space view of the two rules.

Figure 11 recasts the comparison above in probability space. Panel (a) verifies the premise of this section: high entropy corresponds to low realized token probability, so $H_{t}$ is a faithful, smoothly varying proxy for the confidence level $q$ . Panel (c) then exposes the magnitude bias of a fixed threshold from a complementary angle. Inverting $|\delta_{t}|\leq c$ shows that the absolute probability drift needed to trigger the mask scales linearly with $\mu_{t}$ : at confident positions the rule tolerates drifts an order of magnitude larger than at uncertain ones, which is exactly how the amplified low-entropy noise of Eq. (36) slips through. The entropy-scaled boundary instead keeps this trigger sensitivity bounded across the entire probability range, tightening where confidence is high. Panel (b) confirms that this reallocation is not achieved by masking more overall: the two rules mask comparable fractions at every probability level, and the fixed threshold is in fact more aggressive only at the lowest probabilities—the exploratory tokens it should preserve. The entropy-scaled rule thus redistributes a similar masking budget from legitimate exploration to amplified noise, consistent with the geometric analysis of the Strict Generalization property below.

Appendix E Proof of the Strict Generalization Property

Proposition 4 (Strict Generalization)

The fixed-threshold keep rule is the degenerate, position-invariant special case of the ESTR keep rule; conversely, ESTR is a strict, entropy-dependent generalization of it.

Setup.

The ESTR keep rule accepts a token when

\frac{\delta_{t}^{2}}{H_{t}+\epsilon}\leq\tau\quad\Longleftrightarrow\quad|\delta_{t}|\leq\sqrt{\tau\,(H_{t}+\epsilon)},

(58)

so its acceptance boundary in the $(H,|\delta|)$ plane is the entropy-dependent curve

b_{\mathrm{ESTR}}(H_{t})\;\triangleq\;\sqrt{\tau\,(H_{t}+\epsilon)}.

(59)

Fixed thresholding as a special case.

Suppose the scale is forced to be position-invariant, i.e.,

H_{t}+\epsilon\equiv C

(60)

for some constant $C>0$ (the homoscedastic assumption). Substituting into (58) collapses the rule to

|\delta_{t}|\leq\sqrt{\tau C}\;\triangleq\;c,

(61)

which is exactly the fixed-threshold mask $|\delta_{t}|\leq c$ employed by prior methods such as IcePop. Hence fixed-threshold masking is recovered whenever the entropy dependence is removed, establishing it as the constant-scale degenerate case of ESTR.

(a) Gradient norm

(b) Interaction turns

(d) Training reward

(e) Training-side log-perplexity

(f) Per-step wall-clock time

Figure 12: Training dynamics on BrowseComp-Plus (Qwen3-30B-A3B). Top row: (a) gradient norm, (b) number of interaction turns, and (c) rollout response length. Bottom row: (d) training reward, (e) training-side log-perplexity on the generated tokens, and (f) per-step wall-clock time (ESTR vs. the synchronous run). Panels (a)–(d) compare the asynchronous methods (ESTR, KPop, IcePop, and vanilla GRPO) under the same configuration; panel (e) compares the three keep-rule methods (ESTR, KPop, IcePop).

Geometric comparison.

Now compare the ESTR boundary (59) against the constant boundary $|\delta|=c$ with the same $\tau$ . Equating the two,

\sqrt{\tau\,(H_{t}+\epsilon)}=\sqrt{\tau C}\quad\Longleftrightarrow\quad H_{t}=C-\epsilon\;\triangleq\;H^{\star},

(62)

so the two boundaries intersect at the single entropy level $H^{\star}$ . Since $b_{\mathrm{ESTR}}$ is strictly increasing in $H_{t}$ ,

	$\displaystyle H_{t}<H^{\star}\;$	$\displaystyle\Longrightarrow\;b_{\mathrm{ESTR}}(H_{t})<c,$		(63)
	$\displaystyle H_{t}>H^{\star}\;$	$\displaystyle\Longrightarrow\;b_{\mathrm{ESTR}}(H_{t})>c.$		(64)

That is, relative to the fixed threshold, ESTR is strictly tighter at low entropy—rejecting the amplified sampling noise of the low-entropy regime, whose scale $\frac{1-q}{q}$ diverges as entropy vanishes—and strictly wider at high entropy, retaining the genuine exploration characterized by the entropy–ratio scaling law. In the two limits,

	$\displaystyle\lim_{H_{t}\to 0}b_{\mathrm{ESTR}}(H_{t})=\sqrt{\tau\epsilon},$		(65)
	$\displaystyle b_{\mathrm{ESTR}}(H_{t})\sim\sqrt{\tau H_{t}}\to\infty\quad(H_{t}\to\infty),$		(66)

so the admissible deviation contracts to a small floor at near-zero entropy and grows without bound as entropy increases.

Conclusion.

The fixed-threshold rule is thus the single horizontal slice $H^{\star}=C-\epsilon$ of the entropy-opening ESTR boundary. ESTR reduces to it exactly under the homoscedastic assumption, and otherwise strictly departs from it in an entropy-dependent manner, which proves the claim. $\blacksquare$

Appendix F Derivation of the ESTR Policy Gradient

We derive the gradient of the ESTR objective and verify the clip indicator $C_{i,t}$ used in the main text. Let the per-token surrogate be

g_{i,t}=\min\!\big(r_{i,t}A_{i,t},\ \mathrm{clip}(r_{i,t},1-\epsilon_{\mathrm{low}},1+\epsilon_{\mathrm{high}})\,A_{i,t}\big),

(67)

so that $\mathcal{L}_{\mathrm{ESTR}}(\theta)=-\mathbb{E}\big[\tfrac{1}{\sum_{i}|o_{i}|}\sum_{i,t}M_{i,t}\,g_{i,t}\big]$ .

Two elementary facts.

First, the behavior policy $\mu_{i,t}$ that produced the rollout is fixed with respect to $\theta$ , so

	$\displaystyle\nabla_{\theta}r_{i,t}$	$\displaystyle=\nabla_{\theta}\frac{\pi_{\theta}(o_{i,t}\mid s_{i,t})}{\mu_{i,t}(o_{i,t})}$		(68)
		$\displaystyle=r_{i,t}\,\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid s_{i,t}).$		(68)

Second, the keep mask $M_{i,t}=\mathbf{1}[\delta_{i,t}^{2}/(H_{i,t}+\epsilon)\leq\tau]$ is piecewise constant in $\theta$ , hence $\nabla_{\theta}M_{i,t}=0$ almost everywhere, and we treat it as a stop-gradient gate.

Gradient of the clipped surrogate.

Write $u_{i,t}\triangleq\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid s_{i,t})$ . The term $g_{i,t}$ is piecewise, and its gradient vanishes precisely when the $\min$ selects the flat clipped branch. A case analysis on the sign of $A_{i,t}$ and the position of $r_{i,t}$ relative to the clip interval shows that this happens only on the two flat branches, so

\nabla_{\theta}g_{i,t}=\begin{cases}0,&A_{i,t}>0,\ r_{i,t}>1+\epsilon_{\mathrm{high}},\\[2.15277pt] 0,&A_{i,t}<0,\ r_{i,t}<1-\epsilon_{\mathrm{low}},\\[2.15277pt] r_{i,t}A_{i,t}\,u_{i,t},&\text{otherwise.}\end{cases}

(69)

Collecting the two flat branches into the indicator

C_{i,t}=1-\mathbf{1}\!\left[A_{i,t}>0,\ r_{i,t}>1+\epsilon_{\mathrm{high}}\right]\\ -\mathbf{1}\!\left[A_{i,t}<0,\ r_{i,t}<1-\epsilon_{\mathrm{low}}\right]\in\{0,1\},

(70)

all cases collapse to the single expression $\nabla_{\theta}g_{i,t}=C_{i,t}\,r_{i,t}A_{i,t}\,u_{i,t}$ .

Full gradient.

Multiplying by the stop-gradient mask $M_{i,t}$ and taking the expectation gives

\nabla_{\theta}\mathcal{L}_{\mathrm{ESTR}}(\theta)=-\,\mathbb{E}_{x\sim\mathcal{D},\ \{o_{i}\}\sim\mu(\cdot\mid x)}\Bigg[\frac{1}{\sum_{i}|o_{i}|}\\ \times\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}M_{i,t}\,C_{i,t}\,r_{i,t}\,A_{i,t}\,u_{i,t}\Bigg],

(71)

which is the main-text policy gradient. In particular a token with $M_{i,t}=0$ contributes exactly zero, so the entropy-scaled mask removes the gradient of every token that violates the trust region, without affecting the gradient of the retained tokens. $\blacksquare$

Appendix G Training Configuration

All tasks share GRPO advantages, no KL penalty ( $\beta_{\mathrm{KL}}{=}0$ ), asymmetric clipping $(0.20,0.28)$ , token-mean loss, zero entropy bonus, and a constant learning rate $1\!\times\!10^{-6}$ (AdamW). ESTR is a token-level keep rule $\delta_{t}^{2}/(H_{t}+\epsilon)\leq\tau$ ( $\epsilon{=}0.01$ ) applied on top of GRPO, with rollout log-probabilities as the proximal anchor (bypass mode) and full parameter/gradient/optimizer offloading with activation recomputation. The complete per-task configuration is summarized in Table 3.

Appendix H Hyperparameter Sensitivity

(a) Token entropy

(b) Masked token fraction

Figure 14: Training dynamics on multi-turn GSM8K (Qwen2.5-7B). (a) Token entropy: vanilla GRPO and IcePop blow up, while ESTR stays stable and tracks the synchronous run. (b) Masked token fraction: IcePop’s blows up by orders of magnitude; ESTR keeps it low and stable. (c) Validation accuracy (mean@4): ESTR tracks the synchronous run, whereas the asynchronous baselines collapse once their entropy destabilizes.

We probe the sensitivity of ESTR to its two most relevant knobs, varying one at a time while holding all other settings fixed as in Table 3. Tables 4 and 5 report the effect of the keep threshold $\tau$ on BrowseComp-Plus and DAPO-Math, and Table 6 reports the effect of the global batch size on DAPO-Math. Across a broad range of $\tau$ ( $0.7$ – $1.6$ ) performance stays within roughly one point on both tasks, indicating that ESTR is not sensitive to the exact threshold; a mild optimum appears around $\tau\!\approx\!1.0$ on BrowseComp-Plus and $\tau\!\approx\!1.6$ on the harder AIME splits. Increasing the global batch size improves DAPO-Math accuracy and largely saturates beyond $128$ prompts. All AIME numbers are reported as mean@4 (4 samples per problem).

	$\tau{=}0.7$	$\tau{=}1.0$	$\tau{=}1.6$
Accuracy	35.97	37.34	37.15
Reward	29.87	32.25	30.62

Table 4: Sensitivity to the ESTR keep threshold

\tau

on BrowseComp-Plus (Qwen3-30B-A3B); other settings as in Table 3.

$\tau$	$0.7$	$1.0$	$1.6$
AIME-2024 (mean@4)	19.67	19.85	20.03
AIME-2025 (mean@4)	14.33	16.12	15.64
AIME-2026 (mean@4)	13.49	15.21	15.46

Table 5: Sensitivity to

\tau

on DAPO-Math (Qwen2.5-7B); other settings as in Table 3.

Global batch (prompts)	$32$	$64$	$128$	$256$
AIME-2024 (mean@4)	16.97	18.26	20.03	19.87
AIME-2025 (mean@4)	13.23	14.19	15.64	15.58
AIME-2026 (mean@4)	12.94	14.47	15.46	16.32

Table 6: Sensitivity to global batch size (prompts) on DAPO-Math (Qwen2.5-7B);

\tau{=}1.6

, other settings as in Table 3.

Appendix I Additional Training Dynamics

To complement the main-text results, we report additional training-dynamics curves for each task. Unless otherwise noted, all curves show the asynchronous methods (ESTR, KPop, IcePop, and vanilla GRPO) under the same configuration as the corresponding main-text experiment, and raw values are smoothed with an exponential moving average for readability.

I.1 BrowseComp-Plus

Figure 12 tracks six signals on BrowseComp-Plus (Qwen3-30B-A3B). On the gradient norm (Figure 12(a)), vanilla GRPO spikes sharply (early version switches and late bursts) and IcePop/KPop drift upward with the off-policy gap, whereas ESTR stays low and stable—local entropy normalization suppresses the noise-driven updates that inflate the gradient. This stability is productive, not conservative: ESTR reaches the fewest turns (Figure 12(b)) and shortest responses (Figure 12(c)) yet the highest final reward (Figure 12(d)), with vanilla GRPO lowest and IcePop/KPop in between.

Log-perplexity (Figure 12(e)) is not “lower is better”: ESTR rises smoothly to the highest value alongside its highest reward, IcePop spikes late (tracking instability), and KPop stays lowest (weaker learning). ESTR is also faster: unlike the synchronous baseline that stalls at every step (Figure 12(f)), it overlaps rollout with training and settles at a substantially lower, stable per-step time.

Mask evolution over training.

Figure 13 visualizes how the entropy-scaled keep rule operates in the $(H_{t},|\delta_{t}|)$ plane as training progresses, providing a dynamic counterpart to the static boundary of Figure 1 in the main text. Three observations stand out. First, at every stage the kept population fills precisely the region beneath the entropy-scaled boundary: the spread of $|\delta_{t}|$ widens with $H_{t}$ exactly as predicted by the scaling law of Proposition 2, confirming that the boundary tracks the natural scale of the deviation rather than an arbitrary cutoff. Second, the masked tokens (red) are dominated by the low-entropy outliers that trace the concave arcs analyzed earlier for the low-entropy regime—the amplified train-inference noise— whereas large deviations at high entropy remain almost entirely inside the boundary and are retained as legitimate exploration. Third, the masked fraction is self-regulating: it stays below $1\%$ through most of training and rises only to $4.8\%$ at step $800$ , when the accumulated off-policy gap inflates deviations across the batch. The rule thus tightens its effect exactly when and where harmful deviations emerge, without any schedule or staleness-specific tuning, which explains the order-of-magnitude lower masking rates reported in the main text.

(a) Sequence masked fraction

(b) Token entropy

(d) Mask vs. batch size

Figure 15: Training dynamics on DAPO-Math (Qwen2.5-7B). (a) Sequence-level masked fraction (ESTR, IcePop, KPop) and (b) token entropy (ESTR, vanilla GRPO, IcePop, KPop) for the asynchronous methods. (c)–(d) ESTR’s sensitivity to the global batch size (

32/64/128/256

, darker = larger): (c) training reward (mapped

[-1,1]\!\to\![0,1]

) and (d) the entropy-scaled masked fraction (

\times 10^{-3}

). Curves are EMA-smoothed and truncated at

800

steps.

I.2 Multi-turn GSM8K

Figures 14(a) and 14(b) report token entropy and masking on multi-turn GSM8K (Qwen2.5-7B). IcePop is the most unstable: its entropy spikes until the run destabilizes, and its masked fraction drifts up by orders of magnitude, discarding a large share of tokens. Vanilla GRPO also loses control of its entropy, spiking out of the healthy range late in training. ESTR instead keeps both entropy and masked fraction low and stable throughout, discarding only the harmful noise rather than genuine exploration.

This carries over to accuracy (Figure 14(c)): ESTR tracks the synchronous run within about one point and reaches the highest final accuracy among asynchronous methods, whereas vanilla GRPO, IcePop, and KPop each collapse as their entropy destabilizes and never recover. ESTR is thus the only asynchronous method that stays stable and matches the synchronous accuracy trajectory.

I.3 DAPO-Math

Figures 15(a) and 15(b) expose how each method trades stability against exploration on DAPO-Math (Qwen2.5-7B). IcePop masks $40\%$ – $65\%$ of sequences throughout and KPop shows large intermittent spikes, whereas ESTR keeps the sequence-level masked fraction below $\sim$ 5% (Figure 15(a)): fixed thresholds cannot separate genuine high-entropy exploration from noise, so they either over-discard or oscillate, while ESTR’s entropy-scaled boundary removes only the amplified low-entropy noise. The entropy curves (Figure 15(b)) confirm this: vanilla GRPO’s entropy explodes upward without bound, IcePop and KPop collapse toward zero through over-suppression, and ESTR alone sustains a gradual, controlled rise—masking not less but the right tokens.

Sweeping the global batch over $32/64/128/256$ (Figures 15(c) and 15(d)), training reward improves monotonically ( $\approx$ $0.33\!\to\!0.44$ ) with diminishing returns beyond $128$ , driven by lower gradient variance. Crucially, the masked fraction stays negligible (below $\sim$ $0.6\times 10^{-3}$ ) across the entire sweep, so larger batches help through variance reduction rather than by altering the trust region—matching the accuracy sensitivity in Table 6.

Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

Abstract

1 Introduction

2 Related Work

3 Motivation

3.1 Ratio Magnitude Relates to Entropy

3.2 Dual Phenomena under Asynchronous Dynamics

3.3 An Entropy-Scaled Design Principle

4 Method

4.1 A Second-Order Trust Region

4.2 An Entropy-Standardized Budget

4.3 Entropy-Scaled Keep Mask

Proposition 1 (Strict Generalization)

4.4 The ESTR Objective

5 Experiment

5.1 Setup

Tasks.

Baselines.

Implementation.

5.2 Long-Horizon Multi-Turn Tool Use

5.3 Mathematical Reasoning

Mask behavior and variance control.

Robustness to each staleness component.

5.4 Training Efficiency

6 Conclusion

References

Appendix A Proof of the Entropy–Ratio Scaling Law

Setup and notation.

Proposition 2 (Entropy–ratio scaling)

Proof.

Switch-point invariance of the standardized deviation.

Proposition 3 (Switch-point invariance)

Proof.

Empirical symmetry of the signed deviation.

Appendix B Low-Entropy Regime: Amplified Off-Mode Noise and the Concave Arc

Variance law via the delta method.

The concave binary-entropy envelope.

Appendix C Second-Order Expansion of the KL Divergence

Appendix D Joint Monotonicity of Hbin​(q)H_{\mathrm{bin}}(q) and 1−qq\frac{1-q}{q}

Monotonicity of the variance scale.

Monotonicity of the binary entropy.

Monotone reparameterization.

Why entropy is the preferred scale.

Probability-space view of the two rules.

Appendix E Proof of the Strict Generalization Property

Proposition 4 (Strict Generalization)

Setup.

Fixed thresholding as a special case.

Geometric comparison.

Conclusion.

Appendix F Derivation of the ESTR Policy Gradient

Two elementary facts.

Gradient of the clipped surrogate.

Full gradient.

Appendix G Training Configuration

Appendix H Hyperparameter Sensitivity

Appendix I Additional Training Dynamics

I.1 BrowseComp-Plus

Mask evolution over training.

I.2 Multi-turn GSM8K

I.3 DAPO-Math

Appendix D Joint Monotonicity of $H_{\mathrm{bin}}(q)$ and $\frac{1-q}{q}$