FlashREINFORCE and Score Centering address training with data generated by a policy that differs from the learner. Their relationship becomes clearer when we examine the gradient estimator itself.
The central observation is that FlashREINFORCE’s unclipped token-level importance-sampling core already satisfies the zero-mean condition targeted by weighted Score Centering. Under exact importance sampling, the additional centering term is identically zero. Importance sampling also corrects the local action weighting, which Score Centering alone generally does not.
This article derives that relationship and distinguishes it from the separate effects of trajectory selection and long-horizon distribution mismatch.
For trajectory $i$, let $h_{i,t}$ denote a stored prefix and $a_{i,t}$ its sampled token. Write $q_i$ for the behavior policy that generated the trajectory and $p_\theta$ for the current learner. Define
$$ r_{i,t}=\frac{p_\theta(a_{i,t}\mid h_{i,t})}{q_i(a_{i,t}\mid h_{i,t})}, \qquad s_{i,t}=\nabla_\theta\log p_\theta(a_{i,t}\mid h_{i,t}). $$
FlashREINFORCE’s core gradient estimator combines these token ratios with batch-centered advantages, sequence admission, and sample-mean aggregation:[2]
$$ \widehat g_{\mathrm{Flash}} =\frac1B\sum_{i=1}^{B}\frac{M_iA_i}{T_i} \sum_{t=1}^{T_i}r_{i,t}s_{i,t}. $$
Here $A_i$ is the batch/group-centered advantage, $M_i$ admits or rejects an entire trajectory using its mean sampled-action Bernoulli-KL proxy, and $T_i$ counts policy tokens. The reduction averages within each trajectory and then across the batch. Rejected trajectories remain in the original batch denominator.
The published reference loss implements the same update through detached ratios multiplying log-probabilities. Its base method does not clip the token importance ratios, and it contains no explicit SC correction.[3] We analyze this core without optional entropy-based token filtering.
This separates two operations: the sequence gate determines which trajectories contribute; the token ratio determines the importance weight of each contributing token. They have different mathematical effects.
Fix a prefix $h$ and suppress it temporarily. Let $a\sim q$, let $s(a)=\nabla_\theta\log p_\theta(a)$, and let $A(a)$ be a fixed local utility or detached advantage function. This local formulation isolates action-distribution correction from history and continuation mismatch.
The score has zero mean under the policy whose log-probability is differentiated:
$$ \mathbb E_p[s] =\sum_a p_\theta(a)\nabla_\theta\log p_\theta(a) =\nabla_\theta\sum_a p_\theta(a)=0. $$
Under $q$, however, its mean need not be zero. The uncorrected expected update decomposes as
$$ \mathbb E_q[As] =\operatorname{Cov}q(A,s) +\underbrace{\mathbb E_q[A]\,\mathbb E_q[s]}{\text{drift}}. $$
The drift term couples the average advantage to the average score. It can produce an update even when all actions have the same utility. Global reward centering does not generally eliminate it because the conditional mean advantage at each prefix need not vanish.
SC subtracts the conditional mean score:[1]
$$ \widehat g_{\mathrm{SC}}=A\left(s-\mathbb E_q[s]\right), \qquad \mathbb E_q[\widehat g_{\mathrm{SC}}]=\operatorname{Cov}_q(A,s). $$