KL Divergence
Notes on exact KL, k₁, k₃, and KL constraints and aggregation in GRPO.
After SFT, a model has been trained at the token level, making its token probability distribution favor responses that follow the intended format. RL can further improve how it answers and the accuracy of its responses. During RL training, we do not want the model to stray far from the response format established by SFT, so we use KL divergence as a constraint to limit how far the model moves from the post-SFT model while trying to further improve accuracy.
Let the complete response generated by the model be $o=(o_1,o_2,\ldots,o_{|o|})$. The content generated before step $t$ is written as $o_{\lt t}=(o_1,o_2,\ldots,o_{t-1})$. The context prefix at step $t$ is then $h_t=(q,o_{\lt t})$.
Under the same context prefix, the probability that the reference policy model—the post-SFT model whose tone and format we want the model to follow—selects the same candidate token is $\pi_{\mathrm{ref}}(a\mid h_t)=\pi_{\mathrm{ref}}(a\mid q,o_{\lt t})$. When the candidate position is written as a dot, $\pi_\theta(\cdot\mid h_t),\pi_{\mathrm{ref}}(\cdot\mid h_t)$, providing a specific token gives a probability value; going through all candidate tokens gives a list of probabilities.
For a fixed context prefix $h$, assume that the two policies have the same support. Their common candidate set is:
$$ \begin{aligned} \mathcal A_h &=\{a:\pi_\theta(a\mid h)>0\}\\ &=\{a:\pi_{\mathrm{ref}}(a\mid h)>0\}. \end{aligned} $$Write the size of the set as $m_h$, and define two probabilities: the probability of selecting a token from the model currently being trained and from the reference model, respectively:
$$ p_a=\pi_\theta(a\mid h), \qquad b_a=\pi_{\mathrm{ref}}(a\mid h). $$Here, $p,b\in\mathbb R^{m_h}$. Both lists are normalized separately: $\sum_{a\in\mathcal A_h}p_a=1,\sum_{a\in\mathcal A_h}b_a=1$.
Exact KL
For a fixed prefix $h_t$, KL is defined as:
$$ \begin{aligned} &D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot\mid h_t) \Vert \pi_{\mathrm{ref}}(\cdot\mid h_t) \right)\\ &\quad= \mathbb E_{a\sim\pi_\theta(\cdot\mid h_t)} \left[ \log\frac{\pi_\theta(a\mid h_t)} {\pi_{\mathrm{ref}}(a\mid h_t)} \right]. \end{aligned} $$The left-hand side compares two complete next-token distributions. The expectation on the right means taking a weighted average of the values in brackets, weighted by the current policy’s probability of selecting each token. For a finite candidate set, the expectation can be expanded directly into a sum:
$$ \begin{aligned} &D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot\mid h_t) \Vert \pi_{\mathrm{ref}}(\cdot\mid h_t) \right)\\ &\quad= \sum_{a\in\mathcal A_{h_t}} \pi_\theta(a\mid h_t) \log\frac{\pi_\theta(a\mid h_t)} {\pi_{\mathrm{ref}}(a\mid h_t)}. \end{aligned} $$This step uses only the definition of the expectation of a discrete random variable. $\log\frac{\pi_\theta(a\mid h_t)}{\pi_{\mathrm{ref}}(a\mid h_t)}$ is the token’s log probability ratio; an individual term can be positive or negative. Only after summing over all candidates, weighted by their current probabilities, do we obtain a nonnegative KL penalty. If the two distributions are identical, all ratios are 1, taking $\log$ gives 0, and there is no penalty.
The sum for exact KL cannot be negative, so it can serve as a penalty
Use the inequality that holds for all positive numbers:
$$ \log u\leq u-1,\qquad u>0. $$Let:
$$ u=\frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)}. $$Rearranging gives:
$$ \log\frac{\pi_\theta(a\mid h)} {\pi_{\mathrm{ref}}(a\mid h)} \geq 1-\frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)}. $$Multiply both sides by the current probability, then sum over all candidates:
$$ \begin{aligned} &D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot\mid h) \Vert \pi_{\mathrm{ref}}(\cdot\mid h) \right)\\ &\quad\geq \sum_{a\in\mathcal A_h} \left[ \pi_\theta(a\mid h) -\pi_{\mathrm{ref}}(a\mid h) \right]\\ &\quad=1-1=0. \end{aligned} $$This proof also uses the fact that each probability distribution sums to 1. The basic inequality above holds with equality only when $u$ equals 1. Therefore, under the common-support assumption, KL is 0 if and only if the two distributions are identical across all candidates.
$k_1$
For a fixed prefix $h$ and a sampled token $a$, define:
$$ k_1(a,h;\theta) = \log\frac{\pi_\theta(a\mid h)} {\pi_{\mathrm{ref}}(a\mid h)}. $$If we use indices for “response $i$, token $t$,” first define:
$$ h_{i,t}=(q,o_{i,\lt t}), $$Then write:
$$ k_{1,i,t}(\theta) = \log \frac{\pi_\theta(o_{i,t}\mid h_{i,t})} {\pi_{\mathrm{ref}}(o_{i,t}\mid h_{i,t})}. $$This is simply the log probability ratio of the sampled token. For a fixed prefix and a token sampled from the current policy, we have:
$$ \begin{aligned} &\mathbb E_{a\sim\pi_\theta(\cdot\mid h)} \left[k_1(a,h;\theta)\right] = D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot\mid h) \Vert \pi_{\mathrm{ref}}(\cdot\mid h) \right). \end{aligned} $$This follows directly from the definition of KL. An individual sample of $k_1$ can be negative, while the KL of the entire distribution is nonnegative. If we repeatedly sample $k_1$, its overall expectation is also the same as exact KL, making it an unbiased estimator:
Keep the prefix fixed and independently sample $N$ tokens:
$$ a^{(1)},\ldots,a^{(N)} \overset{\mathrm{iid}}{\sim} \pi_\theta(\cdot\mid h). $$$$ \begin{aligned} &\widehat D_{\mathrm{KL}}^{(k_1,N)}(h;\theta) = \frac{1}{N}\sum_{n=1}^{N} \log \frac{\pi_\theta(a^{(n)}\mid h)} {\pi_{\mathrm{ref}}(a^{(n)}\mid h)}. \end{aligned} $$Take the expectation of this random estimator, using linearity of expectation:
$$ \begin{aligned} &\mathbb E\!\left[ \widehat D_{\mathrm{KL}}^{(k_1,N)}(h;\theta) \right]\\ &\quad= \frac{1}{N}\sum_{n=1}^{N} \mathbb E\!\left[ \log \frac{\pi_\theta(a^{(n)}\mid h)} {\pi_{\mathrm{ref}}(a^{(n)}\mid h)} \right]\\ &\quad= D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot\mid h) \Vert \pi_{\mathrm{ref}}(\cdot\mid h) \right). \end{aligned} $$The KL Constraint for a Complete Response
Treat the EOS end marker as a token as well, and use the following rules:
- A response ends when it generates EOS.
- Once a response has ended, all subsequent positions are deterministically padded with EOS.
- At position $T$, both policies are forced to select EOS.
Below, $|o|$ denotes the valid length, including the first EOS but excluding padded positions.
A response that ends early might be ${o=(\text {hi,how,are,you,today,?,EOS,EOS,EOS})}$. The first EOS is the decision to stop; the later EOS tokens are padding added to make the lengths uniform. Both conditional probabilities at padded positions equal 1:
$$ \pi_\theta(\mathrm{EOS}\mid h_t) = \pi_{\mathrm{ref}}(\mathrm{EOS}\mid h_t) =1. $$To clearly distinguish the levels, use capital letters for the probabilities of a complete response:
$$ P_\theta(o\mid q), \qquad P_{\mathrm{ref}}(o\mid q). $$For a specified complete response, the token and preceding context at every step are fixed. The probability that the current policy produces this response is:
$$ P_\theta(o\mid q) = \prod_{t=1}^{T} \pi_\theta(o_t\mid h_t). $$The probability that the reference policy produces the same response is:
$$ P_{\mathrm{ref}}(o\mid q) = \prod_{t=1}^{T} \pi_{\mathrm{ref}}(o_t\mid h_t). $$Expanding the product for the current policy gives:
$$ \begin{aligned} P_\theta(o\mid q) &=\pi_\theta(o_1\mid q)\\ &\quad\times\pi_\theta(o_2\mid q,o_1)\\ &\quad\times\cdots\\ &\quad\times \pi_\theta(o_T\mid q,o_1,\ldots,o_{T-1}). \end{aligned} $$This does not assume that the tokens are independent: every term explicitly depends on the tokens that came before it. First take the ratio of the two probabilities for the complete response, then take the logarithm:
$$ \begin{aligned} &\log\frac{P_\theta(o\mid q)} {P_{\mathrm{ref}}(o\mid q)}\\ &\quad= \log \frac{\prod_{t=1}^{T}\pi_\theta(o_t\mid h_t)} {\prod_{t=1}^{T}\pi_{\mathrm{ref}}(o_t\mid h_t)}\\ &\quad= \log\prod_{t=1}^{T} \frac{\pi_\theta(o_t\mid h_t)} {\pi_{\mathrm{ref}}(o_t\mid h_t)}\\ &\quad= \sum_{t=1}^{T} \log\frac{\pi_\theta(o_t\mid h_t)} {\pi_{\mathrm{ref}}(o_t\mid h_t)}. \end{aligned} $$For a fixed question $q$, denote the set of possible responses that follow the stopping rules above by $\Omega_T(q)$. We have:
$$ \begin{aligned} &D_{\mathrm{KL}}\!\left( P_\theta(\cdot\mid q) \Vert P_{\mathrm{ref}}(\cdot\mid q) \right)\\ &\quad= \sum_{o\in\Omega_T(q)} P_\theta(o\mid q) \log\frac{P_\theta(o\mid q)} {P_{\mathrm{ref}}(o\mid q)}\\ &\quad= \mathbb E_{o\sim P_\theta(\cdot\mid q)} \left[ \log\frac{P_\theta(o\mid q)} {P_{\mathrm{ref}}(o\mid q)} \right]. \end{aligned} $$The structure is the same as KL at a single position, but the candidate changes from “one token” to “one complete response.” The weight also changes from the next-token probability to the probability that the current policy generates the complete response. Similarly, KL for a complete response can be written in several other forms, such as:
- Rewrite the logarithm of the product as a sum:
- Move the finite sum outside the expectation:
$k_3$
Continue to use $h_{i,t}$ for the context in response $i$ before token $t$, and let $u_{i,t}(\theta)=\frac{\pi_{\mathrm{ref}}(o_{i,t}\mid h_{i,t})}{\pi_\theta(o_{i,t}\mid h_{i,t})}$. Our per-token penalty estimate is then:
$$ \hat d_{i,t}(\theta) = u_{i,t}(\theta) -\log u_{i,t}(\theta)-1. $$$$ \begin{aligned} \hat d_{i,t}(\theta) &= \frac{\pi_{\mathrm{ref}}(o_{i,t}\mid h_{i,t})} {\pi_\theta(o_{i,t}\mid h_{i,t})} - \log \frac{\pi_{\mathrm{ref}}(o_{i,t}\mid h_{i,t})} {\pi_\theta(o_{i,t}\mid h_{i,t})} -1. \end{aligned} $$We call the expression above $k_3$. When discussing a candidate token under a fixed prefix, with the prefix denoted by $h$ and the token by $a$, define:
$$ \begin{aligned} k_3(a,h;\theta) &= \frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)} - \log\frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)} -1. \end{aligned} $$Unlike $k_1$, every sample of $k_3$ is nonnegative.
For any positive number $u$, consider the function:
$$ g(u)=u-\log u-1. $$Its derivative is:
$$ g'(u)=1-\frac{1}{u}. $$When $u$ is less than 1, the derivative is negative; when $u$ is greater than 1, the derivative is positive. The function therefore reaches its minimum when $u$ equals 1:
$$ g(1)=1-\log 1-1=0. $$Thus:
$$ u-\log u-1\geq0,\qquad u>0. $$Similarly, when sampling from the current policy, $k_3$ is also an unbiased estimator of exact KL:
$$ \begin{aligned} &\mathbb E_{a\sim\pi_\theta(\cdot\mid h)} \left[ \frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)} \right]\\ &\quad= \sum_{a\in\mathcal A_h} \pi_\theta(a\mid h) \frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)}\\ &\quad= \sum_{a\in\mathcal A_h} \pi_{\mathrm{ref}}(a\mid h)\\ &\quad=1. \end{aligned} \tag{1} $$$$ -\log \frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)} = \log \frac{\pi_\theta(a\mid h)} {\pi_{\mathrm{ref}}(a\mid h)}. \tag{2} $$Substitute equations (1) and (2) into the expectation of $k_3$:
$$ \begin{aligned} &\mathbb E_{a\sim\pi_\theta(\cdot\mid h)} \Biggl[ \frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)} - \log\frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)} -1 \Biggr]\\ &\quad= 1+ \mathbb E_{a\sim\pi_\theta(\cdot\mid h)} \left[ \log\frac{\pi_\theta(a\mid h)} {\pi_{\mathrm{ref}}(a\mid h)} \right]-1\\ &\quad= D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot\mid h) \Vert \pi_{\mathrm{ref}}(\cdot\mid h) \right). \end{aligned} $$We can see that $k_3$ can be viewed as adding a correction term with mean 0 to $k_1$. The correction changes each sample’s value to make it nonnegative, but does not change the expectation:
$$ \begin{aligned} &\frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)} -\log\frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)}-1\\ &\quad= \log\frac{\pi_\theta(a\mid h)} {\pi_{\mathrm{ref}}(a\mid h)} +\left( \frac{\pi_{\mathrm{ref}}(a\mid h)} {\pi_\theta(a\mid h)}-1 \right). \end{aligned} $$Bringing KL into GRPO
The clipped objective uses the probability ratio between the current policy and the old policy:
$$ \rho_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})} {\pi_{\mathrm{old}}(o_{i,t}\mid q,o_{i,\lt t})}. $$$k_3$ uses the probability ratio between the reference policy and the current policy:
$$ u_{i,t}(\theta) = \frac{\pi_{\mathrm{ref}}(o_{i,t}\mid q,o_{i,\lt t})} {\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})}. $$The former measures how much the current model has changed since this batch of samples was collected, and is used in the clipped reward-advantage term. The latter is used to construct the penalty for deviating from the reference model. Their denominators, numerators, and comparison purposes are all different.
In the outcome-supervised GRPO considered here, each token’s contribution is defined as:
$$ \begin{aligned} \ell_{i,t}(\theta) &= \min\Bigl( \rho_{i,t}(\theta)\hat A_i, \operatorname{clip}\bigl( \rho_{i,t}(\theta),1-\epsilon,1+\epsilon \bigr)\hat A_i \Bigr) -\beta\,\hat d_{i,t}(\theta). \end{aligned} $$The GRPO optimization objective is:
$$ \begin{aligned} &J_{\mathrm{GRPO}}(\theta) = \mathbb E_{ \substack{ q\sim\mathcal D\\ \{o_i\}_{i=1}^{G} \sim P_{\mathrm{old}}(\cdot\mid q) } } \left[ \frac{1}{G}\sum_{i=1}^{G} \frac1{|o_i|} \sum_{t=1}^{|o_i|} \ell_{i,t}(\theta) \right]. \end{aligned} $$The unbiasedness of $k_3$ above requires sampling from the current policy; here, the responses come from the old policy, so the term used is a KL surrogate.
Training maximizes this objective. If an implementation negates the objective and minimizes it as a loss, the KL penalty appears with a positive sign in that loss. This is simply the sign relationship between maximizing an objective and minimizing its negative. For a group of responses that has already been sampled, the average penalty estimate used inside the objective is:
$$ \frac{1}{G}\sum_{i=1}^{G} \frac1{|o_i|} \sum_{t=1}^{|o_i|} \hat d_{i,t}(\theta). $$Substitute the full probability expression for $k_3$:
$$ \begin{aligned} &\frac{1}{G}\sum_{i=1}^{G} \frac1{|o_i|} \sum_{t=1}^{|o_i|} \Biggl[ \frac{\pi_{\mathrm{ref}}(o_{i,t}\mid h_{i,t})} {\pi_\theta(o_{i,t}\mid h_{i,t})} - \log \frac{\pi_{\mathrm{ref}}(o_{i,t}\mid h_{i,t})} {\pi_\theta(o_{i,t}\mid h_{i,t})} -1 \Biggr]. \end{aligned} $$In the objective being maximized, this is then multiplied by negative $\beta$. The expression retains the full response indices, token indices, context conditions, probability ratios, and two levels of averaging. The larger $\beta$ is, the higher the cost of deviating from the reference distribution; the smaller $\beta$ is, the more easily reward-driven changes can dominate.
For comparison, when sampling from the current policy, the KL for complete responses is:
$$ \begin{aligned} &D_{\mathrm{KL}}\!\left( P_\theta(\cdot\mid q) \Vert P_{\mathrm{ref}}(\cdot\mid q) \right)\\ &\quad= \mathbb E_{o\sim P_\theta(\cdot\mid q)} \left[ \sum_{t=1}^{|o|} \log \frac{\pi_\theta(o_t\mid h_t)} {\pi_{\mathrm{ref}}(o_t\mid h_t)} \right]. \end{aligned} $$If this is changed to:
$$ \mathbb E_{o\sim P_\theta(\cdot\mid q)} \left[ \frac1{|o|} \sum_{t=1}^{|o|} \log \frac{\pi_\theta(o_t\mid h_t)} {\pi_{\mathrm{ref}}(o_t\mid h_t)} \right], $$Each response’s sum is additionally multiplied by the reciprocal of its own length. This generally no longer equals the original KL for complete responses.
If all responses have the same fixed length, averaging by length introduces only a fixed scaling factor. If lengths differ, the factor varies with the response and cannot be taken outside the expectation as a single constant. Longer responses may therefore accumulate a larger sequence penalty when the deviation at each step is similar; averaging within each response changes this length effect. Length normalization can be a training design choice, but the results need to be interpreted in terms of the quantity it actually computes.
Workflow
Estimation method: exact KL, $k_1$, or $k_3$
The differences between the three can be summarized as follows:
| Calculation method | Information used | Properties with the model and prefix fixed |
|---|---|---|
| Exact conditional KL | Both probability distributions over all candidate tokens | Deterministic value, theoretically nonnegative |
| $k_1$ | Current and reference probabilities of the token actually selected | Varies with sampling, can be negative |
| $k_3$ | Current and reference probabilities of the token actually selected | Varies with sampling, each sample is theoretically nonnegative |
Aggregation method:
Suppose there are $G$ responses to the same question included in the calculation, each padded to the same maximum length. Define the valid-token mask as $m_{i,t}\in\{0,1\}$, where 1 means that the position is included and 0 means that it is excluded. Below, assume that every response included in the average contains at least one valid token.
Using the per-token KL penalty estimate from above, $\hat d_{i,t}(\theta)$, the same set of per-token values can be aggregated in different ways. The difference is whether to normalize by response length first, and how much weight each response or token receives.
- Average over tokens within each response, then take an equally weighted average over responses:
With this method, each response has the same weight. A longer response does not receive greater total weight within the group merely because it contains more tokens; correspondingly, an individual token in a shorter response receives greater weight.
- Pool all valid tokens and average over them:
With this method, each valid token has the same weight. A longer response contains more valid tokens and therefore receives greater weight in the final average.
- First calculate the sequence sum for each response, then take an equally weighted average over responses:
This method preserves the effect of length on the sequence total. When per-token penalties are similar, longer responses usually accumulate a larger total penalty.
The original GRPO formulation discussed in this article uses method 1 for the KL surrogate term.
Comments