大模型对齐算法合集(一)

大模型对齐算法合集

DPO

定理：
max⁡μEx∼μ(x){f(x)}+H(μ),s.t. ∑xμ(x)=1\max_{\mu} \mathbb{E}_{x \sim \mu(x)} \left\{ f(x) \right\} + H(\mu), \quad \text{s.t. } \sum_x \mu(x) = 1 μmax​Ex∼μ(x)​{f(x)}+H(μ),s.t. x∑​μ(x)=1

$$\Rightarrow {\mu }^{\ast }(x)=\frac{e^{f(x)}}{{\sum }_x{e}^{f(x)}}\Rightarrow \text{Boltzmann distribution}$$

证明：
max⁡μEx∼μ(x){f(x)}+H(μ)=max⁡μEx∼μ(x){f(x)−log⁡μ(x)}=min⁡μEx∼μ(x){log⁡μ(x)−log⁡exp⁡(f(x))}=min⁡μEx∼μ(x){log⁡μ(x)exp⁡(f(x))}=min⁡μEx∼μ(x){log⁡μ(x)/zˉexp⁡(f(x))/zˉ},zˉ=∑xexp⁡(f(x))=min⁡μEx∼μ(x){log⁡μ(x)1zˉexp⁡(f(x))−log⁡zˉ}=min⁡μEx∼μ(x){log⁡μ(x)1zˉexp⁡(f(x))}⏟DKL(μ(x)∥1zˉexp⁡(f(x)))\begin{aligned} &\max_{\mu} \mathbb{E}_{x \sim \mu(x)} \left\{ f(x) \right\} + H(\mu) \\ &= \max_{\mu} \mathbb{E}_{x \sim \mu(x)} \left\{ f(x) - \log \mu(x) \right\} \\ &= \min_{\mu} \mathbb{E}_{x \sim \mu(x)} \left\{ \log \mu(x) - \log \exp(f(x)) \right\} \\ &= \min_{\mu} \mathbb{E}_{x \sim \mu(x)} \left\{ \log \frac{\mu(x)}{\exp(f(x))} \right\} \\ &= \min_{\mu} \mathbb{E}_{x \sim \mu(x)} \left\{ \log \frac{\mu(x)/\bar{z}}{\exp(f(x))/\bar{z}} \right\}, \quad \bar{z} = \sum_x \exp(f(x)) \\ &= \min_{\mu} \mathbb{E}_{x \sim \mu(x)} \left\{ \log \frac{\mu(x)}{\frac{1}{\bar{z}} \exp(f(x))} - \log \bar{z} \right\} \\ &= \min_{\mu} \underbrace{\mathbb{E}_{x \sim \mu(x)} \left\{ \log \frac{\mu(x)}{\frac{1}{\bar{z}} \exp(f(x))} \right\}}_{D_{KL}\left( \mu(x) \,\|\, \frac{1}{\bar{z}} \exp(f(x)) \right)} \end{aligned} ​μmax​Ex∼μ(x)​{f(x)}+H(μ)=μmax​Ex∼μ(x)​{f(x)−logμ(x)}=μmin​Ex∼μ(x)​{logμ(x)−logexp(f(x))}=μmin​Ex∼μ(x)​{logexp(f(x))μ(x)​}=μmin​Ex∼μ(x)​{logexp(f(x))/zˉμ(x)/zˉ​},zˉ=x∑​exp(f(x))=μmin​Ex∼μ(x)​{logzˉ1​exp(f(x))μ(x)​−logzˉ}=μmin​DKL​(μ(x)∥zˉ1​exp(f(x)))Ex∼μ(x)​{logzˉ1​exp(f(x))μ(x)​}​​​
$$\Rightarrow {\mu }^{\ast }(x)=\frac{1}{\bar{z}}\exp (f(x))=\frac{e^{f(x)}}{{\sum }_x{e}^{f(x)}}$$

RLHF: max⁡πθEx∼DEy∼πθ(y∣x){rϕ(x,y)}−βDKL(πθ(y∣x)∥πref(y∣x))\text{RLHF: } \max_{\pi_\theta} \mathbb{E}_{x \sim D} \mathbb{E}_{y \sim \pi_\theta(y|x)} \left\{ r_\phi(x, y) \right\} - \beta \, D_{KL}\left( \pi_\theta(y|x) \,\|\, \pi_{\text{ref}}(y|x) \right) RLHF: πθ​max​Ex∼D​Ey∼πθ​(y∣x)​{rϕ​(x,y)}−βDKL​(πθ​(y∣x)∥πref​(y∣x))$$={\mathbb{E}}_{x\sim D}\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{y\sim {\pi }_{\theta }(y|x)}\left\{r_{\varphi }(x,y)-\beta \log \frac{{\pi }_{\theta }(y|x)}{{\pi }_{\text{ref}}(y|x)}\right\}$$=Ex∼Dmax⁡πθ{Ey∼πθ(y∣x){rϕ(x,y)+βlog⁡πref(y∣x)}+βH(πθ(y∣x))}= \mathbb{E}_{x \sim D} \max_{\pi_\theta} \left\{ \mathbb{E}_{y \sim \pi_\theta(y|x)} \left\{ r_\phi(x, y) + \beta \log \pi_{\text{ref}}(y|x) \right\} + \beta \, H(\pi_\theta(y|x)) \right\} =Ex∼D​πθ​max​{Ey∼πθ​(y∣x)​{rϕ​(x,y)+βlogπref​(y∣x)}+βH(πθ​(y∣x))}$$⟸\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{y\sim {\pi }_{\theta }(y|x)}\left\{\underset{f(y|x)}{\underbrace{\frac{1}{\beta }{r}_{\varphi }(x,y)+\log {\pi }_{\text{ref}}(y|x)}}\right\}+\underset{H(\mu (y|x))}{\underbrace{H({\pi }_{\theta }(y|x))}}$$

$$\Rightarrow {\pi }_{\theta }^{\ast }(y|x)=\frac{e^{\frac{1}{\beta }{r}_{\varphi }(x,y)}\cdot {\pi }_{\text{ref}}(y|x)}{Z(x)}\Rightarrow g^{\ast }(x,y)=\beta \left(\log \frac{{\pi }_{\theta }^{\ast }(y|x)}{{\pi }_{\text{ref}}(y|x)}\right)+\beta \log Z$$$$\Rightarrow \text{BT: }p(y_m>y_n\mid x)=\sigma \left\{\beta \left(\log \frac{{\pi }_{\theta }(y_m|x)}{{\pi }_{\text{ref}}(y_m|x)}-\log \frac{{\pi }_{\theta }(y_n|x)}{{\pi }_{\text{ref}}(y_n|x)}\right)\right\}$$

---

Token-level PPO:

$$\underset{\theta }{\max }{\mathbb{E}}_{x\sim D}{\mathbb{E}}_{y\sim {\pi }_{\theta }(y|x)}\left\{\sum _{t=1}^{T}r(s_t,y_t)\right\}-\beta D_{KL}\left({\pi }_{\theta }(y|x)\Vert {\pi }_{\text{ref}}(y|x)\right)$$

其中：
$ (s_t, y_t) = (x, y^{<t}) $

$$=\underset{\theta }{\max }{\mathbb{E}}_{x\sim D}{\mathbb{E}}_{y\sim {\pi }_{\theta }(y|x)}\left\{\sum _{t=1}^{T}r(s_t,y_t)-\beta \log \frac{{\pi }_{\theta }(y_t\mid y^{<t},x)}{{\pi }_{\text{ref}}(y_t\mid y^{<t},x)}\right\}$$$$={\mathbb{E}}_{x\sim D}\underset{\theta }{\max }{\mathbb{E}}_{y\sim {\pi }_{\theta }(y|x)}\left\{\sum _{t=1}^T\underset{r^{\prime }(s_t,y_t)}{\underbrace{r(s_t,y_t)+\beta \log {\pi }_{\text{ref}}(y_t\mid s_t)}}-\beta \log {\pi }_{\theta }(y_t\mid s_t)\right\}$$=Ex∼Dmax⁡θ∑t=1TEyt∼πθ(yt∣st){r′(st,yt)−βlog⁡πθ(yt∣st)}= \mathbb{E}_{x \sim D} \max_{\theta} \sum_{t=1}^T \mathbb{E}_{y_t \sim \pi_\theta(y_t|s_t)} \left\{ r'(s_t, y_t) - \beta \log \pi_\theta(y_t \mid s_t) \right\} =Ex∼D​θmax​t=1∑T​Eyt​∼πθ​(yt​∣st​)​{r′(st​,yt​)−βlogπθ​(yt​∣st​)}⟸max⁡θ∑t=1T{Eyt∼πθ(yt∣st){r′(st,yt)}+βH(πθ(yt∣st))}(Max-Entropy Soft Q-learning)\Longleftarrow \max_{\theta} \sum_{t=1}^T \left\{ \mathbb{E}_{y_t \sim \pi_\theta(y_t|s_t)} \left\{ r'(s_t, y_t) \right\} + \beta \, H(\pi_\theta(y_t \mid s_t)) \right\} \quad \text{(Max-Entropy Soft Q-learning)} ⟸θmax​t=1∑T​{Eyt​∼πθ​(yt​∣st​)​{r′(st​,yt​)}+βH(πθ​(yt​∣st​))}(Max-Entropy Soft Q-learning)

在时刻 $t$ 优化 ${\pi }_{\theta }(y_t|s_t)$:

=max⁡θEyt∼πθ(yt∣st)⏟μ{r′(st,yt)⏟f(yt∣st)+∑k=t+1T{Eyk∼πθ∗(yk∣sk){r′(sk,yk)}+βH(πθ∗(yk∣sk))}⏟1βQ∗(st,yt)+βH(πθ(yt∣st))}= \max_{\theta} \underbrace{\mathbb{E}_{y_t \sim \pi_\theta(y_t|s_t)}}_{\mu} \left\{ \underbrace{r'(s_t, y_t)}_{f(y_t|s_t)} + \underbrace{\sum_{k=t+1}^T \left\{ \mathbb{E}_{y_k \sim \pi_\theta^*(y_k|s_k)} \left\{ r'(s_k, y_k) \right\} + \beta H(\pi_\theta^*(y_k|s_k)) \right\}}_{\frac{1}{\beta} Q^*(s_t, y_t)}+ \beta H(\pi_\theta(y_t|s_t)) \right\} =θmax​μEyt​∼πθ​(yt​∣st​)​​​⎩⎨⎧​f(yt​∣st​)r′(st​,yt​)​​+β1​Q∗(st​,yt​)k=t+1∑T​{Eyk​∼πθ∗​(yk​∣sk​)​{r′(sk​,yk​)}+βH(πθ∗​(yk​∣sk​))}​​+βH(πθ​(yt​∣st​))⎭⎬⎫​

⇒πθ∗(yt∣st)=exp⁡{1βr′(st,yt)+1β∑k=t+1T{Eyk∼πθ∗(yk∣sk){r′(sk,yk)}+βH(πθ∗(yk∣sk))}}Z(yt∣st)\Rightarrow \pi_\theta^*(y_t|s_t) = \frac{ \exp \left\{ \frac{1}{\beta} r'(s_t, y_t) + \frac{1}{\beta} \sum_{k=t+1}^T \left\{ \mathbb{E}_{y_k \sim \pi_\theta^*(y_k|s_k)} \left\{ r'(s_k, y_k) \right\} + \beta H(\pi_\theta^*(y_k|s_k)) \right\} \right\} }{Z(y_t|s_t)} ⇒πθ∗​(yt​∣st​)=Z(yt​∣st​)exp{β1​r′(st​,yt​)+β1​∑k=t+1T​{Eyk​∼πθ∗​(yk​∣sk​)​{r′(sk​,yk​)}+βH(πθ∗​(yk​∣sk​))}}​

其中令⇒1βQ∗(st,yt)=1βr′(st,yt)+1β∑k=t+1T{Eyk∼πθ∗(yk∣sk){r′(sk,yk)}+βH(πθ∗(yk∣sk))}\Rightarrow \frac{1}{\beta}Q^*(s_t,y_t) = \frac{1}{\beta} r'(s_t, y_t) + \frac{1}{\beta} \sum_{k=t+1}^T \left\{ \mathbb{E}_{y_k \sim \pi_\theta^*(y_k|s_k)} \left\{ r'(s_k, y_k) \right\} + \beta H(\pi_\theta^*(y_k|s_k)) \right\}⇒β1​Q∗(st​,yt​)=β1​r′(st​,yt​)+β1​∑k=t+1T​{Eyk​∼πθ∗​(yk​∣sk​)​{r′(sk​,yk​)}+βH(πθ∗​(yk​∣sk​))}

$$\begin{array}{rl}Z(y_t|s_t)& =\sum _{y_t}\exp \left\{\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right\}\\ & =\exp \left\{\frac{1}{\beta }{V}^{\ast }(s_t)\right\}\end{array}$$

📌 说明与解释：

• ${\sum }_{y_t}\exp \left(\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right)$：这是对所有可能动作 $y_t$ 的软 Q 值指数和，即 partition function（配分函数），记作 $Z(s_t)$。
• $V^{\ast }(s_t)$：软价值函数（soft value function），定义为：$$V^{\ast }(s_t)=\beta \log \sum _{y_t}\exp \left(\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right)$$因此有：$$\sum _{y_t}\exp \left(\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right)=\exp \left(\frac{1}{\beta }{V}^{\ast }(s_t)\right)$$
• 这个等式是最大熵强化学习中 Soft Bellman 方程 推导的关键一步，用于归一化最优策略：$${\pi }_{\theta }^{\ast }(y_t|s_t)=\frac{\exp \left(\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right)}{{\sum }_{y_t^{\prime }}\exp \left(\frac{1}{\beta }{Q}^{\ast }(s_t,y_t^{\prime })\right)}=\frac{\exp \left(\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right)}{\exp \left(\frac{1}{\beta }{V}^{\ast }(s_t)\right)}$$

其中：

• $Z(y_t|s_t)$ 是归一化常数（partition function）。
• 上述形式体现了 递归最优策略 的结构，类似于 Soft Q-learning 或 最大熵强化学习 中的策略更新。

---

{πθ∗(yt∣st)=exp⁡{1β(Q∗(st,yt)−V∗(st))}Q∗(st,yt)=r′(st,yt)+∑k=t+1T{Eyk∼πθ∗(yk∣sk){r′(sk,yk)}+βH(πθ∗(yk∣sk))}\left\{ \begin{aligned} \pi_\theta^*(y_t|s_t) &= \exp \left\{ \frac{1}{\beta} \left( Q^*(s_t, y_t) - V^*(s_t) \right) \right\} \\ Q^*(s_t, y_t) &= r'(s_t, y_t) + \sum_{k=t+1}^T \left\{ \mathbb{E}_{y_k \sim \pi_\theta^*(y_k|s_k)} \left\{ r'(s_k, y_k) \right\} + \beta H(\pi_\theta^*(y_k|s_k)) \right\} \end{aligned} \right. ⎩⎨⎧​πθ∗​(yt​∣st​)Q∗(st​,yt​)​=exp{β1​(Q∗(st​,yt​)−V∗(st​))}=r′(st​,yt​)+k=t+1∑T​{Eyk​∼πθ∗​(yk​∣sk​)​{r′(sk​,yk​)}+βH(πθ∗​(yk​∣sk​))}​

$$\Rightarrow V^{\ast }(s_t)=\beta \log \left\{\sum _{y_t}\exp \left\{\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right\}\right\}$$

$\Rightarrow Q^{\ast }-V^{\ast }=\beta \log {\pi }_{\theta }^{\ast }$

---

📌 说明与解释：

• ${\pi }_{\theta }^{\ast }(y_t|s_t)$：在状态 $s_t$ 下选择动作 $y_t$ 的最优策略（概率分布），服从 Boltzmann 分布。

  • 这是最大熵强化学习中的经典形式：策略正比于 $\exp (Q-V)$。

• $Q^{\ast }(s_t,y_t)$：软 Q 值函数（soft Q-function），表示在状态 $s_t$ 采取动作 $y_t$ 后的期望累积回报（含熵项）。

  • 包括即时奖励 $r^{\prime }(s_t,y_t)$ 和未来期望回报（包含后续状态下的奖励和熵）。

• $V^{\ast }(s_t)$：软价值函数（soft value function），表示在状态 $s_t$ 下的最大期望累积回报（考虑熵）：

  $$V^{\ast }(s_t)=\underset{\pi }{\max }{\mathbb{E}}_{\pi }\left[\sum _{k=t}^T{r}^{\prime }(s_k,y_k)+\beta H(\pi (y_k|s_k))\right]$$

  • 在最优策略下，有：

    $$V^{\ast }(s_t)=\beta \log \sum _{y_t}\exp \left(\frac{1}{\beta }{Q}^{\ast }(s_t,y_t)\right)$$

    即对所有可能动作取 softmax 的 log-sum-exp 形式。

• $Q^{\ast }-V^{\ast }$：这个差值恰好等于 $\beta \log {\pi }_{\theta }^{\ast }$，即：

  $$Q^{\ast }(s_t,y_t)-V^{\ast }(s_t)=\beta \log {\pi }_{\theta }^{\ast }(y_t|s_t)$$

  • 这是最大熵 RL 中的一个重要恒等式，也称为 Soft Bellman 方程 的核心关系。

---

$Q^{\ast }$化简

=r′(st,yt)+βH(πθ∗(yt+1∣st+1))+Eyt+1∼πθ∗(yt+1∣st+1){r′(st+1,yt+1)}+∑k=t+2T{Eyk∼πθ∗(yk∣sk){r′(sk,yk)}+βH(πθ∗(yk∣sk))}=r′(st,yt)+βEyt+1∼πθ∗(yt+1∣st+1){log⁡exp⁡(1βQ∗(st+1,yt+1))πθ∗(yt+1∣st+1)}⏟KL 项展开=βEyt+1∼πθ∗(yt+1∣st+1){log⁡exp⁡(1βV∗(st+1))}=V∗(st+1)\begin{aligned} &= r'(s_t, y_t) + \beta H(\pi_\theta^*(y_{t+1}|s_{t+1})) + \mathbb{E}_{y_{t+1} \sim \pi_\theta^*(y_{t+1}|s_{t+1})} \left\{ r'(s_{t+1}, y_{t+1}) \right\} \\ &\quad + \sum_{k=t+2}^T \left\{ \mathbb{E}_{y_k \sim \pi_\theta^*(y_k|s_k)} \left\{ r'(s_k, y_k) \right\} + \beta H(\pi_\theta^*(y_k|s_k)) \right\} \\ &= r'(s_t, y_t) + \underbrace{\beta \mathbb{E}_{y_{t+1} \sim \pi_\theta^*(y_{t+1}|s_{t+1})} \left\{ \log \frac{\exp\left( \frac{1}{\beta} Q^*(s_{t+1}, y_{t+1}) \right)}{\pi_\theta^*(y_{t+1}|s_{t+1})} \right\}}_{\text{KL 项展开}} \\ &= \beta \mathbb{E}_{y_{t+1} \sim \pi_\theta^*(y_{t+1}|s_{t+1})} \left\{ \log \exp\left( \frac{1}{\beta} V^*(s_{t+1}) \right) \right\} = V^*(s_{t+1}) \end{aligned}​=r′(st​,yt​)+βH(πθ∗​(yt+1​∣st+1​))+Eyt+1​∼πθ∗​(yt+1​∣st+1​)​{r′(st+1​,yt+1​)}+k=t+2∑T​{Eyk​∼πθ∗​(yk​∣sk​)​{r′(sk​,yk​)}+βH(πθ∗​(yk​∣sk​))}=r′(st​,yt​)+KL 项展开βEyt+1​∼πθ∗​(yt+1​∣st+1​)​⎩⎨⎧​logπθ∗​(yt+1​∣st+1​)exp(β1​Q∗(st+1​,yt+1​))​⎭⎬⎫​​​=βEyt+1​∼πθ∗​(yt+1​∣st+1​)​{logexp(β1​V∗(st+1​))}=V∗(st+1​)​

$$\Rightarrow Q^{\ast }(s_t,y_t)=r^{\prime }(s_t,y_t)+V^{\ast }(s_{t+1})$$$$\text{其中：}{Q}^{\ast }(s_t,y_t)=\{\begin{array}{ll}r(s_t,y_t)+\beta \log {\pi }_{\text{ref}}(y_t|s_t)+V^{\ast }(s_{t+1}),& y_t\ne \text{EOS}\\ r(s_t,y_t)+\beta \log {\pi }_{\text{ref}}(y_t|s_t),& y_t=\text{EOS}\end{array}\text{\hspace{1em}(*)}$$

---

BT-Model:

$P(y_w>y_l\mid x)=\frac{\exp (r(x,y_w))}{\exp (r(x,y_w))+\exp (r(x,y_l))}$
$=\sigma (r(x,y_w)-r(x,y_l))$

Token-level BT

KaTeX parse error: Can't use function '$' in math mode at position 125: …{t+1}) \right) $̲ $ = V^*(s_1) …

---

Token-level DPO:

$$\Rightarrow P(y_w>y_l\mid x)=\sigma \left(\sum _{t=1}^{T_1}r(s_t^w,y_t^w)-\sum _{t=1}^{T_2}r(s_t^l,y_t^l)\right),s_1^w=s_1^l=x$$$$=\sigma \left(\sum _{t=1}^{T_1}\beta \log \frac{{\pi }_{\theta }^{\ast }(y_t^w|s_t^w)}{{\pi }_{\text{ref}}(y_t^w|s_t^w)}-\sum _{t=1}^{T_2}\beta \log \frac{{\pi }_{\theta }^{\ast }(y_t^l|s_t^l)}{{\pi }_{\text{ref}}(y_t^l|s_t^l)}\right)$$

📌 说明与解释：

• Token-level DPO：指在生成序列时，对每个 token（即每一步）使用 Direct Preference Optimization (DPO) 进行建模。
• $P(y_w>y_l\mid x)$：在输入 $x$ 下，样本 $y_w$（winner）被判断优于 $y_l$（loser）的概率。
• $r(s_t,y_t)$：奖励函数，通常由奖励模型给出。在最大熵框架下，可表示为：$$r(s_t,y_t)=\beta \log \frac{{\pi }_{\theta }^{\ast }(y_t|s_t)}{{\pi }_{\text{ref}}(y_t|s_t)}$$
• ${\pi }_{\theta }^{\ast }(y_t|s_t)$：最优策略，服从 Boltzmann 分布，由软 Q-learning 或最大熵 RL 推导得到。
• ${\pi }_{\text{ref}}(y_t|s_t)$：参考策略（如初始语言模型），用于正则化。
• $\sigma (z)$：sigmoid 函数，定义为：$$\sigma (z)=\frac{1}{1+e^{-z}}$$
• $s_1^w=s_1^l=x$：两个序列从相同的输入 $x$ 开始。
• 最终形式：总奖励差可以分解为各 token 上的对数似然比之和，这正是 DPO 中常用的损失函数形式。

---

reward等价性

$$(\ast )\Rightarrow \beta \log \frac{{\pi }_{\theta }^{\ast }(y_t|s_t)}{{\pi }_{\text{ref}}(y_t|s_t)}=\underset{\tilde{r}}{\underbrace{r(y_t,s_t)}}+\underset{\Phi (s_{t+1})-\Phi (s_t)}{\underbrace{V^{\ast }(s_{t+1})-V^{\ast }(s_t)}}\text{(reward shaping)}$$$$\Rightarrow \tilde{r}\text{ and }r\text{ are equivalent.}$$$$\underset{\text{token-level reward}}{\underbrace{\beta \log \frac{{\pi }_{\theta }^{\ast }(y_t|s_t)}{{\pi }_{\text{ref}}(y_t|s_t)}}}\Rightarrow \text{can be generalized to step-wise reward}$$

📌 说明与解释：

• $\beta \log \frac{{\pi }_{\theta }^{\ast }(y_t|s_t)}{{\pi }_{\text{ref}}(y_t|s_t)}$：这是在最大熵强化学习框架下定义的 token-level 奖励（即每一步生成 token 时的奖励）。
• $r(y_t,s_t)$：原始奖励函数（如来自奖励模型），记作 $\tilde{r}$。
• $V^{\ast }(s_t)$：软价值函数（soft value function），表示从状态 $s_t$ 出发的最大期望回报（考虑熵）。
• $\Phi (s_t)=V^{\ast }(s_t)$：势函数（potential function），用于 reward shaping（奖励塑形）。
• 奖励塑形（Reward Shaping）：
  • 根据 Kaelbling’s Theorem，如果奖励被修改为：$$\tilde{r}(s_t,y_t)=r(s_t,y_t)+\Phi (s_{t+1})-\Phi (s_t)$$那么最优策略保持不变。
  • 在此处，$\tilde{r}=\beta \log \frac{{\pi }_{\theta }^{\ast }(y_t|s_t)}{{\pi }_{\text{ref}}(y_t|s_t)}$ 是一个等价于原始奖励 $r$ 的新奖励形式。
• 结论：
  • $\tilde{r}$ 和 $r$ 在优化目标上是 等价的
  • 因此，可以将这个 token-level 奖励推广到 step-wise reward（逐步奖励），用于训练语言模型。

---

Reference

• DPO: https://arxiv.org/abs/2305.18290
• r2Q*: https://arxiv.org/abs/2404.12358
• Step-DPO: https://arxiv.org/abs/2406.18629
• RTO: https://arxiv.org/abs/2404.18922
• TDPO: https://arxiv.org/abs/2404.11999
• SimPO: http://arxiv.org/abs/2405.14734
• ORPO: http://arxiv.org/abs/2403.07691
• DMPO: https://arxiv.org/pdf/2406.14868
• DAPO: https://arxiv.org/pdf/2503.14476
• GSPO: https://arxiv.org/abs/2507.18071
• GMPO: https://arxiv.org/abs/2507.20673
• CISPO: https://arxiv.org/abs/2506.13585
• VAPO: https://arxiv.org/abs/2504.05118
• TRPO: https://arxiv.org/abs/1502.05477