$\beta$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

Research identifies a structural issue in on-policy self-distillation (OPSD) for LLMs, proposing $\beta$-OPSD as a more stable policy optimization approach.