Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

Research characterizes the sparsity and geometry of parameter updates in on-policy distillation, a post-training recipe for language and vision-language models.