Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss

Factual evidence

What the source reports

Researchers introduced an efficient knowledge distillation method using offline top-K logits and a fused chunked KL loss to train small LLMs.