DRKL: Diversity-Aware Reverse KL Divergence Fixes Overcon…
DRKL: Diversity-Aware Reverse KL Divergence Fixes Overconfidence in LLM Distillation
A new paper proposes Diversity-aware Reverse KL (DRKL), a fix for the overconfidence and reduced diversity caused by the popular Reverse KL divergence in LLM distillation. DRKL consistently outperforms existing objectives across multiple benchmarks.


























