Distilled 'student' models inherit their teacher's confident mistakes, new preprint finds
A team at the fictional Ashby College trained small models on a larger model's answers and found they copied its wrong-but-sure replies almost exactly. The authors suggest filtering the teacher's low-confidence answers before training.
via Preprint Patrol