After reading Build Your Own Benchmark I turned it into a two-week class project.
My students wrote 30 real questions they'd ask a lab partner: why the titration went pink too early, what a control variable is, why their graph looks like a staircase. They wrote a model answer for each, plus a rubric: correct, clear, admits uncertainty, doesn't just give the answer.
Then we ran three assistants blind, with answers labelled A, B and C, and the class scored them.
Findings:
- All three were mostly correct on facts.
- Huge differences on "doesn't just give the answer". One assistant solved the whole problem every time.
- The students' favourite was not the one with the highest accuracy. It was the one that asked them what they'd already tried.
The best part was the students arguing about the rubric. They now read AI answers like markers, not fans.
