Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs

Sun, 19 Jul 2026 00:00:00 +0000

Awarded Best Student Paper!

The unjudged document problem, where systems that did not contribute to the original judgement pool may retrieve documents without a relevance judgement, is a key obstacle to the reuseability of test collections in information retrieval. This is commonly dealt with by treating unjudged documents as non-relevant, and recently, the use of large language models (LLMs) as a relevance judge (LLM-as-a-judge) emerged. However, this has been criticized, among other things, as circular, since the same LLM can be used as the ranker and the judge. We propose to train topic-specific relevance classifiers instead: By finetuning monoT5 with independent LoRA weight adaptation on the judgments of a single assessor for a single topic’s pool, we align it to that assessor’s notion of relevance for that topic. The system rankings obtained through our classifier’s relevance judgments achieve a Spearmans’ ρ correlation of >0.94 with ground truth system rankings. As little as 128 initial human judgments per topic suffice to improve the comparability of models, while achieving more reliability than existing LLM-as-a-judge approaches, and maintaining human judgments as the gold standard for retrieval evaluation. Code, models, and data are available.

Andrew Yates | Harry Scells

Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs