Pierre Achkar | Harry Scells

A Pipeline to Bootstrap the Evaluation of Retrieval-Augmented Generation for the Automation of Systematic Reviews in Computer Science

Wed, 01 Jul 2026 00:00:00 +0000

Automating systematic reviews (SRs), i.e., evidence-driven analyses under explicit protocol constraints, is a natural target for retrieval-augmented generation and deep research agents, yet existing benchmarks evaluate isolated subtasks or assume fixed evidence inputs. We introduce RAG4SR-CS-200, a benchmark of 200 computer science systematic reviews designed for protocol-driven systematic review automation. Each instance comprises review objectives, research questions, eligibility criteria, cleaned full-text review structure, references, and extracted tables. These elements support evaluation across key tasks in systematic review creation such as literature retrieval, eligibility screening, citation-grounded review generation, and structured table generation, in both stage-wise and end-to-end settings. RAG4SR-CS-200 provides a foundation for developing more reliable and diagnosable deep research agents for scientific evidence synthesis. Code and data are publicly available.

UP2DATE: Shared Task on Systematic Review Updates

Thu, 26 Mar 2026 00:00:00 +0000

In all areas of science, the volume of literature has become overwhelming, and staying current with the bleeding edge of research is increasingly unmanageable. This problem is particularly acute in medicine, where studies are published at a rate of approximately two per minute, and up-to-date information is crucial for decisionmaking processes. Maintaining the evidence base by updating systematic reviews is a costly and time-consuming endeavour, yet there is comparatively little research that tackles this important problem via computational means. We propose a shared task that addresses this challenge in (biomedical) scholarly information access. Specifically, we propose three tasks involved in updating systematic reviews: determining when a review should be updated, retrieving literature for the review update, and classifying whether each piece of retrieved literature should be included in the review. These are difficult problems that have received limited research attention, and improved solutions could have considerable impact.