Stopping criteria offer a reliable way to decide when to stop machine-learning-assisted prioritised screening for a systematic map, review, or for document review in e-discovery.
Such a method is vital in order to use machine learning safely (by estimating the risk of missing documents). Our method describes statistical confidence levels that any given level of recall has not been achieved. As such, it offers a robust data-driven method to decide when to stop, and a clear and transparent way of communicating the risks of stopping at any given point. Check the resources below for more details on how this works or check out our papers for a full description of the criteria, including theory and evaluations.
Additional resources
Buscar, short for Biased Urn based Stopping Criteria for technology Assisted Review, is an R and Python package, and a web app. Please note, that Buscar is not validated yet, other than for bias=1 (see Callaghan, 2020, also known as CMH). The documentation for BuscarPy offers additional helpful examples to better understand the approach.
- BuscarR R package
- BuscarPy Python package
- Buscar Web application (Source code)
- Evaluation framework of stopping methods
- Stopping method game (work in progress)
- Reference implementation for CMH stopping method
References
Tim Repke, Francesca Tinsdeall, Lena Schmidt, et al. 2025. "Don’t stop me now, ’cause I’m having a good me screening: Evaluation of stopping methods for safe use of priority screening in systemaic maps and reviews". Preprint https://doi.org/10.22541/au.175074486.66241013/v1
Callaghan, Max, and Finn Müller-Hansen. 2020. “Statistical Stopping Criteria for Automated Screening in Systematic Reviews.” Systematic Reviews. https://doi.org/10.21203/rs.2.18218/v2.
