Posted: September 2026
Contact: Jonas Weisser, Tim Repke
"in the past", we annotated a lot of data so we can train and evaluate classifiers (e.g. basic ones from sklearn or fine-tuning transformers). Nowadays, frameworks like Jev from TypeSafe AI promise an alternative approach: Just provide a structured schema with a description of how outputs look like (which could be a bool yes/no classification). At the same time, we are working on DEET, a toolkit to provide a convenient tool to evaluate prompts against existing annotations.
In either case, the promise is that we can use significantly less annotated data (just some for good enough evaluation) than we would need for traditional supervised classifiers. It would be nice to know exactly how this stacks up and where the resource trade-offs are. Resources being human labour or compute/token costs.
We have many real-world annotated datasets ready-to-go for evaluation. We also have most of a framework ready that can tests all sorts of sklearn and transformer models, does hyperparameter tuning, and runs all that in nested cross-validation loops storing the results. What we are looking for is to put all this together and to stack results from "novel alternatives" over those results. We'd also need a way to simulate the traditional models with varying numbers of training data to determine how much is needed for a "good" model. Ideally, we store all results in a standardised form and have ways to integrate new results nicely, so that testing and comparing a new "frontier" revolution is as easy as possible.
This project can start from a solid foundation, requires some upfront engineering, and can explore many downstream research questions if desired or focus on refining the framework for easy extension and exploration/presentation of results.
