LLM Evals

Evaluations (“evals”) are the methods used to measure an AI model’s behaviour, capabilities, or alignment. Evals provide an evidentiary basis for decision-makers. In the case of the EU AI Act Code of Practice, external evaluations for systemic risk are mandated.

AISI prioritises the following areas in their evaluations:

Misuse: measure how AI can help actors to cause harm (e.g. chemical and biological capabilities, and cyber offence capabilities).
Societal impacts: measure the direct impact of frontier AI systems on both individuals and society (including psychological risks like the extent to which people’s beliefs can be manipulated or whether AI advice is safe).
Autonomy: measure the degree to which AI systems can autonomously execute sequences of actions in ways that might limit human oversight.
Safeguards: measure the efficacy of safety components of frontier AI systems against threats that may circumvent those safeguards.

Evaluations are typically done via:

benchmarking: measuring agnostically (models the average case scenario)
red-teaming & elicitation: actively looking for a property in an interactive manner (models the worst case scenario). This technique currently always succeeds at getting models to comply with harmful requests so it remains a secondary tool.

Often they boil down to creating a large test set of questions, feeding them to a LLM, recording the received output and using a LLM from a different model family to evaluate the answers according to a custom rubric or scoring criteria. Often this involves the grading LLM examining the chain-of-thought generated by the model under evaluation.

Good evaluations are characterised by clarity in what they are measuring (ask why should I care about what they’re measuring), with tests that actually measure this (unbiased prompts, covering the range of relevant and realistic scenarios).

Results from evaluations can be muddied when the dataset is flawed, or examples of the tests are incorporated into the training data for subsequent models (“test set contamination” - one way of flagging this is having both a public test set and a private test set, and monitoring whether any models deviate substantially between those two test sets).

The landscape for evals is burgeoning, best practices are still being developed. Complexity arises when models are run as agents using a scaffolding (or harness) or other software frameworks around the models.

Examples of evaluations:

Victoria Krakovna, et al. “Realistic honeypot evaluations for scheming propensity.” arXiv, 2026.
METR, “MirrorCode: Evidence that AI can already do some weeks-long coding tasks”
AISI, “Our evaluation of Claude Mythos Preview’s cyber capabilities”