Bringing a common language to AI evaluation

Benchmarking may be the most important tool the AI industry has to measure how well models can write code, recognize speech, or carry out instructions. But it’s not always clear how much benchmark scores can be trusted.

Unlike the SAT or GRE, exams for AI models are typically unproctored and feature a static set of questions year after year, increasing the odds that models have already seen the material they’re being tested on. Then there’s the benchmarking process itself. A mishmash of reporting formats and runtime settings can make comparing models, replicating results, and reusing expensive evaluation data nearly impossible.

All these factors can make choosing a model based on its benchmark score a bit dicey. “The evaluation decisions people make are almost always based on gut feeling,” said Leshem Choshen, an IBM researcher focused on AI evaluation. “You would never evaluate models based on feelings, but we evaluate evaluations this way because the data is so scattershot.”

The goal of EveryEvalEver is to raise the scientific bar. Led by researchers at IBM, Hugging Face, Technical University of Munich, and elsewhere, the community project introduces a common language for reporting benchmark scores, and a crowdsourced database for sharing results with the world. Hosted on Hugging Face, the database already lists more than 22,000 model results across 2,200 benchmarks (many of them variations on popular ones), which were translated from 31 evaluation formats.

In a field as competitive as AI, model reputations can rise and fall on the strength of their test scores. Behind the neat rows of reported numbers, however, is a highly fragmented process. More than two dozen ‘harnesses’ for evaluating AI models exist. A harness is the code that runs models or agents on a benchmark and grades their performance. Not only are scores tallied differently from harness to harness, but the results themselves can turn up anywhere from leaderboards to academic papers to blog posts.

It's not uncommon for a nominally identical evaluation to generate scores that vary by as much as 20 percentage points. In one case, Hugging Face researchers noticed conflicting results for the same LlaMA model on the Massive Multitask Language Understanding (MMLU) benchmark. After a lengthy investigation, they concluded that different evaluation harnesses had been used.

Multiply these inconsistencies across thousands of benchmarks, and tens of thousands of models, and you can start to understand why the field has grown more critical of AI benchmarking practices. In a recent paper titled “Benchmarking is Broken,” a group of researchers wrote, “Why do we hold machines to lower evaluative standards than we do for humans in high-stakes environments?”

For all their flaws, benchmarks are still the best tools industry has to track progress and define what progress even looks like. “Whether you’re in industry or academia, evaluation is everything,” said Jan Batzner, a PhD student at TU Munich who co-leads the project. “Our question is, how can we document evaluations in a more rigorous way?”

A community-driven benchmarking standard

The quest for a common benchmarking language began at NeurIPS 2024, with a workshop called Evaluating Evaluations. Led by Choshen and others, the workshop established an open, iterative design process modeled after the one behind Croissant, the open-source standard for dataset documentation.

The team eventually settled on a JSON evaluation format that, like Croissant, consolidates metadata into a single record. They gathered feedback from dozens of experts in benchmarking, evaluation frameworks, and leaderboards, and further refined the format with discussion over GitHub and Slack.

The result is a modular format with four blocks of metadata explaining who produced a result and how (source metadata); which model was tested and whether it was accessed through a hosted API or a local inference engine (model information); what generation settings were used, (configuration); and how the scores should be interpreted (results and metric semantics). Instance-level data like a model's prompt-by-prompt outputs can also be added.

Complete records aren’t required. EveryEvalEver allows fields to be left blank to keep valuable data inside the system. To make it easy to contribute, they created a tool that automatically translates results into JSON from three widely used evaluation harnesses— HELM, lm-eval-harness, and Inspect AI. They created a similar tool to translate results from popular leaderboards, including AlpacaEval, RewardBench, and SWE-bench. A validation tool checks each submission before posting it to the EveryEvalEver repository on Hugging Face.

“Once you have everything in one place, the discrepancies will come to light,” said Choshen, one of the project's senior leaders.

To further improve access to benchmark documentation, the team also created EvaluationCards, built atop the BenchmarkCards project that IBM and University of Notre Dame launched last year. Results for nearly 5,800 models across 633 benchmarks have been captured so far.

A common language won’t solve all of benchmarking’s problems, but it can move the industry toward more transparent reporting. “Once the community sets a new standard, reporting practices shift,” said Batzner. “More metadata can be beneficial to everyone, especially small model providers.”

What aggregated scores can reveal

With about 230,000 evaluation results so far, EveryEvalEver doesn’t quite live up to its name yet. But it’s still comprehensive enough to expose broader trends.

One point that became clear for the team after analyzing the data was just how thin documentation is currently. Model providers identified which platform was used to serve their model just 2% of the time and disclosed important generation settings like ‘temperature,’ which is the randomness of an LLM’s outputs, less than 25% of the time. The data also showed a bias toward U.S. commercial models, with five companies accounting for 23 of the 24 most frequently evaluated AI systems.

Additional trends jumped out. AI models are often compressed or quantized to reduce their computational costs, with something called a "perplexity" score reporting how shrinkage impacts performance. Across EveryEvalEver, the researchers discovered perplexity scores reported three different ways — by token, word, and byte — complicating model-to-model comparisons.

As LLM usage transitions from chatbots to agents, the cost of running (and evaluating) them is growing rapidly. Researchers see EveryEvalEver as a useful way to gauge cost-accuracy tradeoffs among agents and the software “scaffolds” used to run them. When the team ran models on different scaffolds, they found that the choice of scaffold had a significant impact on cost without necessarily improving its performance.

To look at reproducibility, they re-ran models on HELM leaderboard tasks to see if their results would match the official records. They converted both sets of scores to EveryEvalEver’s JSON format to ensure a fair comparison. Overall, the results aligned nicely, and where they diverged, the team pinpointed procedural defects that led to the mismatch.

Eventually, the EveryEvalEver dataset could provide insights into how benchmarking could be improved generally. If you can identify the most predictive questions, for example, can the test be compressed into something smaller that costs less to run? Where are the field’s blind spots? “It can help you get into the nooks and crannies of benchmarking and see what's missing,” said Choshen.

One of those nooks and crannies is the bias noted above toward U.S. commercial models. In a separate project called the Next Billion AI Index, IBM researchers are helping to shift the conversation away from what models can do toward how easy they are to adopt, especially in regulated or resource-constrained environments. “We’d like to broaden the evaluation lens to include affordability, deployability, openness, and validation by stakeholders in user communities,” said Ambrish Rawat, a senior research scientist at IBM who is co-leading the project.

An evolving community resource

Like Wikipedia, EveryEvalEver has controls in place to prevent crowdsourced contributions from slipping into chaos. Each submitted record receives a unique identifier and can’t be deleted or discretely overwritten. When model evaluations conflict (as they did with LlaMA), both remain visible in the metadata.

Coverage so far is strongest for text-based, single-model evaluations. Multi-modal evaluations, human-preference judgments (as in ChatbotArena’s pairwise competitions), and multi-agent settings are currently only partially supported. How useful it proves to be will depend on its broader adoption. Model providers and leaderboard operators can still withhold important metadata, like generation settings for proprietary systems, but at least now it can be explicitly flagged as missing.

Evaluations are only getting more expensive as AI systems evolve. EveryEvalEver could help to lower costs by making prior results easier to reuse. The team estimates it would hypothetically cost up to $370,000 to reproduce the aggregated results in their database which don’t include agentic evaluations, reasoning models, and repeated runs, which are far more computationally intensive.

“If we can create a system that is reusable and trustworthy, people don’t need to run these expensive evaluations again and again,” said Avijit Ghosh, a technical AI policy researcher at Hugging Face who oversees the project. "EveryEvalEver makes these results accessible to everyone."