ConsensusBench

Methodology

Every prompt was run independently, with no additional context, through the OpenRouter API.

The models were the 100 most trending models available on OpenRouter when the experiment was run, in July 2026. Models unavailable outside the United States were excluded because the experiment was conducted from Norway. Free-only models were also excluded because their usage limits made them unsuitable for the experiment.

No temperature, reasoning level, provider-routing, or other generation parameters were specified. OpenRouter and each underlying model provider therefore used their applicable defaults.

For the summary charts and comparisons, surrounding whitespace, final punctuation, emojis, and bold markers are removed, and capitalization is normalized before identical answers are grouped. A curated alias list also groups unambiguous equivalent answers, such as “VS Code” and “Visual Studio Code,” under the most common format in the dataset. The downloadable dataset preserves every model's original output.

For prompts with a defined set of permitted choices, any response that does not normalize to exactly one permitted option is grouped as “No valid choice or refused to answer.” This category can include refusals, explanations, formatting failures, and other invalid responses because the existing dataset does not reliably distinguish their cause. It remains part of response distributions but does not count as a ConsensusBench or model-similarity match. Open-ended prompts are not classified this way.

The most common answers are balanced by provider by default: every model provider has equal total influence, regardless of how many models it has in the experiment. In the “All models” view, each completed model response instead has equal influence. Tied answers are shown jointly.

Download the data

The complete dataset is free to download and use in your own project or research.

View and download the dataset on GitHub