mt-bench
**MT-Bench** (Multi-Turn Bench) is an evaluation benchmark designed to assess LLMs on **multi-turn conversational ability** — testing not just single-response quality but how well models handle follow-up questions, maintain context, and engage in sustained dialogue.
**Benchmark Design**
- **80 High-Quality Questions**: Covering 8 categories with 10 questions each — **writing**, **roleplay**, **reasoning**, **math**, **coding**, **extraction**, **STEM**, and **humanities**.
- **Two-Turn Format**: Each question has a **first turn** (initial question) and a **second turn** (follow-up question that builds on the first). This tests context retention and instruction following.
- **Automated Judging**: A strong LLM (GPT-4) scores each response on a **1–10 scale**, providing reasoning for its judgment.
**Example**
- **Turn 1**: "Compose a short poem about the beauty of mathematics."
- **Turn 2**: "Now rewrite the poem so that every line starts with a letter that spells out the word 'MATH'." (Tests instruction following + context awareness)
**Scoring**
- **Per-Category Scores**: Models receive average scores for each of the 8 categories, revealing strengths and weaknesses.
- **Overall Score**: Average across all categories. Frontier models typically score **8.5–9.5** out of 10.
- **Turn-by-Turn**: Separate scores for first and second turns, showing how well models handle follow-ups.
**Significance**
- **Multi-Turn Gap**: MT-Bench revealed that many models that perform well on single-turn evaluations **struggle with follow-ups** — failing to maintain context or follow complex instructions.
- **Category Insights**: Models often excel at writing and humanities but struggle more with math, coding, and precise reasoning.
- **Complementary to Arena**: MT-Bench provides controlled, reproducible evaluation while the Chatbot Arena provides open-ended human preference signals.
**Developed By**: The **LMSYS team** at UC Berkeley, alongside the Chatbot Arena. MT-Bench is part of their comprehensive evaluation framework for instruction-tuned LLMs.