prompt ensemble

**Prompt ensemble** is the technique of **combining predictions from multiple different prompts** for the same task — leveraging the diversity of prompt formulations to produce more robust and accurate outputs than any single prompt alone. **Why Prompt Ensembles?** - LLM outputs are **sensitive to prompt phrasing** — different wordings of the same question can produce different answers. - No single prompt is reliably optimal across all inputs — some prompts work better for certain examples. - **Ensembling** reduces the variance of predictions by averaging out the idiosyncrasies of individual prompts. **Prompt Ensemble Methods** - **Majority Voting**: Run the same input through $k$ different prompts. Each prompt produces a prediction. The **most common answer** wins. - Example for sentiment classification: 3 prompts say "positive," 2 say "negative" → final answer is "positive." - Simple and effective for classification tasks. - **Weighted Voting**: Assign weights to prompts based on their validation accuracy. Better prompts contribute more to the final decision. - $\hat{y} = \arg\max_c \sum_i w_i \cdot \mathbb{1}[p_i = c]$ - **Probability Averaging**: Average the probability distributions over classes from each prompt. Choose the class with highest average probability. - $P(c|x) = \frac{1}{K} \sum_{i=1}^{K} P_i(c|x)$ - Smoother than voting — uses confidence information. - **Verbalizer Ensemble**: For classification, use multiple verbalizer mappings (e.g., "positive"/"negative" vs. "good"/"bad" vs. "favorable"/"unfavorable") and combine predictions. **Ensemble Prompt Diversity** - **Template Diversity**: Different instruction phrasings — "Classify this text," "Is this positive or negative?," "What sentiment does this express?" - **Format Diversity**: Different output formats — "Answer with positive/negative," "Rate 1-5," "Explain and then classify." - **Perspective Diversity**: Different reasoning angles — "As a movie critic...," "Consider the emotional tone...," "Focus on the factual claims..." - **Few-Shot Diversity**: Different sets of demonstration examples — each prompt uses a different subset of few-shot examples. **Benefits** - **Accuracy**: Ensembles typically improve accuracy by **2–5%** over the best single prompt. - **Robustness**: Less sensitive to individual prompt failures or adversarial inputs. - **Reliability**: More consistent performance across diverse inputs. - **Calibration**: Ensemble probabilities tend to be better calibrated than single-prompt probabilities. **Costs** - **Compute**: Each prompt requires a separate model inference — $k$ prompts means $k×$ the compute cost. - **Latency**: Sequential execution multiplies latency. Parallel execution helps but requires more resources. - **Diminishing Returns**: Beyond 5–10 diverse prompts, additional prompts provide minimal improvement. Prompt ensembles are one of the most **reliable techniques for improving LLM accuracy** — they exploit the observation that different prompts capture different aspects of the task, and combining them produces a more complete and robust understanding.