example ordering
**Example ordering** (also called **demonstration ordering**) is the arrangement of in-context learning examples within a prompt to **maximize model performance** — because the order in which demonstrations are presented significantly affects how well the language model extracts and applies the task pattern.
**Why Order Matters**
- LLMs process text sequentially — the position of each example in the context creates different attention patterns and different inductive biases.
- Research shows that **reordering the same examples** can cause accuracy to vary by **10–15%** or more — sometimes the difference between random and state-of-the-art performance.
- The model may give more weight to examples near the end of the prompt (recency bias) or near the beginning (primacy bias), depending on the model and task.
**Ordering Effects**
- **Recency Bias**: Many models weigh later examples more heavily — the last few demonstrations before the test input have outsized influence on the prediction.
- **Primacy Bias**: Some models (especially with shorter contexts) are more influenced by the first few examples.
- **Label Bias**: If the last several examples all have the same label, the model may be biased toward predicting that label for the test input.
- **Pattern Recognition**: Certain orderings make the task pattern more obvious to the model — for example, grouping similar examples together vs. alternating.
**Ordering Strategies**
- **Random Ordering**: Shuffle demonstrations randomly. Simple baseline, but suboptimal.
- **Similarity-Based Ordering**: Place the most similar example to the test input **last** (closest to the test input) — leverages recency bias to maximize the influence of the most relevant demonstration.
- **Reverse Similarity**: Place the most similar example first — works better for models with strong primacy bias.
- **Difficulty Ordering**: Arrange from easy to hard — starts with clear examples to establish the pattern, then shows more nuanced cases.
- **Label Alternation**: Alternate between different labels/categories — prevents label bias from consecutive same-label examples.
- **Curriculum-Style**: Start with diverse, representative examples and end with examples similar to the test input.
**Optimal Ordering Methods**
- **Entropy-Based**: Choose the ordering that minimizes the model's prediction entropy on a validation set — the ordering that makes the model most confident.
- **Beam Search**: Try multiple orderings and evaluate each — select the best. Computationally expensive but effective.
- **Learned Ordering**: Train a model to predict the optimal ordering — using validation performance as the training signal.
**Practical Guidelines**
- **Put the most relevant example last** (works for most models).
- **Alternate labels** to avoid label bias.
- **Use consistent formatting** across all examples — inconsistency confuses the model.
- **Test multiple orderings** on a validation set if performance is critical.
- **Fix the ordering** once determined — don't randomly shuffle at inference time.
Example ordering is an **often overlooked** but highly impactful aspect of few-shot prompting — the same examples in different orders can produce dramatically different results, making ordering optimization a critical step in prompt engineering.