In-Context Learning Prompt Engineering
# In-Context Learning & Prompt Engineering
## Introduction & Motivation
In-context learning (ICL) refers to the remarkable ability of large language models to adapt their behavior to a new task purely from examples or instructions provided within the input prompt, without any gradient-based weight updates. A model shown a handful of example input-output pairs for a novel task, followed by a new input, will often produce a plausible output for that new input by inferring the underlying task pattern, a capability that was first documented prominently in GPT-3's original paper and has since become one of the defining characteristics distinguishing large language models from earlier generations of task-specific supervised learning systems. Prompt engineering is the applied practice of designing the instructions, examples, and formatting given to a language model to reliably elicit the desired behavior, and has emerged as a critical skill for practitioners deploying LLMs, sitting somewhere between software engineering, applied linguistics, and empirical science. The motivation for studying ICL and prompt engineering rigorously, rather than treating prompt design as pure trial-and-error craft, stems from the practical reality that prompt formulation can produce performance differences as large as those from switching to a substantially different model or fine-tuning approach, while requiring far less computational cost and engineering overhead than either alternative. Understanding why and how ICL works also has significant implications for AI safety and interpretability, since much of an LLM's task-relevant "reasoning" happens implicitly within the forward pass conditioned on prompt content, rather than through any explicit, inspectable training update.
## Core Concepts & Theory
In-context learning is typically categorized by the number of examples provided in the prompt: zero-shot prompting provides only a task instruction with no examples, few-shot prompting provides a small number (typically two to a few dozen) of input-output example pairs before the actual query, and one-shot is the special case of exactly one example. The examples provided in a few-shot prompt are often called "demonstrations," and their selection, ordering, and formatting have all been empirically shown to substantially affect downstream task performance, sometimes as much as the choice of examples' content itself. Chain-of-thought (CoT) prompting instructs or demonstrates that the model should produce intermediate reasoning steps before arriving at a final answer, dramatically improving performance on tasks requiring multi-step arithmetic, logical, or commonsense reasoning compared to prompting for a direct answer, and can be elicited either via explicit worked-example demonstrations (few-shot CoT) or via a simple instruction like "let's think step by step" (zero-shot CoT). Instruction tuning and RLHF (covered in depth elsewhere) shape a base language model into one that reliably follows natural-language instructions and produces well-formatted, helpful responses even in zero-shot settings, and the interaction between instruction-tuned model behavior and further in-context prompting is an active area of both practical prompt engineering and theoretical study. The broader theoretical question of what ICL actually is mechanistically remains actively debated: competing accounts frame it as implicit Bayesian inference over latent tasks learned during pretraining, as a form of gradient descent implicitly implemented within the forward pass via attention mechanisms, or as pattern matching and retrieval of pretraining-time task templates.
## Mathematical Formulation
Formally, given a pretrained language model with parameters theta (held fixed, unmodified during ICL), a prompt is constructed as a sequence combining an optional instruction I, a set of k demonstration pairs, and a query input, and the model produces an output by sampling or selecting the highest-probability continuation:
$$ p_ heta(y \mid x_{query}, \{(x_1, y_1), \ldots, (x_k, y_k)\}, I) $$
where the demonstration pairs and instruction are simply concatenated into the model's input context and processed through the same fixed forward pass used for any other text, with no parameter updates occurring at any point. One influential theoretical framing, from Xie et al.'s "implicit Bayesian inference" account, models ICL as approximate posterior inference over a latent task concept z, where pretraining exposes the model to a distribution of documents each generated by some latent underlying "concept," such that observing the demonstration examples allows the model to infer which concept is being invoked at inference time:
$$ p(y \mid x_{query}, ext{demonstrations}) \approx \int p(y \mid x_{query}, z) \, p(z \mid ext{demonstrations}) \, dz $$
Separately, a mechanistic account (von Oswald et al. and related work) shows that a single layer of linear self-attention, given appropriately structured input, can implement an update mathematically equivalent to one step of gradient descent on a linear regression objective defined by the in-context demonstrations, suggesting that multi-layer Transformers may implement something functionally analogous to iterative optimization within their forward pass, though the extent to which this mechanism explains ICL in large, realistic language models trained on natural text remains an open empirical question rather than settled theory.
## Advanced Theory & Extensions
Self-consistency improves upon basic chain-of-thought prompting by sampling multiple independent reasoning paths for the same query at a non-zero decoding temperature, then selecting the final answer via majority vote across the sampled paths, substantially improving accuracy on reasoning-heavy tasks at the cost of multiple forward passes per query. Tree-of-thought and related structured reasoning frameworks generalize chain-of-thought further by allowing the model to explore, evaluate, and backtrack across multiple branching intermediate reasoning steps rather than committing to a single linear reasoning chain, framing complex problem-solving as a search process over a tree of partial solutions. Least-to-most prompting decomposes a complex problem into a sequence of progressively harder subproblems, solving and incorporating the answer to each subproblem before attempting the next, which has shown particular benefit on compositional generalization tasks where the full problem is substantially harder than any of its individual components. Automatic prompt optimization techniques (e.g., APE, OPRO, DSPy's compiled prompt pipelines) treat prompt wording itself as an optimizable artifact, using an LLM (or a dedicated search procedure) to iteratively propose, evaluate, and refine candidate prompts against a validation set, effectively automating what was previously a manual, intuition-driven engineering process. Retrieval-augmented in-context learning dynamically selects which demonstration examples to include in the prompt based on similarity to the current query (rather than using a fixed, static set of demonstrations for every query), often producing meaningfully better performance than a fixed demonstration set, particularly on datasets with high input diversity.
## Computational Considerations
The computational cost of in-context learning scales directly with prompt length, since self-attention's quadratic complexity with respect to sequence length means that longer few-shot prompts with many demonstrations incur substantially higher inference latency and cost than shorter zero-shot prompts, a direct and often underappreciated trade-off against the accuracy gains that additional demonstrations typically provide. Chain-of-thought and self-consistency prompting further multiply inference cost, since generating explicit intermediate reasoning tokens (potentially several times longer than a direct answer) and, for self-consistency, sampling multiple independent completions per query, can increase total token generation and therefore latency and API cost by an order of magnitude or more relative to direct prompting. Prompt caching, supported by several major LLM API providers, allows the computational cost of processing a long, static prefix (e.g., a lengthy system instruction or a fixed set of few-shot demonstrations) to be amortized across many subsequent queries that share that prefix, substantially reducing the marginal cost of few-shot prompting in production settings where the same demonstrations are reused across many queries. Context window limits impose a hard ceiling on how many demonstrations or how much retrieved context can be included in a single prompt, though this ceiling has expanded dramatically across model generations (from a few thousand tokens in early GPT-3 to context windows exceeding one hundred thousand or even millions of tokens in some modern models), shifting but not eliminating the underlying length-versus-cost trade-off. Automatic prompt optimization techniques introduce their own substantial computational overhead during the optimization phase itself, since evaluating each candidate prompt typically requires running it against a validation set, meaning prompt optimization is generally performed offline as a development-time process rather than at inference time in production.
## Practical Implementation Strategies
Demonstration selection should prioritize examples that are representative of the diversity of inputs the model will encounter at inference time, and empirically, demonstrations that are semantically similar to the current query (via retrieval-augmented selection) generally outperform a fixed, randomly chosen demonstration set, particularly for tasks with high input variability. Demonstration ordering matters more than intuition might suggest: models can be sensitive to the order in which few-shot examples are presented, sometimes exhibiting a recency bias toward the most recently shown examples, and practitioners should empirically validate ordering choices rather than assuming order is inconsequential. Explicit output format specification, including clear delimiters, structured formats (e.g., requesting JSON output with a specified schema), and example outputs that exactly match the desired format, substantially reduces downstream parsing errors and formatting inconsistency in production LLM pipelines. Iterative, empirical prompt refinement against a held-out validation set, treating prompt engineering as an empirical optimization process rather than a one-shot design exercise, is essential given how sensitive LLM behavior can be to seemingly minor wording changes; version-controlling prompts and tracking their performance over model updates is increasingly recognized as a software engineering best practice rather than an afterthought. For complex, multi-step tasks, decomposing a single large prompt into a pipeline of smaller, more constrained prompts (each handling a well-defined subtask) generally produces more reliable and debuggable behavior than attempting to elicit the entire complex behavior from a single, densely packed prompt, an approach formalized in frameworks like DSPy and LangChain's prompt-chaining abstractions.
## Benchmark Datasets & Evaluation
BIG-Bench and its focused hard subset BIG-Bench Hard (BBH) aggregate a very large, diverse collection of tasks specifically designed to probe the boundaries of language model capability, widely used to evaluate how few-shot and chain-of-thought prompting affect performance across a broad task distribution rather than any single narrow benchmark. GSM8K, a dataset of grade-school-level math word problems, has become a standard benchmark for evaluating chain-of-thought reasoning specifically, since these problems require correct multi-step arithmetic reasoning that direct-answer prompting handles poorly but chain-of-thought prompting substantially improves. MMLU (Massive Multitask Language Understanding), spanning 57 subjects from elementary mathematics to professional law and medicine, is commonly used to evaluate zero-shot and few-shot performance across a broad academic knowledge distribution, and is frequently reported alongside both prompting styles to characterize how much few-shot examples improve knowledge-intensive task performance. Evaluation of prompt engineering techniques should account for variance across multiple prompt phrasings and demonstration samples rather than reporting a single point estimate, since published results have shown that performance on a fixed benchmark can vary substantially (sometimes by ten or more percentage points) simply from minor, semantically equivalent rewordings of an otherwise identical prompt. Human evaluation remains important for open-ended generation tasks where automatic metrics like exact-match accuracy do not apply, and structured rubric-based human or LLM-as-judge evaluation protocols have become increasingly standard for assessing prompt engineering improvements on tasks like summarization, dialogue, and creative writing where there is no single correct output to match against.
## Key Challenges & Limitations
Prompt sensitivity, referring to the sometimes dramatic performance swings caused by seemingly trivial changes in prompt wording, formatting, or example ordering, undermines the reliability and reproducibility of prompt-engineered systems and makes it genuinely difficult to distinguish a robust improvement from noise without careful, statistically grounded evaluation across multiple prompt variants. In-context learning does not reliably generalize to genuinely novel reasoning patterns not represented in the model's pretraining distribution, and chain-of-thought prompting, while improving performance on many reasoning tasks, does not guarantee that the generated intermediate reasoning steps are faithful explanations of the actual computation the model performed, an important caveat for any application relying on chain-of-thought output for interpretability or auditability purposes. The number of demonstrations does not monotonically improve performance in all cases; beyond a task- and model-dependent point, additional demonstrations can provide diminishing or even negative returns, particularly if later demonstrations introduce noise, redundancy, or subtly conflicting patterns relative to earlier ones. Prompt injection, where adversarial or unintended text embedded in retrieved documents, user input, or other externally sourced content is interpreted by the model as an instruction rather than as data to be processed, represents both a practical reliability challenge and a genuine security vulnerability for LLM applications that incorporate untrusted external content into their prompts. Finally, the boundary between what a model "knows" from pretraining versus what it infers purely from in-context demonstrations is often difficult to disentangle empirically, complicating efforts to cleanly attribute observed task performance to genuine in-context learning versus latent pretrained knowledge that the prompt merely activates or surfaces.
## Hyperparameter Tuning
The number of few-shot demonstrations (commonly referred to as "k" in k-shot prompting) is a primary tunable parameter, with optimal values varying substantially by task and model, typically requiring empirical sweep across a small range (e.g., zero, one, four, and eight-shot) against a validation set rather than assuming more demonstrations are always better. Decoding temperature directly affects the diversity and reliability of generated outputs, with lower temperatures (near zero) producing more deterministic, typically higher-precision outputs suited to tasks with a single correct answer, and higher temperatures producing more diverse outputs useful for creative generation tasks or for generating the multiple diverse reasoning paths required by self-consistency prompting. For self-consistency, the number of sampled reasoning paths trades off compute cost against the reliability of the majority-vote final answer, with returns typically diminishing beyond roughly five to twenty samples depending on task difficulty and the base model's per-sample reasoning accuracy. System prompt and instruction phrasing, while not a numeric hyperparameter in the traditional sense, functions analogously and should be treated as a tunable component subject to the same empirical validation discipline as any other prompt engineering choice, since instruction phrasing has been repeatedly shown to materially affect downstream task performance. For automatic prompt optimization pipelines, the size of the validation set used to score candidate prompts and the number of optimization iterations both trade off search thoroughness against computational cost, with larger validation sets reducing the risk of overfitting the optimized prompt to a small, potentially unrepresentative sample of task instances.
## Real-World Applications & Case Studies
Customer support and conversational AI systems rely heavily on carefully engineered system prompts and few-shot examples to constrain LLM behavior to an appropriate tone, scope, and set of allowed actions, with prompt engineering serving as a primary lever for behavior control before or alongside more expensive fine-tuning interventions. Code generation and coding assistants (GitHub Copilot, Cursor, and similar tools) use in-context learning extensively, providing relevant surrounding code, file context, and sometimes explicit examples within the prompt to guide the model toward generating code consistent with the specific codebase's conventions and APIs, connecting directly to the broader retrieval-augmented generation literature for selecting relevant context. Data extraction and structured output generation pipelines (e.g., extracting structured records from unstructured documents, converting natural language to database queries) rely on careful prompt engineering with explicit format specifications and few-shot examples to achieve the high reliability required for automated, unsupervised production use. Chain-of-thought and self-consistency prompting have been applied in high-stakes reasoning domains including mathematical problem solving, legal document analysis, and clinical decision support tools, where the explicit intermediate reasoning trace, while not a guaranteed faithful explanation, provides at least a partial audit trail that supports human review before final decisions are made. Enterprises building internal LLM-powered tools increasingly maintain structured prompt libraries and evaluation harnesses, treating prompts as versioned, tested software artifacts subject to the same engineering rigor (code review, regression testing, staged rollout) as traditional application code, reflecting the maturation of prompt engineering from ad hoc practice toward standard software engineering discipline.
## Integration with Other Methods
In-context learning and fine-tuning are complementary rather than mutually exclusive: a common production pattern fine-tunes a model on a moderate amount of task-specific data to establish strong baseline behavior, then uses in-context prompting and few-shot examples at inference time to handle edge cases, style adjustments, or rapidly evolving requirements that would be impractical to address through repeated fine-tuning cycles. Retrieval-augmented generation and in-context learning are tightly coupled in practice, since RAG's retrieved context is itself a form of in-context conditioning, and techniques for selecting, ordering, and formatting retrieved passages draw directly on prompt engineering research regarding demonstration selection and context construction. LLM agent architectures build extensively on in-context learning and prompt engineering, using carefully structured prompts (often including few-shot examples of tool use, reasoning traces, and expected output formats) to guide an LLM's tool selection, planning, and multi-step task execution behavior within an agentic loop. Prompt engineering techniques are also foundational to the automatic evaluation and reward-modeling pipelines used in RLHF and constitutional AI approaches, where carefully engineered prompts elicit consistent, well-calibrated preference judgments or self-critiques from either human annotators or LLM-based evaluators, feeding into the broader alignment training pipeline. Distillation of prompt-engineered behavior into a fine-tuned model, sometimes called "prompt distillation," trains a model directly on outputs generated via an elaborate, expensive prompting strategy (e.g., extensive chain-of-thought with self-consistency), aiming to internalize that behavior into a model that can then produce similar-quality outputs with a much simpler, cheaper prompt or even zero-shot at inference time.
## Future Research Directions
Developing a more rigorous mechanistic and theoretical understanding of what in-context learning actually is, reconciling the competing implicit-Bayesian-inference and implicit-gradient-descent accounts (or identifying additional mechanisms) with empirical behavior in large, realistically-trained language models, remains a foundational open research question with implications for both capability prediction and safety. Improving the faithfulness of chain-of-thought reasoning, ensuring that a model's stated intermediate reasoning steps genuinely reflect the computation underlying its final answer rather than being a post-hoc, potentially misleading rationalization, is an active and important direction connecting prompt engineering research to the broader interpretability and AI safety literature. Reducing prompt sensitivity and improving robustness to semantically equivalent prompt rephrasing, through both training-time interventions (e.g., training models to be more robust to prompt variation) and inference-time techniques (e.g., automatic prompt ensembling), would substantially improve the reliability and trustworthiness of prompt-engineered production systems. Automating prompt engineering further, extending current automatic prompt optimization techniques to more complex, multi-step agentic and pipeline settings beyond single-prompt optimization, is a practically important direction given the substantial engineering effort currently required to develop and maintain effective prompts by hand. Finally, defending against prompt injection and related adversarial manipulation of in-context instructions represents an increasingly urgent security research direction as LLM applications are deployed with access to sensitive data, external tools, and consequential real-world actions, where a successful prompt injection attack can have concrete, harmful consequences beyond simply producing an incorrect answer.
## Summary & Key Takeaways
In-context learning allows large language models to adapt to new tasks purely through prompt content, without any weight updates, spanning zero-shot, few-shot, and chain-of-thought prompting styles that elicit progressively more structured reasoning behavior from the model. The mechanistic basis of ICL remains an active research question, with competing theoretical accounts framing it as implicit Bayesian task inference or as an implicit optimization process implemented within the Transformer's forward pass via self-attention. Practical prompt engineering, encompassing demonstration selection and ordering, explicit output formatting, chain-of-thought elicitation, and increasingly automated prompt optimization, has matured from ad hoc trial-and-error into an empirical discipline requiring systematic validation given the substantial and sometimes counterintuitive sensitivity of LLM behavior to prompt wording. Key limitations include prompt sensitivity undermining reproducibility, the non-guaranteed faithfulness of chain-of-thought explanations, and prompt injection as a genuine security concern for production systems incorporating untrusted external content. As LLM applications mature into agentic, tool-using, and retrieval-augmented systems, prompt engineering increasingly functions as a core software engineering discipline, with prompts treated as versioned, tested artifacts integrated into broader machine learning and software development pipelines.
Keywords: in-context learning, ICL, prompt engineering, few-shot learning, zero-shot prompting, chain-of-thought, CoT prompting, self-consistency, tree-of-thought, least-to-most prompting, automatic prompt optimization, demonstration selection, prompt sensitivity, prompt injection, instruction tuning, implicit Bayesian inference, GSM8K, BIG-Bench, LLM agents, retrieval-augmented prompting
---
## Appendix: Practical Labs
### Lab 1: Simulating Few-Shot Demonstration Selection via Similarity
import numpy as np
def embed_text(text, vocab, embedding_dim=16, seed=0):
"""Toy embedding function (bag-of-words hash average), standing in for
a real sentence embedding model for demonstration purposes."""
rng = np.random.RandomState(seed)
word_vectors = {w: rng.randn(embedding_dim) for w in vocab}
words = [w for w in text.lower().split() if w in word_vectors]
if not words:
return np.zeros(embedding_dim)
return np.mean([word_vectors[w] for w in words], axis=0)
def select_few_shot_demonstrations(query, demonstration_pool, vocab, k=3, embedding_dim=16):
"""Selects the k demonstrations most similar to the query via cosine
similarity, mirroring retrieval-augmented in-context learning."""
query_embed = embed_text(query, vocab, embedding_dim)
scored = []
for demo_input, demo_output in demonstration_pool:
demo_embed = embed_text(demo_input, vocab, embedding_dim)
sim = np.dot(query_embed, demo_embed) / (
np.linalg.norm(query_embed) * np.linalg.norm(demo_embed) + 1e-8
)
scored.append((sim, demo_input, demo_output))
scored.sort(key=lambda x: -x[0])
return scored[:k]
def test_demonstration_selection():
demonstration_pool = [
("classify sentiment: this movie was fantastic", "positive"),
("classify sentiment: terrible waste of time", "negative"),
("classify sentiment: the acting was superb and moving", "positive"),
("translate to french: good morning", "bonjour"),
("translate to french: thank you very much", "merci beaucoup"),
("classify sentiment: boring and predictable plot", "negative"),
]
vocab = set(" ".join(d for d, _ in demonstration_pool).lower().split())
query = "classify sentiment: an absolutely brilliant and delightful film"
selected = select_few_shot_demonstrations(query, demonstration_pool, vocab, k=3)
print(f"Query: {query!r}")
print("Selected demonstrations:")
for sim, demo_in, demo_out in selected:
print(f" sim={sim:.3f} {demo_in!r} -> {demo_out!r}")
selected_outputs = [d[2] for d in selected]
sentiment_demos = sum(1 for out in selected_outputs if out in ("positive", "negative"))
assert sentiment_demos >= 2, "Similarity-based selection should favor same-task demonstrations"
print("Few-shot demonstration selection test passed.")
if __name__ == "__main__":
test_demonstration_selection()### Lab 2: Self-Consistency via Majority Voting Over Sampled Reasoning Paths
import random
from collections import Counter
def simulate_llm_reasoning_sample(problem, seed):
"""Simulates a language model sampling one reasoning path and final
answer for a math word problem, with realistic per-sample error rates
that self-consistency is designed to average out."""
rng = random.Random(seed)
true_answer = problem["true_answer"]
error_rate = problem["difficulty_error_rate"]
if rng.random() < error_rate:
# Simulate a plausible but incorrect answer (a common arithmetic slip).
wrong_answer = true_answer + rng.choice([-2, -1, 1, 2, 3])
return wrong_answer
return true_answer
def self_consistency_answer(problem, n_samples=15, seed=0):
"""Samples multiple independent reasoning paths and returns the
majority-vote final answer, along with the vote distribution."""
samples = [simulate_llm_reasoning_sample(problem, seed=seed * 1000 + i) for i in range(n_samples)]
vote_counts = Counter(samples)
majority_answer, majority_count = vote_counts.most_common(1)[0]
return majority_answer, vote_counts, majority_count / n_samples
def single_sample_answer(problem, seed=0):
return simulate_llm_reasoning_sample(problem, seed=seed)
def test_self_consistency_improves_accuracy():
# A moderately difficult problem where any single reasoning path has a
# meaningful chance of arithmetic error, but the correct answer is still
# the single most common outcome across many samples.
problem = {"true_answer": 42, "difficulty_error_rate": 0.4}
n_trials = 200
single_sample_correct = 0
self_consistency_correct = 0
for trial in range(n_trials):
single_answer = single_sample_answer(problem, seed=trial)
if single_answer == problem["true_answer"]:
single_sample_correct += 1
sc_answer, votes, confidence = self_consistency_answer(problem, n_samples=15, seed=trial)
if sc_answer == problem["true_answer"]:
self_consistency_correct += 1
single_accuracy = single_sample_correct / n_trials
sc_accuracy = self_consistency_correct / n_trials
print(f"Single-sample accuracy: {single_accuracy:.3f}")
print(f"Self-consistency accuracy: {sc_accuracy:.3f}")
assert sc_accuracy >= single_accuracy, "Self-consistency should match or improve upon single-sample accuracy"
print("Self-consistency majority voting test passed.")
if __name__ == "__main__":
test_self_consistency_improves_accuracy()### Lab 3: Prompt Template Construction and Format Validation
import json
import re
class PromptTemplate:
"""A structured prompt template supporting instruction, few-shot
demonstrations, and a query, with explicit output format specification."""
def __init__(self, instruction, output_schema, demonstrations=None):
self.instruction = instruction
self.output_schema = output_schema # dict describing expected keys/types
self.demonstrations = demonstrations or []
def render(self, query):
parts = [self.instruction, ""]
if self.demonstrations:
parts.append("Examples:")
for demo_input, demo_output in self.demonstrations:
parts.append(f"Input: {demo_input}")
parts.append(f"Output: {json.dumps(demo_output)}")
parts.append("")
parts.append(f"Input: {query}")
parts.append("Output:")
return "
".join(parts)
def validate_output(self, raw_output):
"""Checks that a model's raw output string is valid JSON matching
the expected schema keys, a common production reliability check."""
try:
parsed = json.loads(raw_output)
except json.JSONDecodeError:
return False, "Output is not valid JSON"
if not isinstance(parsed, dict):
return False, "Output JSON must be an object"
missing_keys = set(self.output_schema.keys()) - set(parsed.keys())
if missing_keys:
return False, f"Missing required keys: {missing_keys}"
for key, expected_type in self.output_schema.items():
if not isinstance(parsed[key], expected_type):
return False, f"Key '{key}' should be type {expected_type.__name__}"
return True, parsed
def test_prompt_template_and_validation():
template = PromptTemplate(
instruction="Extract the entity and sentiment from the input as JSON.",
output_schema={"entity": str, "sentiment": str, "confidence": float},
demonstrations=[
("I loved the new iPhone camera", {"entity": "iPhone", "sentiment": "positive", "confidence": 0.95}),
("The battery life on my laptop is disappointing", {"entity": "laptop battery", "sentiment": "negative", "confidence": 0.88}),
],
)
prompt = template.render("The customer service at that restaurant was outstanding")
print("Rendered prompt:")
print(prompt)
print()
assert "Examples:" in prompt, "Prompt should include the few-shot demonstrations section"
assert prompt.count("Input:") == 3, "Prompt should show 2 demo inputs plus 1 query input"
# Simulate a well-formed model response.
good_response = json.dumps({"entity": "restaurant service", "sentiment": "positive", "confidence": 0.92})
is_valid, result = template.validate_output(good_response)
print(f"Valid response check: {is_valid}, parsed={result}")
assert is_valid, "Well-formed JSON matching the schema should validate successfully"
# Simulate a malformed model response (missing required key).
bad_response = json.dumps({"entity": "restaurant service"})
is_valid_bad, error = template.validate_output(bad_response)
print(f"Invalid response check: {is_valid_bad}, error={error}")
assert not is_valid_bad, "Response missing required schema keys should fail validation"
print("Prompt template and output validation test passed.")
if __name__ == "__main__":
test_prompt_template_and_validation()### Lab 4: Simple Automatic Prompt Optimization Loop
import random
def evaluate_prompt_variant(prompt_variant, validation_set, seed):
"""Simulates evaluating a candidate prompt variant against a validation
set, where each prompt variant has an intrinsic 'quality' that
determines its expected accuracy, with realistic per-example noise."""
rng = random.Random(seed)
correct = 0
for example in validation_set:
success_prob = prompt_variant["quality"]
if rng.random() < success_prob:
correct += 1
return correct / len(validation_set)
def generate_candidate_variants(base_variant, n_candidates, rng):
"""Simulates an LLM proposing prompt rewrites: each candidate's quality
is a small random perturbation of the base variant's quality, modeling
the idea that most rewrites are similar in effectiveness but some are
meaningfully better or worse."""
candidates = []
for _ in range(n_candidates):
delta = rng.uniform(-0.15, 0.15)
new_quality = min(0.98, max(0.05, base_variant["quality"] + delta))
candidates.append({"name": f"variant_of_{base_variant['name']}", "quality": new_quality})
return candidates
def automatic_prompt_optimization(initial_prompt, validation_set, n_iterations=5, n_candidates_per_iter=4, seed=0):
rng = random.Random(seed)
best_prompt = initial_prompt
best_score = evaluate_prompt_variant(best_prompt, validation_set, seed=seed)
history = [(best_prompt["name"], best_score)]
for iteration in range(n_iterations):
candidates = generate_candidate_variants(best_prompt, n_candidates_per_iter, rng)
for candidate in candidates:
score = evaluate_prompt_variant(candidate, validation_set, seed=seed + iteration)
if score > best_score:
best_prompt, best_score = candidate, score
history.append((best_prompt["name"], best_score))
return best_prompt, best_score, history
def test_automatic_prompt_optimization():
validation_set = list(range(50)) # 50 synthetic validation examples
initial_prompt = {"name": "v0_basic_instruction", "quality": 0.55}
best_prompt, best_score, history = automatic_prompt_optimization(
initial_prompt, validation_set, n_iterations=8, n_candidates_per_iter=5, seed=7
)
print("Optimization history (best score per iteration):")
for name, score in history:
print(f" {name}: {score:.3f}")
print(f"
Final best prompt quality score: {best_score:.3f}")
assert best_score >= history[0][1], "Optimization loop should never regress below the initial score"
assert best_score > history[0][1] * 1.05 or best_score > 0.9, "Optimization should meaningfully improve or reach high performance"
print("Automatic prompt optimization test passed.")
if __name__ == "__main__":
test_automatic_prompt_optimization()