Prompt Versioning and Management

Why Version Prompts? Prompts are code. Track changes, roll back issues, and collaborate effectively.

Git-Based Versioning

<svg viewBox="0 0 275 264" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto" role="img"><rect x="0" y="0" width="275" height="264" rx="12" fill="#0d1117"/><g font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,&quot;Liberation Mono&quot;,monospace" font-size="14"><text xml:space="preserve" x="20" y="31.7"><tspan fill="#c9d1d9">prompts/</tspan></text><text xml:space="preserve" x="20" y="50.7"><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> customer_support/</tspan></text><text xml:space="preserve" x="20" y="69.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> main_prompt.md</tspan></text><text xml:space="preserve" x="20" y="88.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> escalation_prompt.md</tspan></text><text xml:space="preserve" x="20" y="107.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> metadata.yaml</tspan></text><text xml:space="preserve" x="20" y="126.7"><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> summarization/</tspan></text><text xml:space="preserve" x="20" y="145.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> v1/</tspan></text><text xml:space="preserve" x="20" y="164.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> prompt.md</tspan></text><text xml:space="preserve" x="20" y="183.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> v2/</tspan></text><text xml:space="preserve" x="20" y="202.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">       </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> prompt.md</tspan></text><text xml:space="preserve" x="20" y="221.7"><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> tests/</tspan></text><text xml:space="preserve" x="20" y="240.7"><tspan fill="#c9d1d9">    </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> prompt_tests.yaml</tspan></text></g></svg>

Metadata Format

# metadata.yaml
name: customer_support_main
version: 2.3.1
author: team-ai
created: 2024-01-15
updated: 2024-03-20

model_requirements:
  min_context: 4096
  recommended_model: gpt-4o

evaluation:
  last_eval_date: 2024-03-18
  accuracy: 0.94
  latency_p50: 1.2s

Prompt Template Management

from jinja2 import Environment, FileSystemLoader

env = Environment(loader=FileSystemLoader("prompts"))

def load_prompt(name, version=None, **kwargs):
    if version:
        path = f"{name}/v{version}/prompt.md"
    else:
        path = f"{name}/prompt.md"

    template = env.get_template(path)
    return template.render(**kwargs)

# Usage
prompt = load_prompt("summarization", version=2, max_length=100)

LangSmith/LangChain Hub

from langchain import hub

# Push prompt
hub.push("my-org/customer-support-v2", prompt_template)

# Pull prompt
prompt = hub.pull("my-org/customer-support-v2")

A/B Testing Prompts

import random

class PromptExperiment:
    def __init__(self, prompts, weights=None):
        self.prompts = prompts
        self.weights = weights or [1/len(prompts)] * len(prompts)

    def get_prompt(self, user_id):
        # Deterministic assignment based on user
        bucket = hash(user_id) % 100
        cumulative = 0
        for prompt, weight in zip(self.prompts, self.weights):
            cumulative += weight * 100
            if bucket < cumulative:
                return prompt
        return self.prompts[-1]

Prompt Registry

class PromptRegistry:
    def __init__(self, storage):
        self.storage = storage

    def register(self, name, prompt, version, metadata):
        key = f"{name}:{version}"
        self.storage.set(key, {
            "prompt": prompt,
            "metadata": metadata,
            "created_at": datetime.now()
        })

    def get(self, name, version="latest"):
        if version == "latest":
            version = self.get_latest_version(name)
        return self.storage.get(f"{name}:{version}")

Best Practices

• Use semantic versioning (major.minor.patch)
• Include evaluation metrics with versions
• Document changes in changelog
• Test prompts before deploying
• Keep production prompts immutable
• A/B test significant changes