Pre-training | ChipFoundryServices

Home› Knowledge Base› Pre-training

Pre-training is the initial training phase where models learn general patterns from large unlabeled datasets — creating foundation models that capture broad language or vision understanding, which can then be fine-tuned for specific downstream tasks with much less data and compute.

What Is Pre-Training?

Definition: Training on large, general datasets before specialization.
Objective: Learn universal representations (language patterns, visual features).
Scale: Billions of tokens/images, weeks-months of compute.
Output: Foundation model or base model.

Why Pre-Training Works

Transfer Learning: General knowledge transfers to specific tasks.
Data Efficiency: Fine-tuning needs much less task-specific data.
Emergence: Capabilities arise from scale that can't be directly trained.
Cost Amortization: One expensive pre-train, many cheap fine-tunes.
Better Representations: Self-supervised learning captures structure.

Pre-Training Objectives

Language Models:

Objective             | Description
----------------------|----------------------------------
Causal LM (GPT)       | Predict next token: P(x_t | x_{<t})
Masked LM (BERT)      | Predict masked tokens: P(x_mask | context)
Span corruption (T5)  | Predict multiple masked spans
Prefix LM             | Bidirectional attention on prefix

Vision Models:

Objective             | Description
----------------------|----------------------------------
Contrastive (CLIP)    | Match images to text descriptions
MAE                   | Reconstruct masked image patches
SimCLR                | Match augmented views of same image
DINO                  | Self-distillation without labels

Pre-Training at Scale

Modern LLM Training:

Component         | Typical Scale
------------------|---------------------------
Data              | 1-15 trillion tokens
Parameters        | 7B-405B parameters
Compute           | 10^23-10^25 FLOPs
Hardware          | 1000s-10000s of GPUs
Time              | Weeks to months
Cost              | $1M-$100M+

Dataset Composition:

Source              | Percentage
--------------------|------------
Web crawl (CommonCrawl)| 60-80%
Code (GitHub)       | 5-15%
Books               | 5-10%
Wikipedia           | 2-5%
Scientific papers   | 2-5%
Curated/synthetic   | Variable

Pre-Training Pipeline

Stages:

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Pre-Training — Building Foundation Models</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">self-supervised learning on trillions of tokens to learn general representations</text>

  <!-- Main: pretraining pipeline -->
  <rect x="30" y="65" width="700" height="175" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="380" y="84" fill="#e6edf3" font-size="11" font-weight="600" text-anchor="middle">Pre-Training Pipeline</text>

  <!-- Data -->
  <rect x="50" y="100" width="110" height="115" rx="6" fill="#0b1220" stroke="#60a5fa" stroke-width="1.2"/>
  <text x="105" y="118" fill="#93c5fd" font-size="9" font-weight="600" text-anchor="middle">Web Corpus</text>
  <text x="105" y="136" fill="#8b98a5" font-size="7.5" text-anchor="middle">CommonCrawl</text>
  <text x="105" y="150" fill="#8b98a5" font-size="7.5" text-anchor="middle">Wikipedia</text>
  <text x="105" y="164" fill="#8b98a5" font-size="7.5" text-anchor="middle">Books / Code</text>
  <text x="105" y="178" fill="#8b98a5" font-size="7.5" text-anchor="middle">arXiv / Stack</text>
  <text x="105" y="198" fill="#6b7684" font-size="7.5" text-anchor="middle">10–15 T tokens</text>
  <text x="105" y="210" fill="#6b7684" font-size="7" text-anchor="middle">(deduplicated, filtered)</text>

  <path d="M162,157 L190,157" fill="none" stroke="#3a4453" stroke-width="1.2"/>
  <polygon points="188,154 194,157 188,160" fill="#3a4453"/>

  <!-- Objective -->
  <rect x="196" y="100" width="130" height="115" rx="6" fill="#0b1220" stroke="#f59e0b" stroke-width="1.2"/>
  <text x="261" y="118" fill="#fbbf24" font-size="9" font-weight="600" text-anchor="middle">Self-Supervised Obj.</text>
  <text x="261" y="140" fill="#f59e0b" font-size="8" text-anchor="middle">Next-Token Prediction</text>
  <text x="261" y="158" fill="#8b98a5" font-size="7.5" text-anchor="middle">P(x_t | x_1...x_{t-1})</text>
  <text x="261" y="178" fill="#6b7684" font-size="7.5" text-anchor="middle">cross-entropy loss</text>
  <text x="261" y="194" fill="#6b7684" font-size="7.5" text-anchor="middle">causal LM (decoder-only)</text>
  <text x="261" y="210" fill="#6b7684" font-size="7" text-anchor="middle">or MLM (encoder: BERT)</text>

  <path d="M328,157 L356,157" fill="none" stroke="#3a4453" stroke-width="1.2"/>
  <polygon points="354,154 360,157 354,160" fill="#3a4453"/>

  <!-- Training infrastructure -->
  <rect x="362" y="100" width="140" height="115" rx="6" fill="#0b1220" stroke="#a78bfa" stroke-width="1.2"/>
  <text x="432" y="118" fill="#c4b5fd" font-size="9" font-weight="600" text-anchor="middle">Distributed Training</text>
  <text x="432" y="138" fill="#8b98a5" font-size="7.5" text-anchor="middle">1000–16000 GPUs</text>
  <text x="432" y="153" fill="#8b98a5" font-size="7.5" text-anchor="middle">FSDP + TP + PP</text>
  <text x="432" y="168" fill="#8b98a5" font-size="7.5" text-anchor="middle">bf16 mixed precision</text>
  <text x="432" y="183" fill="#8b98a5" font-size="7.5" text-anchor="middle">weeks to months</text>
  <text x="432" y="200" fill="#f87171" font-size="7.5" text-anchor="middle">$10M–$100M+ compute</text>
  <text x="432" y="212" fill="#6b7684" font-size="7" text-anchor="middle">H100/B200 clusters</text>

  <path d="M504,157 L532,157" fill="none" stroke="#3a4453" stroke-width="1.2"/>
  <polygon points="530,154 536,157 530,160" fill="#3a4453"/>

  <!-- Output: Foundation model -->
  <rect x="538" y="100" width="170" height="115" rx="6" fill="#0b1220" stroke="#34d399" stroke-width="1.2"/>
  <text x="623" y="118" fill="#6ee7b7" font-size="9" font-weight="600" text-anchor="middle">Foundation Model</text>
  <text x="623" y="138" fill="#8b98a5" font-size="7.5" text-anchor="middle">general-purpose LLM</text>
  <text x="623" y="153" fill="#8b98a5" font-size="7.5" text-anchor="middle">emergent capabilities</text>
  <text x="623" y="168" fill="#8b98a5" font-size="7.5" text-anchor="middle">in-context learning</text>
  <text x="623" y="183" fill="#8b98a5" font-size="7.5" text-anchor="middle">few-shot reasoning</text>
  <text x="623" y="200" fill="#34d399" font-size="7.5" text-anchor="middle">→ fine-tune for any task</text>
  <text x="623" y="212" fill="#6b7684" font-size="7" text-anchor="middle">GPT-4, Llama 3, Claude</text>

  <!-- Scaling laws chart -->
  <rect x="30" y="252" width="345" height="175" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="202" y="272" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">Scaling Laws (Chinchilla)</text>

  <!-- Axes -->
  <line x1="75" y1="400" x2="340" y2="400" stroke="#3a4453" stroke-width="0.8"/>
  <line x1="75" y1="400" x2="75" y2="285" stroke="#3a4453" stroke-width="0.8"/>
  <text x="207" y="416" fill="#6b7684" font-size="7.5" text-anchor="middle">compute budget (FLOPs)</text>
  <text x="60" y="340" fill="#6b7684" font-size="7" text-anchor="middle" transform="rotate(-90,60,340)">loss</text>

  <!-- Loss curve (decreasing with more compute) -->
  <path d="M90,295 C130,310 170,340 220,360 C270,378 310,388 335,392" fill="none" stroke="#60a5fa" stroke-width="2"/>

  <!-- Annotation -->
  <circle cx="130" cy="310" r="4" fill="#f59e0b"/>
  <text x="140" y="305" fill="#fbbf24" font-size="7">7B</text>
  <circle cx="200" cy="350" r="4" fill="#a78bfa"/>
  <text x="210" y="345" fill="#c4b5fd" font-size="7">70B</text>
  <circle cx="290" cy="385" r="4" fill="#34d399"/>
  <text x="300" y="380" fill="#6ee7b7" font-size="7">405B</text>

  <text x="202" y="425" fill="#8b98a5" font-size="7.5" text-anchor="middle">L(N,D) ∝ (N/N₀)^α + (D/D₀)^β — loss is predictable from scale</text>

  <!-- Right: Key numbers -->
  <rect x="390" y="252" width="340" height="175" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="560" y="272" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">Pre-Training by the Numbers</text>

  <text x="410" y="296" fill="#60a5fa" font-size="9" font-weight="600">Llama 3 405B:</text>
  <text x="410" y="312" fill="#8b98a5" font-size="8">15.6T tokens · 16K H100s · 54 days</text>
  <text x="410" y="326" fill="#6b7684" font-size="7.5">3.8×10²⁵ FLOPs · context 8K→128K</text>

  <text x="410" y="350" fill="#34d399" font-size="9" font-weight="600">GPT-4 (estimated):</text>
  <text x="410" y="366" fill="#8b98a5" font-size="8">~13T tokens · ~25K A100s · ~90 days</text>
  <text x="410" y="380" fill="#6b7684" font-size="7.5">MoE ~1.8T params (220B active)</text>

  <text x="410" y="404" fill="#f59e0b" font-size="9" font-weight="600">Chinchilla rule:</text>
  <text x="410" y="420" fill="#8b98a5" font-size="8">optimal: tokens ≈ 20× params</text>

  <text x="380" y="452" fill="#6b7684" font-size="11" text-anchor="middle">Pre-training is the most expensive and irreversible step — it determines what the model can ever learn to do.</text>
</svg>

Code Example:

from transformers import (
    AutoModelForCausalLM, 
    AutoTokenizer,
    Trainer, 
    TrainingArguments,
    DataCollatorForLanguageModeling,
)

# Load base architecture
model = AutoModelForCausalLM.from_config(config)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Pre-training data
def tokenize(examples):
    return tokenizer(examples["text"], truncation=True, max_length=2048)

tokenized_dataset = dataset.map(tokenize, batched=True)

# Training arguments for pre-training
training_args = TrainingArguments(
    output_dir="./pretrained-model",
    per_device_train_batch_size=8,
    gradient_accumulation_steps=16,
    learning_rate=3e-4,
    warmup_steps=2000,
    max_steps=500000,
    bf16=True,
    save_steps=5000,
)

# Data collator for causal LM
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    data_collator=data_collator,
)

trainer.train()

Pre-Training vs. Fine-Tuning

Aspect          | Pre-Training      | Fine-Tuning
----------------|-------------------|------------------
Data            | Billions tokens   | Thousands-millions
Compute         | $1M+              | $10-$10K
Time            | Weeks-months      | Hours-days
Objective       | General LM        | Task-specific
Who does it     | AI labs           | Everyone
Learning rate   | Higher (1e-4)     | Lower (1e-5)

Pre-training is the foundation of modern AI — by investing massive resources once to create powerful general-purpose models, the community enables efficient specialization through fine-tuning, democratizing access to capabilities that would be impossible to train from scratch.

pretrainingfoundationbase modelcorpusscalingtransfer

Explore 500+ Semiconductor & AI Topics

From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.

🔍 Search Topics 💬 Ask CFSGPT 📚 Browse All