scibert
**SciBERT** is a **BERT language model pre-trained from scratch on 1.14 million scientific papers from Semantic Scholar, with a custom scientific vocabulary that efficiently tokenizes domain-specific terminology** — outperforming general-purpose BERT on scientific NLP tasks including paper classification, citation intent prediction, Named Entity Recognition (NER) for chemicals and proteins, and relation extraction from biomedical and computer science literature.
**What Is SciBERT?**
- **Definition**: A domain-adapted BERT model trained on full-text scientific papers rather than Wikipedia and BookCorpus — using a custom WordPiece vocabulary optimized for scientific terminology, enabling efficient tokenization of terms like "acetylcholine," "backpropagation," and "endoplasmic reticulum" that general BERT breaks into meaningless subword fragments.
- **Custom Vocabulary**: Standard BERT's vocabulary is built from Wikipedia — it tokenizes "acetylcholine" as ["ace", "##ty", "##lch", "##oline"] (4 tokens). SciBERT's vocabulary treats it as a single token, preserving semantic meaning and reducing sequence length.
- **Training Data**: 1.14 million papers from Semantic Scholar — 18% computer science, 82% biomedical — totaling 3.17 billion tokens of full-text scientific content.
- **Architecture**: Same BERT-base architecture (110M parameters, 12 layers, 768 hidden) — the improvement comes entirely from domain-specific pretraining and vocabulary, demonstrating that data quality and domain match matter more than architectural changes.
**Performance on Scientific NLP Tasks**
| Task | SciBERT | BERT-base | Improvement |
|------|---------|-----------|------------|
| Paper Classification (SciCite) | 85.5% | 83.1% | +2.4% |
| NER - Chemicals (BC5CDR) | 90.1% F1 | 87.2% F1 | +2.9% |
| NER - Proteins (JNLPBA) | 77.3% F1 | 74.8% F1 | +2.5% |
| Relation Extraction (ChemProt) | 76.8% F1 | 73.4% F1 | +3.4% |
| Citation Intent (SciCite) | 84.0% | 82.1% | +1.9% |
**Why SciBERT Matters**
- **Vocabulary Efficiency**: Scientific terms that consume 3-5 tokens in general BERT use 1-2 tokens in SciBERT — effectively doubling the useful context length for scientific documents within BERT's 512-token limit.
- **Semantic Understanding**: SciBERT understands that "model" in a ML paper means "neural network" while in a biology paper it means "organism representation" — contextual disambiguation trained on domain text.
- **Transfer Learning**: SciBERT serves as a superior starting point for fine-tuning on any scientific NLP task — chemical NER, drug interaction extraction, paper recommendation, and research topic classification.
- **Reproducibility**: Fully open-source with pretrained weights on Hugging Face Hub (`allenai/scibert_scivocab_uncased`) — directly usable with the Transformers library.
**SciBERT vs. Domain BERT Models**
| Model | Domain | Training Data | Vocabulary | Key Strength |
|-------|--------|------|------|------|
| **SciBERT** | Science (CS + Bio) | 1.14M papers | Custom scientific | Broad scientific coverage |
| BioBERT | Biomedical only | PubMed abstracts | BERT vocab | Biomedical NER |
| ClinicalBERT | Clinical notes | MIMIC-III | BERT vocab | EHR understanding |
| MatSciBERT | Materials science | Materials papers | Custom | Materials NLP |
**SciBERT is the foundational domain-adapted language model for scientific text processing** — proving that pre-training on domain-specific data with a custom vocabulary produces substantial improvements on scientific NLP tasks, and establishing the methodology that spawned dozens of subsequent domain-adapted BERT variants across medicine, law, finance, and materials science.