LexGLUE is the legal language understanding benchmark suite — aggregating six established legal NLP datasets into a unified evaluation framework modeled after GLUE and SuperGLUE, enabling systematic comparison of general and domain-adapted language models on the classification, multi-label prediction, and NLI tasks that constitute the core of automated legal document processing.

What Is LexGLUE?

• Origin: Chalkidis et al. (2021,2022) from the University of Copenhagen.
• Tasks: 6 legal NLP datasets spanning multiple jurisdictions and document types.
• Evaluation: Macro-F1 for classification tasks; accuracy for NLI tasks; combined LexGLUE score as geometric mean.
• Purpose: Provide a single, reproducible leaderboard for comparing legal language models — replacing fragmented per-paper evaluation with a unified standard.

The 6 LexGLUE Tasks

Task 1 — ECtHR (Article Prediction):

• Predict which European Convention on Human Rights articles are violated in a court judgment.
• Input: ECHR case description. Output: Multi-label violation set (e.g., Article 3, Article 6, Article 8).
• Scale: 11,000 cases; 10 frequently violated articles.

Task 2 — SCOTUS (Issue Area Classification):

• Classify US Supreme Court decisions into 14 legal issue areas (Criminal Procedure, Civil Rights, First Amendment, etc.).
• Scale: 9,300 decisions from 1946-2020.

Task 3 — EUR-Lex (Subject Matter Categorization):

• Multi-label classification of EU legislation into EUROVOC subject categories.
• Scale: 65,000 EU documents; 100 fine-grained labels.

Task 4 — LEDGAR (Contract Provision Classification):

• Classify contract provision paragraphs into 100 legal provision types (indemnification, termination, assignment, etc.).
• Scale: 100,000 contract provisions; source: SEC EDGAR filings.

Task 5 — UNFAIR-ToS (Unfair Clause Detection):

• Identify potentially unfair or unlawful clauses in Terms of Service agreements.
• Multi-label: 8 unfairness categories (unilateral change, arbitration clause, content removal, etc.).
• Scale: 9,400 ToS paragraphs.

Task 6 — CaseHOLD (Holding Identification):

• Multiple-choice selection of correct legal holding from citing context (53,137 examples).

Performance Results

Key Findings

• Domain Adaptation Value: Legal-BERT and LexLM consistently outperform general models of equal scale on legal-specific tasks — validating specialized pretraining.
• GPT-4 Zero-Shot Pattern: GPT-4 zero-shot exceeds fine-tuned BERT on CaseHOLD (reasoning task) but falls below on EUR-Lex (taxonomy familiarity task) — illustrating different competence profiles.
• Multi-label Difficulty: EUR-Lex and UNFAIR-ToS (multi-label tasks) remain hardest — models struggle with rare label combinations.

Why LexGLUE Matters

• Legal AI Standardization: LexGLUE enabled the legal NLP community to stop measuring progress on isolated datasets and start tracking comprehensive capability improvements.
• Product Evaluation Framework: Legal tech companies (Kira Systems, Luminance, Relativity) can use LexGLUE to evaluate whether new models improve on the commercial legal tasks their products perform.
• Multi-Jurisdiction Coverage: Combining ECHR, SCOTUS, and EU tasks in one benchmark surfaces models that generalize across legal systems vs. those that specialize narrowly.
• Regulatory Compliance AI: EUR-Lex categorization and UNFAIR-ToS detection are directly deployable in regulatory compliance scanning tools.

LexGLUE is the GLUE benchmark for legal AI — providing the unified six-task evaluation suite that enables fair, reproducible comparison of general and domain-specific legal language models, establishing the empirical standard for measuring progress in automated legal document understanding.