glue (general language understanding evaluation)

GLUE (General Language Understanding Evaluation) is a benchmark suite of nine natural language understanding tasks designed to evaluate and compare the general linguistic capabilities of NLP models, serving as a standardized test bed that drove significant progress in language model development from 2018 to 2020. The nine GLUE tasks span diverse linguistic phenomena: CoLA (Corpus of Linguistic Acceptability — judging grammaticality of sentences), SST-2 (Stanford Sentiment Treebank — binary sentiment classification of movie reviews), MRPC (Microsoft Research Paraphrase Corpus — determining if two sentences are paraphrases), STS-B (Semantic Textual Similarity Benchmark — rating sentence similarity on a 1-5 continuous scale), QQP (Quora Question Pairs — identifying duplicate questions), MNLI (Multi-Genre Natural Language Inference — determining entailment, contradiction, or neutral between premise and hypothesis across genres), QNLI (Question Natural Language Inference — derived from SQuAD), RTE (Recognizing Textual Entailment — binary entailment classification), and WNLI (Winograd Natural Language Inference — pronoun resolution requiring commonsense reasoning). The GLUE score is the average performance across all tasks, providing a single number for model comparison. GLUE was introduced by Wang et al. in 2018 and quickly became the standard benchmark for evaluating pre-trained models — BERT, RoBERTa, ALBERT, DeBERTa, and others were directly compared on GLUE. However, rapid progress meant that models surpassed human baseline performance on all GLUE tasks by 2019, leading to the creation of SuperGLUE with more challenging tasks. Despite being largely "solved," GLUE remains historically important as it established the evaluation paradigm for language understanding: a multi-task benchmark measuring diverse capabilities through a unified score, inspiring similar benchmarks for other domains and languages.