hellaswag

HellaSwag is a benchmark for evaluating commonsense natural language inference — specifically, the ability to predict the most plausible continuation of an event description. The name stands for "Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations." Introduced by Zellers et al. in 2019, HellaSwag presents a context (a partial description of a situation or activity) followed by four possible continuations, and the model must select the one that most plausibly follows. The key innovation is the use of Adversarial Filtering (AF) to generate challenging incorrect options: candidate wrong endings are generated by a language model and then filtered to select those that are difficult for state-of-the-art models but easy for humans — eliminating trivially wrong options that contain grammatical errors or obvious semantic inconsistencies. This adversarial construction makes HellaSwag significantly harder than previous commonsense benchmarks. Contexts are drawn from two sources: ActivityNet Captions (describing activities in videos like cooking, sports, and household tasks) and WikiHow articles (describing step-by-step procedures). The correct continuation comes from the actual next sentence in the source, while distractors are model-generated and adversarially filtered. At release, BERT achieved only ~47.3% accuracy (near random chance at 25% for 4-way classification), while humans scored ~95.6%, revealing a massive gap in commonsense understanding. This gap has narrowed significantly — GPT-4 achieves ~95.3%, approaching human performance. HellaSwag remains widely used because it tests grounded commonsense reasoning about physical activities and everyday situations, capabilities that require understanding causality, temporal sequences, physical constraints, and social norms rather than just linguistic patterns. It is a standard component of evaluation suites like the Open LLM Leaderboard.