RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics
Zhengyang Qi, Charles Dickens, Derek Pham, Amanda Dsouza, Armin Parchami, Frederic Sala, Paroma Varma · 2026 · arXiv
TL;DR
Grounded-theory taxonomy of 8 rubric-evaluation failure modes across 3 categories (reliability, content validity, consequential validity), developed from rubrics in instruction following, code generation, creative writing, and deep research; paired with automated diagnostic metrics.
Abstract
Rubric-based evaluation is widely used in LLM benchmarks and training pipelines for open-ended, less verifiable tasks. We introduce RIFT: RubrIc Failure mode Taxonomy, a taxonomy for systematically characterizing failure modes in rubric composition and design. RIFT consists of eight failure modes organized into three high-level categories: Reliability Failures, Content Validity Failures, and Consequential Validity Failures. RIFT is developed using grounded theory by iteratively annotating rubrics drawn from five diverse data sources spanning general instruction following, code generation, creative writing, and expert-level deep research, until no new failure modes are identified. We evaluate the consistency of the taxonomy by measuring agreement among independent human annotators (87% pairwise agreement, 0.64 average Cohen's kappa) and propose automated rubric quality metrics aligned with human annotations, achieving up to 0.925 F1.
Editorial notes
Complements SAICA-KG's FailureMode axis from the evaluation/rubric angle rather than the agent-trace angle. Informs how FailureMode tags interact with evaluator quality (meta-evaluation). This record describes arXiv v2 (2026-04-20): eight failure modes in three categories, first author Zhengyang Qi. From v3 (2026-10-01) the same arXiv id carries a rewritten paper, "The Hitchhikers Guide to Rubric Quality Understanding and Enrichment", first author Ankit Aich, ten authors; its abstract describes RIFT with nine failure modes under reliability and content validity.