myWiki/raw/papers/rubrics-survey-2026.md

title, created, updated, type, source_url, github, authors, affiliation, date, tags

title	created	updated	type	source_url	github	authors	affiliation	date	tags
The Rules of the Game: A Survey of Rubrics for Large Language Models	2026-06-27	2026-06-27	raw-paper	https://8421bcd.github.io/_pages/Rubrics_Survey.pdf	https://github.com/8421BCD/Rubrics_Survey	Wenhan Liu Jiajie Jin Zhaoheng Huang Tongyu Wen Guanting Dong Ziliang Zhao Yutao Zhu Zhicheng Dou Ji-Rong Wen	Renmin University of China	2026-05-22	rubric evaluation reward-modeling survey

The Rules of the Game: A Survey of Rubrics for Large Language Models

Abstract

As LLMs evolve from general text generators into systems for reasoning, decision-making, tool use, and long-horizon problem solving, the question of how to specify, optimize, and evaluate model responses has become critical. Simple correctness signals and unconstrained LLM-based judgments are often insufficient for open-ended and high-stakes tasks where response quality depends on multiple criteria (factuality, completeness, safety, reasoning soundness, evidence grounding, practical utility).

Rubrics have emerged as a mechanism for making evaluation standards explicit and operational. This survey:

1. Defines rubrics and distinguishes them from reward models, verifiable rewards, and LLM-as-a-judge
2. Organizes rubric construction into four paradigms: direct generation, contrastive generation, iterative refinement, and online/co-evolving generation
3. Examines rubric-driven training for both policy models (standard RL, advanced reward design, policy guidance) and reward models (interpretability, reward signals, data construction)
4. Summarizes rubric-driven evaluation across general tasks (reasoning, deep research, agent capability, alignment) and domain-specific tasks (intermediate trajectories, final outputs)
5. Discusses open challenges: rubric reward hacking, generalization of rubric-based reward models, bias in rubric-based evaluation, personalized rubrics, and rubric safety

Key Contributions

• First comprehensive survey on rubrics for LLMs
• Systematic taxonomy of rubric construction methods (4 categories)
• Comprehensive review of rubrics for model training and evaluation
• Discussion of open challenges and future directions (personalized rubrics, rubric safety, RIPD)

Formal Definition

A rubric is a set of rubric items: R = {(dⱼ, wⱼ)}ᵏⱼ₌₁

• dⱼ: natural-language description of the j-th rubric item
• wⱼ: its weight
• Score: cⱼ(x, y) ∈ [0, 1]
• Aggregated rubric score: S_R(x, y) = Σwⱼcⱼ(x,y) / Σwⱼ