55 lines
2.6 KiB
Markdown
55 lines
2.6 KiB
Markdown
---
|
|
title: "The Rules of the Game: A Survey of Rubrics for Large Language Models"
|
|
created: 2026-06-27
|
|
updated: 2026-06-27
|
|
type: raw-paper
|
|
source_url: "https://8421bcd.github.io/_pages/Rubrics_Survey.pdf"
|
|
github: "https://github.com/8421BCD/Rubrics_Survey"
|
|
authors:
|
|
- "Wenhan Liu"
|
|
- "Jiajie Jin"
|
|
- "Zhaoheng Huang"
|
|
- "Tongyu Wen"
|
|
- "Guanting Dong"
|
|
- "Ziliang Zhao"
|
|
- "Yutao Zhu"
|
|
- "Zhicheng Dou"
|
|
- "Ji-Rong Wen"
|
|
affiliation: "Renmin University of China"
|
|
date: "2026-05-22"
|
|
tags:
|
|
- rubric
|
|
- evaluation
|
|
- reward-modeling
|
|
- survey
|
|
---
|
|
|
|
# The Rules of the Game: A Survey of Rubrics for Large Language Models
|
|
|
|
## Abstract
|
|
|
|
As LLMs evolve from general text generators into systems for reasoning, decision-making, tool use, and long-horizon problem solving, the question of how to specify, optimize, and evaluate model responses has become critical. Simple correctness signals and unconstrained LLM-based judgments are often insufficient for open-ended and high-stakes tasks where response quality depends on multiple criteria (factuality, completeness, safety, reasoning soundness, evidence grounding, practical utility).
|
|
|
|
Rubrics have emerged as a mechanism for making evaluation standards explicit and operational. This survey:
|
|
|
|
1. **Defines rubrics** and distinguishes them from reward models, verifiable rewards, and LLM-as-a-judge
|
|
2. **Organizes rubric construction** into four paradigms: direct generation, contrastive generation, iterative refinement, and online/co-evolving generation
|
|
3. **Examines rubric-driven training** for both policy models (standard RL, advanced reward design, policy guidance) and reward models (interpretability, reward signals, data construction)
|
|
4. **Summarizes rubric-driven evaluation** across general tasks (reasoning, deep research, agent capability, alignment) and domain-specific tasks (intermediate trajectories, final outputs)
|
|
5. **Discusses open challenges**: rubric reward hacking, generalization of rubric-based reward models, bias in rubric-based evaluation, personalized rubrics, and rubric safety
|
|
|
|
## Key Contributions
|
|
|
|
- First comprehensive survey on rubrics for LLMs
|
|
- Systematic taxonomy of rubric construction methods (4 categories)
|
|
- Comprehensive review of rubrics for model training and evaluation
|
|
- Discussion of open challenges and future directions (personalized rubrics, rubric safety, RIPD)
|
|
|
|
## Formal Definition
|
|
|
|
A rubric is a set of rubric items: R = {(dⱼ, wⱼ)}ᵏⱼ₌₁
|
|
- dⱼ: natural-language description of the j-th rubric item
|
|
- wⱼ: its weight
|
|
- Score: cⱼ(x, y) ∈ [0, 1]
|
|
- Aggregated rubric score: S_R(x, y) = Σwⱼcⱼ(x,y) / Σwⱼ
|