---
title: "The Rules of the Game: A Survey of Rubrics for Large Language Models"
created: 2026-06-27
updated: 2026-06-27
type: raw-paper
source_url: "https://8421bcd.github.io/_pages/Rubrics_Survey.pdf"
github: "https://github.com/8421BCD/Rubrics_Survey"
authors:
  - "Wenhan Liu"
  - "Jiajie Jin"
  - "Zhaoheng Huang"
  - "Tongyu Wen"
  - "Guanting Dong"
  - "Ziliang Zhao"
  - "Yutao Zhu"
  - "Zhicheng Dou"
  - "Ji-Rong Wen"
affiliation: "Renmin University of China"
date: "2026-05-22"
tags:
  - rubric
  - evaluation
  - reward-modeling
  - survey
---

# The Rules of the Game: A Survey of Rubrics for Large Language Models

## Abstract

As LLMs evolve from general text generators into systems for reasoning, decision-making, tool use, and long-horizon problem solving, the question of how to specify, optimize, and evaluate model responses has become critical. Simple correctness signals and unconstrained LLM-based judgments are often insufficient for open-ended and high-stakes tasks where response quality depends on multiple criteria (factuality, completeness, safety, reasoning soundness, evidence grounding, practical utility).

Rubrics have emerged as a mechanism for making evaluation standards explicit and operational. This survey:

1. **Defines rubrics** and distinguishes them from reward models, verifiable rewards, and LLM-as-a-judge
2. **Organizes rubric construction** into four paradigms: direct generation, contrastive generation, iterative refinement, and online/co-evolving generation
3. **Examines rubric-driven training** for both policy models (standard RL, advanced reward design, policy guidance) and reward models (interpretability, reward signals, data construction)
4. **Summarizes rubric-driven evaluation** across general tasks (reasoning, deep research, agent capability, alignment) and domain-specific tasks (intermediate trajectories, final outputs)
5. **Discusses open challenges**: rubric reward hacking, generalization of rubric-based reward models, bias in rubric-based evaluation, personalized rubrics, and rubric safety

## Key Contributions

- First comprehensive survey on rubrics for LLMs
- Systematic taxonomy of rubric construction methods (4 categories)
- Comprehensive review of rubrics for model training and evaluation
- Discussion of open challenges and future directions (personalized rubrics, rubric safety, RIPD)

## Formal Definition

A rubric is a set of rubric items: R = {(dⱼ, wⱼ)}ᵏⱼ₌₁
- dⱼ: natural-language description of the j-th rubric item
- wⱼ: its weight
- Score: cⱼ(x, y) ∈ [0, 1]
- Aggregated rubric score: S_R(x, y) = Σwⱼcⱼ(x,y) / Σwⱼ
