20260706:新增一些文章

raw/papers/rubrics-survey-2026.md

@@ -0,0 +1,54 @@
+---
+title: "The Rules of the Game: A Survey of Rubrics for Large Language Models"
+created: 2026-06-27
+updated: 2026-06-27
+type: raw-paper
+source_url: "https://8421bcd.github.io/_pages/Rubrics_Survey.pdf"
+github: "https://github.com/8421BCD/Rubrics_Survey"
+authors:
+  - "Wenhan Liu"
+  - "Jiajie Jin"
+  - "Zhaoheng Huang"
+  - "Tongyu Wen"
+  - "Guanting Dong"
+  - "Ziliang Zhao"
+  - "Yutao Zhu"
+  - "Zhicheng Dou"
+  - "Ji-Rong Wen"
+affiliation: "Renmin University of China"
+date: "2026-05-22"
+tags:
+  - rubric
+  - evaluation
+  - reward-modeling
+  - survey
+---
+
+# The Rules of the Game: A Survey of Rubrics for Large Language Models
+
+## Abstract
+
+As LLMs evolve from general text generators into systems for reasoning, decision-making, tool use, and long-horizon problem solving, the question of how to specify, optimize, and evaluate model responses has become critical. Simple correctness signals and unconstrained LLM-based judgments are often insufficient for open-ended and high-stakes tasks where response quality depends on multiple criteria (factuality, completeness, safety, reasoning soundness, evidence grounding, practical utility).
+
+Rubrics have emerged as a mechanism for making evaluation standards explicit and operational. This survey:
+
+1. **Defines rubrics** and distinguishes them from reward models, verifiable rewards, and LLM-as-a-judge
+2. **Organizes rubric construction** into four paradigms: direct generation, contrastive generation, iterative refinement, and online/co-evolving generation
+3. **Examines rubric-driven training** for both policy models (standard RL, advanced reward design, policy guidance) and reward models (interpretability, reward signals, data construction)
+4. **Summarizes rubric-driven evaluation** across general tasks (reasoning, deep research, agent capability, alignment) and domain-specific tasks (intermediate trajectories, final outputs)
+5. **Discusses open challenges**: rubric reward hacking, generalization of rubric-based reward models, bias in rubric-based evaluation, personalized rubrics, and rubric safety
+
+## Key Contributions
+
+- First comprehensive survey on rubrics for LLMs
+- Systematic taxonomy of rubric construction methods (4 categories)
+- Comprehensive review of rubrics for model training and evaluation
+- Discussion of open challenges and future directions (personalized rubrics, rubric safety, RIPD)
+
+## Formal Definition
+
+A rubric is a set of rubric items: R = {(dⱼ, wⱼ)}ᵏⱼ₌₁
+- dⱼ: natural-language description of the j-th rubric item
+- wⱼ: its weight
+- Score: cⱼ(x, y) ∈ [0, 1]
+- Aggregated rubric score: S_R(x, y) = Σwⱼcⱼ(x,y) / Σwⱼ