66 lines
2.1 KiB
Markdown
66 lines
2.1 KiB
Markdown
---
|
||
title: "Rubric-Based Reward Modeling"
|
||
created: 2026-06-27
|
||
updated: 2026-06-27
|
||
type: concept
|
||
tags:
|
||
- rubric
|
||
- reward-modeling
|
||
- rl
|
||
- training
|
||
sources:
|
||
- "rubrics-survey-2026"
|
||
---
|
||
|
||
# Rubric-Based Reward Modeling
|
||
|
||
## 定义
|
||
|
||
将 rubric 用作**奖励信号**来训练 policy model 或 reward model。Rubric 将多维度评估标准转化为结构化的训练监督信号，弥补传统标量奖励在开放任务中的不足。
|
||
|
||
## Policy Model Training 的三种模式
|
||
|
||
### 1. Standard Rubric-based RL
|
||
直接用 rubric 定义 RL 中的奖励函数，标准 RL 优化。
|
||
- 代表：RLCF, ResearchPlanGen, RaR, OpenRS, Chasing the Tail, RLAC, rDPO, AutoRubric-R1V, RLCER, DR-Tulu, OralGPT-Plus
|
||
- 流程：rubric 打分 → 聚合为 scalar reward → RL (PPO/DPO/GRPO)
|
||
|
||
### 2. Advanced Reward Design
|
||
在标准 RL 基础上设计更精细的奖励机制：
|
||
- RBR：rubric-based reward 重塑
|
||
- Rubicon：rubric 加权奖励
|
||
- SYNTHAGENT：轨迹级 rubric 奖励
|
||
- RuCL：rubric 驱动的课程学习
|
||
- ARL-RR、PAPO、RTT、DRO
|
||
- 关键思想：rubric 不仅是"总分"，还可以塑造奖励的**形状**和**时序**分布
|
||
|
||
### 3. Rubrics as Policy Guidance
|
||
不让 rubric 仅作为外部奖励，而是将其融入模型的推理/规划过程：
|
||
- RuscaRL：rubric-conditioned RL
|
||
- Think-with-Rubrics：推理时参考 rubric
|
||
- HeRL：分层 rubric 指导
|
||
- 本质：rubric 从"评分标准"变为"思考框架"
|
||
|
||
## Reward Model Training 的三种角色
|
||
|
||
### 1. Rubrics for Interpretability
|
||
用 rubric 维度使 reward model 的输出可解释：
|
||
- R3, mR3：rubric-agnostic reward reasoning
|
||
- CDPRM, RRM, OMNI-RRM, C2, ArmoRM, DeltaRubric, MR-RML
|
||
- 输出形式：多维评分向量而非单一标量
|
||
|
||
### 2. Rubrics for Reward Signals
|
||
用 LLM + rubric 直接生成奖励信号（替代标注数据训练 RM）：
|
||
- METAJUDGE, Proxy-GRM
|
||
|
||
### 3. Rubrics for Data Construction
|
||
用 rubric 构造训练数据：
|
||
- CROME：从 rubric 生成偏好数据
|
||
|
||
## 参考
|
||
|
||
- [[rubrics-for-llms|Rubrics for LLMs]]
|
||
- [[rubrics-survey-2026|Rubrics Survey (2026)]]
|
||
- [[rubric-construction]]
|
||
- [[reward-hacking]]
|