20260706:新增一些文章

raw/papers/safe-equilibrium-exploration.md

@@ -0,0 +1,61 @@
+---
+title: "Safe Equilibrium Exploration (Raw)"
+source: https://arxiv.org/abs/2602.00636
+authors: Yujie Yang, Zhilong Zheng, Shengbo Eben Li
+institution: Tsinghua University, School of Vehicle and Mobility
+arxiv: 2602.00636v2
+category: cs.LG
+venue: IEEE TPAMI 48(7), 8344-8360 (2026)
+doi: 10.1109/TPAMI.2026.3669907
+date: January 31, 2026
+---
+
+# On the Equilibrium between Feasible Zone and Uncertain Model in Safe Exploration
+
+## Abstract
+
+Ensuring the safety of environmental exploration is a critical problem in RL. While limiting exploration to a feasible zone has become widely accepted, key questions remain: what is the maximum feasible zone achievable through exploration, and how can it be identified? This paper reveals that the goal of safe exploration is to find the equilibrium between the feasible zone and the environment model. These two components are interdependent: a larger feasible zone leads to a more accurate environment model, and a more accurate model enables exploring a larger zone. We propose Safe Equilibrium Exploration (SEE), which alternates between finding the maximum feasible zone and the least uncertain model. Using a graph formulation, we prove the uncertain model is monotonically refined, feasible zones monotonically expand, and both converge to the equilibrium. Experiments on classic control tasks show zero constraint violation and equilibrium within a few iterations.
+
+## 1. Introduction
+
+**Training modes for safe RL**:
+- **OTOI** (Offline Training Online Implementation): Train in simulator, deploy online — no safe exploration needed, but requires high-fidelity sim
+- **SOTI** (Simultaneous Online Training Implementation): Learn through online interaction — requires safe exploration
+
+**Key insight**: Safe exploration isn't about maximizing feasible zone alone, but finding equilibrium with the uncertain model.
+
+## 2. Related Work
+
+**Safety filter methods**:
+- Dalal et al. (2018): Action correction under instantaneous constraints
+- Pham et al. (2018): QP layer for distance/velocity constraints
+- Cheng et al. (2019): State-affine CBF + QP safety filter
+- Control Barrier Function (CBF) based methods
+- Safety Index method (Liu et al.)
+
+**Limitation**: Safety filter methods rely on human-designed constraints → feasible zones are conservative and incomplete.
+
+## 3. Problem Formulation
+
+- MDP with safety constraints
+- Feasible zone Z: subset of state-action space where constraint-satisfying policy exists
+- Uncertain model M: estimated transition dynamics with quantified uncertainty
+- Equilibrium condition: Z is the maximum feasible zone given M, and M is the least uncertain model given data collected within Z
+
+## 4. SEE Algorithm
+
+Alternating optimization:
+1. **Zone expansion**: Given current model M, find maximum feasible zone Z
+2. **Model refinement**: Collect data within Z, update model M
+3. Repeat until convergence
+
+## 5. Theoretical Analysis
+
+- Monotonic refinement of uncertain model
+- Monotonic expansion of feasible zones
+- Convergence to equilibrium
+- Graph formulation as the analytical framework
+
+## 6. Experiments
+
+Classic control tasks: SEE achieves zero constraint violation with monotonic zone expansion, reaching equilibrium within few iterations.