myWiki/raw/papers/safe-equilibrium-exploration.md

title, source, authors, institution, arxiv, category, venue, doi, date

title	source	authors	institution	arxiv	category	venue	doi	date
Safe Equilibrium Exploration (Raw)	https://arxiv.org/abs/2602.00636	Yujie Yang, Zhilong Zheng, Shengbo Eben Li	Tsinghua University, School of Vehicle and Mobility	2602.00636v2	cs.LG	IEEE TPAMI 48(7), 8344-8360 (2026)	10.1109/TPAMI.2026.3669907	January 31, 2026

On the Equilibrium between Feasible Zone and Uncertain Model in Safe Exploration

Abstract

Ensuring the safety of environmental exploration is a critical problem in RL. While limiting exploration to a feasible zone has become widely accepted, key questions remain: what is the maximum feasible zone achievable through exploration, and how can it be identified? This paper reveals that the goal of safe exploration is to find the equilibrium between the feasible zone and the environment model. These two components are interdependent: a larger feasible zone leads to a more accurate environment model, and a more accurate model enables exploring a larger zone. We propose Safe Equilibrium Exploration (SEE), which alternates between finding the maximum feasible zone and the least uncertain model. Using a graph formulation, we prove the uncertain model is monotonically refined, feasible zones monotonically expand, and both converge to the equilibrium. Experiments on classic control tasks show zero constraint violation and equilibrium within a few iterations.

1. Introduction

Training modes for safe RL:

• OTOI (Offline Training Online Implementation): Train in simulator, deploy online — no safe exploration needed, but requires high-fidelity sim
• SOTI (Simultaneous Online Training Implementation): Learn through online interaction — requires safe exploration

Key insight: Safe exploration isn't about maximizing feasible zone alone, but finding equilibrium with the uncertain model.

2. Related Work

Safety filter methods:

• Dalal et al. (2018): Action correction under instantaneous constraints
• Pham et al. (2018): QP layer for distance/velocity constraints
• Cheng et al. (2019): State-affine CBF + QP safety filter
• Control Barrier Function (CBF) based methods
• Safety Index method (Liu et al.)

Limitation: Safety filter methods rely on human-designed constraints → feasible zones are conservative and incomplete.

3. Problem Formulation

• MDP with safety constraints
• Feasible zone Z: subset of state-action space where constraint-satisfying policy exists
• Uncertain model M: estimated transition dynamics with quantified uncertainty
• Equilibrium condition: Z is the maximum feasible zone given M, and M is the least uncertain model given data collected within Z

4. SEE Algorithm

Alternating optimization:

1. Zone expansion: Given current model M, find maximum feasible zone Z
2. Model refinement: Collect data within Z, update model M
3. Repeat until convergence

5. Theoretical Analysis

• Monotonic refinement of uncertain model
• Monotonic expansion of feasible zones
• Convergence to equilibrium
• Graph formulation as the analytical framework

6. Experiments

Classic control tasks: SEE achieves zero constraint violation with monotonic zone expansion, reaching equilibrium within few iterations.