20260625:很多新内容

raw/papers/mozer-topological-trouble-transformers-2026.md

@@ -0,0 +1,41 @@
+---
+title: "The Topological Trouble With Transformers"
+source: arXiv
+source_id: 2604.17121
+authors:
+  - Michael C. Mozer (Google DeepMind)
+  - Shoaib Ahmed Siddiqui (Google DeepMind)
+  - Rosanne Liu (Google DeepMind)
+published: 2026-04-18
+updated: 2026-06-03
+categories:
+  - cs.LG
+  - cs.AI
+venue: Preprint
+---
+
+# The Topological Trouble With Transformers
+
+## Abstract
+Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking—the iterative updating of latent variables reflecting an evolving environment—involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth.
+
+While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. The authors argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures.
+
+## Core Contributions
+1. **Topological analysis** of why feedforward Transformers fundamentally cannot track state indefinitely
+2. **Taxonomy of recurrent Transformer architectures** along two dimensions: recurrence axis (depth vs step) and input-tokens-per-recurrence-step ratio
+3. **Identification of empty cells** in the taxonomy as promising research directions
+4. **Critique of Chain-of-Thought as workaround** — it externalizes what should be implicit
+5. **Roadmap** for enhanced SSMs, coarse recurrence, representational alignment, and efficient recurrence training
+
+## Key Concepts
+- state tracking, belief state, depth dilemma
+- recurrent transformer architectures (depth/step/both)
+- recurrence taxonomy: axis × ratio
+- attractor dynamics, latent thought models
+- enhanced state-space models (DeltaNet, RWKV-7, PaTH attention)
+- representational alignment, coarse-grained recurrence
+- sequential dependency, autoregressive unrolling
+
+## URL
+https://arxiv.org/abs/2604.17121