24 lines
592 B
Markdown
24 lines
592 B
Markdown
---
|
||
title: Mixture of Attention Schemes (MoAS)
|
||
created: 2025-04-15
|
||
updated: 2026-05-01
|
||
type: concept
|
||
tags: []
|
||
sources: []
|
||
---
|
||
|
||
# Mixture of Attention Schemes (MoAS)
|
||
|
||
**注意力方案混合路由**，根据 Token 复杂度动态分配注意力类型。
|
||
|
||
## 核心思想
|
||
|
||
"简单" Token 用廉价 [[multi-query-attention|MQA]]，"困难" Token 用强大 [[multi-head-attention|MHA]]，实现条件计算。
|
||
|
||
## 相关概念
|
||
|
||
- [[multi-head-attention]] — MHA
|
||
- [[grouped-query-attention]] — GQA
|
||
- [[duo-attention]] — 另一种分类方案
|
||
- [[llm-attention-survey-2026]] — 综述参考
|