40 lines
1.8 KiB
Markdown
40 lines
1.8 KiB
Markdown
---
|
|
title: "KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls"
|
|
authors:
|
|
- Kailin Jiang
|
|
- Hongbo Jiang
|
|
- Ning Jiang
|
|
- Zhi Gao
|
|
- Jinhe Bi
|
|
- Yuchen Ren
|
|
- Bin Li
|
|
- Yuntao Du
|
|
- Lei Liu
|
|
- Qing Li
|
|
date: 2026
|
|
arxiv: "2510.19316"
|
|
venue: "ICML 2026"
|
|
domain: "Multimodal Learning, Knowledge Injection, Continual Learning"
|
|
type: paper
|
|
source: "https://arxiv.org/abs/2510.19316"
|
|
---
|
|
|
|
# KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
|
|
|
|
**Authors**: Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, Qing Li
|
|
|
|
**Venue**: ICML 2026
|
|
|
|
**arXiv**: 2510.19316
|
|
|
|
## Abstract
|
|
|
|
KORE is a synergistic method centered around Knowledge-Oriented Controls for injecting new knowledge into LMMs while preserving old knowledge. It implements a two-stage optimization: (1) KORE-AUGMENTATION converts individual knowledge items into structured multi-round dialogues and instruction tasks, building a "knowledge tree" that enables internalization; (2) KORE-CONSTRAINT stores previous knowledge in the covariance matrix of linear layer activations and initializes a LoRA adapter by projecting original weights into the matrix's null space, defining a fine-tuning direction that minimally interferes with previous knowledge.
|
|
|
|
## Key Contributions
|
|
|
|
1. **KORE-AUGMENTATION**: Structured knowledge augmentation pipeline — multi-round dialogues (trunk) + instruction tasks (branches) = knowledge tree
|
|
2. **KORE-CONSTRAINT**: Null space projection via covariance matrix SVD — freezes adapter A in null space, fine-tunes only B
|
|
3. **HARS metric**: Harmonized Adaptation-Retention Score for unified evaluation
|
|
4. **State-of-the-art**: Outperforms 9 baselines on EVOKE benchmark across LLaVA-v1.5 (7B/13B) and Qwen2.5-VL (7B)
|