1.8 KiB
title, authors, date, arxiv, venue, domain, type, source
| title | authors | date | arxiv | venue | domain | type | source | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls |
|
2026 | 2510.19316 | ICML 2026 | Multimodal Learning, Knowledge Injection, Continual Learning | paper | https://arxiv.org/abs/2510.19316 |
KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
Authors: Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, Qing Li
Venue: ICML 2026
arXiv: 2510.19316
Abstract
KORE is a synergistic method centered around Knowledge-Oriented Controls for injecting new knowledge into LMMs while preserving old knowledge. It implements a two-stage optimization: (1) KORE-AUGMENTATION converts individual knowledge items into structured multi-round dialogues and instruction tasks, building a "knowledge tree" that enables internalization; (2) KORE-CONSTRAINT stores previous knowledge in the covariance matrix of linear layer activations and initializes a LoRA adapter by projecting original weights into the matrix's null space, defining a fine-tuning direction that minimally interferes with previous knowledge.
Key Contributions
- KORE-AUGMENTATION: Structured knowledge augmentation pipeline — multi-round dialogues (trunk) + instruction tasks (branches) = knowledge tree
- KORE-CONSTRAINT: Null space projection via covariance matrix SVD — freezes adapter A in null space, fine-tunes only B
- HARS metric: Harmonized Adaptation-Retention Score for unified evaluation
- State-of-the-art: Outperforms 9 baselines on EVOKE benchmark across LLaVA-v1.5 (7B/13B) and Qwen2.5-VL (7B)