Feature/kimi linear support #17592

cacaview · 2025-11-29T15:10:52Z

Make sure to read the contributing guidelines before submitting a PR
This is the current work progress：
#16930 (comment)

- Implement KDA layer (linear attention with gates and decay) - Implement MLA layer (multi-head latent attention with KV compression) - Support MoE FFN with shared experts - Add TikToken tokenizer support for Kimi models - Fix vocab loading for large vocabularies - Model loads and runs inference (27 layers, 603 tensors)

- Add missing MoE metadata to GGUF conversion: - moe_intermediate_size (1024) - num_shared_experts (1) - first_k_dense_replace (1) - routed_scaling_factor (2.446) - expert_gating_func (sigmoid) - Fix MoE gating function default to SIGMOID (was SOFTMAX) - Add expert_weights_scale loading with default 2.446 - Enable moe_renormalize (norm_w=true) in build_moe_ffn - Add fallback for exp_probs_b tensor suffix compatibility

- Add KDA (Kimi Delta Attention) CUDA kernel (kda-scan.cu) - Fix recurrence order: decay first, then retrieval - Verify CPU/CUDA implementation consistency - Support head_dim=128, L2 normalization for Q/K

CISC · 2025-11-29T16:52:10Z

convert_hf_to_gguf.py

+# KimiLinearModel is defined later in this file (line ~5140) as a TextModel subclass
+# This old definition has been removed to avoid conflicts
+
+
 @ModelBase.register(


Suggested change

# KimiLinearModel is defined later in this file (line ~5140) as a TextModel subclass

# This old definition has been removed to avoid conflicts

@ModelBase.register(

@ModelBase.register(

CISC · 2025-11-29T16:53:57Z

convert_hf_to_gguf.py

                (self.format_tensor_name(gguf.MODEL_TENSOR.ATTN_Q, bid), q),
                (self.format_tensor_name(gguf.MODEL_TENSOR.ATTN_K, bid), k),
                (self.format_tensor_name(gguf.MODEL_TENSOR.ATTN_V, bid), v),
            ]


Suggested change

]

]

else:

return [(self.map_tensor_name(name), data_torch)]

CISC · 2025-11-29T16:55:59Z

convert_hf_to_gguf.py

+@ModelBase.register("KimiLinearModel", "KimiLinearForCausalLM")
+class KimiLinearModel(TextModel):
+    """Kimi-Linear model with hybrid MLA+KDA architecture"""
+    model_arch = gguf.MODEL_ARCH.KIMI


Suggested change

model_arch = gguf.MODEL_ARCH.KIMI

model_arch = gguf.MODEL_ARCH.KIMI_LINEAR

CISC · 2025-11-29T17:02:23Z

convert_hf_to_gguf.py

+    _experts: list[dict[str, Tensor]] | None = None
+
+    def set_gguf_parameters(self):
+        self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])


Suggested change

self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])

super().set_gguf_parameters()

self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])

CISC · 2025-11-29T17:03:02Z

convert_hf_to_gguf.py

+        # Use find_hparam for context length
+        # Kimi uses model_max_length
+        n_ctx = self.find_hparam(["max_position_embeddings", "model_max_length", "n_ctx", "n_positions"], optional=True)
+        if n_ctx is not None:
+            self.gguf_writer.add_context_length(n_ctx)
        else:
-            return [(self.map_tensor_name(name), data_torch)]
+            # Default to 4096 if not found
+            logger.warning("No context length found in config, defaulting to 4096")
+            self.gguf_writer.add_context_length(4096)


Add model_max_length to TextModel.set_gguf_parameters instead, the fallback is not necessary.

CISC · 2025-11-29T17:36:31Z

src/llama-model.cpp

        case LLM_ARCH_ARCTIC:
        case LLM_ARCH_DEEPSEEK:
        case LLM_ARCH_DEEPSEEK2:
+        case LLM_ARCH_KIMI:


Suggested change

case LLM_ARCH_KIMI:

case LLM_ARCH_KIMI_LINEAR:

CISC · 2025-11-29T17:36:56Z

src/llama-quant.cpp

-    if (qs.n_attention_wv != 0 && !is_clip_model)
+    // Skip this check for Kimi models which have hybrid KDA+MLA architecture
+    // (only MLA layers have attn_kv_b weights, KDA layers don't)
+    if (qs.n_attention_wv != 0 && !is_clip_model && model.arch != LLM_ARCH_KIMI)


Suggested change

if (qs.n_attention_wv != 0 && !is_clip_model && model.arch != LLM_ARCH_KIMI)

if (qs.n_attention_wv != 0 && !is_clip_model && model.arch != LLM_ARCH_KIMI_LINEAR)

CISC · 2025-11-29T17:37:28Z

src/models/kimi-linear.cpp

Rename to kimi-linear.cpp

CISC · 2025-11-29T17:37:44Z

src/models/kimi.cpp

@@ -0,0 +1,429 @@
+#include "models.h"
+
+llm_build_kimi::llm_build_kimi(const llama_model & model, const llm_graph_params & params) : llm_graph_context_mamba(params), model(model) {


Suggested change

llm_build_kimi::llm_build_kimi(const llama_model & model, const llm_graph_params & params) : llm_graph_context_mamba(params), model(model) {

llm_build_kimi::llm_build_kimi_linear(const llama_model & model, const llm_graph_params & params) : llm_graph_context_mamba(params), model(model) {

CISC · 2025-11-29T17:38:17Z

src/models/models.h

+struct llm_build_kimi : public llm_graph_context_mamba {
+    llm_build_kimi(const llama_model & model, const llm_graph_params & params);


Suggested change

struct llm_build_kimi : public llm_graph_context_mamba {

llm_build_kimi(const llama_model & model, const llm_graph_params & params);

struct llm_build_kimi_linear : public llm_graph_context_mamba {

llm_build_kimi_linear(const llama_model & model, const llm_graph_params & params);

…view/llama.cpp into feature/kimi-linear-support

cacaview · 2025-11-30T03:52:05Z

I have fixed these errors in the commit at cacaview@780dd78

cacaview and others added 5 commits November 28, 2025 23:42

Add Kimi Linear model conversion support

7c0334e

feat(kimi): add KDA CUDA kernel and optimize recurrence implementation

6b20da1

- Add KDA (Kimi Delta Attention) CUDA kernel (kda-scan.cu) - Fix recurrence order: decay first, then retrieval - Verify CPU/CUDA implementation consistency - Support head_dim=128, L2 normalization for Q/K

Merge branch 'ggml-org:master' into feature/kimi-linear-support

1b29643

cacaview requested review from CISC and ggerganov as code owners November 29, 2025 15:10

github-actions bot added model Model specific Nvidia GPU Issues specific to Nvidia GPUs python python script changes ggml changes relating to the ggml tensor library for machine learning labels Nov 29, 2025

loci-dev mentioned this pull request Nov 29, 2025

UPSTREAM PR #17592: Feature/kimi linear support auroralabs-loci/llama.cpp#363

Open

CISC reviewed Nov 29, 2025

View reviewed changes

cacaview and others added 3 commits November 30, 2025 11:43

fix: ggml-org#17592 (review)

780dd78

Merge branch 'feature/kimi-linear-support' of https://github.com/caca…

3a7e87f

…view/llama.cpp into feature/kimi-linear-support

Merge branch 'ggml-org:master' into feature/kimi-linear-support

02d3d8d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/kimi linear support #17592

Feature/kimi linear support #17592

cacaview commented Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

CISC Nov 29, 2025

Uh oh!

cacaview commented Nov 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	model_arch = gguf.MODEL_ARCH.KIMI
	model_arch = gguf.MODEL_ARCH.KIMI_LINEAR

	self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])
	super().set_gguf_parameters()
	self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])

	if (qs.n_attention_wv != 0 && !is_clip_model && model.arch != LLM_ARCH_KIMI)
	if (qs.n_attention_wv != 0 && !is_clip_model && model.arch != LLM_ARCH_KIMI_LINEAR)

		@@ -0,0 +1,429 @@
		#include "models.h"

		llm_build_kimi::llm_build_kimi(const llama_model & model, const llm_graph_params & params) : llm_graph_context_mamba(params), model(model) {

	llm_build_kimi::llm_build_kimi(const llama_model & model, const llm_graph_params & params) : llm_graph_context_mamba(params), model(model) {
	llm_build_kimi::llm_build_kimi_linear(const llama_model & model, const llm_graph_params & params) : llm_graph_context_mamba(params), model(model) {

		struct llm_build_kimi : public llm_graph_context_mamba {
		llm_build_kimi(const llama_model & model, const llm_graph_params & params);

Feature/kimi linear support #17592

Are you sure you want to change the base?

Feature/kimi linear support #17592

Conversation

cacaview commented Nov 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cacaview commented Nov 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants