Introduction

🎯 Overview

KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT.

🔥 Updates

• Sep 13, 2026: Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts.
• Aug 26, 2026: Added native support for GLM-5.3-flash, bringing 1M-token context and multimodal input to consumer GPUs. (Tutorial)
• Aug 25, 2026: Uploaded a new easy-to-use KTransformers × LlamaFactory MoE Fine-Tuning Cookbook, covering hardware checks, installation, BF16/FP8/INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting.
• Aug 17, 2026: LoRA fine-tuning now supports compatible AVX512 x86 CPUs, including AMD servers, without requiring AMX. (v0.7.0 Release Notes)
• Aug 16, 2026: DeepSeek-V4-Flash on a single Ascend NPU with CPU expert offload. (Tutorial)
• Aug 5, 2026: Introduced native block-FP8 LoRA fine-tuning (PR #2141)
• July 23, 2026: Added end-to-end BF16 full-parameter fine-tuning for MoE models, including complete checkpoint saving. (PR #2094)
• June 21, 2026: MiniMax-M3 Day0 Support! (Tutorial)
• June 17, 2026: GLM-5.2 Day0 Support! (Tutorial)
• May 6, 2026: KTransformers at GOSIM Paris 2026 — “Agentic AI on Edge” track. We’ll present KT’s inference performance on consumer hardware.
• May 02, 2026: DeepSeek-V4-Flash Support! (Tutorial)
• Apr 30, 2026: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points.
• Mar 26, 2026: Support AVX2-only CPU backend for KT-Kernel inference. (Tutorial)
• Feb 13, 2026: MiniMax-M2.5 Day0 Support! (Tutorial)
• Feb 12, 2026: GLM-5 Day0 Support! (Tutorial)
• Jan 27, 2026: Kimi-K2.5 Day0 Support! (Tutorial) (SFT Tutorial)
• Jan 22, 2026: Support CPU-GPU Expert Scheduling, Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference
• Dec 24, 2025: Support Native MiniMax-M2.1 inference. (Tutorial)
• Dec 22, 2025: Support RL-DPO fine-tuning with LLaMA-Factory. (Tutorial)
• Dec 5, 2025: Support Native Kimi-K2-Thinking inference (Tutorial)
• Nov 6, 2025: Support Kimi-K2-Thinking inference (Tutorial) and fine-tune (Tutorial)
• Nov 4, 2025: KTransformers Fine-Tuning × LLaMA-Factory Integration. (Tutorial)
• Oct 27, 2025: Support Ascend NPU. (Tutorial)
• Oct 10, 2025: Integrating into SGLang. (Roadmap, Blog)
• Sept 11, 2025: Support Qwen3-Next. (Tutorial)
• Sept 05, 2025: Support Kimi-K2-0905. (Tutorial)
• July 26, 2025: Support SmallThinker and GLM4-MoE. (Tutorial)
• July 11, 2025: Support Kimi-K2. (Tutorial)
• June 30, 2025: Support 3-layer (GPU-CPU-Disk) prefix cache reuse.
• May 14, 2025: Support Intel Arc GPU (Tutorial).
• Apr 29, 2025: Support AMX-Int8、 AMX-BF16 and Qwen3MoE (Tutorial)
• Apr 9, 2025: Experimental support for LLaMA 4 models (Tutorial).
• Apr 2, 2025: Support Multi-concurrency. (Tutorial).
• Mar 15, 2025: Support ROCm on AMD GPU (Tutorial).
• Mar 5, 2025: Support unsloth 1.58/2.51 bits weights and IQ1_S/FP8 hybrid weights. Support 139K Longer Context for DeepSeek-V3 and R1 in 24GB VRAM.
• Feb 25, 2025: Support FP8 GPU kernel for DeepSeek-V3 and R1; Longer Context.
• Feb 15, 2025: Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed （+15%, up to 16 Tokens/s), update docs and online books.
• Feb 10, 2025: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see here.
• Aug 28, 2024: Decrease DeepseekV2’s required VRAM from 21G to 11G.
• Aug 15, 2024: Update detailed tutorial for injection and multi-GPU.
• Aug 14, 2024: Support llamfile as linear backend.
• Aug 12, 2024: Support multiple GPU; Support new model: mixtral 8*7B and 8*22B; Support q2k, q3k, q5k dequant on gpu.
• Aug 9, 2024: Support windows native.

📦 Capabilities

🚀 Inference - High-Performance kt-kernel Serving

CPU-optimized kernel operations for heterogeneous LLM inference.

Key Features:

• AMX/AVX Acceleration: Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
• MoE Optimization: Efficient Mixture-of-Experts inference with NUMA-aware memory management
• Quantization Support: CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support
• Easy Integration: Clean Python API for SGLang and other frameworks

Quick Start:

cd kt-kernel
pip install .

Use Cases:

• CPU-GPU hybrid inference for large MoE models
• Integration with SGLang for production serving
• Heterogeneous expert placement (hot experts on GPU, cold experts on CPU)

Performance Examples:

👉 Full Documentation →

🎓 SFT - Fine-Tuning with LlamaFactory

KTransformers × LlamaFactory integration for ultra-large MoE model fine-tuning. The new Cookbook provides an easy-to-use path from hardware checks and installation to BF16/FP8/INT8 configuration, LoRA/full training, resource planning, and troubleshooting.

Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts. Follow the PyPI installation, training, resume and SGLang serving guide (中文). End-to-end validation uses Kimi K2.5; K2.6 follows the same model-architecture path.

Key Features:

• Multi-Backend Support: CPU/GPU hybrid fine-tuning with INT8/INT4 quantization
• Ultra-Large MoE Support: Fine-tune models like DeepSeek-V3/R1 on limited GPU memory
• Faster than ZeRO-Offload: 6-12x training speedup in benchmarked MoE SFT workloads
• Lower CPU Memory: About half the CPU memory of the previous KT SFT path in the benchmarked setup
• LlamaFactory Integration: Seamless integration with the popular fine-tuning framework

Quick Start:

cd /path/to/LLaMA-Factory
python -m pip install -e .
python -m pip install "ktransformers[sft]==0.7.0"
python -m pip install "sglang-kt==0.7.0"
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
  --config_file examples/ktransformers/accelerate/fsdp2_kt_int8.yaml \
  src/train.py \
  examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml

👉 Easy-to-Use Fine-Tuning Cookbook → 👉 Quick Start → 👉 Full Documentation →

🔥 Citation

If you use KTransformers in your research, please cite our paper:

@inproceedings{10.1145/3731569.3764843,
  title = {KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models},
  author = {Chen, Hongtao and Xie, Weiyu and Zhang, Boxin and Tang, Jingqi and Wang, Jiahao and Dong, Jianwei and Chen, Shaoyuan and Yuan, Ziwei and Lin, Chen and Qiu, Chengyu and Zhu, Yuening and Ou, Qingliang and Liao, Jiaqi and Chen, Xianglin and Ai, Zhiyuan and Wu, Yongwei and Zhang, Mingxing},
  booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles},
  year = {2025}
}

👥 Contributors & Team

Developed and maintained by:

• MADSys Lab @ Tsinghua University
• Approaching.AI
• 9#AISoft
• Community contributors

We welcome contributions! Please feel free to submit issues and pull requests.

💬 Community & Support

• GitHub Issues: Report bugs or request features
• WeChat Group: See archive/WeChatGroup.png

📦 KT original Code

The original integrated KTransformers framework has been archived to the archive/ directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.

For the original documentation with full quick-start guides and examples, see:

• archive/README.md (English)
• archive/README_ZH.md (中文)

KT-Kernel

High-performance kernel operations for KTransformers, featuring CPU-optimized MoE inference with AMX, AVX, KML and blis (amd library) support.

• Note
• Features
• Installation
  • Option 1: Install from PyPI (Recommended for Most Users)
  • Option 2: Install from Source (For Local Use or Custom Builds)
• Verification
• KT CLI Overview
• Integration with SGLang
  • Installation Steps
  • Complete Example: Qwen3-30B-A3B
  • KT-Kernel Parameters
• Direct Python API Usage
  • Advanced Options
  • Manual Configuration (Advanced)
• Build Configuration
  • Manual Installation (Without install.sh)
• Error Troubleshooting
  • CUDA Not Found
  • hwloc Not Found
• Weight Quantization
• Before Commit!

Note

Current Support Status:

• ✅ Native Precision with AVX512/AMX: Supported with AVX512 CPUs in FP8, BF16 and RAWINT4 format - Guide
• ✅ Intel CPUs with AMX: Fully supported (using weights converted to INT4/INT8 format)
• ✅ Universal CPU (llamafile backend): Supported (using GGUF-format weights)
• ✅ AMD CPUs with BLIS: Supported (for int8 prefill & decode) - Guide

KT-CLI

We are developing a simpler way to use KTransformers. Check out the KT-CLI Guide for more details.

Features

• CPU-Optimized MoE Kernels: High-throughput MoE expert kernels optimized for instruction sets.
• AVX512 Native Precision Backend: FP8 / BF16 / INT4 native MoE backend for AVX512-capable servers.
• AMX INT4/INT8 Backend: INT4 / INT8 quantized expert inference backend for AMX-capable servers.
• Llamafile CPU Backend: AVX2/AVX512-based MoE backend built on Llamafile for universal CPU deployment.
• NUMA-Aware Execution: Thread pool and memory layout designed for multi-socket / multi-NUMA machines.

Installation

Option 1: Install from PyPI (Recommended for Most Users)

Install the latest version with a single command:

pip install kt-kernel

Note: Check the latest version on PyPI

Features:

• ✅ Automatic CPU detection: Detects your CPU and loads the optimal kernel variant
• ✅ CPU multi-variant support: Includes AMX, AVX512 (Base/VNNI/VBMI/BF16), and AVX2 variants
• ✅ CUDA support included: GPU acceleration for NVIDIA GPUs (SM 80, 86, 89, 90)
• ✅ No compilation needed: Pre-built wheels for Python 3.10, 3.11, 3.12
• ✅ Static CUDA runtime: No CUDA toolkit installation required
• ✅ Works on CPU-only systems: CUDA features automatically disabled when GPU not available

Requirements:

• Python 3.10, 3.11, or 3.12
• Linux x86-64 (manylinux_2_17 compatible)
• CPU with AVX2 support (Intel Haswell 2013+, AMD Zen+)
• Optional: NVIDIA GPU with compute capability 8.0+ for CUDA features

CUDA Installation (GPU Acceleration)

For NVIDIA GPU-accelerated inference:

pip install kt-kernel

Features:

• ✅ Multi-architecture support: Single wheel supports SM 80/86/89/90 (Ampere, Ada, Hopper)
• ✅ Static CUDA runtime: No CUDA toolkit installation required
• ✅ Broad compatibility: Works with CUDA 11.8+ and 12.x drivers
• ✅ PyTorch compatible: Works with any PyTorch CUDA variant (cu118, cu121, cu124)

Requirements:

• Python 3.10, 3.11, or 3.12
• Linux x86-64 (manylinux_2_17 compatible)
• NVIDIA GPU with compute capability 8.0+ (Ampere or newer)
  • ✅ Supported: A100, RTX 3000/4000 series, H100
  • ❌ Not supported: V100, P100, GTX 1000/2000 series (too old)
• NVIDIA driver with CUDA 11.8+ or 12.x support (no CUDA toolkit needed)

GPU Compatibility Matrix:

CUDA Driver Compatibility (for GPU features):

• CUDA 11.8, 11.9, 12.0-12.6+: Full support
• CUDA 11.0-11.7: Not supported (upgrade driver or use CPU-only)

CPU Variants Included:

The wheel includes 6 optimized variants that are automatically selected at runtime based on your CPU:

Verify installation:

import kt_kernel

# Check which CPU variant was loaded
print(f"CPU variant: {kt_kernel.__cpu_variant__}")
print(f"Version: {kt_kernel.__version__}")

# Check CUDA support
from kt_kernel import kt_kernel_ext
cpu_infer = kt_kernel_ext.CPUInfer(4)
has_cuda = hasattr(cpu_infer, 'submit_with_cuda_stream')
print(f"CUDA support: {has_cuda}")

print("✓ kt-kernel installed successfully!")

Environment Variables:

# Override automatic CPU detection (for testing or debugging)
export KT_KERNEL_CPU_VARIANT=avx2  # Force specific variant

# Enable debug output to see detection process
export KT_KERNEL_DEBUG=1
python -c "import kt_kernel"

Option 2: Install from Source (For Local Use or Custom Builds)

Build from source for local installation or when you need AMD (BLIS), ARM (KML), or custom CUDA versions.

Prerequisites

First, initialize git submodules and create a conda environment:

git submodule update --init --recursive
conda create -n kt-kernel python=3.11 -y
conda activate kt-kernel

Quick Installation (Recommended)

Simply run the install script - it will auto-detect your CPU and optimize for best performance:

./install.sh

What happens automatically:

• Auto-detects CPU capabilities (AMX, AVX512_VNNI, AVX512_BF16)
• Installs system dependencies (cmake, libhwloc-dev, pkg-config)
• Builds optimized binary for your CPU only (using -march=native)
• Software fallbacks: Automatically enabled for CPUs without VNNI/BF16

Optional: Two-step installation

./install.sh deps   # Install dependencies only
./install.sh build  # Build and install kt-kernel

CPU Requirements by Backend:

Software Fallback Support (AVX512 backends):

• ✅ VNNI fallback: Uses AVX512BW instructions
• ✅ BF16 fallback: Uses AVX512F instructions
• ✅ Older AVX512 CPUs (Skylake-X, Cascade Lake) can run RAWINT4 with fallbacks

⚠️ Portability Note: The default build is optimized for your specific CPU and may not work on different/older CPUs. For portable builds or binary distribution, see Manual Configuration below.

⚠️ AMD BLIS backend users: See installation guide for AMD-specific setup.

Verification

After installation, verify that the CLI is working:

kt version

Expected output:

KTransformers CLI v0.x.x

  Python:        3.11.x
  Platform:      Linux 5.15.0-xxx-generic
  CUDA:          12.x
  kt-kernel:     0.x.x (amx)
  sglang:        0.x.x

You can also verify the Python module directly:

python -c "from kt_kernel import KTMoEWrapper; print('✓ kt-kernel installed successfully')"

KT CLI Overview

The kt command-line tool provides a unified interface for running and managing KTransformers models:

Quick Start Example:

# Start a model server (auto-detects hardware and applies optimal settings)
kt run m2

# In another terminal, chat with the model
kt chat

# Check system compatibility
kt doctor

Run kt --help for more options, or kt <command> --help for command-specific help.

Integration with SGLang

KT-Kernel can be used standalone via Direct Python API or integrated with SGLang for production deployment. This section describes SGLang integration to enable CPU-GPU heterogeneous inference, where “hot” experts run on GPU and “cold” experts run on CPU for optimal resource utilization.

Installation Steps

1. Install SGLang

Install the kvcache-ai fork of SGLang (required for kt-kernel support):

# Option A: One-click install (from ktransformers root, installs sglang + kt-kernel)
./install.sh

# Option B: pip install
pip install kt-kernel sglang-kt

# Option C: From source (editable mode)
git clone --recursive https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
pip install -e "third_party/sglang/python[all]"

Important: Use sglang-kt (kvcache-ai fork), not the official sglang package. If you have the official version installed, uninstall it first: pip uninstall sglang -y

2. Prepare Weights

You need both GPU weights and CPU-side expert weights for heterogeneous inference. The exact format depends on the backend:

GPU Weights (for all backends):
Use the model weights required by SGLang for GPU inference (for example, the original or already-quantized model directory from Hugging Face).

CPU Weights (AMX backend: AMXINT4 / AMXINT8): Quantize weights to AMX-optimized INT4/INT8 format using the provided script:

python scripts/convert_cpu_weights.py \
  --input-path /path/to/model \
  --input-type bf16 \
  --output /path/to/cpu-weights \
  --quant-method int8  # or int4 or moe_int8 (for amd now) 

• --input-path: Path to GPU-side original weights
• --input-type: Depends on your GPU weights type (fp8, fp16, or bf16)

In SGLang integration, --kt-weight-path should point to this converted CPU weights directory.

Supported input formats: FP8, FP16, BF16 → INT4/INT8.

CPU Weights (LLAMAFILE backend: LLAMAFILE): LLAMAFILE uses pre-quantized GGUF weights on the CPU side directly, without running convert_cpu_weights.py. You need to:

• Download a GGUF model directly from the web (e.g., GGUF repos on Hugging Face / Modelscope);
• In SGLang integration, use that GGUF directory as --kt-weight-path. KT-Kernel supports multiple GGUF quantization formats such as Q4_KM, Q4_K, Q5_K, etc. Choose based on your latency and accuracy requirements.

3. Launch SGLang Server

Start the SGLang server with your normal SGLang parameters, and add the following KT-Kernel specific parameters to enable CPU-GPU heterogeneous inference:

KT-Kernel Parameters to Add:

• --kt-method: Backend method (AMXINT4, AMXINT8, or LLAMAFILE)
• --kt-weight-path: Path to the converted CPU weights
• --kt-cpuinfer: Number of CPU inference threads (set to physical cores)
• --kt-threadpool-count: Number of thread pools (set to NUMA node count)
• --kt-num-gpu-experts: Number of experts to keep on GPU
• --kt-max-deferred-experts-per-token: Deferred experts for pipelined execution

Example:

python -m sglang.launch_server \
  [your normal SGLang parameters...] \
  --kt-method AMXINT8 \
  --kt-weight-path /path/to/cpu-weights \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

See KT-Kernel Parameters section below for detailed parameter tuning guidelines.

Complete Example: Qwen3-30B-A3B

This example demonstrates the full workflow from downloading weights to launching the server, showing Native backend, AMX backend and LLAMAFILE backend options.

Hardware Configuration:

• GPU: NVIDIA RTX 4090 24GB
• CPU: 2x Intel Xeon Gold 6454S (64 physical cores total, 128 threads, 2 NUMA nodes)
• Model: Qwen3-30B-A3B

How to verify your system configuration:

# Check CPU configuration
lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core|Socket\(s\)|NUMA node\(s\)"
# Expected output example:
CPU(s):                                  128
Thread(s) per core:                      2
Socket(s):                               2
NUMA node(s):                            2
# → Physical cores = CPU(s) / Thread(s) per core = 128 / 2 = 64

Parameter Rationale:

• --kt-cpuinfer 64: Set to physical cores (64), not hyperthreads (128)
• --kt-threadpool-count 2: 2 NUMA nodes detected (dual-socket system)
• --kt-num-gpu-experts 32: With 24GB GPU memory, we can fit ~32 experts on GPU for this model (varies by model architecture and actual memory usage)
• --kt-max-deferred-experts-per-token 2: Enable pipelined execution; allows CPU to process next batch while GPU completes current batch
• --kt-gpu-prefill-token-threshold 2048: Use layerwise prefill strategy when token count exceeds 2048 (for native backends only)

Option A: Native Backend (BF16)

For AVX512 CPUs with BF16 support.

Step 1: Download model weights

# Install huggingface-cli if not already installed
pip install huggingface-hub
# Download model from Hugging Face  
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B

Step 2: Launch SGLang server

python -m sglang.launch_server \
    --host 0.0.0.0 \
    --port 30000 \
    --model /mnt/data/models/Qwen3-30B-A3B \
    --kt-weight-path /mnt/data/models/Qwen3-30B-A3B \
    --kt-cpuinfer 64 \
    --kt-threadpool-count 2 \
    --kt-num-gpu-experts 32 \
    --kt-method BF16 \
    --attention-backend flashinfer \
    --trust-remote-code \
    --mem-fraction-static 0.80 \
    --chunked-prefill-size 16384 \
    --max-running-requests 4 \
    --served-model-name Qwen3 \
    --enable-mixed-chunk \
    --tensor-parallel-size 1 \
    --enable-p2p-check \
    --disable-shared-experts-fusion \
    --kt-gpu-prefill-token-threshold 4096 \
    --kt-enable-dynamic-expert-update

Option B: AMX Backend (AMXINT8)

For Intel CPUs with AMX instruction set support.

Step 1: Download model weights

# Install huggingface-cli if not already installed
pip install huggingface-hub

# Download model from Hugging Face
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B

Step 2: Convert to CPU weights (AMXINT8)

python scripts/convert_cpu_weights.py \
  --input-path /mnt/data/models/Qwen3-30B-A3B \
  --input-type bf16 \
  --output /mnt/data/models/Qwen3-30B-A3B-INT8 \
  --quant-method int8

Step 3: Launch SGLang server

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 8000 \
  --model /mnt/data/models/Qwen3-30B-A3B \
  --trust-remote-code \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 4096 \
  --served-model-name Qwen3-30B-A3B \
  --enable-mixed-chunk \
  --kt-method AMXINT8 \
  --kt-weight-path /mnt/data/models/Qwen3-30B-A3B-INT8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

Option C: LLAMAFILE Backend (GGUF)

For universal CPUs (no AMX required), using pre-quantized GGUF weights directly.

Step 1: Download GPU weights (original model)

pip install huggingface-hub

huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B

Step 2: Download CPU weights (GGUF format)

huggingface-cli download Qwen/Qwen3-30B-A3B-GGUF Qwen3-30B-A3B-Q4_K_M.gguf \
  --local-dir /mnt/data/models/Qwen3-30B-A3B-Q4_K_M

Step 3: Launch SGLang server

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 8000 \
  --model /mnt/data/models/Qwen3-30B-A3B \
  --trust-remote-code \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 4096 \
  --served-model-name Qwen3-30B-A3B \
  --enable-mixed-chunk \
  --kt-method LLAMAFILE \
  --kt-weight-path /mnt/data/models/Qwen3-30B-A3B-Q4_K_M \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

KT-Kernel Parameters

Parameter Guidelines:

• kt-method: Choose based on your CPU and weight format:

  • AMXINT4: Best performance on AMX CPUs with INT4 quantized weights (May cause huge accuracy drop for some models, e.g., Qwen3-30B-A3B)
  • AMXINT8: Higher accuracy with INT8 quantized weights on AMX CPUs
  • RAWINT4: Native INT4 weights shared by CPU and GPU (currently supports Kimi-K2-Thinking model). See Kimi-K2-Thinking Native Tutorial for details.
  • FP8, FP8_PERCHANNEL: FP8 weights shared by CPU and GPU
  • BF16: BF16 weights shared by CPU and GPU
  • LLAMAFILE: GGUF-based backend

• kt-cpuinfer: Set to the number of physical CPU cores (not hyperthreads).

  • Check physical cores: lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core"
  • Physical cores = CPU(s) / Thread(s) per core
  • Example: If CPU(s)=128 and Thread(s) per core=2, then physical cores = 64
  • Important: Do NOT set to hyperthread count - this will degrade performance

• kt-threadpool-count: Set to the number of NUMA nodes.

  • Check NUMA count: lscpu | grep "NUMA node(s)"
  • Or use: numactl --hardware | grep "available"
  • Note: NUMA node count is NOT necessarily the number of physical CPUs
    • It represents memory domains, which may be divided within a single CPU or across multiple CPUs
    • Use the NUMA node count from lscpu, regardless of physical CPU count
  • Typical values: 1-2 for single-socket, 2-4 for dual-socket systems
  • This enables better memory bandwidth utilization across NUMA domains

• kt-num-gpu-experts: Determine based on GPU memory and profiling:

  • More GPU experts = lower latency but higher GPU memory usage (May cause OOM)

• kt-max-deferred-experts-per-token: Enables pipelined execution:

  • 0: Synchronous execution (simpler, higher latency)
  • 1-4: Deferred execution (recommended range; good latency/quality balance, requires tuning)
  • 5-7: Highest latency reduction but may introduce noticeable accuracy loss; use with care

• kt-gpu-prefill-token-threshold (FP8 and RAWINT4 only): Controls prefill strategy for native FP8 and INT4 inference:

  • ≤ threshold: Uses hybrid CPU+GPU prefill. No extra VRAM needed, but performance degrades slowly as token count increases.
  • > threshold: Uses layerwise GPU prefill. Performance scales better with longer sequences, but requires one MoE layer extra VRAM (e.g., ~9GB+ for Kimi-K2-Thinking and ~3.6GB for MiniMax-M2.1).
  • Only applicable when --kt-method RAWINT4 or --kt-method FP8 is used.

• kt-enable-dynamic-expert-update: Enables dynamic expert placement updates during inference.

  • During layerwise prefill, the system collects actual routing statistics and redistributes GPU experts accordingly.
  • Requires --kt-gpu-prefill-token-threshold to be set, and prefill length must be ≥ the threshold value.
  • Particularly effective at lower GPU expert ratios (10%-70%), where it can significantly outperform static strategies.
  • See Expert Scheduling Tutorial for benchmarks and details.

• kt-expert-placement-strategy: Determines which experts are placed on GPU at server startup.

  • uniform: Distributes GPU experts evenly across all MoE layers. Default option, no prior statistics needed.
  • frequency: Places the most frequently activated experts on GPU. Best performance when activation statistics are available; requires --init-expert-location pointing to a .pt statistics file.
  • front-loading: Fills GPU experts from the first MoE layer onwards.
  • random: Randomly selects experts with a fixed seed (42).
  • See Expert Scheduling Tutorial for strategy comparison.

Direct Python API Usage

For standalone usage without SGLang, you can use KT-Kernel directly via Python API:

from kt_kernel import KTMoEWrapper

# Initialize the MoE wrapper
wrapper = KTMoEWrapper(
    layer_idx=0,
    num_experts=8,
    num_experts_per_tok=2,
    hidden_size=4096,
    moe_intermediate_size=14336,
    num_gpu_experts=2,
    cpuinfer_threads=32,
    threadpool_count=2,
    weight_path="/path/to/weights",
    chunked_prefill_size=512,
    method="AMXINT4"  # Options: "AMXINT4", "AMXINT8", "LLAMAFILE"
)

# Load weights (from disk - pre-quantized)
wrapper.load_weights(physical_to_logical_map)

# Or load weights from tensors (online quantization)
wrapper.load_weights_from_tensors(gate_proj, up_proj, down_proj, physical_to_logical_map)

# Run inference
output = wrapper.forward(hidden_states, topk_ids, topk_weights, cuda_stream)

# Or use async API for better performance
wrapper.submit_forward(hidden_states, topk_ids, topk_weights, cuda_stream)
# ... do other work ...
output = wrapper.sync_forward(hidden_states, cuda_stream)

Advanced Options

# Initialize with additional options
wrapper = KTMoEWrapper(
    layer_idx=0,
    num_experts=8,
    num_experts_per_tok=2,
    hidden_size=4096,
    moe_intermediate_size=14336,
    num_gpu_experts=2,
    cpuinfer_threads=32,
    threadpool_count=2,
    weight_path="/path/to/weights",
    chunked_prefill_size=512,
    method="AMXINT4",
    cpu_save=False,  # Keep weights in CPU memory after loading
    max_deferred_experts_per_token=0  # Number of experts to defer (for pipelined execution)
)

# Pre-allocate buffers for specific batch sizes (improves performance)
KTMoEWrapper.set_capture_batch_sizes([1, 2, 4, 8, 16])

# Query captured batch sizes
batch_sizes = KTMoEWrapper.get_capture_batch_sizes()

# Clear buffer cache to free memory
KTMoEWrapper.clear_buffer_cache()

Manual Configuration (Advanced)

For portable builds, binary distribution, or cross-machine deployment, you need to manually specify target instruction sets:

# General distribution (works on any AVX512 CPU from 2017+)
export CPUINFER_CPU_INSTRUCT=AVX512
export CPUINFER_ENABLE_AMX=OFF
./install.sh build --manual

# Maximum compatibility (works on any CPU from 2013+)
export CPUINFER_CPU_INSTRUCT=AVX2
export CPUINFER_ENABLE_AMX=OFF
./install.sh build --manual

# Modern CPUs only (Ice Lake+, Zen 4+)
export CPUINFER_CPU_INSTRUCT=FANCY
export CPUINFER_ENABLE_AMX=OFF
./install.sh build --manual

Optional: Override VNNI/BF16 detection

# Force enable/disable VNNI and BF16 (for testing fallbacks)
export CPUINFER_ENABLE_AVX512_VNNI=OFF
export CPUINFER_ENABLE_AVX512_BF16=OFF
./install.sh

Optional: oneDNN INT8 BRGEMM on non-AMX AVX512 CPUs

# Requires an installed oneDNN >= 3.9 package:
export CPUINFER_ENABLE_ONEDNN_VNNI=ON

# Or build against a oneDNN source tree:
export CPUINFER_ONEDNN_SOURCE_DIR=/path/to/oneDNN
./install.sh

# The compiled wheel defaults to oneDNN; native remains available for A/B tests.
export KT_INT8_VNNI_BACKEND=auto  # auto | onednn | native

See ./install.sh --help for all available options.

Build Configuration

Manual Installation (Without install.sh)

If you prefer manual installation without the install.sh script:

1. Install System Dependencies

Prerequisites:

• cmake (recommended: conda install -y cmake)
• libhwloc-dev and pkg-config

2. Set Build Configuration

Core Options:

Instruction Set Details:

Example Configurations:

# Local use - maximum performance (default behavior)
export CPUINFER_CPU_INSTRUCT=NATIVE
export CPUINFER_ENABLE_AMX=ON  # or OFF

# Distribution build - works on any AVX512 CPU
export CPUINFER_CPU_INSTRUCT=AVX512
export CPUINFER_ENABLE_AMX=OFF

# Maximum compatibility - works on CPUs since 2013
export CPUINFER_CPU_INSTRUCT=AVX2
export CPUINFER_ENABLE_AMX=OFF

# Debug build
export CPUINFER_BUILD_TYPE=Debug
export CPUINFER_VERBOSE=1

3. Build and Install

# Editable installation (for development)
pip install -e .

# Standard installation
pip install .

Error Troubleshooting

CUDA Not Found

 -- Looking for a CUDA compiler - NOTFOUND
  CMake Error at CMakeLists.txt:389 (message):
    KTRANSFORMERS_USE_CUDA=ON but CUDA compiler not found

Make sure you have the CUDA toolkit installed and nvcc is in your system PATH.

Try export CMAKE_ARGS="-D CMAKE_CUDA_COMPILER=$(which nvcc)" and reinstall again.

hwloc Not Found

Run sudo apt install libhwloc-dev if on a Debian-based system or build from source: https://www.open-mpi.org/projects/hwloc/.

wget https://download.open-mpi.org/release/hwloc/v2.12/hwloc-2.12.2.tar.gz
tar -xzf hwloc-2.12.2.tar.gz
cd hwloc-2.12.2
./configure
make
sudo make install

Weight Quantization

For AMX backends (AMXINT4 / AMXINT8), CPU-side experts must be converted to AMX-friendly INT4/INT8 format using the provided script:

python scripts/convert_cpu_weights.py \
  --input-path /path/to/model \
  --input-type bf16 \
  --output /path/to/output \
  --quant-method int4

Supported formats: FP8, FP16, BF16 → INT4/INT8

For LLAMAFILE backend (LLAMAFILE), CPU-side experts are loaded directly from GGUF weights. You do not need to run the AMX conversion script; instead, download a GGUF model from the web (e.g., a GGUF repo on Hugging Face) and point weight_path / SGLang --kt-weight-path (or --model when appropriate) to that GGUF directory. KT-Kernel supports multiple GGUF quantization types such as Q4_KM, Q4_K, Q5_K, etc.

For detailed documentation, advanced options, and low-memory mode, see scripts/README.md.

Before Commit!

Commit messages should follow the Conventional Commits specification: https://www.conventionalcommits.org/

Please format your code before committing:

cmake -B build
cd build
make format

You may need a newer clang-format (at least version 18). In a conda environment:

conda install -c conda-forge clang-format=18
rm -rf build

It’s also recommended to install black for Python code formatting:

conda install black

• Introduction
  • Fine-Tuning Results (Examples)
• Quick to Start
  • Environment Setup
  • Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models
  • Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter)
  • Core Feature 3: Batch inference + metrics (base + LoRA adapter)
• KT Fine-Tuning Speed (User-Side View)
  • End-to-End Performance
  • GPU/CPU Memory Footprint
• Conclusion

KTransformers Fine-Tuning × LLaMA-Factory Integration – User Guide

MadSys Lab, KVCache-AI Team, Approaching AI, LLaMA-Factory Team

Introduction

From DeepSeek-V3/R1 to Qwen3-MoE and Kimi-K2, each wave of open-sourced large models brings leaps in performance and scale. However, many researchers and developers are constrained by expensive GPUs and models with tens or even hundreds of billions of parameters, making it hard to fine-tune very large models under limited resources. To bridge this gap, we propose a practical approach: combining KTransformers with LLaMA-Factory. With just 2–4 RTX 4090s and a high-memory CPU, you can fine-tune ultra-large MoE models like DeepSeek-671B.

Our goal is to give resource-constrained researchers a local path to explore fine-tuning ultra-large models, and also a fast way to customize smaller models (e.g., 14B/30B) for specific scenarios. We validate the setup using stylized dialogue, Westernized translation tone, and medical Q&A as representative tasks, showing that personalized adaptation can be achieved within hours.

As shown below, LLaMA-Factory is the unified orchestration/configuration layer for the whole fine-tuning workflow—handling data, training scheduling, LoRA injection, and inference interfaces. KTransformers acts as a pluggable high-performance backend that takes over core operators like Attention/MoE under the same training configs, enabling efficient GPU+CPU heterogeneous cooperation.

Within LLaMA-Factory, we compared LoRA fine-tuning with HuggingFace, Unsloth, and KTransformers backends. KTransformers is the only workable 4090-class solution for ultra-large MoE models (e.g., 671B) and also delivers higher throughput and lower GPU memory on smaller MoE models (e.g., DeepSeek-14B).

† 1400 GB is a theoretical FP16 full-parameter resident footprint (not runnable). 70 GB is the measured peak with KT strategy (Attention on GPU + layered MoE offload).

Fine-Tuning Results (Examples)

Stylized Dialogue (CatGirl tone)

Dataset: NekoQA-10K. Goal: improve style consistency and recognizability.

The figure compares responses from the base vs. fine-tuned models. The fine-tuned model maintains the target tone and address terms more consistently (red boxes), validating the effectiveness of style-transfer fine-tuning.

Benchmarks

We use:

(1) Translational-Style-ChatLLM, which asks for an exaggerated, Westernized translation tone—clear, stylized customization.

(2) AfriMed-QA (ACL 2025), a medical dataset for African contexts with strong domain specificity, including multiple-choice and short-answer sub-tasks—well-suited for vertical fine-tuning evaluation.

The tables show metrics before vs. after LoRA fine-tuning. We observe large improvements across metrics, verifying fine-tuning effectiveness:

Even for ultra-large MoE models, KTransformers-backed fine-tuning achieves strong task performance quickly.

Quick to Start

This section shows how to install and use LLaMA-Factory + KTransformers for fine-tuning and inference:

• Environment setup
• Fine-tune ultra-large MoE models with KTransformers backend
• Load the fine-tuned model (base + LoRA adapter) for chat/inference
• Batch inference and metric evaluation

Environment Setup

According to the following example, install both the KTransformers and LLaMA-Factory environments simultaneously. This time, to simplify the installation process of KTransformers, use the PyPI packages to avoid local compilation. The detailed installation steps are as follows: (Note: Make sure your local Python version, Torch version, and CUDA version are compatible with the installed packages.)

# 1. Create a conda environment
conda create -n Kllama python=3.12 # choose from : [3.11, 3.12, 3.13]
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime

# 2. Install the LLaMA-Factory environment
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .

# 3. Install the KTransformers SFT packages
pip install "ktransformers[sft]"

# 4. Install flash-attention, download the corresponding file based on your Python and Torch versions from: https://github.com/Dao-AILab/flash-attention/releases
pip install flash-attn --no-build-isolation
# abi=True/False can find from below
# import torch
# print(torch._C._GLIBCXX_USE_CXX11_ABI)

# 5. (Optional) If you want to use flash_infer (otherwise it defaults to triton)
git clone https://github.com/kvcache-ai/custom_flashinfer.git
pip install custom_flashinfer/

Usage tip: In LLaMA-Factory YAML, set use_kt: true and pick a kt_optimize_rule file to have KTransformers handle the core compute. The features below show typical configs.

Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models

Run the command: USE_KT=1 ACCELERATE_USE_KT=true accelerate launch --config_file examples/ktransformers/accelerate/fsdp2_kt_bf16.yaml -m llamafactory.cli train examples/ktransformers/train_lora/deepseek_v3_lora_sft_kt.yaml.

Note: You must provide a BF16 model. DeepSeek-V3-671B is released in FP8 by default; convert with DeepSeek-V3/inference/fp8_cast_bf16.py.

### model
model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all

### dataset
dataset: identity
template: deepseek
cutoff_len: 2048
max_samples: 100000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output
output_dir: saves/Kllama_deepseekV3
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

### ktransformers
use_kt: true # use KTransformers as LoRA sft backend
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
cpu_infer: 32
chunk_size: 8192

kt_optimize_rule controls placement strategy. See also ktransformers/optimize_rules. Naming hints (* = wildcard):

Example: DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml = V3-Chat fine-tuning with AMX and 2-GPU model parallel.

We recommend AMX acceleration where available (lscpu | grep amx). AMX supports BF16/INT8. Example:

- match:
    name: "^model\\.layers\\..*\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert parallelism
    kwargs:
      prefill_device: "cpu"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op: "KSFTExpertsCPU"
      out_device: "cuda"
      backend: "AMXInt8" # or "AMXBF16" or "llamafile" (default)

Outputs go to output_dir in safetensors format plus adapter metadata for later loading.

Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter)

Run the command: llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml.

Use the safetensors adapter trained with KT for inference.

model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
adapter_name_or_path: saves/Kllama_deepseekV3
template: deepseek
infer_backend: ktransformers  # choices: [huggingface, vllm, sglang, ktransformers]
trust_remote_code: true

use_kt: true # use KTransformers as LoRA sft backend to inference
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
cpu_infer: 32
chunk_size: 8192

We also support GGUF adapters: for safetensors, set the directory; for GGUF, set the file path in adapter_name_or_path.

During loading, LLaMA-Factory maps layer names to KT’s naming. You’ll see logs like Loaded adapter weight: XXX -> XXX:

Core Feature 3: Batch inference + metrics (base + LoRA adapter)

Run the command: API_PORT=8000 llamafactory-cli api examples/inference/qwen3_lora_sft.yaml. Invoke the KT fine-tuned adapter to provide the API; the usage logic of other APIs is consistent with the native LLaMA-Factory approach.

model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
adapter_name_or_path: saves/Kllama_deepseekV3
template: deepseek
infer_backend: ktransformers  # choices: [huggingface, vllm, sglang, ktransformers]
trust_remote_code: true

use_kt: true # use KTransformers as LoRA sft backend to inference
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
cpu_infer: 32
chunk_size: 8192

KT Fine-Tuning Speed (User-Side View)

End-to-End Performance

Definitions

• step_time: wall-clock time for a full optimization step (tensor movement + Attention + MoE + other compute).
• tokens_per_step = GAS × qlen; token/s = tokens_per_step / step_time.

Settings: GAS=16, qlen=512 (→ tokens_per_step = 8192); LoRA (r=8, alpha=32, dropout=0.1); AMX enabled; GPU: RTX 4090, CPU: Intel Xeon Platinum 8488C.

Measured

• DeepSeek-V3-671B: step_time = 203 s → token/s ≈ 8192 / 203 ≈ 40.35
• DeepSeek-V2-Lite-14B: step_time = 36 s → token/s ≈ 8192 / 36 ≈ 227.6

GPU/CPU Memory Footprint

• DeepSeek-V3 (671B; 61 layers with 58 MoE): ~70 GB total GPU VRAM (multi-GPU), ~1.2–1.3 TB CPU RAM.
• DeepSeek-V2-Lite (14B; 27 layers with 26 MoE): ~5.5 GB GPU VRAM, ~30 GB CPU RAM.

Conclusion

By integrating KTransformers LoRA fine-tuning into LLaMA-Factory, we provide a practical guide for efficient training and deployment of MoE LLMs. KT brings cutting-edge optimizations (DeepSeek/Qwen/Kimi support with AMX-accelerated kernels), and LoRA enables customization under very low GPU memory. LLaMA-Factory offers a friendly, unified interface.

This integration (akin to Unsloth-style speedups) means even models with tens to hundreds of billions of parameters can be fine-tuned and deployed with low latency on commodity hardware. You get memory savings, speed-ups, and usability together. We encourage you to try LLaMA-Factory + KT for your next MoE project and follow this guide. Feedback is welcome!

KTransformers SFT Docs

• Kimi K2.5: PyPI Install, LoRA Training, Resume and Serving (中文)
• KTransformers × LlamaFactory MoE Fine-Tuning Cookbook
• KTransformers × LlamaFactory MoE Fine-Tuning Cookbook (中文)
• v0.6.1 Quick Start
• Fine-Tuning User Guide
• KT-FT Fine-Tuning and Inference Loop
• Developer Technical Notes
• DPO Tutorial
• Injection Tutorial

KTransformers × LlamaFactory MoE Fine-Tuning Cookbook

From Qwen3.5 to DeepSeek-V4, Kimi-K3, and GLM-5.2, each new ultra-large open model brings a major leap in capability and scale. However, the cost of high-end GPUs still prevents many researchers and developers from fine-tuning these models under constrained resources. KTransformers and LlamaFactory provide a practical alternative: with 1–4 RTX 4090 GPUs and a CPU platform with sufficient memory, users can fine-tune trillion-parameter MoE (Mixture of Experts) models such as the DeepSeek-V3/V4 family, Kimi-K2.5, and GLM-5.2.

KTransformers integrates with LlamaFactory while preserving its familiar training workflow. LlamaFactory remains the unified configuration and orchestration layer for data processing, training, LoRA injection, and inference integration. The GPU runs Attention and Shared Expert modules, while KTransformers manages Routed Experts in CPU memory for heterogeneous GPU–CPU execution.

Technical overview: KTransformers places compute-intensive Attention and shared modules on the GPU, while storing the large, sparsely activated Routed Experts in CPU memory. This avoids repeatedly transferring large Expert weights over PCIe and lets the CPU runtime select an optimized backend according to the weight format and available instruction set.

Guide at a glance: For a first run, start with native BF16 + LoRA or full fine-tuning + kt_backend: auto. This path requires no weight conversion and starts with two YAML files. If the model provides a native FP8 checkpoint and the training method is LoRA, KTransformers can load the native FP8 Expert weights directly to reduce host-memory usage. Use converted INT8 or AMXINT4 weights only when host memory remains the limiting resource and the matching converted artifacts are available.

Contents

• Check hardware and install
• Choose a fine-tuning setup
• Custom YAML: advanced settings
• Performance and resource estimates
• Appendix: complete configurations and Q&A

1. Check hardware and install

1.1 Check hardware resources

Check the CPU instruction set first to identify the available acceleration backend:

lscpu | grep -i -E 'Model name|Socket|NUMA|avx512|amx'

Then check GPU memory, host memory, and disk capacity:

nvidia-smi
free -h
df -h /data

Confirm two requirements:

• Instruction-set compatibility: Native FP8 requires a compatible AMD/x86 CPU with AVX512F, AVX512_BF16, AVX512_VNNI, and AVX512_VBMI. For INT8, auto can select an AMX-INT8 or AVX512-VNNI implementation. AMXINT4 requires AMX and matching converted weights. Native BF16 performs best when AMX is available.
• Sufficient capacity: Host memory must hold the Expert weights and leave room for activations, gradients, and optimizer states. Disk capacity must also cover checkpoints and temporary files.

1.2 Install the environment

Use a clean Python 3.11 environment. Pin PyTorch 2.9.1 before installing LlamaFactory, and install the KT dependencies last so standard transformers or accelerate packages do not overwrite the KT variants:

conda create -n kt-sft python=3.11 -y
conda activate kt-sft

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory

python -m pip install torch==2.9.1 torchaudio==2.9.1 torchvision==0.24.1
python -m pip install -e .
python -m pip install "ktransformers[sft]==0.7.0"
python -m pip install "sglang-kt==0.7.0"

After installation, check dependency consistency and the versions that Python actually imports:

python -m pip check

python - <<'PY'
from importlib.metadata import version

import accelerate
import ktransformers
import kt_kernel
import torch
import transformers

for name, module in {
    "torch": torch,
    "transformers": transformers,
    "accelerate": accelerate,
    "kt_kernel": kt_kernel,
    "ktransformers": ktransformers,
}.items():
    print(f"{name:14s} {getattr(module, '__version__', 'unknown')}")

print(f"{'sglang-kt':14s} {version('sglang-kt')}")
print(f"{'transformers-kt':14s} {version('transformers-kt')}")
print(f"{'accelerate-kt':14s} {version('accelerate-kt')}")

from accelerate.utils.dataclasses import KTransformersPlugin  # noqa: F401
PY

The compatible environment should include these versions:

No broken requirements found.
torch          2.9.1
transformers   5.6.0
accelerate     1.14.0
kt_kernel      0.7.0
ktransformers  0.7.0
sglang-kt      0.7.0
transformers-kt 5.6.0.post2
accelerate-kt  1.14.0.post2

2. Choose a fine-tuning setup

Choose the Expert weight format first, then select LoRA or full fine-tuning. After making both choices, use the corresponding training YAML and Accelerate YAML below.

2.1 Choose the Expert weight format

Native versus converted precision describes the Expert weight format. AMX (Advanced Matrix Extensions) and AVX512 (Advanced Vector Extensions 512-bit) describe CPU instruction sets and backends; they are a separate dimension.

Prefer native BF16 or native FP8 when host memory allows. Use INT8 or AMXINT4 only when matching converted weights are available and reducing Expert memory is necessary.

2.2 Choose LoRA or full fine-tuning

Expert weight format and parameter-update scope are separate configuration dimensions, but the supported combinations have clear boundaries. Native BF16 supports both LoRA and full fine-tuning. Native FP8, INT8, and AMXINT4 recipes use LoRA with frozen base Experts. Full fine-tuning uses BF16.

2.3 Responsibilities of the two YAML files

Each run uses two YAML files. The training YAML defines the task and owns every KTransformers setting. The Accelerate YAML defines only distributed execution and FSDP2. Current LlamaFactory rejects kt_config in the Accelerate YAML.

Check three relationships before launch:

1. The number of GPUs in CUDA_VISIBLE_DEVICES must equal num_processes.
2. If kt_config.kt_model_max_length is set manually in the training YAML, it must cover cutoff_len plus the runtime token margin.
3. Write the LoRA rank only at the top level of the training YAML. LlamaFactory derives the internal KT fields. Remove LoRA-only fields for full fine-tuning.

2.4 Base YAML for four common setups

Copy the complete configurations from Appendix A.1 and A.2, then replace the fields shown for the selected setup. Each block labels the training YAML and Accelerate YAML sections. kt_config always belongs in the training YAML.

2.4.1 Native BF16 + LoRA

# File 1: training YAML
model_name_or_path: /data/models/Your-BF16-Model
finetuning_type: lora
lora_rank: 8
use_kt: true
# Do not set kt_weight_path for native BF16
kt_config:
  kt_backend: auto
  kt_expert_weight_format: bf16

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.4.2 Native FP8 + LoRA

# File 1: training YAML
model_name_or_path: /data/models/Your-Native-FP8-Model
finetuning_type: lora
lora_rank: 8
use_kt: true
# Do not set kt_weight_path for native FP8
kt_config:
  kt_backend: auto
  kt_expert_weight_format: fp8

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.4.3 INT8 / AMXINT4 + LoRA

# File 1: training YAML
finetuning_type: lora
lora_rank: 8
use_kt: true
# INT8 setup
kt_weight_path: /data/models/Your-Routed-Experts-INT8
kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16
kt_config:
  kt_backend: auto
  kt_expert_weight_format: int8
  kt_weight_lifecycle: persistent

# For AMXINT4, use matching paths and replace kt_config above with:
# kt_weight_path: /data/models/Your-Model-AMXINT4
# kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16
# kt_config:
#   kt_backend: AMXINT4

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.4.4 Native BF16 + full fine-tuning

# File 1: training YAML
finetuning_type: full
learning_rate: 1.0e-5
use_kt: true
# Remove lora_rank, lora_alpha, lora_dropout, lora_target, and other LoRA fields
kt_config:
  kt_backend: auto
  kt_expert_weight_format: bf16

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.5 Launch

CUDA_VISIBLE_DEVICES=0,1 accelerate launch --main_process_port 0 --config_file qwen35_fsdp2_2gpu.yaml src/train.py qwen35_397b_bf16_lora.yaml

3. Custom YAML: advanced settings

After a base setup launches successfully, tune the following parameters according to the training objective and resource bottleneck. Keep the recommended values unless there is a clear reason to change them, and avoid changing several parameter groups at once.

(1) LoRA capacity: lora_rank, lora_alpha, lora_target, lora_dropout

Change these four fields only at the top level of the training YAML. LlamaFactory automatically derives the rank, alpha, and dropout used internally by KTransformers. Do not repeat them in kt_config or the Accelerate YAML. Start with lora_rank: 8, lora_alpha: 16, lora_target: all, and lora_dropout: 0.0.

If task adaptation is insufficient, increase the rank to 16 and then 32 while keeping alpha at about twice the rank. If a small dataset shows clear overfitting, increase dropout to 0.05.

(2) Sequence length: cutoff_len

Set this field in the training YAML according to the effective token length of most samples. Do not use the maximum dataset length solely for a small number of long samples. Longer sequences increase activation and KT buffer usage, which raises both host-memory and GPU-memory requirements.

The system derives the KT model-length capacity from cutoff_len. To increase it manually, set kt_model_max_length in the training YAML’s kt_config. It must be greater than cutoff_len and include room for additional runtime tokens.

(3) Batch: per_device_train_batch_size, gradient_accumulation_steps

Set both fields in the training YAML. For ultra-large MoE models, start with per_device_train_batch_size: 1. Increase the micro batch only after confirming sufficient GPU and host memory. If GPU memory is insufficient, keep the micro batch at 1 and use gradient accumulation to increase the effective batch.

effective batch size = per_device_train_batch_size × number of GPUs × gradient_accumulation_steps

(4) Activation recomputation and CPU Activation Reuse: disable_gradient_checkpointing, kt_cpu_activation

In the training YAML, disable_gradient_checkpointing: false enables gradient checkpointing to trade additional computation for lower activation memory. Setting it to true retains more intermediate results and may improve speed, but uses more memory.

When host memory is sufficient, add this top-level field to the same training YAML:

kt_cpu_activation: retain

This option retains CPU Expert activations during checkpoint recomputation, using more host memory to reduce repeated CPU work. When gradient checkpointing is enabled and the field is omitted, CPU and GPU activations are recomputed. kt_cpu_activation controls CPU Expert activation retention, while disable_gradient_checkpointing controls the overall checkpoint-recomputation policy; do not treat them as one switch.

(5) CPU Expert reuse and threads: kt_share_backward_bb, kt_num_threads

Both fields belong in the training YAML’s kt_config. Keep kt_share_backward_bb: true to reuse the buffer required by CPU Expert backward computation. It is separate from kt_cpu_activation; do not disable it unless the configuration for the current model explicitly requires that change.

Set kt_num_threads to the number of physical CPU cores available to the job, not the total number of logical cores including SMT threads. If data preprocessing shares the same CPUs, reserve cores for the DataLoader and system processes.

The system derives kt_threadpool_count from the NUMA topology. Change it manually only after confirming the CPU socket and NUMA layout.

(6) Resume from a checkpoint: resume_from_checkpoint

Set this field in the training YAML to the target checkpoint-* directory. Omit it for a new run. Keep output_dir pointed at the output directory for the current job, never at the base-model directory, and keep overwrite_output_dir: false to avoid overwriting existing results.

4. Performance and resource estimates

The table below uses a 2K context (sequence length = 2048), per-GPU batch size 1, and gradient accumulation 1. It includes only accepted KTransformers BF16 runs that completed real LoRA training. Throughput includes forward, loss, backward, and optimizer steps.

The GPU-memory column reports the highest single-GPU peak from the measured run. The host-memory column reports peak CPU RSS for the training process tree, normalized to GiB. These values are minimum references for the measured configurations, not safety margins. Reserve additional capacity for data loading, caches, checkpoint saving, and system processes.

Results with different GPU counts should not be used to infer linear scaling directly. Increasing cutoff_len, batch size, or KT cache depth increases resource usage. Full fine-tuning also requires additional capacity for gradients, master weights, optimizer states, and complete checkpoints.

Appendix

A.1 Qwen3.5-397B BF16 LoRA training YAML

Save as qwen35_397b_bf16_lora.yaml:

### model
model_name_or_path: /data/models/Qwen3.5-397B-A17B
trust_remote_code: true
disable_gradient_checkpointing: false

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.0
lora_target: all

### dataset
dataset: your_dataset
template: qwen3_5
cutoff_len: 2048
packing: false
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output
output_dir: /data/output/qwen35-bf16-lora
logging_steps: 10
save_strategy: steps
save_steps: 500
plot_loss: true
overwrite_output_dir: false
save_only_model: false
report_to: none

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 1.0e-4
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### ktransformers
use_kt: true
# kt_cpu_activation: retain  # Optional: use more host memory to reduce CPU Expert recomputation
# kt_weight_path: /data/models/Your-Routed-Experts-INT8  # Converted INT8 Routed Experts
# kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16  # Must match the path above
kt_config:
  kt_expert_weight_format: bf16
  kt_backend: auto
  kt_num_threads: 64
  kt_tp_enabled: true
  kt_threadpool_count: 2
  kt_max_cache_depth: 2
  kt_share_backward_bb: true

A.2 FSDP2 Accelerate YAML

Save as qwen35_fsdp2_2gpu.yaml:

compute_environment: LOCAL_MACHINE
distributed_type: FSDP
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_reshard_after_forward: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_version: 2
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
use_cpu: false

A.3 Q&A

A.3.1 Why can’t a native FP8 checkpoint use the converted INT8 configuration?

The weight formats and loading paths are different. Native FP8 preserves the FP8 weights and scales from the checkpoint and uses the AVX512 native-precision backend. The INT8 recipe reads a mutually matched pair of INT8 Routed Expert weights and a BF16 non-expert cache, then lets auto select an AMX-INT8 or AVX512-VNNI implementation. Do not pass a native FP8 checkpoint through kt_weight_path.

A.3.2 auto did not select the expected backend

Save the lscpu output and startup log, then confirm that the container or virtualization environment did not hide the required instruction set. Native BF16 can select AMXBF16 when AMX is available. Native FP8 should select AVX512. Do not force a backend that the CPU or weight format does not support.

A.3.3 Insufficient host memory

First reduce concurrent data loading, sequence length, and cache depth, and confirm that two copies of the Expert weights are not loaded. If memory is still insufficient, prepare converted INT8 or AMXINT4 weights. For full fine-tuning, estimate the additional memory for gradients, master weights, and optimizer states separately.

A.3.4 GPU OOM

OOM (Out of Memory) means GPU or host memory is insufficient. Reduce cutoff_len, per-GPU batch size, and activation usage first, then increase gradient_accumulation_steps to preserve the global batch size. num_processes, the visible GPU count, and the actual model placement must agree.

A.3.5 kt_model_max_length does not match cutoff_len

When increasing cutoff_len, increase kt_model_max_length as well and leave room for the effective runtime sequence length. A value that is too small causes buffer or shape errors; an unnecessarily large value increases host-memory usage.

KTransformers × LlamaFactory MoE 微调 Cookbook

从 Qwen3.5 到 DeepSeek-V4、Kimi-K3、GLM-5.2，每一次超大模型的开源都带来性能与规模上的巨大跃升。然而，多数研究者与开发者受限于昂贵的显卡，难以在资源受限条件下微调超大模型。面对这种差距，我们提出了一种更具可行性的方案：通过 KTransformers 与 LlamaFactory 的结合，仅需1~4张RTX 4090与较高内存CPU，便可微调 DeepSeek-V3&V4系列/Kimi-K2.5/GLM-5.2等1T规模的 MoE（Mixture of Experts，混合专家）模型。

为给大家提供便捷高效的使用方式，KTransformers 与 LlamaFactory 合作，保持您的工作流不受影响，最大化支持矩阵。如下图所示，LlamaFactory 是整个微调流程的统一调度与配置框架，负责数据处理、训练调度、LoRA 插入与推理接口管理；GPU 运行 Attention、Shared Expert 等模块，KTransformers 接管位于 CPU 与大内存中的 Routed Experts，实现 GPU+CPU 异构协同。

**技术简介：**如下图，KTransformers 通过将 Attention、共享模块等计算密集部分放在 GPU，将体积大但每次只激活少量的 Routed Experts 放入 CPU 内存。避免了大量专家权重在 PCIe 上面反复传输，通过 GPU+CPU 协同计算的方式，最大化性能。CPU 则根据权重格式和硬件能力自动适配了各种后端。

**指南简介：**首次上手建议直接选择 原精度 BF16（16 位浮点）+ LoRA/全量 + kt_backend: auto：无需转换权重，修改两份 YAML 配置文件即可启动。模型提供原生 FP8 checkpoint 且选择 LoRA 时，可直接使用原生 FP8 Expert 权重降低主机内存占用。仅在系统内存仍然紧张且已准备好 KT 转换权重时，再考虑 INT8 或 AMXINT4 量化方案。

阅读导航

• 硬件检查与安装
• 选择微调方案
• Custom YAML：高级参数
• 性能与资源估算
• 附录：完整配置与 Q&A

1. 硬件检查与安装

1.1 检查硬件资源

先检查 CPU 指令集，确认可用的加速后端：

lscpu | grep -i -E 'Model name|Socket|NUMA|avx512|amx'

再检查 GPU 显存、系统内存与磁盘空间：

nvidia-smi
free -h
df -h /data

根据检查结果确认两件事：

• 指令集匹配： 原生 FP8 需要具备 AVX512F、AVX512_BF16、AVX512_VNNI、AVX512_VBMI 的兼容 AMD/x86 CPU；INT8 可由 auto 在 AMX-INT8 与 AVX512-VNNI 实现之间选择；AMXINT4 需要 AMX 与匹配的转换权重；原生 BF16 在支持 AMX 时性能最佳。
• 容量充足： 系统内存要容纳 Expert 权重，同时为 activation（激活值）、梯度和 optimizer state（优化器状态）留出空间；磁盘还要容纳 checkpoint 与临时文件。

1.2 安装环境

推荐使用干净的 Python 3.11 环境。先固定 PyTorch 2.9.1，再安装 LlamaFactory；KT 依赖必须最后安装，避免标准版 transformers 或 accelerate 覆盖 KT 定制版本：

conda create -n kt-sft python=3.11 -y
conda activate kt-sft

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory

python -m pip install torch==2.9.1 torchaudio==2.9.1 torchvision==0.24.1
python -m pip install -e .
python -m pip install "ktransformers[sft]==0.7.0"
python -m pip install "sglang-kt==0.7.0"

安装完成后检查依赖关系，并核对实际导入版本：

python -m pip check

python - <<'PY'
from importlib.metadata import version

import accelerate
import ktransformers
import kt_kernel
import torch
import transformers

for name, module in {
    "torch": torch,
    "transformers": transformers,
    "accelerate": accelerate,
    "kt_kernel": kt_kernel,
    "ktransformers": ktransformers,
}.items():
    print(f"{name:14s} {getattr(module, '__version__', 'unknown')}")

print(f"{'sglang-kt':14s} {version('sglang-kt')}")
print(f"{'transformers-kt':14s} {version('transformers-kt')}")
print(f"{'accelerate-kt':14s} {version('accelerate-kt')}")

from accelerate.utils.dataclasses import KTransformersPlugin  # noqa: F401
PY

当前兼容版本的输出应包括：

No broken requirements found.
torch          2.9.1
transformers   5.6.0
accelerate     1.14.0
kt_kernel      0.7.0
ktransformers  0.7.0
sglang-kt      0.7.0
transformers-kt 5.6.0.post2
accelerate-kt  1.14.0.post2

2. 选择微调方案

先确定 Expert 权重格式，再决定使用 LoRA 还是全量微调。选定后，直接使用本节对应的训练 YAML 和 Accelerate YAML 基础配置。

2.1 选择 Expert 权重格式

“原精度”与“量化”描述 Expert 权重格式；AMX（Advanced Matrix Extensions）与 AVX512（Advanced Vector Extensions 512-bit）描述 CPU 后端指令集，两者不是同一个维度。

系统内存允许时优先选择原生 BF16 或原生 FP8。只有已经准备好匹配的转换权重，并且确实需要降低 Expert 内存占用时，再选择 INT8 或 AMXINT4。

2.2 选择 LoRA 或全量微调

Expert 权重格式与参数更新范围是两个配置维度，但当前可用组合有明确边界。原生 BF16 支持 LoRA 与全量微调；原生 FP8、INT8 和 AMXINT4 配方用于冻结基座 Expert 的 LoRA。全量微调使用 BF16。

2.3 两份 YAML 的分工

一次训练使用两份 YAML。训练 YAML 描述训练任务并持有全部 KTransformers 设置；Accelerate YAML 只描述分布式与 FSDP2 运行方式。当前 LlamaFactory 会拒绝 Accelerate YAML 中的 kt_config。

启动前核对三个关系：

1. CUDA_VISIBLE_DEVICES 中的 GPU 数必须等于 num_processes；
2. 手动设置训练 YAML 中的 kt_config.kt_model_max_length 时，它必须覆盖 cutoff_len，并为运行时额外 token 留出余量；
3. LoRA rank 只写训练 YAML 顶层，LlamaFactory 会派生 KT 内部字段；全量微调删除 LoRA 专用字段。

2.4 四种方案的基础 YAML

先从附录 A.1 和 A.2 复制完整配置，再按所选方案替换下面的字段。每个代码块用注释区分训练 YAML 与 Accelerate YAML；kt_config 始终放在训练 YAML。

2.4.1 原生 BF16 + LoRA

# 文件 1：训练 YAML
model_name_or_path: /data/models/Your-BF16-Model
finetuning_type: lora
lora_rank: 8
use_kt: true
# 原生 BF16 不设置 kt_weight_path
kt_config:
  kt_backend: auto
  kt_expert_weight_format: bf16

# 文件 2：Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.4.2 原生 FP8 + LoRA

# 文件 1：训练 YAML
model_name_or_path: /data/models/Your-Native-FP8-Model
finetuning_type: lora
lora_rank: 8
use_kt: true
# 原生 FP8 不设置 kt_weight_path
kt_config:
  kt_backend: auto
  kt_expert_weight_format: fp8

# 文件 2：Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.4.3 INT8 / AMXINT4 + LoRA

# 文件 1：训练 YAML
finetuning_type: lora
lora_rank: 8
use_kt: true
# INT8 方案
kt_weight_path: /data/models/Your-Routed-Experts-INT8
kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16
kt_config:
  kt_backend: auto
  kt_expert_weight_format: int8
  kt_weight_lifecycle: persistent

# AMXINT4 方案改为匹配的权重目录，并将上面的 kt_config 替换为：
# kt_weight_path: /data/models/Your-Model-AMXINT4
# kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16
# kt_config:
#   kt_backend: AMXINT4

# 文件 2：Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.4.4 原生 BF16 + 全量微调

# 文件 1：训练 YAML
finetuning_type: full
learning_rate: 1.0e-5
use_kt: true
# 删除 lora_rank、lora_alpha、lora_dropout、lora_target 等 LoRA 字段
kt_config:
  kt_backend: auto
  kt_expert_weight_format: bf16

# 文件 2：Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.5 启动

CUDA_VISIBLE_DEVICES=0,1 accelerate launch --main_process_port 0 --config_file qwen35_fsdp2_2gpu.yaml src/train.py qwen35_397b_bf16_lora.yaml

3. Custom YAML：高级参数

基础方案能够启动后，再按训练目标和资源瓶颈调整下面的参数。没有明确需求时保留推荐值，不要同时修改多组参数。

（1）LoRA 容量：lora_rank、lora_alpha、lora_target、lora_dropout

这四项只在训练 YAML 顶层修改，LlamaFactory 会自动派生 KTransformers 内部使用的 rank、alpha 和 dropout；不要在 kt_config 或 Accelerate YAML 中重复填写。建议从 lora_rank: 8、lora_alpha: 16、lora_target: all、lora_dropout: 0.0 开始。

任务适配不足时依次把 rank 提高到 16、32，并让 alpha 保持约为 rank 的两倍。小数据集出现明显过拟合时，再把 dropout 提高到 0.05。

（2）序列长度：cutoff_len

在训练 YAML 中设置，建议按大多数样本的有效 token 长度选择，不要为了少量超长样本直接采用数据集最大长度。序列越长，中间激活值和 KT 缓冲区占用越高，也就是需要的内存和显存越高。

系统会根据 cutoff_len 生成 KT 的模型长度容量。需要手动扩大时，在训练 YAML 的 kt_config 中设置 kt_model_max_length，并让它大于 cutoff_len、保留额外 token 的运行余量。

（3）Batch：per_device_train_batch_size、gradient_accumulation_steps

这两项在训练 YAML 中修改。超大 MoE 模型优先使用 per_device_train_batch_size: 1；确认显存和内存均仍有余量后再增加 micro batch，显存不足时则保持为 1，通过梯度累积扩大有效 batch。

有效 batch size = per_device_train_batch_size × GPU 数量 × gradient_accumulation_steps

（4）Activation 重计算与 CPU Activation Reuse：disable_gradient_checkpointing、kt_cpu_activation

训练 YAML 中设置 disable_gradient_checkpointing: false 会启用 gradient checkpointing，以额外计算换取更低的 activation 内存；设为 true 会保留更多中间结果，速度可能更高，但内存占用也更大。

主机内存充足时，还可以在同一份训练 YAML 顶层设置：

kt_cpu_activation: retain

该选项在 checkpoint 重计算期间保留 CPU Expert activation，以增加主机内存占用换取更少的 CPU 重复计算。启用 gradient checkpointing 且未填写该字段时，CPU 与 GPU activation 默认重计算。kt_cpu_activation 与 disable_gradient_checkpointing 分别控制 CPU Expert activation 保留和整体 checkpoint 重计算，不要将两者合并成一个开关。

（5）CPU Expert 复用与线程：kt_share_backward_bb、kt_num_threads

这两项位于训练 YAML 的 kt_config。保持 kt_share_backward_bb: true，复用 CPU Expert 反向计算所需的缓冲；它不等同于 kt_cpu_activation。除非当前模型的配置说明明确要求，否则不要关闭。

kt_num_threads 填写本次作业实际可用的物理 CPU 核数，不要直接填写包含超线程的逻辑核总数。若数据预处理也使用同一批 CPU，需为 DataLoader 和系统进程预留部分核心。

kt_threadpool_count 与 NUMA 拓扑由系统生成；只有确认 CPU socket 和 NUMA 划分后才手动修改。

（6）断点续训：resume_from_checkpoint

在训练 YAML 中设置，值指向需要恢复的 checkpoint-* 目录；新训练不填写。output_dir 应继续指向本次任务的输出目录，不得与 base model 目录重叠，并保留 overwrite_output_dir: false，避免覆盖已有结果。

4. 性能与资源估算

下表统一采用 2K context（sequence length = 2048）、每卡 batch size 1、梯度累积 1，并只统计完成真实 LoRA 训练的 KTransformers BF16 结果。吞吐量包含 forward、loss、backward 和 optimizer。

显存列取该次运行中占用最高的单卡峰值；系统内存列取训练进程树 CPU RSS 峰值，并统一换算为 GiB。它们是已测配置的最低容量参考，不是安全余量：正式训练还要为数据加载、缓存、保存 checkpoint 和系统进程预留空间。

不同 GPU 数的结果不能直接用来判断线性扩展效率。增大 cutoff_len、batch size 或 KT cache 深度都会增加资源占用；全量微调还需要额外容纳梯度、master weight、optimizer state 与完整 checkpoint。

附录

A.1 Qwen3.5-397B BF16 LoRA 训练 YAML

保存为 qwen35_397b_bf16_lora.yaml：

### model
model_name_or_path: /data/models/Qwen3.5-397B-A17B
trust_remote_code: true
disable_gradient_checkpointing: false

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.0
lora_target: all

### dataset
dataset: your_dataset
template: qwen3_5
cutoff_len: 2048
packing: false
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output
output_dir: /data/output/qwen35-bf16-lora
logging_steps: 10
save_strategy: steps
save_steps: 500
plot_loss: true
overwrite_output_dir: false
save_only_model: false
report_to: none

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 1.0e-4
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### ktransformers
use_kt: true
# kt_cpu_activation: retain  # 可选：以更多主机内存减少 CPU Expert 重计算
# kt_weight_path: /data/models/Your-Routed-Experts-INT8  # 转换 INT8 Routed Experts
# kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16  # 与上面权重匹配
kt_config:
  kt_expert_weight_format: bf16
  kt_backend: auto
  kt_num_threads: 64
  kt_tp_enabled: true
  kt_threadpool_count: 2
  kt_max_cache_depth: 2
  kt_share_backward_bb: true

A.2 FSDP2 Accelerate YAML

保存为 qwen35_fsdp2_2gpu.yaml：

compute_environment: LOCAL_MACHINE
distributed_type: FSDP
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_reshard_after_forward: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_version: 2
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
use_cpu: false

A.3 Q&A

A.3.1 原生 FP8 为什么不能选择 INT8 转换权重配置

两者的权重格式和加载方式不同。原生 FP8 保留 checkpoint 中的 FP8 权重与 scale，使用 AVX512 原精度后端；INT8 配方读取相互匹配的 Routed Expert INT8 权重与 BF16 non-expert cache，并由 auto 选择 AMX-INT8 或 AVX512-VNNI 实现。不要把原生 FP8 checkpoint 填入 kt_weight_path。

A.3.2 auto 没有选择预期后端

保存 lscpu 输出和启动日志，确认容器或虚拟化环境没有屏蔽指令集。原生 BF16 可在支持 AMX 时选择 AMXBF16；原生 FP8 应选择 AVX512。不要强制指定本机或权重格式不支持的后端。

A.3.3 CPU 系统内存不足

优先降低并发数据加载、序列长度和 cache 深度，并确认没有重复加载两份 Expert 权重。若仍不足，再考虑准备 INT8 / AMXINT4 转换权重。全量微调还要单独估算梯度、master weight 与 optimizer state。

A.3.4 GPU OOM

OOM（Out of Memory）表示显存或系统内存不足。先降低 cutoff_len、每卡 batch size 和 activation 占用，再增加 gradient_accumulation_steps 维持全局 batch size。num_processes、可见 GPU 数与模型实际放置必须一致。

A.3.5 kt_model_max_length 与 cutoff_len 不一致

增大 cutoff_len 时同步提高 kt_model_max_length，并按分布式运行时的有效序列长度留出余量。值过小会导致 buffer 或 shape 错误，过大会增加系统内存占用。

Qwen3.5 MoE KT LoRA Serving with SGLang-KT

Last updated: 2026-06-23

This guide documents the current KT-FT loop for Qwen3.5 MoE: train with KT SFT, convert the output once, and serve the fine-tuned result through SGLang with a single merged adapter path.

KT SFT raw output
  -> convert_kt_to_sglang_adapter.py
  -> <MERGED_ADAPTER_DIR>
  -> sglang --lora-paths <name>=<MERGED_ADAPTER_DIR>
  -> server auto-splits expert / non-expert internally
  -> request model=<served_model>:<name>

Training-side KT SFT docs remain separate. This page focuses on the bridge from trained LoRA artifacts to online inference.

1. Scope

Current supported and validated workflow:

• Base model: Qwen3.5 MoE, for example Qwen3.5-35B-A3B
• KTransformers version: v0.6.3 or newer
• KT expert weights: AMX/BF16 SFT-compatible KT CPU expert path
• User-facing serving input: one converted merged adapter directory
• Runtime split: expert LoRA goes to the KT CPU expert path; non-expert LoRA goes to SGLang’s LoRA manager. This split happens automatically at server startup.
• This workflow is for KT MoE expert LoRA artifacts. Standard dense-model PEFT LoRA adapters usually do not need this converter.

2. Artifacts At Each Stage

Raw KT SFT output

After LLaMA-Factory + KT training, the output directory contains two LoRA artifacts:

<KT_SFT_OUTPUT_DIR>/
  adapter_model.safetensors      # non-expert LoRA
  fused_expert_lora.safetensors  # expert LoRA in KT fused format
  adapter_config.json

Do not pass this raw directory directly to SGLang serving.

Converted merged adapter

Run the converter once to produce the serving input:

<MERGED_ADAPTER_DIR>/
  adapter_config.json
  adapter_model.safetensors

This merged directory contains both expert and non-expert LoRA tensors in one PEFT-style adapter. Pass only this directory to --lora-paths.

3. Convert Once

python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  <KT_SFT_OUTPUT_DIR> \
  <MERGED_ADAPTER_DIR> \
  --base-model-name-or-path /path/to/Qwen3.5-35B-A3B \
  --overwrite

Example:

python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  saves/KT_FT_qwen35B_Moe_custom \
  saves/KT_FT_qwen35B_Moe_custom_sglang \
  --base-model-name-or-path /mnt/data3/models/Qwen3.5-35B-A3B \
  --overwrite

The converter reads fused_expert_lora.safetensors and the existing non-expert adapter_model.safetensors, then writes one merged adapter directory.

If the raw KT SFT output does not contain an adapter_config.json with lora_alpha, pass --lora-alpha <value> explicitly. The converter does not fold LoRA scaling into the tensors; runtime scaling remains lora_alpha / r.

Optional split outputs for debugging:

python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  <KT_SFT_OUTPUT_DIR> \
  <MERGED_ADAPTER_DIR> \
  --base-model-name-or-path /path/to/Qwen3.5-35B-A3B \
  --expert-output-dir <EXPERT_ADAPTER_DIR> \
  --nonexpert-output-dir <NONEXPERT_ADAPTER_DIR> \
  --overwrite

For normal serving, only <MERGED_ADAPTER_DIR> is needed.

4. Launch SGLang

Use the KTransformers SGLang fork from this repository and point PYTHONPATH at both kt-kernel/python and third_party/sglang/python.

cd /path/to/ktransformers

PYTHONPATH=/path/to/ktransformers/kt-kernel/python:/path/to/ktransformers/third_party/sglang/python:$PYTHONPATH \
python -m sglang.launch_server \
  --host 127.0.0.1 \
  --port 30006 \
  --model-path /path/to/Qwen3.5-35B-A3B \
  --tokenizer-path /path/to/Qwen3.5-35B-A3B \
  --kt-weight-path /path/to/Qwen3.5-35B-A3B-AMXINT4 \
  --kt-method AMXINT4 \
  --kt-cpuinfer 60 \
  --kt-threadpool-count 2 \
  --kt-numa-nodes 0 1 \
  --kt-num-gpu-experts 0 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 4096 \
  --max-running-requests 2 \
  --max-total-tokens 32000 \
  --served-model-name qwen3.5-kt-ft \
  --enable-mixed-chunk \
  --tensor-parallel-size 4 \
  --enable-p2p-check \
  --disable-cuda-graph \
  --disable-custom-all-reduce \
  --enable-lora \
  --lora-backend triton \
  --lora-paths qwen35b_lora=/path/to/KT_FT_qwen35B_Moe_custom_sglang \
  --log-level info

Important points:

• Pass only one merged adapter through --lora-paths.
• Do not also pass --kt-expert-lora-path in the normal user workflow.
• At startup, the server detects the merged KT MoE adapter, splits it internally, and writes runtime cache directories under $TMPDIR/sglang_kt_lora_cache/ (or $SGLANG_KT_LORA_CACHE_DIR if set).
• Prefer --lora-backend triton for Qwen3.5 full-LoRA generation.

Current constraints:

• single merged KT composite adapter only
• --kt-num-gpu-experts 0
• do not enable --kt-enable-dynamic-expert-update
• do not use --kt-gpu-prefill-token-threshold
• --max-running-requests must be at least 2
• use an AMX/BF16 SFT-compatible KT method such as AMXINT4, AMXINT8, AMXBF16, or BF16

5. Request Semantics

The OpenAI-compatible request model field uses names, not paths.

--served-model-name qwen3.5-kt-ft
--lora-paths qwen35b_lora=/path/to/merged_adapter

Request behavior in the current single-adapter implementation:

model=qwen3.5-kt-ft
=> base + KT expert LoRA

model=qwen3.5-kt-ft:qwen35b_lora
=> base + KT expert LoRA + SGLang non-expert LoRA

The suffix after : must match the left-side name in --lora-paths.

If you need a true base-only comparison, launch a separate server without --lora-paths.

6. Smoke Test

curl -sS http://127.0.0.1:30006/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.5-kt-ft:qwen35b_lora",
    "messages": [{"role": "user", "content": "Explain what LoRA is in one sentence."}],
    "temperature": 0.7,
    "max_tokens": 160,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Startup logs should include lines similar to:

Prepared merged KT LoRA adapter ... for runtime: expert=... nonexpert=...
Loaded KT expert LoRA for layer ...
Using triton as backend of LoRA kernels.

7. Advanced: Manual Split Serving

The older split-runtime contract is still available for debugging:

--kt-expert-lora-path <EXPERT_ADAPTER_DIR> \
--enable-lora \
--lora-paths <NONEXPERT_LORA_NAME>=<NONEXPERT_ADAPTER_DIR>

This is not the recommended user-facing path. Normal users should pass one merged adapter directory through --lora-paths only.

8. Troubleshooting

Got LoRA adapter that has never been loaded: lora0

The adapter name in the request must match the left side of --lora-paths. If you launched with qwen35b_lora=..., request model=qwen3.5-kt-ft:qwen35b_lora, not :lora0.

No visible adapter effect

Make sure you are serving the converted merged adapter directory produced by the converter, not the raw KT SFT output directory or a different test adapter.

connection refused

Check that the server is listening on the port you curl, and remember the example above binds to 127.0.0.1, not 0.0.0.0.

Server resolves upstream SGLang instead of this checkout

python - <<'PY'
import inspect
import sglang.srt.models.qwen3_5 as qwen3_5
print(inspect.getfile(qwen3_5))
PY

The path should come from this repository’s third_party/sglang.

Qwen3.5 MoE KT LoRA 的 SGLang-KT Serving

最后更新：2026-06-23

本文档描述当前 Qwen3.5 MoE 的 KT-FT 闭环：用 KT SFT 完成微调，转换一次输出，再通过 SGLang 用单个 merged adapter path 把微调结果服务化。

KT SFT 原始输出
  -> convert_kt_to_sglang_adapter.py
  -> <MERGED_ADAPTER_DIR>
  -> sglang --lora-paths <name>=<MERGED_ADAPTER_DIR>
  -> server 内部自动拆分 expert / non-expert
  -> 请求 model=<served_model>:<name>

训练侧 KT SFT 文档仍然独立维护；本文重点说明从已训练 LoRA artifacts 到在线推理的连接部分。

1. 范围

当前已验证路径：

• 基座模型：Qwen3.5 MoE，例如 Qwen3.5-35B-A3B
• KTransformers 版本：v0.6.3 或更新版本
• KT expert 权重：AMX/BF16 SFT 兼容的 KT CPU expert 路径
• 用户侧 serving 输入：一个 converted merged adapter 目录
• Runtime 内部仍会 split：expert LoRA 走 KT CPU expert path，non-expert LoRA 走 SGLang LoRA manager，但这一步对用户不可见
• 该工作流面向 KT MoE expert LoRA 产物；普通 dense 模型的标准 PEFT LoRA 通常不需要使用此转换器（converter）。

2. 各阶段产物

原始 KT SFT 输出

LLaMA-Factory + KT 训练完成后，输出目录里有两个 LoRA 文件：

<KT_SFT_OUTPUT_DIR>/
  adapter_model.safetensors      # non-expert LoRA
  fused_expert_lora.safetensors  # KT fused expert LoRA
  adapter_config.json

不要把 raw 训练目录直接传给 SGLang serving。

Convert 后的 merged adapter

converter 一次性生成 serving 输入：

<MERGED_ADAPTER_DIR>/
  adapter_config.json
  adapter_model.safetensors

这个 merged 目录同时包含 expert 和 non-expert LoRA。正常 serving 只需要传这一个目录。

3. 转换一次

python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  <KT_SFT_OUTPUT_DIR> \
  <MERGED_ADAPTER_DIR> \
  --base-model-name-or-path /path/to/Qwen3.5-35B-A3B \
  --overwrite

示例：

python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  saves/KT_FT_qwen35B_Moe_custom \
  saves/KT_FT_qwen35B_Moe_custom_sglang \
  --base-model-name-or-path /mnt/data3/models/Qwen3.5-35B-A3B \
  --overwrite

converter 会读取 fused_expert_lora.safetensors 和已有的 non-expert adapter_model.safetensors，写出一个 merged adapter 目录。

如果原始 KT SFT 输出目录没有包含带 lora_alpha 的 adapter_config.json，需要显式传入 --lora-alpha <value>。converter 不会把 LoRA scaling 折进 tensor；运行时 scaling 仍然是 lora_alpha / r。

如需调试，也可以额外输出 split 目录：

python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  <KT_SFT_OUTPUT_DIR> \
  <MERGED_ADAPTER_DIR> \
  --base-model-name-or-path /path/to/Qwen3.5-35B-A3B \
  --expert-output-dir <EXPERT_ADAPTER_DIR> \
  --nonexpert-output-dir <NONEXPERT_ADAPTER_DIR> \
  --overwrite

正常 serving 只需要 <MERGED_ADAPTER_DIR>。

4. 启动 SGLang

请使用本仓库的 KTransformers SGLang fork，并把 PYTHONPATH 指向 kt-kernel/python 和 third_party/sglang/python。

cd /path/to/ktransformers

PYTHONPATH=/path/to/ktransformers/kt-kernel/python:/path/to/ktransformers/third_party/sglang/python:$PYTHONPATH \
python -m sglang.launch_server \
  --host 127.0.0.1 \
  --port 30006 \
  --model-path /path/to/Qwen3.5-35B-A3B \
  --tokenizer-path /path/to/Qwen3.5-35B-A3B \
  --kt-weight-path /path/to/Qwen3.5-35B-A3B-AMXINT4 \
  --kt-method AMXINT4 \
  --kt-cpuinfer 60 \
  --kt-threadpool-count 2 \
  --kt-numa-nodes 0 1 \
  --kt-num-gpu-experts 0 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 4096 \
  --max-running-requests 2 \
  --max-total-tokens 32000 \
  --served-model-name qwen3.5-kt-ft \
  --enable-mixed-chunk \
  --tensor-parallel-size 4 \
  --enable-p2p-check \
  --disable-cuda-graph \
  --disable-custom-all-reduce \
  --enable-lora \
  --lora-backend triton \
  --lora-paths qwen35b_lora=/path/to/KT_FT_qwen35B_Moe_custom_sglang \
  --log-level info

要点：

• 用户只需要传一个 merged adapter：--lora-paths <name>=<MERGED_ADAPTER_DIR>
• 正常 workflow 不要再额外传 --kt-expert-lora-path
• server 启动时会自动识别 merged KT MoE adapter，并在 $TMPDIR/sglang_kt_lora_cache/（或 $SGLANG_KT_LORA_CACHE_DIR）下生成 runtime cache
• Qwen3.5 full LoRA 生成优先使用 --lora-backend triton

当前限制：

• 只支持单个 merged KT composite adapter
• --kt-num-gpu-experts 0
• 不启用 --kt-enable-dynamic-expert-update
• 不使用 --kt-gpu-prefill-token-threshold
• --max-running-requests 必须至少为 2
• 使用 AMX/BF16 SFT 兼容 KT method，例如 AMXINT4、AMXINT8、AMXBF16、BF16

5. 请求语义

OpenAI-compatible 请求里的 model 字段用 name，不用 path。

--served-model-name qwen3.5-kt-ft
--lora-paths qwen35b_lora=/path/to/merged_adapter

当前 single-adapter 实现的请求语义：

model=qwen3.5-kt-ft
=> base + KT expert LoRA

model=qwen3.5-kt-ft:qwen35b_lora
=> base + KT expert LoRA + SGLang non-expert LoRA

冒号后的 adapter 名必须和 --lora-paths 左侧注册名一致。

如果需要 true base-only 对照，请单独启动一个不带 --lora-paths 的服务。

6. Smoke Test

curl -sS http://127.0.0.1:30006/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.5-kt-ft:qwen35b_lora",
    "messages": [{"role": "user", "content": "用一句话解释什么是 LoRA。"}],
    "temperature": 0.7,
    "max_tokens": 160,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

启动日志里应能看到类似输出：

Prepared merged KT LoRA adapter ... for runtime: expert=... nonexpert=...
Loaded KT expert LoRA for layer ...
Using triton as backend of LoRA kernels.

7. 高级：手动 split serving

旧 split runtime 仍可用于调试：

--kt-expert-lora-path <EXPERT_ADAPTER_DIR> \
--enable-lora \
--lora-paths <NONEXPERT_LORA_NAME>=<NONEXPERT_ADAPTER_DIR>

这不是推荐的用户路径。正常用户只需要通过 --lora-paths 传一个 merged adapter 目录。

8. Troubleshooting

Got LoRA adapter that has never been loaded: lora0

请求里的 adapter 名必须和 --lora-paths 左侧一致。如果启动时写的是 qwen35b_lora=...，请求应使用 model=qwen3.5-kt-ft:qwen35b_lora，而不是 :lora0。

看不出 adapter 效果

确认 serving 使用的是 converter 生成的 merged adapter 目录，而不是原始 KT SFT 输出目录或其他测试 adapter。

connection refused

确认 server 监听的端口与 curl 一致；上面的示例绑定的是 127.0.0.1，不是 0.0.0.0。

Server 解析到了上游 SGLang，而不是当前 checkout

python - <<'PY'
import inspect
import sglang.srt.models.qwen3_5 as qwen3_5
print(inspect.getfile(qwen3_5))
PY

路径应来自本仓库的 third_party/sglang。

Tutorial: Inject Operator Step by Step

Author: Azure-Tang

TL;DR

This tutorial will guide you through the process of injecting custom operators into a model using the KTransformers framework. We will use the DeepSeekV2-Chat model as an example to demonstrate how to inject custom operators into the model step by step. The tutorial will cover the following topics:

• TL;DR
• How to Write Injection Rules
• Understanding Model Structure
• Matrix Absorption-based MLA Injection
• Injection of Routed Experts
• Injection of Linear Layers
• Injection of Modules with Pre-calculated Buffers
• Specifying Running Devices for Modules
• Muti-GPU
• How to Write a New Operator and Inject into the Model

How to Write Injection Rules

The basic form of the injection rules for the Inject framework is as follows:

- match:
    name: "^model\\.layers\\..*\\.*$"  # Target module name
    class: torch.nn.Linear  # Target module
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda:0"
      # your_op_param_1: 1234
      # your_op_param_2: 5678
  recursive: True

• match: This field marks the matching rules, which can appear in two forms, name and class. These two matching rules can appear together or separately; they only match when both criteria are met.
• replace:
  • class: Python class that can be imported to replace the target module. If no replacement is desired, set to default.
  • kwargs: List of parameters needed for module initialization.
    • generate_device: The device for this module, can be set to “cpu”, “cuda”, “cuda:1”, etc.
• recursive: Whether to recursively inject this module’s submodules, default is True.

For the recursive field: Some modules contain multiple submodules, such as the Self-attention module typically includes q/k/v/o four linear modules. If we replace the self-attention module but do not want the internal linear modules to be covered by other rules, set this rule to False.

Understanding Model Structure

Using deepseek-ai/DeepSeek-V2-Lite-Chat as an example, we can follow the above rules step by step to inject our custom module and run it. KTransformers offers a high degree of flexibility, allowing you to replace/experiment with basic operators. However, it also requires users to clearly understand the structure of the model they are running.

Fortunately, knowing the structure of a model is very simple. Open the file list on the deepseek-ai/DeepSeek-V2-Lite homepage, and you can see the following files:

From the .saftensors file, we can see the name of each layer’s weights, corresponding to the match.name attribute in the injection rules. From the modeling_deepseek.py file, we can see the specific implementation of each module class, with the class name corresponding to the match.class attribute in the injection rules.

The structure of the DeepSeekV2 model from the .saftensors and modeling_deepseek.py files is as follows:

Supported operators and their corresponding classes are as follows:

Then we start step-by-step injection of custom modules, our targets are:

• Replace the linear module with custom Marlin linear module.
• Replace the self-attention module with a custom Absorption-based MLA module.
• Replace the experts module with a custom Experts module.
• Replace the MoE module with a custom MoE module.
• Replace the RoPE module with a custom RoPE module.
• Set the running device for each module.

The full implementation of the injection rules can be found in the here.

Matrix Absorption-based MLA Injection

For the injection of the Attention module, we only need to use a regular expression to match the module names used in transformers and replace them with our own MLA module implementation. The YAML injection rule is as follows:

- match:
    name: "^model\\.layers\\..*\\.self_attn$"  # Regular expression
  replace:
    class: ktransformers.operators.attention.KDeepseekV2Attention # Optimized MLA implementation

As you can see, each rule in the YAML file has two parts: match and replace. The match part specifies the module to be replaced, and the replace part specifies the module to be injected into the model along with the initialization keywords.

Injection of Routed Experts

For Routed Experts (corresponding to the exps in the diagram), the module we inject is CPUInfer, which is wrapped in the wrapper module KTransformersExperts. KTransformersExperts has multiple implementations, and we need to specify keywords to tell the wrapper module which implementation we want to use and how we plan to use it.

In the source code of the transformer, MoE is implemented using nn.ModuleList. We do not want KTransformers to traverse all submodules in the list and inject them one by one, so in this rule, we set recursive: False to prevent recursive injection into the submodules of this module. The YAML rule is as follows:

- match:
    name: "^model\\.layers\\..*\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # Custom MoE kernel with expert parallelism
    kwargs:
      generate_device: "cpu"
      generate_op: "MLPCPUExperts"
      out_device: "cuda"
  recursive: False # Don't recursively inject submodules of this module

If we inject Routed Experts as a custom module, we cannot use the interfaces in the original nn.ModuleList. Therefore, it is necessary to modify the forward function in the FFN module. The simplest method is to implement a new module with a custom forward function and inject it.

- match:
    class: ktransformers.models.modeling_deepseek.DeepseekV2MoE
  replace:
    class: ktransformers.operators.experts.KDeepseekV2MoE     # MLP module with custom forward function

Injection of Linear Layers

For the remaining linear layer modules, we aim to use quantized operators to save storage space while improving performance. Since there is no current research on using MLA and quantization together, we do not want to inject linear into the MLA operator. Therefore, we can modify the regular expression and add a type check in the match part of the rule. Only modules that match both the name and class simultaneously will be injected. We also need to pass some keywords similar to the injection of Routed Experts. The YAML rule is as follows:

- match:
    name: "^model\\.layers\\.(?!.*self_attn).*$"  # Regular expression
    class: torch.nn.Linear  # Only match modules matching name and class simultaneously
  replace:
    class: ktransformers.operators.linear.KTransformersLinear  # Optimized kernel on quantized data types
    kwargs:
      generate_device: "cuda"
      generate_op: "QuantizedLinearMarlin"

Injection of Modules with Pre-calculated Buffers

To avoid occupying resources when initializing the injected original model, we use torch’s meta device to initialize the original model. The RoPE module pre-calculates some buffers during initialization, but no calculations are performed when using the meta device. Therefore, we need to compensate for the calculation of the buffer when loading the model. Simply, we inject a custom module into the rotary embedding module, which performs pre-calculation during loading. The YAML rule is as follows:

- match:
    class: ktransformers.models.modeling_deepseek.DeepseekV2YarnRotaryEmbedding
  replace:
    class: ktransformers.operators.RoPE.YarnRotaryEmbedding

Specifying Running Devices for Modules

Finally, we set a fallback basic attribute generate_device for all modules:

- match:
    name: "^model\\.layers\\..*\\.|^lm_head"
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda"
  
- match:
    name: "^model.embed_tokens"
  replace:
    class: "default"
    kwargs:
        generate_device: "cpu"

Through these two rules, we place all previously unmatched layers (and their submodules) and lm_head on cuda, and the embedding on cpu. Note that the properties of a module will be determined by the first rule it matches. For example, if you later set a new replace.kwargs.generate_device in an injected module, the device set earlier will take precedence. If your computer has multiple cards, you can also configure the model to multiple cards.

Muti-GPU

If you have multiple GPUs, you can set the device for each module to different GPUs. DeepseekV2-Chat got 60 layers, if we got 2 GPUs, we can allocate 30 layers to each GPU. Complete multi GPU rule examples here.

First of all, for multi-GPU, we have to inject an new operator KDeepseekV2Model. And set division of the layers to different GPUs. For our case, we have to set the transfer_map in the KDeepseekV2Model operatoras as follows:

- match:
    name: "^model$"
  replace:
    class: "ktransformers.operators.models.KDeepseekV2Model"
    kwargs:
      transfer_map: 
        30: "cuda:1"

And we have to set the device for each module in the model.

For example, for routed experts, the yaml for one GPU is:

- match:
    name: "^model\\.layers\\..*\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # Custom MoE kernel with expert parallelism
    kwargs:
      generate_device: "cuda:0"
      generate_op: "MLPCUDAExperts"
      out_device: "cuda:0"
  recursive: False # Don't recursively inject submodules of this module

But for two GPUs, we need to set the device for each module in the model.

# allcate 0-29 layers‘s out_device to cuda:0
- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:0"
  recursive: False # don't recursively inject submodules of this module

# allocate 30-59 layers‘s out_device to cuda:1
- match:
    name: "^model\\.layers\\.([345][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:1"
  recursive: False # don't recursively inject submodules of this module

For other modules, we can set the device in the same way.

How to Write a New Operator and Inject into the Model

In this section, we will explain how to write an operator that can be injected, using the implementation of a new linear as an example.

First, all injectable operators need to inherit from the BaseInjectedModule class, which inherits some attributes required by our injection framework. Its initialization function needs to meet the following basic format:

class LinearTorchInject(BaseInjectedModule):
    def __init__(
        self,
        key: str,
        gguf_loader: GGUFLoader,
        config: PretrainedConfig,
        orig_module: nn.Module = None,
        generate_device: str = "cuda",
        **kwargs,
    ):
        super().__init__(key, gguf_loader, config, orig_module, generate_device, **kwargs)

If users have other parameters that need to be passed to this class, they can also be included in the init function and re-passed in the kwargs parameter in the yaml file. For example, if our operator wants to pass a parameter my_param, the init function can be written as:

class LinearTorchInject(BaseInjectedModule):
    def __init__(
        self,
        key: str,
        gguf_loader: GGUFLoader,
        config: PretrainedConfig,
        orig_module: nn.Module = None,
        generate_device: str = "cuda",
        my_param: bool = True,
        **kwargs,
    ):
        super().__init__(key, gguf_loader, config, orig_module, generate_device, **kwargs)
        self.my_param = my_param

Then our injection rule can be written as:

- match: 
    name: "^model\\.layers\\..*$"  # Regular expression matches the module name.
    class: torch.nn.Linear  # Type restrictions can be added.
  replace:
    class: ktransformers.operators.linear.LinearTorchInject  # Inject module path
    kwargs: # Extra parameters
      generate_device: "cuda"
      my_param: True

For the linear module, it is also necessary to read weights from a gguf file. We provide the KLinearBase class to help users read weights from gguf files. Users only need to inherit and implement the load, unload, and forward functions. Therefore, a fully injectable linear class would look like this:

class LinearTorchInject(BaseInjectedModule, KLinearBase):
    def __init__(
        self,
        key: str,
        gguf_loader: GGUFLoader,
        config: PretrainedConfig,
        orig_module: nn.Module = None,
        generate_device: str = "cuda",
        **kwargs,
    ):
        super().__init__(key, gguf_loader, config, orig_module, generate_device, **kwargs)
        KLinearBase.__init__(self)
        self.has_bias = False
        self.dtype = torch.get_default_dtype()
        self.w = None
        self.has_bias = False
    
    def load(self, w: dict | nn.Parameter | tuple | None = None, device: str|None = None):
        if device is None: device = self.device
        if w is None: w = self.load_weight(device=device)

        if isinstance(w, nn.Parameter):
            self.w = w.to(dtype=self.dtype).view(self.out_features, self.in_features).T
            self.has_bias = False
        elif isinstance(w, tuple):
            self.w = w[0].to(dtype=self.dtype).view(self.out_features, self.in_features).T
            self.bias = w[1].to(dtype=self.dtype)
            self.has_bias = True
        else:
            raise ValueError("Invalid weight type")
        self.w = self.w.to(device)
        if self.has_bias:
            self.bias = self.bias.to(device)

    def unload(self):
        if self.w is not None:
            self.w = None
        if self.has_bias:
            self.bias = None

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        dtype = x.dtype
        out_device = x.device
        x = x.to(device=self.device, dtype=self.dtype)
        x = x @ self.w
        if self.has_bias:
            x = x + self.bias
        x = x.to(dtype=dtype, device=out_device)
        return x

Note that the self.load_weight function is provided by the KLinearBase class to help users load weights from a gguf file into the module. The implementation details of KLinearBase can be found on GITHUB.

• Introduction
• Overall View of the KT Fine-Tuning Framework
  • Attention (LoRA + KT coexist)
  • MoE (operator encapsulation + backward)
  • Multi-GPU Loading/Training: Placement strategy instead of DataParallel
• KT-LoRA Fine-Tuning Evaluation
  • Setup
  • Results
  • Speed Tests
  • Memory Footprint
• Conclusion

KTransformers Fine-Tuning × LLaMA-Factory Integration – Developer Technical Notes

MadSys Lab, KVCache-AI Team, Approaching AI, LLaMA-Factory Team

Introduction

Recent open-source LLMs—from DeepSeek-V3/R1 to Qwen-MoE and Kimi-K2—have surged in performance and scale. Yet due to compute and memory constraints, it is difficult for typical researchers to fine-tune trillion-parameter-class models. We therefore integrate KTransformers with LLaMA-Factory so that, with 2–4 RTX 4090 GPUs and sufficient CPU memory, one can fine-tune ultra-large Mixture-of-Experts (MoE) models such as DeepSeek-671B.

This architecture bridges resource gaps, enabling local fine-tuning of ultra-large models, while also supporting efficient scenario customization at 14B/30B scales. We validate on stylized dialogue, Westernized translation tone, and medical Q&A, achieving rapid adaptation within hours.

Architecturally, LLaMA-Factory orchestrates data/config/training, LoRA insertion, and inference; KTransformers is a pluggable, high-performance operator backend that takes over Attention and MoE under the same training code, enabling GPU+CPU heterogeneity to accelerate training and reduce GPU memory.

We evaluated LoRA fine-tuning with HuggingFace default, Unsloth, and KTransformers backends (same settings and data). KTransformers is currently the only solution feasible on 2–4×24GB 4090s for 671B-scale MoE, and also shows higher throughput and lower GPU memory for 14B MoEs.

† The 1400 GB is the theoretical FP16 full-resident footprint (not runnable). 70 GB is the measured peak with KT (Attention on GPU + layered MoE offload).

From the table above, it can be seen that for the 14B model, the KTransformers backend achieves approximately 75% higher throughput than the default HuggingFace solution, while using only about one-fifth of the GPU memory. For the 671B model, both HuggingFace and Unsloth fail to run on a single 4090 GPU, whereas KTransformers is able to perform LoRA fine-tuning at 40 tokens/s, keeping the GPU memory usage within 70 GB.

Overall View of the KT Fine-Tuning Framework

We detail how KTransformers takes over core operators in LLaMA-Factory’s fine-tuning framework to optimize Attention and MoE.

DeepSeek-V3/V2 MoE models comprise a small-parameter dense Attention part and a large-parameter sparse MoE part. For illustration, consider layer 2 of DeepSeek-V2-Lite-Chat (from which each layer includes both Attention and MoE). Attention compute and KV cache mainly reside on the GPU; the heavyweight MoE part is primarily executed on the CPU. We first cover Attention replacement and inheritance, then MoE encapsulation and backend interfacing, and finally multi-GPU placement.

Attention (LoRA + KT coexist)

KTransformers provides operator injection (BaseInjectedModule), and PEFT provides LoRA layer insertion. For fine-tuning, we design KTransformersLinearLora, inheriting from both KTransformersLinear and LoraLayer:

• Inheritance: KTransformersLinearLora retains KT’s high-performance paths (prefill_linear/generate_linear) while accepting LoRA parameters (lora_A/lora_B).
• Replacement: During preparation, we replace original KTransformersLinear layers (Q/K/V/O) with KTransformersLinearLora, preserving KT optimizations while enabling LoRA trainability.

After replacement, LoRA is inserted at Q/K/V/O linear transforms (left), and KTransformersLinearLora contains both KT fast paths and LoRA matrices (right).

MoE (operator encapsulation + backward)

Encapsulation

Given large parameters and sparse compute, we encapsulate the expert computation as a differentiable black-box operator—transparent upstream, replaceable downstream.

• Upstream (PyTorch graph): we register a custom Autograd Function so the MoE layer appears as a single node. In the left figure (red box), only KSFTExpertsCPU is visible; on the right, the unencapsulated graph expands routing, dispatch, and FFN experts. Encapsulation makes the MoE layer behave like a standard nn.Module with gradients.
• Downstream (backend): inside the Autograd Function, pybind11 calls C++ extensions for forward/backward. Multiple pluggable backends exist (AMX BF16/INT8; llamafile). The backend can be switched via YAML (e.g., "backend": "AMXBF16" vs. "llamafile").

Backward (CPU)

MoE backward frequently needs the transposed weights $W^\top$. To avoid repeated runtime transposes, we precompute/cache $W^\top$ at load time (blue box). We also cache necessary intermediate activations (e.g., expert projections, red box) to reuse in backward and reduce recomputation. We provide backward implementations for llamafile and AMX (INT8/BF16), with NUMA-aware optimizations.

Multi-GPU Loading/Training: Placement strategy instead of DataParallel

To lower per-GPU memory peaks on 2–4 GPUs, we use model parallelism + explicit placement, not DataParallel (which duplicates the whole model on each GPU).

Key changes:

1. KTrainer: takes over .to(device) to prevent “move whole model to a single GPU”. Using KT’s optimize-rule YAML, each layer declares device: cuda:0/cuda:1/... and is constructed directly on the target GPU (no extra copies).
2. Disable automatic DataParallel: when USE_KT=1, we disable automatic DP wrappers from LLaMA-Factory/HF Trainer to avoid duplication and keep full control over sharding.
3. Gradient aggregation: gradients are reduced to cuda:0. Intermediate activations stay local; only necessary tensors are transferred, cutting communication/activation overhead.

Thus, we keep KT placement strategies under multi-GPU fine-tuning. Users choose a kt_optimize_rule with multi-gpu. For DeepSeek-671B, DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml is a typical 2-GPU plan: KV/attention parts on each GPU; MoE experts sharded on CPU; both GPUs share the workload.

KT-LoRA Fine-Tuning Evaluation

Setup

LLaMA-Factory orchestration, KTransformers backend, LoRA (rank=8, α=32, dropout=0.1, BF16), GAS=16, qlen=512, with the same KT optimize rule as training. We evaluate (a) stylized dialogue transfer and (b) two small-scale representative benchmarks: Translational-Style (generative) and AfriMed-QA (medical vertical; SAQ and MCQ). AMX is enabled; GPUs: 2×48GB RTX 4090; CPU: Intel Xeon Platinum 8488C.

Results

Stylized Dialogue (CatGirl tone)

Dataset: NekoQA-10K. The fine-tuned model consistently exhibits the target style (red boxes) versus neutral/rational base (blue). This shows KT-LoRA injects style features into the generation distribution with low GPU cost.

Translational-Style benchmark (generative)

Dataset: Translational-Style-ChatLLM. Metrics: BLEU-1/2/3/4, ROUGE-1/2/L.

As shown by the test results in the tables above, under a unified workflow and placement strategy, both model scales exhibit consistent gains after fine-tuning, supporting the usability and effectiveness of the “KT backend + LoRA fine-tuning” combination for generative style control. At the same time, this indicates that KT’s heterogeneous placement and operator optimizations can stably support small-sample adaptation in the style domain.

Medical Vertical Benchmark (AfriMed-SAQ/MCQ)

The dataset adopts AfriMed-QA (ACL 2025), a domain-specific dataset for the medical field in Africa with strong scenario customization characteristics, comprising two formats—multiple-choice questions (MCQ) and short-answer questions (SAQ)—which in this case serve as the evaluation for vertical-domain fine-tuning. In terms of evaluation criteria, BLEU/ROUGE are used for SAQ, and Accuracy is used for MCQ.

As shown in the tables above, (1) DeepSeek-V3 (671B) after KT-LoRA fine-tuning achieves clearly higher performance than the fine-tuned DeepSeek-V2-Lite (14B) on both MCQ and SAQ, and it also surpasses the V3 base model. Within our small-scale setting, this preliminarily indicates that KT-LoRA fine-tuning of ultra-large-parameter models has practical significance in vertical domains.

(2) Across both SAQ/MCQ sub-tasks, KT-LoRA delivers consistent gains, indicating that—with KT’s heterogeneous placement and backend operator support—LoRA fine-tuning can effectively inject the key knowledge points of vertical domains such as medicine into the model.

Limitations

At present, most of our testing is conducted on single datasets and at small scale (≤ 20k examples), with the goal of providing existence evidence of system effectiveness for KT-LoRA fine-tuning, rather than drawing generalized conclusions about algorithmic generalization or scaling laws. Our report primarily presents representative figures; to support stronger algorithmic claims, larger sample sizes, multi-lingual/multi-domain datasets, and multi-seed repeated experiments would be required—these are beyond the scope of this work.

We also warmly welcome everyone to join the open-source LLaMA-Factory KT fine-tuning project. If you have additional test results, we especially welcome you to record them in the shared spreadsheet below, and to include the corresponding kt_optimize_rule files, dataset examples, training/evaluation YAMLs, and detailed GPU-memory and CPU configurations for community reference and reproducibility~!

Speed Tests

End-to-End Performance

Definitions

step_time：time per optimization step (tensor movement + Attention + MoE + others).

tokens_per_step = GAS × qlen；token/s = tokens_per_step / step_time。 We use GAS=16, qlen=512 → tokens_per_step=8192.

Measured

MoE Compute (DeepSeek-V3-671B)

Theory

• MoE per-layer, per-token FLOPs (forward+backward) approx.: $$ \text{FLOPs}_{\text{per-layer, per-token}} \approx c \cdot k \cdot H \cdot I $$

​ with $k = 8$（Top-k），$H = 7168$（hidden size），$I = 2048$（intermediate size），$c\approx16$（≈6 forward + ≈10 backward matmuls）。

• Per-step across all MoE layers: $$ \text{FLOPs}{\text{per-step}} \approx c \cdot qlen \cdot k \cdot H \cdot I \cdot L{\text{MoE}} $$

​ Plugging $c=16, qlen=512, k=8, H=7168, I=2048, L_{MoE}=58$，$\text{FLOPs}_{\text{per-step}} \approx 55.8\ \text{TFLOPs}$.

Measured (MoE TFLOPS on CPU)

If the MoE-only time per step is t_moe (seconds), $\text{TFLOPS} = \text{FLOPs}_{\text{per-step}} / \text{step_per_second}.$

Use MoE-phase time, not full step_time, to get MoE throughput.

Memory Footprint

• DeepSeek-V3 (671B; 58 MoE layers out of 61): ~70 GB total GPU, ~1.2–1.3 TB host memory.
• DeepSeek-V2-Lite (14B; 26 MoE layers out of 27): ~5 GB GPU, ~30 GB host memory.

Conclusion

Integrating KTransformers LoRA with LLaMA-Factory provides a practical path to efficiently train and deploy MoE LLMs. KT contributes placement strategies and operator optimizations (DeepSeek/Qwen/Kimi support with AMX-accelerated kernels), and LoRA enables customization with very low GPU memory; LLaMA-Factory supplies a coherent user-level interface.

This means even tens-to-hundreds-of-billion-parameter MoE models can be fine-tuned and served with low latency on ordinary hardware. The approach balances memory savings, speed, and usability, turning ultra-large models into tools that developers can actually wield.

DPO Training with LLaMA-Factory

This tutorial demonstrates how to use Direct Preference Optimization (DPO) to fine-tune a language model using the LLaMA-Factory framework. DPO is a method for training models based on human preferences, allowing for more aligned and user-centric outputs.

Installation

Step 1: Create a conda environment and suit it for KTransformers

conda create -n Kllama python=3.12 # choose from : [3.11, 3.12, 3.13]
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
conda install -y -c nvidia/label/cuda-12.8.0 cuda-runtime

Step 2: Install the LLaMA-Factory environment

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation

Step 3: Install KTransformers

Option 1: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.4

(Note: The CUDA version can differ from that in the wheel filename.)

pip install ktransformers-0.4.4+cu128torch28fancy-cp312-cp312-linux_x86_64.whl

Option 2: Install KTransformers from source

git clone --depth 1 https://github.com/kvcache-ai/ktransformers.git
cd ktransformers/kt-sft
export TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0" # set according to your GPU

pip install -r "requirements-sft.txt"
KTRANSFORMERS_FORCE_BUILD=TRUE pip install -v . --no-build-isolation

Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases

# abi=True/False can find from below
# import torch
# print(torch._C._GLIBCXX_USE_CXX11_ABI)

pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl

Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton)

git clone https://github.com/kvcache-ai/custom_flashinfer.git
pip install custom_flashinfer/

Prepare Models

We use deepseek-ai/DeepSeek-V2-Lite as an example here. You can replace it with other models such as Kimi K2.

How to start

# For LoRA SFT
USE_KT=1 llamafactory-cli train examples/train_lora/deepseek2_lora_dpo_kt.yaml
# For Chat with model after LoRA SFT
llamafactory-cli chat examples/inference/deepseek2_lora_dpo_kt.yaml
# For API with model after LoRA SFT
llamafactory-cli api examples/inference/deepseek2_lora_dpo_kt.yaml

For example, we provide the YAML file as follows:

（1）examples/train_lora/deepseek2_lora_dpo_kt.yaml

### model
model_name_or_path: deepseek-ai/DeepSeek-V2-Lite
trust_remote_code: true

### method
stage: dpo
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid  # choices: [sigmoid (dpo), orpo, simpo]

### dataset
dataset: dpo_en_demo
template: llama3
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output
output_dir: saves/Kllama_deepseekV2_DPO
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-6
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

### ktransformers
use_kt: true # use KTransformers as LoRA sft backend
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V2-Lite-Chat-sft-amx.yaml
cpu_infer: 64
chunk_size: 8192

For more details about –kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md

Then, you can use the lora adapter saved in saves/Kllama_deepseekV2_DPO for inference the same as the sft training. For example,

model_name_or_path: DeepSeek-V2-Lite-Chat 
adapter_name_or_path: saves/Kllama_deepseekV2_DPO
template: deepseek
infer_backend: ktransformers  # choices: [huggingface, vllm, sglang, ktransformers]
trust_remote_code: true

use_kt: true # use KTransformers as LoRA sft backend to inference
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V2-Lite-Chat-sft-amx.yaml
cpu_infer: 32
chunk_size: 8192

kt-kernel Docs

KT-CLI

⚠️ Note: This feature is currently under active development. Many functionalities are not yet complete and are being improved. Please stay tuned for updates.

Design Philosophy

KT-CLI is designed to minimize the burden of reading documentation. Instead of requiring users to read lengthy docs, the CLI provides:

• Interactive Mode: Run commands without arguments to get step-by-step guided prompts
• Direct Mode: Pass arguments directly for automation and scripting

  💡 Tip: The arguments are fully compatible with the previous SGLang + KTransformers approach, so you can migrate seamlessly.

Simply run a command, and the CLI will interactively guide you through the process!

Usage

You can check the usage by kt --help

kt [OPTIONS] COMMAND [ARGS]...

KTransformers CLI - A unified command-line interface for KTransformers.

Options

Commands

GLM-5.3-flash: Native KTransformers Support for 1M Context

1. 1M Context and Native Multimodality

GLM-5.3-flash natively supports a context window of up to 1M tokens. A single request can process a large codebase, a long document, or a long-running agent task without repeatedly splitting the context.

The model also supports images, video, reasoning, and tool calling, and can be used directly by coding agents through an OpenAI-compatible API. GLM-5.3-flash has approximately 321B parameters. Its 45 layers comprise 34 Linear Attention layers and 11 DSA layers.

2. Native-Precision KTransformers Support

KTransformers (KT) reads the official GLM-5.3-flash FP8 weights directly. No model conversion or additional quantization of expert weights is required.

The FP8 model occupies approximately 306 GiB. Reserve at least 350 GB of available system memory. The current implementation supports:

• NVIDIA SM89 and SM120 GPUs (RTX 40 and 50 series)
• The AVX-512 FP8 CPU expert kernel
• Heterogeneous CPU-GPU expert inference and Layerwise Prefill
• A context window of up to 1M tokens
• Multimodality: text, multiple images, video
• Tool calling

3. Installation

Use a clean Python 3.11 environment and run:

pip install "ktransformers[sglang]"

This command installs compatible versions of KT Kernel, SGLang-KT, and Transformers-KT.

4. Launch

Replace /path/to/GLM-5.3-flash with the model directory. The configurations below enable Layerwise Prefill, multimodal input, and decode CUDA Graphs by default. The model supports up to 1M context; the examples use the validated 501025-token configuration.

4.1 Four-GPU Launch

MODEL_PATH=/path/to/GLM-5.3-flash

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --kt-weight-path "$MODEL_PATH" \
  --served-model-name GLM-5.3-flash \
  --host 0.0.0.0 \
  --tp-size 4 \
  --context-length 501025 \
  --mem-fraction-static 0.60 \
  --chunked-prefill-size 4096 \
  --kt-method FP8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 40 \
  --kt-gpu-prefill-token-threshold 2048 \
  --cuda-graph-bs 1 2 4 \
  --limit-mm-data-per-request '{"image":8,"video":1}' \
  --mm-process-config '{"image":{"max_pixels":1254400}}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45

4.2 Single-GPU Launch

MODEL_PATH=/path/to/GLM-5.3-flash

CUDA_VISIBLE_DEVICES=0 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --kt-weight-path "$MODEL_PATH" \
  --served-model-name GLM-5.3-flash \
  --host 0.0.0.0 \
  --tp-size 1 \
  --context-length 501025 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 2048 \
  --kt-method FP8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 0 \
  --kt-gpu-prefill-token-threshold 2048 \
  --cuda-graph-bs 1 2 4 \
  --limit-mm-data-per-request '{"image":8,"video":1}' \
  --mm-process-config '{"image":{"max_pixels":1254400}}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45

When Layerwise Prefill is enabled for GLM-5.3-flash, the current implementation normalizes the resident GPU expert count to zero. The --kt-num-gpu-experts 40 setting in the four-GPU example therefore does not keep 40 experts resident during Layerwise Prefill.

The server listens on http://localhost:30000 by default. Check the endpoint after startup:

curl http://localhost:30000/v1/models

The OpenAI-compatible endpoint is:

http://localhost:30000/v1/chat/completions

5. Multimodal Request Boundaries

• A request may contain text only, or text with up to eight images.
• A request may instead contain text with one video.
• Images and video cannot appear in the same request.
• Video uses the regular mixed-prefill path rather than Layerwise Prefill. Decode CUDA Graphs remain supported.

GLM-5.3-flash：1M 长上下文，KTransformers 原生支持

1. 1M 长上下文与原生多模态

GLM-5.3-flash 原生支持最高 1M tokens 上下文，可以在一次请求中处理大型代码库、长篇文档和长程 Agent 任务，减少频繁切分上下文带来的信息损失。

模型还原生支持图片、视频、Reasoning 和 Tool Calling，可以直接接入使用 OpenAI 兼容接口的 Coding Agent。GLM-5.3-flash 约 321B 参数；45 层网络中包含 34 层 Linear Attention 和 11 层 DSA。

2. KTransformers 原精度支持

KTransformers（KT）直接读取 GLM-5.3-flash 的官方 FP8 权重，不需要转换模型，也不会对 Expert 权重再次量化。

FP8 模型约占 306 GiB，建议至少预留 350 GB 可用系统内存。当前实现支持：

• NVIDIA SM89 和 SM120 GPU（RTX 40、50 系列）
• TP1、TP2、TP4 和 TP8
• AVX-512 FP8 CPU Expert Kernel
• CPU-GPU Expert 异构推理和 Layerwise Prefill
• 文本、多图、单视频、Reasoning 和 Tool Calling

3. 安装

建议使用全新的 Python 3.11 环境，然后执行：

pip install "ktransformers[sglang]"

该命令会自动安装匹配的 KT Kernel、SGLang-KT 和 Transformers-KT，无需克隆源码仓库。

4. 启动

将 /path/to/GLM-5.3-flash 替换为模型目录。下面的配置默认开启 Layerwise Prefill、多模态和 Decode CUDA Graph。模型支持最高 1M 上下文；示例使用已经过验收的 501025 tokens 配置。

4.1 四卡启动

MODEL_PATH=/path/to/GLM-5.3-flash

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --kt-weight-path "$MODEL_PATH" \
  --served-model-name GLM-5.3-flash \
  --host 0.0.0.0 \
  --tp-size 4 \
  --context-length 501025 \
  --mem-fraction-static 0.60 \
  --chunked-prefill-size 4096 \
  --kt-method FP8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 40 \
  --kt-gpu-prefill-token-threshold 2048 \
  --cuda-graph-bs 1 2 4 \
  --limit-mm-data-per-request '{"image":8,"video":1}' \
  --mm-process-config '{"image":{"max_pixels":1254400}}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45

4.2 单卡启动

MODEL_PATH=/path/to/GLM-5.3-flash

CUDA_VISIBLE_DEVICES=0 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --kt-weight-path "$MODEL_PATH" \
  --served-model-name GLM-5.3-flash \
  --host 0.0.0.0 \
  --tp-size 1 \
  --context-length 501025 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 2048 \
  --kt-method FP8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 0 \
  --kt-gpu-prefill-token-threshold 2048 \
  --cuda-graph-bs 1 2 4 \
  --limit-mm-data-per-request '{"image":8,"video":1}' \
  --mm-process-config '{"image":{"max_pixels":1254400}}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45

当 GLM-5.3-flash 开启 Layerwise Prefill 时，当前实现会将 resident GPU experts 归一为 0；因此四卡示例中的 --kt-num-gpu-experts 40 不会在 Layerwise 阶段常驻 40 个 GPU experts。

服务默认监听 http://localhost:30000。启动完成后可检查接口：

curl http://localhost:30000/v1/models

OpenAI 兼容接口为：

http://localhost:30000/v1/chat/completions

5. 多模态边界

• 单个请求可以包含纯文本，或纯文本加最多 8 张图片。
• 单个请求也可以包含纯文本加 1 个视频。
• 图片和视频不能出现在同一个请求中。
• 视频使用普通混合预填充，不走 Layerwise Prefill；解码仍支持 CUDA Graph。

Running KTransformers on AVX2 CPUs

This tutorial explains how to run KTransformers on machines that only support AVX2 (without AVX512 or AMX).

Table of Contents

• Supported Precision Formats
• Hardware Requirements
• Installation
• Verification
• Starting the Inference Server
  • Example: Qwen3-30B-A3B (BF16)
  • Example: Qwen3.5-35B-A3B-FP8 (FP8)
  • Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)
  • Example: Kimi-K2.5 (RAWINT4)
  • Sending Requests
• Performance Tuning
• FAQ

Supported Precision Formats

Hardware Requirements

• CPU: x86-64 + AVX2 + FMA (Intel Haswell 2013+ / AMD Zen+)
• GPU: NVIDIA 24GB+ VRAM (RTX 3090/4090/5090, etc.)
• Memory: At least the size of the model weights (e.g., Qwen3-30B-A3B BF16 requires 64GB+)
• OS: Linux

Installation

Build and install from source (one-click install for kt-kernel + SGLang):

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive

# One-click install
./install.sh

On AVX512 or AMX machines, you can also manually force AVX2 compilation:

export KT_RAWINT4_BACKEND=avx2
export CPUINFER_CPU_INSTRUCT=AVX2
export CPUINFER_ENABLE_AMX=OFF
./install.sh kt-kernel --manual

Verification

# Check if the CPU supports AVX2
lscpu | grep -i avx2

# Check the loaded kt-kernel variant
python -c "import kt_kernel; print(kt_kernel.__cpu_variant__)"
# Expected output: avx2

# System diagnostics
kt doctor

Starting the Inference Server

Use --kt-method BF16, FP8, GPTQ_INT4, or RAWINT4. KT-Kernel will automatically detect the CPU and fall back to the AVX2 backend when AVX512/AMX is unavailable.

Example: Qwen3-30B-A3B (BF16)

# Download the model
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /path/to/Qwen3-30B-A3B

# Check physical core count and NUMA node count
lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core|NUMA node\(s\)"

# Start the server (adjust kt-cpuinfer and kt-threadpool-count based on your hardware)
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3-30B-A3B \
  --kt-weight-path /path/to/Qwen3-30B-A3B \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 32 \
  --kt-method BF16 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.80 \
  --chunked-prefill-size 8192 \
  --max-running-requests 2 \
  --served-model-name Qwen3 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --enable-p2p-check \
  --disable-shared-experts-fusion

Example: Qwen3.5-35B-A3B-FP8 (FP8)

# Download the model
huggingface-cli download Qwen/Qwen3.5-35B-A3B-FP8 --local-dir /path/to/Qwen3.5-35B-A3B-FP8

# Start the server
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3.5-35B-A3B-FP8 \
  --kt-weight-path /path/to/Qwen3.5-35B-A3B-FP8 \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 2 \
  --kt-method FP8 \
  --kt-gpu-prefill-token-threshold 400 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --max-total-tokens 32000 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --disable-shared-experts-fusion

Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)

# Download the model
huggingface-cli download Qwen/Qwen3-30B-A3B-GPTQ-Int4 --local-dir /path/to/Qwen3-30B-A3B-GPTQ-Int4

# Start the server
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
  --kt-weight-path /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 2 \
  --kt-method GPTQ_INT4 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --max-total-tokens 32000 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --disable-shared-experts-fusion

Example: Kimi-K2.5 (RAWINT4)

Note: The following command is optimized for 4x RTX PRO 6000 Blackwell (96GB each) + AMD Threadripper PRO 5995WX (64 cores, 1 NUMA node) + 256GB RAM.

# Download the model
huggingface-cli download moonshotai/Kimi-K2.5 --local-dir /path/to/Kimi-K2.5

# Start the server
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Kimi-K2.5 \
  --kt-weight-path /path/to/Kimi-K2.5 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 228 \
  --kt-enable-dynamic-expert-update \
  --kt-method RAWINT4 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.95 \
  --chunked-prefill-size 8192 \
  --max-running-requests 4 \
  --context-length 262144 \
  --enable-mixed-chunk \
  --tensor-parallel-size 4 \
  --enable-p2p-check \
  --disable-shared-experts-fusion

Sending Requests

# Interactive chat
kt chat

# OpenAI-compatible API
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3","messages":[{"role":"user","content":"Hello"}],"stream":true}'

Performance Tuning

• --kt-cpuinfer: set to the number of physical cores
• --kt-threadpool-count: set to the number of NUMA nodes
• --kt-num-gpu-experts: higher values reduce CPU load but increase GPU VRAM usage
• Memory bandwidth is often the bottleneck; high-frequency DDR5 memory helps significantly

FAQ

GPU OOM

• Reduce --kt-num-gpu-experts, --chunked-prefill-size, --max-total-tokens
• Lower --mem-fraction-static

For more questions, see FAQ.

在 AVX2 CPU 上使用 KTransformers

本教程介绍如何在仅支持 AVX2 的机器上运行 KTransformers（无需 AVX512 或 AMX）。

目录

• 支持的精度格式
• 硬件要求
• 安装
• 验证
• 启动推理服务
  • 示例：Qwen3-30B-A3B (BF16)
  • 示例：Qwen3.5-35B-A3B-FP8 (FP8)
  • 示例：Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)
  • 示例：Kimi-K2.5 (RAWINT4)
  • 发送请求
• 性能调优
• 常见问题

支持的精度格式

硬件要求

• CPU：x86-64 + AVX2 + FMA（Intel Haswell 2013+ / AMD Zen+）
• GPU：NVIDIA 24GB+ 显存（RTX 3090/4090/5090 等）
• 内存：不少于模型权重大小（如 Qwen3-30B-A3B BF16 需 64GB+）
• 系统：Linux

安装

从源码编译安装（一键安装 kt-kernel + SGLang）：

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive

# 一键安装
./install.sh

在AVX512， AMX机器上， 也可以手动强制 AVX2 编译：

export KT_RAWINT4_BACKEND=avx2
export CPUINFER_CPU_INSTRUCT=AVX2
export CPUINFER_ENABLE_AMX=OFF
./install.sh kt-kernel --manual

验证

# 检查 CPU 是否支持 AVX2
lscpu | grep -i avx2

# 检查 kt-kernel 加载的变体
python -c "import kt_kernel; print(kt_kernel.__cpu_variant__)"
# 预期输出：avx2

# 系统诊断
kt doctor

启动推理服务

使用 --kt-method BF16、FP8、GPTQ_INT4 或 RAWINT4，KT-Kernel 会自动检测 CPU 并在缺少 AVX512/AMX 时回退到 AVX2 后端。

示例：Qwen3-30B-A3B (BF16)

# 下载模型
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /path/to/Qwen3-30B-A3B

# 查看物理核心数和 NUMA 节点数
lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core|NUMA node\(s\)"

# 启动服务（按实际硬件调整 kt-cpuinfer 和 kt-threadpool-count）
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3-30B-A3B \
  --kt-weight-path /path/to/Qwen3-30B-A3B \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 32 \
  --kt-method BF16 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.80 \
  --chunked-prefill-size 8192 \
  --max-running-requests 2 \
  --served-model-name Qwen3 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --enable-p2p-check \
  --disable-shared-experts-fusion

示例：Qwen3.5-35B-A3B-FP8 (FP8)

# 下载模型
huggingface-cli download Qwen/Qwen3.5-35B-A3B-FP8 --local-dir /path/to/Qwen3.5-35B-A3B-FP8

# 启动服务
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3.5-35B-A3B-FP8 \
  --kt-weight-path /path/to/Qwen3.5-35B-A3B-FP8 \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 2 \
  --kt-method FP8 \
  --kt-gpu-prefill-token-threshold 400 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --max-total-tokens 32000 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --disable-shared-experts-fusion

示例：Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)

# 下载模型
huggingface-cli download Qwen/Qwen3-30B-A3B-GPTQ-Int4 --local-dir /path/to/Qwen3-30B-A3B-GPTQ-Int4

# 启动服务
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
  --kt-weight-path /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 2 \
  --kt-method GPTQ_INT4 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --max-total-tokens 32000 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --disable-shared-experts-fusion

示例：Kimi-K2.5 (RAWINT4)

说明：以下命令针对 4x RTX PRO 6000 Blackwell（各 96GB）+ AMD Threadripper PRO 5995WX（64 核，1 NUMA 节点）+ 256GB RAM 优化。

# 下载模型
huggingface-cli download moonshotai/Kimi-K2.5 --local-dir /path/to/Kimi-K2.5

# 启动服务
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Kimi-K2.5 \
  --kt-weight-path /path/to/Kimi-K2.5 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 228 \
  --kt-enable-dynamic-expert-update \
  --kt-method RAWINT4 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.95 \
  --chunked-prefill-size 8192 \
  --max-running-requests 4 \
  --context-length 262144 \
  --enable-mixed-chunk \
  --tensor-parallel-size 4 \
  --enable-p2p-check \
  --disable-shared-experts-fusion

发送请求

# 交互聊天
kt chat

# OpenAI 兼容 API
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3","messages":[{"role":"user","content":"你好"}],"stream":true}'

性能调优

• --kt-cpuinfer 设为物理核心数
• --kt-threadpool-count 设为 NUMA 节点数
• --kt-num-gpu-experts 越大 CPU 负担越小，但 GPU 显存占用越高
• 内存带宽往往是瓶颈，DDR5 高频内存有明显帮助

常见问题

GPU OOM

• 减小 --kt-num-gpu-experts、--chunked-prefill-size、--max-total-tokens
• 降低 --mem-fraction-static

更多问题参见 FAQ。

在 Intel iGPU 上使用 SYCL GPTQ INT4 后端

本教程介绍如何使用 KTransformers 的 SYCL_GPTQ_INT4 后端，将 MoE 模型中的对称 GPTQ INT4 experts 放到 Intel 集成显卡（iGPU）上计算。

这是一个异构推理方案：Intel iGPU 负责未放在 CUDA GPU 上的 GPTQ INT4 experts；Attention、Embedding、LM Head 以及通过 --kt-num-gpu-experts 保留的 experts 仍由 CUDA GPU 执行。因此，当前实现不能替代 NVIDIA GPU，也不是纯 Intel GPU 推理后端。

目录

• 支持范围
• 硬件和软件要求
• 安装
• 验证安装
• 下载模型
• 启动推理服务
• 发送请求
• 参数说明
• 常见问题

支持范围

当前 SYCL 后端支持：

• Linux x86-64 平台。
• 通过 Intel Level Zero 驱动访问的 Intel GPU，主要面向共享系统内存的 Intel iGPU。
• MoE expert 的对称 GPTQ INT4 权重。
• sym=true、desc_act=false，权重包含 qweight 和 scales，不包含 qzeros。
• 已重点验证 group_size=128 的模型。
• 单机、tensor_parallel_size=1、以 batch size 1 为主的端侧推理。

不支持把 BF16、FP8 或非对称 GPTQ 权重交给此后端。启动时必须使用：

--kt-method SYCL_GPTQ_INT4

硬件和软件要求

• Intel GPU：支持 Level Zero 和 shared USM 的 Intel iGPU。
• CUDA GPU：一张受当前 PyTorch 和 SGLang 支持的 NVIDIA GPU，用于模型的其余部分。
• CPU：x86-64，支持 AVX2 和 FMA。
• 系统内存：至少能够容纳模型权重、KV Cache 和 iGPU 运行缓冲区。iGPU 会与 CPU 共享系统内存带宽。
• 操作系统：Linux。
• Intel oneAPI：需要包含 icpx、SYCL runtime 和 sycl-ls 的 oneAPI Base Toolkit。
• Intel GPU 驱动：需要安装 Level Zero 用户态驱动。
• CUDA Toolkit：当前 SYCL 构建仍需要 CUDA runtime 提供 SGLang 与 CPUInfer 之间的 host callback 调度。

先确认 Intel GPU 能被 SYCL 正常识别：

source /opt/intel/oneapi/setvars.sh
sycl-ls

输出中应当能看到类似下面的 Level Zero GPU：

[level_zero:gpu:0] ...

还需要确认当前用户可以读写 GPU render node：

ls -l /dev/dri/renderD*
groups

如果当前用户不在 render 用户组中，可以执行：

sudo usermod -aG render "$USER"

修改用户组后需要注销并重新登录。

安装

从源码构建时，CPUINFER_USE_SYCL 默认关闭，需要显式设置为 1：

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive

source /opt/intel/oneapi/setvars.sh
CPUINFER_USE_SYCL=1 ./install.sh

安装脚本会使用 icpx 构建 SYCL GPTQ INT4 后端。普通的 CPU/CUDA 构建不需要 oneAPI；只有启用 CPUINFER_USE_SYCL=1 时才需要上述环境。

SYCL、CUDA、ROCm、MUSA 和 MACA 是互斥的 kt-kernel 编译后端。如果当前 shell 曾经显式设置过其他后端，请先取消对应变量，例如：

unset CPUINFER_USE_CUDA CPUINFER_USE_ROCM CPUINFER_USE_MUSA CPUINFER_USE_MACA

未显式设置 CPUINFER_USE_CUDA 时不需要手动取消，安装脚本会在启用 SYCL 后端时自动关闭 CUDA kernel 后端。CUDA Toolkit 仍会用于链接 host callback 所需的 cudart。

验证安装

安装完成后，检查 Python 扩展是否包含 SYCL 后端：

source /opt/intel/oneapi/setvars.sh
python -c "import kt_kernel_ext.moe as moe; print(hasattr(moe, 'SYCLGPTQInt4_MOE'))"

预期输出：

True

如果机器上存在多个 SYCL GPU，可以在启动服务前指定设备：

export ONEAPI_DEVICE_SELECTOR=level_zero:0

只有一个可用 Intel GPU 时通常不需要设置该变量，后端会优先选择 Level Zero GPU。

下载模型

下面使用 Qwen3.5-35B-A3B-GPTQ-Int4 作为示例：

huggingface-cli download Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 \
  --local-dir /path/to/Qwen3.5-35B-A3B-GPTQ-Int4

使用其他模型时，请先确认它是 MoE 模型，并且 GPTQ 配置为 sym=true、desc_act=false。目前推荐使用已经验证过的 group_size=128 权重。

启动推理服务

下面的命令面向单用户、batch size 1 的端侧推理。请将模型路径和显存相关参数改为适合本机的值：

source /opt/intel/oneapi/setvars.sh

SGLANG_MAMBA_CONV_DTYPE=float16 \
python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 30000 \
  --model /path/to/Qwen3.5-35B-A3B-GPTQ-Int4 \
  --kt-weight-path /path/to/Qwen3.5-35B-A3B-GPTQ-Int4 \
  --served-model-name qwen3.5 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --dtype float16 \
  --attention-backend triton \
  --mem-fraction-static 0.90 \
  --max-running-requests 1 \
  --max-total-tokens 32000 \
  --max-prefill-tokens 16000 \
  --chunked-prefill-size 4096 \
  --watchdog-timeout 1200 \
  --kt-method SYCL_GPTQ_INT4 \
  --kt-cpuinfer 8 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 24 \
  --disable-shared-experts-fusion \
  --disable-cuda-graph

正常使用时不需要关闭 radix cache。--disable-radix-cache 主要用于确保每次请求都执行完整 prefill 的性能测试，不建议加入日常启动命令。

发送请求

服务启动后，可以通过 OpenAI 兼容接口发送请求：

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5",
    "stream": false,
    "messages": [
      {"role": "user", "content": "你好，请介绍一下你自己。"}
    ]
  }'

参数说明

SYCL kernel 的 tile、subgroup 和 prefill 阈值已经使用当前验证过的默认值，不需要额外设置调试或调优环境变量。

常见问题

提示 SYCL_GPTQ_INT4 backend not available

当前安装没有包含 SYCL 扩展。重新加载 oneAPI 环境并构建：

source /opt/intel/oneapi/setvars.sh
CPUINFER_USE_SYCL=1 ./install.sh

找不到 icpx 或 sycl-ls

确认已经安装 Intel oneAPI Base Toolkit，并在当前 shell 执行：

source /opt/intel/oneapi/setvars.sh

无法访问 /dev/dri/renderD*

将当前用户加入 render 用户组，注销并重新登录。不要通过长期设置 chmod 666 绕过设备权限。

提示没有可用的 SYCL GPU

先运行 sycl-ls 检查 Level Zero GPU。如果存在多个设备，可以显式选择：

export ONEAPI_DEVICE_SELECTOR=level_zero:0

CMake 找不到 CUDA Toolkit

当前 SYCL 后端仍需要 cudart 完成 host callback 调度。确认 CUDA Toolkit 已安装；如果不在默认路径，可以设置：

export CUDA_HOME=/usr/local/cuda

然后重新运行安装命令。

提示多个 GPU backend 同时启用

SYCL 与 kt-kernel 的其他 GPU 编译后端互斥。检查并取消之前显式设置的 CPUINFER_USE_CUDA=1、CPUINFER_USE_ROCM=1 等变量。

模型提示 qzeros、sym=false 或 desc_act=true 不受支持

当前后端只支持对称 GPTQ INT4：sym=true、desc_act=false，且不使用 qzeros。需要更换为兼容的模型权重。

CUDA 显存不足

• 降低 --kt-num-gpu-experts。
• 降低 --mem-fraction-static、--max-total-tokens 或 --chunked-prefill-size。
• 降低 --kt-num-gpu-experts 会把更多 expert 计算交给 iGPU，可能降低速度并增加系统内存带宽压力。

iGPU 性能不稳定

• 确认系统处于高性能电源模式，并避免同时运行占用 iGPU 的桌面或视频任务。
• iGPU 与 CPU 共享内存带宽，双通道或更高带宽的内存通常有明显帮助。
• 首次请求包含权重准备和运行时预热，性能比较应在 warmup 后进行。

Frequently Asked Questions

1. SGLang “Using default MoE kernel config” warning at startup

When using kt-kernel with SGLang, you may see a warning like:

[2026-05-15 20:31:38] Using default MoE kernel config. Performance might be sub-optimal!
Config file not found at .../fused_moe_triton/configs/...

This warning is expected and can be safely ignored. kt-kernel replaces SGLang’s built-in MoE implementation with its own CPU/GPU hybrid dispatch, so SGLang’s fused-MoE Triton kernel configuration is never used. The warning is emitted by SGLang before kt-kernel takes over MoE execution and has no impact on performance or correctness.

2. Where can I find more help?

Check the existing issues or open a new one.