With a scale of nearly three trillion parameters and close to 1,000 experts, the way information circulates must be redesigned.
This undertaking guided the development of Moonshot AI’s latest flagship open-weight model: Kimi K3. It unfolded across three dimensions: sequence length, network depth, and model width.
On the first dimension, Moonshot AI leveraged a hybrid attention mechanism. It interleaves, at a 3-to-1 ratio, KDA (Kimi Delta Attention) layers and MLA (Multi-head Latent Attention). The former exploit a fixed-size recurrent state to keep memory usage in check, while the latter preserve global context.
The so-called Attention Residuals mechanism operates on the second dimension. It replaces the traditional residual connection by allowing each layer to selectively retrieve and weight representations from across all preceding blocks.
Stable LatentMoE comes into play on the third dimension. As its name indicates, it stabilizes training at the scale of Moonshot AI’s 896 experts within Kimi K3. Its role: balance the load to prevent some experts from being underutilized or overused. In addition, rather than forcing the entire hidden dimension through each of the 16 experts selected for every token, the architecture compresses them into a lower-dimensional latent space where the experts perform their calculations. The outputs are then re-expanded back to the original dimensionality.
MicroVMs to hold the million tokens
The post-training phase combined supervised fine-tuning, reinforcement learning, and distillation. The context window was progressively extended, ultimately reaching up to one million tokens.
To manage the parallelism of the experts, Moonshot AI built its own library. It leverages statically shaped computations, reserving fixed memory buffers for each token so that the GPU no longer needs to execute dynamic-size change instructions.
To keep the attention trajectory aligned with agentic usage, the training leveraged microVMs capable of suspending sandboxes and restarting them in a few tens of milliseconds. Moonshot AI added a consolidation of the elementary operations of the KDA blocks into a single custom GPU kernel.
All of these techniques, it claims, enabled an efficiency of scale that is 2.5 times higher than that of Kimi K2.
Kimi K3, under a permissive license, but…
Kimi K3 ships with inference tooling for vLLM, SGLang, and TokenSpeed. It carries a highly permissive MIT-like license… except for large enterprises.
Those offering MaaS (models as a service) must negotiate a commercial agreement with Moonshot AI once they have already—subsidiaries included—generated at least $20 million in revenue over a rolling 12-month period.
Another requirement applies. It is not limited to MaaS providers. It mandates clearly displaying “Kimi K3” on the UI of any product or service that counts more than 100 million monthly active users and/or generates at least $20 million in monthly revenue.
Meta imposes a similar constraint for its open-weight Llama models. Anyone operating products or services with more than 700 million monthly active users must obtain a commercial license.
Complementary reading:
Who is Moonshot AI, the publisher of the Kimi models?
Mistral AI shifts its approach to reinforcement learning
AMD truly enters the exascale AI rack game
Agentic AI is changing how work is done: Doctolib tests it with its developers
Cohere and Aleph Alpha: the sovereign AI union
