On this page
- Kimi K3 is enormous, selectively
- The benchmark story is good enough without lying about it
- The keynote explained K3 before K3 existed
- Muon is the part accountants should understand
- KDA and Attention Residuals attack two different bottlenecks
- Moonshot’s version of more with less
- The giant model that makes giant models look silly
Kimi K3 has 2.8 trillion parameters, a one-million-token context window, and a trophy cabinet that already includes third place on OpenLM’s Arena+ leaderboard. Naturally, the parameter count will get the headlines. It is the least interesting number in the release.
The interesting part is how Moonshot AI is trying to turn less wasted computation into frontier capability. K3 activates only 16 of its 896 routed experts at a time, replaces standard attention in key parts of the stack, and arrives with a serving strategy built around caching, quantization, and parallel agents. This is not a cute optimization pass. It is Moonshot’s answer to an industry that increasingly treats data centers as a substitute for ideas.
And yes, K3 is third on OpenLM’s Chatbot Arena+, as retrieved on July 20, 2026. It sits at an Arena Elo of 1506, behind Claude Fable 5 at 1510 and GPT-5.6 Sol at 1509, tied on the displayed score with Claude Opus 4.8 Thinking and GPT-5.5-high. Leaderboards move, ties exist, and pretending otherwise is how benchmark press releases become astrology.
Kimi K3 is enormous, selectively
Moonshot calls K3 the first open 3T-class model. The exact figure is 2.8 trillion total parameters, arranged as a sparse Mixture of Experts model with 896 routed experts. For each routing operation, only 16 are activated.
That distinction matters. A sparse model’s total parameter count is not the same thing as the amount of computation used for every token. Quoting 2.8 trillion without explaining the routing is like describing a hotel by counting every bed, then implying every guest sleeps in all of them simultaneously. Impressive building. Very strange bedtime.
K3 also offers native vision and a one-million-token context window. Moonshot positions it for long-horizon coding, research, and knowledge work, the jobs where an agent must keep a repository, tool history, documents, screenshots, and its own mistakes in view for hours rather than answering one polished trivia question.
The launch is open-weight with an asterisk. K3 is available through Kimi’s products and API now, while Moonshot says the full weights, license details, and technical report will arrive by July 27, 2026. Until those files exist, this is an announced open-weight release, not a model you can already download and inspect. Moonshot also recommends supernodes with 64 or more accelerators for deployment. Open does not mean it will fit under your desk, unless your desk is inside a data center.
The benchmark story is good enough without lying about it
On the July 20 snapshot of OpenLM’s Arena+, Kimi K3 ranks third by displayed Arena Elo. The same table gives it 1562 for coding, 1311 for vision, an Artificial Analysis Intelligence Index score of 78, and 89.3 on MMLU-Pro. OpenLM describes Arena+ as an agent-driven battle platform using an LLM judge to calculate Elo, so this is not the same leaderboard or methodology as LM Arena’s human preference votes.
That last sentence is not pedantry. “Chatbot Arena” has become a generic phrase people fling around as if every arena were one universal football table. It is not. On LM Arena’s separate leaderboards, K3 was first in WebDev, fourth in Agent Arena, and in a crowded pack around ninth in Text when checked on the same date. Different prompts, judges, voters, categories, and confidence intervals produce different ranks. Welcome to evaluation in 2026, where even the scoreboards need model cards.
Moonshot’s own evaluations are more ambitious and less independent. The company reports 67.3 on DeepSWE with mini-SWE-agent and 90.4 on BrowseComp when K3 receives its full million-token context without external context management. It also shows long autonomous runs for GPU kernel optimization, an ASIC proof of concept, and an astrophysics research workflow.
The demonstrations are legitimately interesting. In the ASIC task, K3 reportedly ran for 48 hours, used open-source EDA tools and a Nangate 45 nm process, reached 100 MHz timing closure, and produced a simulated design exceeding 8,700 tokens per second. But it was simulated silicon, not a fabricated chip. The astrophysics run reportedly reviewed more than 20 papers, processed over 300 equations of state, and wrote more than 3,000 lines of Python in roughly two hours. Those are Moonshot’s case studies, not independent lab replications.
Moonshot itself says K3 still trails Claude Fable 5 and GPT-5.6 Sol overall. Good. A company admitting its model is not supreme at everything is so refreshing that someone should check whether marketing has been informed.
The keynote explained K3 before K3 existed
Four months before the launch, Moonshot founder Zhilin Yang gave a GTC 2026 keynote about scaling Kimi K2.5. The model name has changed, but the talk now reads like K3’s architectural manifesto.
Yang divides scaling into three dimensions: token efficiency, context length, and agent count. The standard frontier-lab playbook mostly asks how much more compute, data, and electricity can be poured into the next run. Moonshot asks a nastier question: how much of the existing budget is being converted into learning rather than heat?
At about 1:45 in the talk, Yang describes token efficiency as moving the scaling curve left. The goal is not merely to train faster. It is to reach a lower loss with the same number of tokens. If high-quality data is finite, a twofold improvement in token efficiency is roughly equivalent to discovering twice as much useful training data. No new internet required, which is handy because the old internet is increasingly made of AI summaries of other AI summaries.
The second dimension is context. More context lets an agent run longer and preserve more state, but ordinary attention makes that expensive. The third is agent count: split a hard task into parallel subtasks, let specialized agents search independently, then coordinate their results. Moonshot calls this agent swarms. It is less “one genius thinks harder” and more “a competent team stops waiting for one person to finish every ticket.”
K3 maps directly onto that thesis. Its optimizer and architecture target more learning per token. Kimi Delta Attention targets cheaper long context. Its million-token window targets long-running work. Moonshot’s agent products target parallel execution. This is a coherent research program, not a bag of acronyms shaken over a launch post.
Muon is the part accountants should understand
The transcript occasionally renders Muon as “Meow,” which is objectively a better optimizer name but technically wrong. Muon is a second-order optimizer that transforms gradient updates through orthogonalization rather than relying on AdamW’s usual update geometry.
Moonshot’s published scaling experiments report roughly 2x compute efficiency over AdamW under tested conditions. Yang’s keynote makes the economic point plainly: if you have 50 trillion high-quality tokens and double token efficiency, the result behaves more like 100 trillion tokens under the old recipe. This does not halve every cost in the company. It does mean the same training budget can buy a better model.
Scaling Muon was not a matter of changing one line in a configuration file and going for lunch. Moonshot added weight decay, normalized update magnitudes against AdamW, and distributed optimizer state across GPU clusters. It also ran into attention-logit explosions, where attention scores grow until training destabilizes. The team’s QK-Clip intervention selectively clips problematic query and key projections rather than bludgeoning every attention head.
This is the unglamorous machinery behind “more with less”: identify where computation stops producing useful learning, then remove the waste without breaking a trillion-parameter run. Frontier AI turns out to involve fewer magic sparks and more months spent staring at one chart that should not be going vertical.
KDA and Attention Residuals attack two different bottlenecks
Kimi Delta Attention, or KDA, is Moonshot’s approach to linear attention. Standard softmax attention lets every token compare itself with every other token, which is expressive and brutally expensive as sequences grow. Linear attention maintains a compressed recurrent state, making long sequences cheaper, but that compression can throw away useful detail.
KDA adds a learned gate that controls how much old state gets erased and how much new information gets written. In the keynote, Yang frames this as a middle ground: retain the efficiency of a compact state while recovering more of full attention’s expressiveness. K3 combines KDA with Gated Multi-head Latent Attention, rather than betting the entire model on one attention mechanism.
Attention Residuals, or AttnRes, tackles depth instead of sequence length. Conventional residual connections keep adding every layer’s output into one growing stream. That can dilute useful information as models become deeper. AttnRes lets a layer selectively retrieve earlier representations using attention, giving information a cleaner route through the network.
Then there is Stable LatentMoE, which is meant to make sparse expert routing more stable at K3’s scale. Put the pieces together and Moonshot claims an approximate 2.5x improvement in overall scaling efficiency versus Kimi K2. That is a first-party aggregate claim, and the launch page does not yet publish enough curves or a precise composite definition to reproduce it. Treat it as a serious claim awaiting the technical report, not a commandment delivered from the moon.
Moonshot’s version of more with less
American frontier labs are scaling infrastructure with the delicacy of someone solving a plumbing problem using the Hoover Dam. OpenAI and its partners announced Stargate with an intention to invest $500 billion over four years in US AI infrastructure, starting with $100 billion. Anthropic has also raised and committed enormous capital to compute, even if its engineering culture often looks more efficiency-conscious than OpenAI’s.
Those figures are infrastructure commitments, not the training bill for one model. Moonshot has not disclosed K3’s total training cost, GPU count, training FLOPs, active parameter count, or dedicated team size. So no, the available evidence does not prove that K3 cost one-tenth as much as Claude or GPT. Anyone giving you that ratio has replaced accounting with fan fiction.
What the evidence does show is a company attacking waste at every layer:
- Training: Muon aims to extract more learning from the same tokens and compute.
- Architecture: sparse expert routing avoids activating the full 2.8T parameters for every token.
- Context: KDA compresses long-range state instead of paying full quadratic attention everywhere.
- Depth: AttnRes creates selective paths to useful earlier representations.
- Post-training: quantization-aware SFT targets MXFP4 weights and MXFP8 activations.
- Serving: prefix caching and disaggregated inference reduce repeated work.
- Agents: parallel swarms spend additional compute only when decomposing a task can buy a better result.
This fits the broader argument I made in how Chinese AI is challenging the American capital model. Export constraints and smaller budgets do not magically make Chinese labs virtuous. They make efficiency a survival requirement. American labs can often purchase another cluster. Moonshot has stronger incentives to redesign the engine.
The giant model that makes giant models look silly
K3 is not small. It is not cheap to deploy. It does not prove that compute scaling is dead, and a recommendation for 64-plus accelerators will not trigger a Raspberry Pi port by Friday. The model is, in fact, comically large.
But its architecture makes a useful point: total scale and computational frugality are not opposites. You can build an enormous reservoir while being careful about which valves open, what gets cached, how information travels, and when parallel search is worth paying for.
The next test is not another launch chart. It is whether Moonshot ships the weights and technical report on schedule, whether independent evaluators reproduce the efficiency claims, and whether real deployments can serve a 2.8T model economically. Arena ranks will shuffle by breakfast. Infrastructure economics are harder to charm.
If K3’s ideas hold up, the frontier race gets more interesting. OpenAI and Anthropic can keep building bigger power plants. Moonshot is trying to make every electron do more work. The winner will probably do both, which is bad news for anyone whose entire moat is an electricity bill.
Keep reading