Watch an AI coding assistant fix a bug and you'll see it read tens of thousands of lines of code, logs and error messages before it touches anything. Whether it still remembers what it read near the start often decides whether the fix works. On November 3, 2026, Mistral published a 30-page technical report on Mistral Large 4, posted in the model's Hugging Face repository and on the paper site alphaXiv. It describes a costly detour. To make the model cheap to run, the team cut that memory aggressively, then found the model getting worse at exactly these long tasks, and had to operate on a half-trained model.
Some dates first. Mistral opened a public preview of Large 4 with a blog post on October 6 and released the weights with a second post on October 29. The weights are the model's parameters themselves, and once released anyone can download them. The model has about 1T parameters in total but uses only 52B of them for each word it processes; the model card puts that at 49B if you leave out the input and output layers. It can read up to 1M tokens at once (a token is the basic unit a model reads and writes, roughly a word or part of one). The design details below come from the report and from the configuration file in the model repository.
What the model's memory is, and why it costs so much
Each time a model produces a word, it looks back over the text so far to judge which parts matter most right now. That mechanism is called attention. So it doesn't have to reread everything each time, the model writes a note for every token at every layer as it reads, then consults the notes. Those notes are called the KV cache. They live in GPU memory and grow with the length of the text. How many users one GPU can serve at once, and so what each call costs, depends heavily on how small those notes can be made.
Mistral originally stacked 3 ways to keep fewer notes. The first is called a sliding window: in 3 of every 4 layers, the model doesn't look at the whole text, only the most recent 512 tokens, so it only needs notes for those. The second is cross-layer sharing, where a layer keeps no notes of its own and uses another layer's: among the layers that see the whole text, every other one borrowed the notes of the layer before it. The third is called YOCO, short for "You Only Cache Once": the whole-text layers in the second half of the network kept no notes at all and all read the set left by the last such layer in the first half. Together the 3 tricks cut the notes to 1/16 of the usual design, where every layer sees the whole text and keeps its own notes, known as full attention. Early comparisons, the report says, showed the effect on perplexity and common benchmarks was negligible or small enough to accept for the savings. Perplexity is the standard measure of how well a model guesses the next word, and lower is better.
Just as good at guessing words, worse at fixing bugs
The trouble surfaced while pretraining was still running. Pretraining is the stage where a model reads huge amounts of text and learns to guess the next word. The team took several versions saved partway through training, trained each briefly on example answers, and found knowledge tests and perplexity looked normal. But agent tests, where the model must read a long context and take many steps on its own to finish a task, fell clearly short. The report names SWE-bench and BrowseComp. The first has the model fix bugs in real open-source projects, and the second tests whether it can find hard-to-find information on the web. When the team isolated the cause, SWE-bench Verified scores (the part of SWE-bench whose problems were checked by people) at the same training budget went down when full attention was swapped for sliding windows, and down again when cross-layer sharing was added.
The conclusion the report draws from this is its most important sentence. 2 models equally good at guessing the next word can perform very differently on agent tasks, and Mistral saw no such gap on knowledge tests. Its comparison experiments also suggest longer training can narrow the gap. The report offers no explanation for why, only the measurements. That alone matters, because the quick check most teams run on a new memory-saving design is whether perplexity moved. By Mistral's own data, that check missed the ability it cared about most.
Most of it came out, but one kind of sharing stayed
Rather than start over, the team changed the design of the version saved at the end of the first pretraining phase, ran about 2/3 of the planned second phase, then went on to midtraining. Midtraining is the stretch after pretraining and before training for specific skills. Lifting the sliding window was easy: those layers were simply allowed to see the whole text. Undoing the note sharing was harder. Layers that used to borrow now had to keep their own notes, so the team gave them a starting point by copying the matching parameters from the nearest layer that already kept notes. To help layers that had only ever looked nearby learn to look far, they made 2 adjustments that let the model learn for itself how to tell where words sit relative to each other and how sharply each strand of attention should focus. Afterward, test passages 32,000 tokens long showed the converted layers putting about 55% of their attention on content more than 512 tokens back, against 67% for layers that had always seen the whole text. The report reads this as old habits surviving in part, with the converted layers still learning to look far when needed.
The report titles this chapter "Why ML4 is (almost) a full-attention model," and the word that matters is "almost." In smaller experiments, removing YOCO as well also recovered, though training was less stable right after. The team couldn't rerun the whole second phase for the final model, so they ran a conservative test at a larger scale with only the surgery and midtraining, and the version that kept YOCO did better. The released model therefore still shares memory across some layers. The configuration file spells it out: of 72 layers, 9 (numbered from 0, layers 39, 43, 47 and every 4th layer through 71) keep no notes of their own and all read layer 35's, while the other 63 each keep their own. Nor are the notes stored as they are. Each layer stores only a compressed version and expands it when needed, a method called MLA, and the config sets that compressed version's size at 512. The report does not say how much smaller the released model's notes are than a full-attention model's, and we could not find that figure anywhere else.
Picking what to look at saves compute, not notes
With the notes largely restored, the compute problem remained: in a 1M-token context, every new word has to be compared against every note. Mistral's answer was sparse attention, an approach from DeepSeek in which the model looks closely at only a small share of the notes at each step. A lightweight helper module first picks the 2,048 notes most relevant to the current word, and the main model looks only at those 2,048. Training came in 2 steps. First, with the main model held fixed, the helper modules spent about 15.1B tokens learning to imitate where the main model usually put its attention. Then both trained together in this pick-and-look mode for about 198B tokens. The team also noticed that layers that used to be sliding-window layers picked very similar content, so from layer 16 on, neighboring layers of that kind share one picking module; the config marks 28 layers as sharing.
This is the part that's easy to misread. The report says this method is there to cut the compute cost of reading long text. It does not shrink what has to be stored. Each new word picks a different 2,048, so every note has to stay available. Saying Large 4 no longer compresses its memory is wrong. A precise version: it removed most of its compression, kept part of the sharing in the second half plus MLA's compressed version, and relies on picking what to look at to cut the work at each step.
The storage burden moves to the system that runs the model. The report says Mistral splits the notes across several GPUs instead of keeping a full copy on each, stores the notes in the 8-bit FP8 format, and moves what doesn't fit elsewhere. That frees roughly 20M tokens of note space on 4 Nvidia GB300 GPUs, shared by every request running at the same time. The model itself was slimmed too: the parameters for its different specialties are stored in Nvidia's 4-bit NVFP4 format, so each number takes less space, while everything else stays at training precision. That brings the model file to 606 GB, small enough for those 4 GPUs. Mistral says this slimmed version is on par with the FP8 version on all its tests, and its weights are published separately.
The scores are Mistral's own, and it doesn't lead everywhere
Every number below comes from Mistral's report and is self-reported. In its tables, rows marked AA are evaluations by the independent benchmarking firm Artificial Analysis, and rows marked internal were run by Mistral; we did not check each figure against Artificial Analysis's site. On FrontierCode 2.0, a coding test from the AI coding company Cognition, Large 4 scores 19.7%, against 19.3% for DeepSeek V4.1 Flash and 17.8% for GLM-5.3. On Terminal-Bench 4.0, which tests finishing tasks at the command line, it scores 41% to GLM-5.3's 40.0%. Both margins are thin. Elsewhere it trails: 65.2% on DeepSWE 1.1, a test of long, original engineering tasks, against DeepSeek V4.1 Flash's 74.2%; 50.9% on Artificial Analysis's cybersecurity index against DeepSeek V4.1 Flash's 56.8%; and 17.3% on HLE-Diamond, a set of expert-level academic questions, the lowest of the models with a score in that table.
The numbers have also moved. The October 6 preview post listed 28.3% on Terminal-Bench 4.0. The October 29 weights post says 3 more weeks of reinforcement learning (having the model try repeatedly and improve based on how the results turn out) added 13 percentage points, bringing it to 41%. When you see a Large 4 score, check whether it describes the preview or the open-weight release. On its Terminal-Bench 4.0 leaderboard updated October 29, 2026, the independent evaluator Vals AI scores the open-weight Large 4 at 26.26%, below the 41% that Mistral's report cites from Artificial Analysis's testing. The two ran the model inside different helper programs that let it act: Vals used mini-swe-agent, a minimal general-purpose program that only lets the model run command-line commands, while Artificial Analysis's site labels its Large 4 results as run in Mistral's own coding tool, Mistral Vibe, so the figures are not directly comparable.
When you pick a model, don't stop at how much memory it saves
If you're choosing a model for agent work, the most useful lesson here is that claims like "the KV cache is N times smaller" or "perplexity barely changed" can't tell you whether a model will hold up on long tasks. Mistral's own data shows those things can come apart, so test candidates on long tasks of your own rather than looking only at what the design saves. If you plan to run Large 4 yourself, the NVFP4 version runs on 4 GB300s. The license is Mistral's modified MIT license, which allows commercial and non-commercial use, but if you want to package it as an API and sell access to others, you need a separate commercial license from Mistral.
The report closes by saying this experience pushed the team to study other, cheaper kinds of attention design, with agent tests guiding design throughout training, and that the results will come in later releases. Mistral's next model will very likely try to save memory again. When it arrives, look at its long-task scores before you look at how much it saved.