73.6 on SWE-bench, and Not One Way to Check It Yet

DiffuSpace has released Clover-35B-A3B-Preview, a downloadable coding model built on diffusion, and says it scores 73.6 on SWE-bench Verified and generates 515 tokens per second on a single chip ^[1].

The release is confirmed. The weights and code sit on Hugging Face and GitHub, and anyone can pull them down ^[1][2][5]. The performance figures are another matter. They come from the company's own launch post, which does not give enough detail for an outsider to reproduce them ^[1].

That is my reading of the story: what you can verify today is the licence and the files, and what you cannot yet verify is the scores. Both deserve attention, and they deserve different amounts of trust.

What the model is

Clover is the first model in a new DiffuSpace line. It is text-only and derived from Qwen3.6-35B-A3B, which the model card names as its base ^[1][5]. It has 35 billion parameters in total, but only about three billion are active for any given token.

A token is a small piece of text, often a word or part of one. That is the mixture-of-experts design, in which the network is split into specialist parts and each token consults only a few of them ^[1][2][5]. The stated context limit is 131,072 tokens, shared between the prompt and the response ^[1][2][5].

The unusual part is how it writes. Most language models are autoregressive, meaning they produce text left to right, one token at a time, each choice conditioned on everything before it. Think of a typist who cannot go back.

A diffusion model works more like an editor with a draft full of blanks. It starts with a block of masked placeholders, fills in the positions it is confident about, and repeats until the block is complete.

Clover does this in blocks of 32 tokens by default in its Hugging Face implementation, with a confidence threshold deciding which positions get filled at each pass ^[1][5]. In principle, filling many positions at once should be faster than going one by one. That is the bet the company is making, and the 515 figure is its evidence.

The licence is the solid part

Clover ships under Apache License 2.0 ^[2][4]. This is one of the most permissive licences in common use. It allows use, copying, modification, distribution, sublicensing and sale.

The conditions are modest: a redistributor must include a copy of the licence and keep the applicable notices ^[2][4]. The text also carries patent and warranty provisions ^[4].

The model card adds that the accompanying licence preserves the upstream Apache licence and copyright statement from the base model, and that the implementation derives from the Hugging Face Transformers Qwen3.5-MoE code ^[5]. A developer or a company's lawyer can read all of that today and know where they stand. That is rarer than it sounds in a field where "open" is often a marketing word.

But a licence settles what you may do with a file, not what the file can do. Apache 2.0 does not make the benchmark claim true, and it does not make deployment cheap. By the repository's own estimate, running the weights in BF16, a 16-bit number format that keeps the model at full working precision, takes about 69 GB of GPU memory. That is before counting inference state, the extra memory the machine uses while it generates text ^[2][5]. It puts the full-precision model beyond any ordinary desktop graphics card.

The project is also unfinished. The GitHub page says supervised fine-tuning code and training instructions are coming, and that backend code and setup guides will be added ^[2].

So you can use the model and, in principle, change it, but the recipe for how it was made is not all there yet. The company recommends its official SGLang implementation for serving and evaluation. SGLang is the software that loads a model and runs it for other programs and for testing. The model card warns that small numerical differences between backends can lead to different generated text ^[2][5].

A score with no method behind it

SWE-bench Verified is a better test than most coding leaderboards. It holds 500 software issues that humans have reviewed, taken from real repositories. A system is given the code and the issue description and must produce a patch that fixes the problem ^[3]. If the 73.6 is a plain share of those 500, it means about 368 issues resolved, and each issue is worth 0.2 percentage points. That is a lot of working patches.

The trouble is that the number belongs to a system, not a model. A model does not fix software on its own. It needs an agent scaffold, the software wrapper that lets it read files, call tools, run code, retry and take actions. The wrapper is sometimes called a harness. The benchmark's maintainers separate evaluations that use a common mini-SWE-agent setup, a standard, deliberately simple wrapper that lets a model read a code project and attempt a fix, from a broader leaderboard that admits different agent systems. They also warn that results from different versions of their own standard setup are not necessarily comparable ^[3].

If the maintainers say their own versions can drift, an unpublished setup deserves a question mark.

The launch post lists scores. It also reports 51.7 on Terminal-Bench 2.1 and 65.3 on τ³ Overall ^[1]. Terminal-Bench is a test of how well an agent handles tasks in a command-line terminal, and the τ-bench family tests agents that use tools across multi-step tasks.

What the launch post does not give is the scaffold, the tools the model could call, the number of attempts allowed, or the details of the runs ^[1]. Those are the knobs that move a score. A model allowed several tries inside a good harness can beat the same model given one shot and a bare one, and a reader of the scores cannot tell which this is.

There is a second reason for care. Diffusion decoding has settings, among them block size and the confidence threshold, and the model card itself says backend differences can change the text produced ^[5]. How much the score wobbles across backends, settings and repeated runs is not stated, and neither is whether 73.6 is a single run or an average. A single headline number hides that spread.

The speed claim has the same problem

The company reports 515 tokens per second on agent tasks on one Nvidia H200, against 365 for a similarly sized Qwen model accelerated with DFlash2 ^[1]. That is a gap of 150 tokens per second, or about 41 percent.

A second figure, 680 tokens per second on one Nvidia GB300, comes from internal agent workloads at BF16 precision and batch size one ^[1]. Batch size one means one request at a time, with no other users sharing the chip. It is the setting that favours how fast a single user sees text appear, and it says little about a busy server handling many users at once.

The H200 and GB300 figures are different chips, so they are not a trend line. And the launch post does not say enough to know what the 515 means.

A per-request rate, how fast one user sees text appear, is a different thing from aggregate throughput across many users at once. Neither says whether the runs shared the same tasks, hardware conditions and quality bar ^[1].

Speed also is not the same as getting work done. A fast agent that fails more often can burn through its retries and finish later than a slow one that gets it right the first time. Parallel block decoding is a real idea, and it may well be faster. Whether that survives a full agent loop is what the number has to show, and for now it does not show it.

The strongest objection

The fair objection is that I am asking for too much from a preview. DiffuSpace labelled this a preview, it says it plans further validation on a broader range of real-world tasks ^[1], and it has put the weights out so others can test them. That is more than many companies do.

A closed lab posting the same scores with no downloadable model would deserve harsher treatment.

I grant all of that. The openness is real and it counts. But it cuts the other way too. A company that publishes weights invites checking, and the checking cannot start with a score if the method is withheld.

The missing piece is small and cheap to supply: the scaffold, the tools, the attempt count, the runtime and precision, the input and output lengths, the concurrency. Publishing those would turn a claim into a test anyone could run. Until then the strongest sentence the evidence supports is that the company says so.

What to ask of any benchmark

The questions travel beyond this one release. Which version of the benchmark, and which track of it? Was the score a single run or an average? For a speed figure, what exactly was measured, and against which baseline run under which conditions?

A set of scores that answers none of these is an advertisement. One that answers all of them is a result.

Open files make this cheaper than it used to be. Anyone with the hardware can load Clover, and independent evaluators can run it under a fixed, disclosed setup.

If the SWE-bench maintainers or another outside group reproduce something near 73.6, the claim becomes a finding, and a diffusion model has a real argument against the one-token-at-a-time habit. If the score drops well below it, the lesson will be about the scaffold and not the model.

Watch for three things. First, whether DiffuSpace publishes the fine-tuning code and backend guides it has promised ^[2]. Second, whether it fills in the evaluation details. Third, whether anyone outside the company posts a number of their own.