Weight-space programming on medicine: what a hand-made weight edit can and cannot fix

A weight edit written to stop a small model's math loops, with zero medical data in it, made that model better at medical exam questions. On the 1,273 MedQA-USMLE test questions, Qwen3.5-0.8B went from 24.4% to 32.9%. Two random edits of the same size and shape scored 25.5% and 25.2%.

That surprised us, so we went further. We rebuilt the setup for clinical questions and let a frontier model edit the weights of four small open models directly, with no training. Every run found the same obstacle, and it was not medicine.

Below: the method in plain words, the edits, the controls and the limits. The models, exact diffs and per-question results are open.

What weight-space programming is

In "Sparks of Weight-Space Programming", Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL) show that a frontier model can read a small model's weights and change them directly to fix a behaviour. It works like debugging a program you can only inspect:

• A notebook over the weights. The editor model gets a persistent Python kernel holding the small model's tensors: CPU only, no internet, gradients disabled.
• A small budget of forward passes. It can request real generations under the evaluation's sampling, or have the model read texts and return hidden states, activations and attention.
• Rules. No backward passes, no optimizers, no fitting to a loss, no simulating the network in the kernel. The final norm, the embeddings and whole-tensor rescaling are off limits.
• The job. Diagnose how the model fails, find the components responsible, write a targeted change, confirm it on questions it has not used.

"By hand" means the change is computed and written directly, the way you would patch code, not learned by gradient descent.

The paper's published math edit changed two MLP down-projections of Qwen3.5-0.8B (layers 15 and 19, 64 output columns each, 131,072 weights) to stop a thinking loop. In the paper, MATH-500 went from 7.9% to 48.1%. Hugging Face reproduced it with its ML Intern, shared by Niels Rogge, as ML-Intern-lab/Qwen3.5-0.8B-weight-space-programming. We checked that its two edited tensors are byte-identical to the authors' published diff and that every other tensor equals the release.

Step 1: the math edit on medical questions

Every number below uses the paper's evaluation protocol:

• Sampling. Thinking on, temperature 0.6, top_p 0.95, top_k 20, presence penalty 1.5, at most 32,768 new tokens, one sample per question, seed 0 (MedQA also with seeds 1 and 2 for the edited models).
• Grading. OpenMed's fixed MedQA prompt, graded strictly on the last Answer: X line after the thinking. Answers cut at the token limit or with no gradable line count as wrong.
• Benchmarks. MedQA-USMLE test (1,273), a fixed random 1,000 MedMCQA validation questions, and the six MMLU medical subsets (1,089).

The math edit gains +8.5 points on MedQA (95% CI +5.7 to +11.2, exact McNemar p = 4.4e-9); over three seeds the medians are 23.1% and 32.9%. MMLU medical rises from 43.4% to 52.4%. MedMCQA moves from 30.6% to 33.3%, not significant (p = 0.11).

The reason is the loops. As released, 18.3% of MedQA answers were still thinking when they hit the token limit. With the math edit, 0.1% were.

The edit knows nothing about medicine. It removed a failure that was hiding answers. The math side reproduces too: MATH-500 went from 7.4% to 50.6% and IFEval from 20.2% to 61.4%, close to the paper's 48.1% and 62.3%.

Step 2: rebuilding the setup for clinical questions

The paper publishes its prompts and rules but not its harness code, so we rebuilt the harness from them, with the paper's task text retargeted from math to clinical questions. We ran it on Qwen3-0.6B, Qwen3.5-0.8B, Qwen3-1.7B and Qwen3.5-2B.

• Editor. A frontier model, in the paper's role.
• Budget per model. 12 forward passes and 40 Python runs.
• Practice data. Training splits only: 10,178 MedQA-USMLE and 10,000 MedMCQA train questions. The editor never saw the test sets.
• Controls. For each model, two random patches: the same tensors, rows and columns, the same singular values, random directions.
• Release gate, pre-registered. On MedQA, beat the release and both random patches with p < 0.01 and a 95% CI above zero; MedMCQA and MMLU medical not significantly worse. All four models pass.

What the edits actually were

Our prompt ends with "finish with one last line exactly like: Answer: ". Under this sampling, the small models often copied the placeholder instead of filling it in: Answer: <B>, or literally Answer: <letter>. No strict grader reads that as a letter. All four editors found it in their first diagnostic generation.

The editors traced the copy to late attention heads. At the final Answer: position, these heads attend to the < in the prompt and write the direction that raises the logit of the < token. The presence penalty makes it worse: the four letters have usually appeared in the thinking already, while < has not.

Each edit subtracts that output direction from those heads' columns of the attention output projection: W_h <- W_h - s * U U^T W_h. U is the output direction of < (for Qwen3-0.6B, of < and <; for the two Qwen3.5 models, made orthogonal to the letters A to D). A strength s of 1 removes the component; larger values reverse part or all of it. Three of the editors picked s from dose-response tests on practice questions.

The edited rows and columns cover 0.039% to 0.240% of each model's weights. An independent check confirms it: in every edited tensor, one of the top two directions of the change has cosine 0.91 or more with the output direction of < (direction_check.json in each repo).

The editors' own estimates, measured on practice questions they had not used to choose the edit, pointed the same way as the test set they never saw:

Results, with controls

The random patches stay within 2.4 points of the release, so the direction matters, not the size of the change. Strict scores also rise on the other two benchmarks:

Nothing else broke: no model changed significantly on MATH-500 or IFEval (paired McNemar). The largest move was Qwen3.5-2B on IFEval, 62.1% to 58.4% (p = 0.14).

Read in any format, most answers were already there

The strict score only reads Answer: X. We also scored the same answers with a format-agnostic reader that accepts Answer: <B>, (b) and \boxed{B}.

For three of the four models, the paired any-format difference is not significant. Only Qwen3.5-0.8B gains beyond format: +6.5 points (95% CI +3.5 to +9.4). A likely reason is that many of its failures had no letter inside the placeholder at all: in the editor's diagnostic run on practice questions, 67 answers were Answer: <letter> and 56 were like Answer: <C>.

This is the paper's own conclusion: an edit can remove an obstacle that hides an ability, but it cannot add the ability. Our first read of the strict numbers said four models got better at medicine; the any-format column corrected us, and we now print it next to every small model's strict score. For scale, Qwen3.5-4B scores 90.4% on the same exam with no placeholder failures.

Stacking the math edit with one clinical head

The fifth model, Qwen3.5-0.8B-WSP-Math-Clinical, starts from the paper's math edit and adds one clinical edit to head 4 of layer 23, found in a longer run of 21 forward passes. It scores 45.4% on MedQA (median of 3 seeds 45.4%, range 44.1% to 47.0%): +12.5 points over the math edit alone (95% CI +9.7 to +15.3), while same-size random patches on top of the math edit score 32.4% and 34.0%. Its clinical edit is the same placeholder fix, in one head; read in any format, it goes from 39.4% to 45.4%.

What failed

• Loops in the clinical-only Qwen3.5-0.8B edit. 19.6% of its MedQA answers still hit the token limit. The editor tried scaling down four heads that sustain the loops and dropped it: it barely loosened them and hurt ordinary reasoning text more than three times as much. The paper's math edit is what fixes the loops, which is why the Math-Clinical model carries both.
• No knowledge repair. The longer run looked for a second mechanism (letter bias, negation questions, option binding, sampling noise) and found none; the remaining errors looked like knowledge errors with no localized signature. On Qwen3-1.7B, EXCEPT and NOT questions stayed harder (45% vs 58% for the others, with the edit), with no component to repair.

Limits

• Small models, one sample. 0.6B to 2B models, one sample per question; MedQA also ran with three seeds for each edit.
• Exams are not clinical practice. Multiple-choice scores say nothing about clinical safety. These models are for research on model editing and evaluation, not for clinical use.
• What the controls show. The random patches show that the direction of the edit matters, not its size. They do not show that medical knowledge was added; the any-format score shows most of the gain is format.
• One prompt. Our prompt literally says Answer: <letter>. With a prompt that shows no placeholder, the edits have less to fix.
• Side effects. The edits lower the probability of < wherever those heads write, not only in the answer line. The Qwen3-1.7B edit writes "<" less often in its thinking ("ANC <500": 10 vs 32 times on the same questions).

Try the models

Each card's quick start ran on the Hub checkpoints. From OpenMed/Qwen3.5-2B-WSP-Clinical:

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer


def main():
    repo = "OpenMed/Qwen3.5-2B-WSP-Clinical"
    tok = AutoTokenizer.from_pretrained(repo)
    llm = LLM(model=repo, dtype="bfloat16", max_model_len=34816, language_model_only=True, generation_config="vllm")

    # a MedQA-USMLE practice question (train split, id medqa_train_4541; the key is B)
    question = ("A 23-year-old male with a homozygous CCR5 mutation is found to be immune to HIV infection. "
                "The patient's CCR5 mutation interferes with the function of which viral protein?")
    options = {"A": "Reverse transcriptase", "B": "gp120", "C": "gp41", "D": "pp17"}
    user = ("Which option is the best answer to this clinical question?\n\n" + question + "\n\n"
            + "\n".join(f"{k}. {v}" for k, v in options.items())
            + "\n\nReason as much as you need, then finish with one last line exactly like: Answer: <letter>")
    prompt = tok.apply_chat_template([{"role": "user", "content": user}], tokenize=False,
                                     add_generation_prompt=True, enable_thinking=True)
    sp = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, presence_penalty=1.5, max_tokens=32768, seed=0)
    text = llm.generate([prompt], sp)[0].outputs[0].text
    print(text.split("</think>")[-1].strip())


if __name__ == "__main__":  # vLLM starts worker processes
    main()

Swap repo for any of the five models:

• OpenMed/Qwen3-0.6B-WSP-Clinical
• OpenMed/Qwen3.5-0.8B-WSP-Clinical
• OpenMed/Qwen3-1.7B-WSP-Clinical
• OpenMed/Qwen3.5-2B-WSP-Clinical
• OpenMed/Qwen3.5-0.8B-WSP-Math-Clinical

Each repo has the edit as an exact, hash-checked diff in edit/ and per-question results, scores and a run receipt in eval/. Everything is in the Clinical weight-space programming collection.

Credits

Thank you to Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL for the method and for publishing their prompts, rules, logs and diffs (lm_understand-logs); the math edit inside the Math-Clinical model is theirs, used under CC BY 4.0. Thank you to Hugging Face and Niels Rogge for the reproduction that started this, and to the Qwen team for the Apache-2.0 base models. Data: MedQA-USMLE, MedMCQA and MMLU.