Honest results from a small GRPO lab

This is not a leaderboard claim or a reproduction recipe. It is a small reasoning-model study about measurement, failure classification, and knowing when a successful intervention does not transfer. A public research snapshot is on github.

Reinforcement learning from verifiable rewards is the idea behind a lot of the recent reasoning-model progress. On math, you do not need a reward model guessing whether an answer is good. You can check it. The answer is right or it is not. That makes math a clean place to actually study what the training does, because the reward is not a vibe, it is a unit test.

I did not set out to beat anything. I set out to build a harness I could trust, run small models through it, and read every sample by hand. The interesting findings came out of one specific problem: the answer boundary.

The answer-boundary problem

When you grade a reasoning model on GSM8K, you have to extract its final answer from a wall of reasoning. If the model writes the right number but then keeps talking, or buries it mid-paragraph, a strict grader marks it wrong even though the model knew the answer. That is not a reasoning failure. It is a formatting failure, and the two get tangled together if you are not careful about how you measure.

So I tracked three things separately on every run: exact correctness, whether the final line was clean and parseable, and whether there was trailing text after the answer. Pulling those apart is what made the rest of the experiment readable.

smaller reasoning model │ ├─ separate correctness from answer formatting └─ targeted boundary intervention │ ▼ improvement on the smaller model │ ▼ transfer check on a stronger model └─ intervention rejected after regression

What worked on 3B

The smaller model needed help with answer boundaries. Left alone it knew many answers but lost credit by not stopping cleanly. A targeted intervention removed the trailing-text failure in the held-out check while preserving useful accuracy. The main result was not a new reasoning ability. It was cleaner, more measurable output.

What got rejected on 7B

The obvious next move is to take the recipe that helped the 3B and apply it to the 7B. It did not work, and I kept the result instead of quietly dropping it.

The stronger base model did not have the boundary problem in the first place. It already stopped cleanly. Applying the smaller model's fix was therefore solving a problem that did not exist, and the adapted version came out slightly behind the base model on exact accuracy. A paired uncertainty check supported rejecting the change.

So the honest write-up says: the boundary recipe is a real, stable win on 3B, and it is the wrong tool for the stronger model. Reporting "it worked once" without "it regressed elsewhere" would have been the easier story and the false one. The negative result is part of the public research record.

Where the 7B errors actually live

If the 7B is not losing points on formatting, where is it losing them? I built a failure taxonomy over the wrong answers to find out. Nearly every incorrect response still had a clean, parseable final-answer line. The model was not fumbling the format. It was getting the math wrong. Those are genuine reasoning errors, not extraction artifacts, and that distinction changes what you would do next. You do not fix reasoning errors with a formatting pass.

Why I build it this way

Verifiable rewards keep you honest. The grader is a real check, not a model's opinion, so you can argue with the number.
Separate the failure modes. Exact, final-line, and trailing-text are three different things. Collapsing them hides exactly the signal you need.
Keep the negative results. The rejected transfer is useful because it shows where the intervention stops applying.
Estimate uncertainty around deltas. A small accuracy difference can be noise, so promotion decisions need more than a point estimate.

The point of the lab is not a leaderboard score. It is a workflow I can run cheaply, read end to end, and scale to bigger models without changing how I reason about the results. Evidence over claims, with the evidence committed next to the claim.

Public research snapshot on github. This thread connects to Lethe, which applies the same evidence-first instinct to agent reliability.