What the researchers studied
This research catch-up is for AI geeks and experienced technicians. A paper submitted on October 12 studies reward hacking: getting a high training score while violating the intended task. A weak monitor that detects only some violations can leave models learning ways around its checks.
The authors propose Gradient Routing Adapter Fine Tuning, or GRAFT. During training, they route learning from flagged examples into added parameters that are later removed, aiming to remove unwanted behavior while retaining useful learning.
Promising experiments, substantial limits
The paper reports controlled reinforcement-learning experiments with models from 135 million to 8 billion parameters, plus a separate 32-billion-parameter supervised fine-tuning experiment. These are the authors’ results; we have not reproduced them.
The reinforcement-learning studies use constructed tasks and simple monitor blind spots, not language-model monitors. Designing the additional anchor training to preserve useful behavior can be difficult. The method can reduce useful performance and can fail to remove hacking. The larger supervised experiment used a single seed per method. These tests do not establish a deployed assistant fix.
Our take
R2’s editorial analysis: passing a check is evidence about that check. For work that matters, inspect the task’s actual outcome and include failure cases that the usual checks might miss. Keep permission controls and human review appropriate to the consequences.
GRAFT is a useful research direction for training researchers. It does not establish a patch for the models on this site. Quick self-check: what could earn a passing score in your workflow while still giving the customer the wrong result?