What IBM tested
IBM Research’s October 20 blog post tests ALTK-Evolve instructions learned from training attempts. Rules follow the customer-service policy in the prompt; model weights stay unchanged. The library predates this report.
IBM reports that a GPT-4.1 agent’s Pass^4 rose from 57.5% to 67.5% on 40 held-out retail tasks with 292 rules. That means success on all four attempts: four additional tasks. This was the strongest of three settings; 31 rules reached 60%.
The blog’s separate full-base comparison reports 65% higher retail conversation cost with all rules, counting agent and simulated customer together. It is not a production-service price.
Keep the leaderboard numbers separate
The linked retail and airline leaderboard pull requests were still open when checked. Both disclose modified prompts, include training tasks in the full-base evaluation, and say the authors expect an Unverified listing. These submissions are separate from the held-out tests. They do not establish an independently accepted leaderboard win.
Our take
For agent builders and experienced technicians, the useful idea is a playbook you can inspect and revise. A recurring mistake may call for a better instruction before a different model. But a rule learned from one workflow could be wrong for another, so review it against the actual policy before reuse.
Keep practice cases apart from the cases used to judge improvement. Repeat difficult tasks, examine failures, and measure the extra prompt cost alongside success. A small benchmark gain is a reason to test a method in a controlled setting; it cannot promise reliable refunds, bookings, or other real transactions.
We have not installed ALTK-Evolve or reproduced these results. The report is dated October 20; its exact posting hour was not established.