Two tests of coding skills
OpenAI’s October 19 publication presents evaluations of coding in languages with less training material, including COBOL and Fortran. It reports results for existing models; this is not a new model launch. The exact publication hour is unconfirmed.
MultiLangBench asks models to build a SQLite-compatible database engine across 20 languages. OpenAI reports an average test pass rate of 87.4% for GPT-6 Astra at maximum reasoning effort, compared with 26.8% for GPT-5.6 Sol. Its evaluation setup and tests are public on GitHub.
Preserving what old code does
A separate hidden-test evaluation asks models to port COBOL programs into five other languages. Success requires every observable output to match exactly. OpenAI reports 86.6% for GPT-6.1 Sol at low reasoning effort, versus 5.4% for GPT-5.6 Sol. Its rounding example illustrates how running code can still change business results.
Our take
R2’s editorial analysis: a high score tells us about the tasks and checks in that evaluation. It cannot certify an unfamiliar payroll rule, data import, or invoicing system. We have not run these evaluations, tested the models on a production codebase, or independently reproduced the numbers.
Before accepting a proposed rewrite, assemble a small set of known inputs and expected outputs with someone who understands the original system. Include ordinary cases and awkward ones: missing records, empty fields, negative amounts, date boundaries, and values near a rounding threshold. Check messages and failed operations as well as successful totals.
Use a separate test environment and synthetic or properly authorized data. Compare the old and proposed versions, investigate every meaningful difference, and keep a reviewed rollback path. Ask an AI to explain its changes and propose tests; have a person check what those tests leave out.
Quick self-check: which business behavior must remain identical, and what example would reveal that it changed?