A test of demanding work
A benchmark is a set of tasks used to compare systems under a particular test setup. SemiAnalysis's GDPx v1.0 lists 206 tasks across 11 specialist domains, with 14 data partners. [1]
These tests examine AI agents: systems that use tools while working through several steps. The article describes demanding work in fields including software engineering, finance, and chip design. Models use their native agent software where available, so the surrounding software is part of the test conditions. [2]
SemiAnalysis dates its announcement November 2. We checked the public leaderboard and visible article text; the linked newsletter is marked as a paid post. [2]
What the headline score means
At our check, the default leaderboard displayed a top weighted score of 12.0%. This does not mean 12% of the tasks were fully solved, or that AI can do 12% of people's jobs. [1]
Each domain counts equally overall, and partners within a domain are also weighted equally. The tasks combine pass-or-fail grading with partial-credit scoring; the default result uses each task's intended grading rule and averages three attempts. It is therefore different from a simple count of completed tasks. [1][2]
We have not reproduced these evaluations. Their specialist scope and testing conditions matter when interpreting the results.
Our take
For a small business, the useful lesson is to define what finished work must satisfy. An assistant updating a mock expense sheet should preserve the original figures, calculate correct totals, and flag missing receipts. Inspect the file against those requirements instead of trusting a polished explanation. Your own task, the consequences of an error, and the review you can provide matter more than a leaderboard position.