GPT-6 Astra solved 99.9 per cent of the tasks in the ARC-AGI-3 benchmark using the OpenAI-optimised setup. When assessed using the standard evaluation tool for the same test, the model only managed to solve 62.7% of the tasks. The difficulty of proving that general artificial intelligence has arrived lies precisely in the gap between these two scores.
At the launch of GPT-6 Astra on 3 September, OpenAI’s president, Greg Brockman, suggested that in years to come, this model might be seen as the moment when artificial general intelligence, or AGI, was truly created. The term refers – although there is no universally accepted scientific definition – to artificial intelligence capable of learning, reasoning and adapting to a wide variety of tasks.
To underpin this ambition, OpenAI has published a series of spectacular results. Notably, Astra is reported to have achieved a score of 99.9% on ARC-AGI-3, a test designed to measure a system’s ability to acquire new skills in unknown abstract environments. The model also demonstrated high performance in mathematics, cyber security, programming and the autonomous use of software.
However, the ARC-AGI-3 score was achieved using a ‘Provider Adapter’, a proprietary software layer optimised for the model. When using the benchmark’s standard setup, the result drops to 62.7%. The ARC Prize Foundation itself points out that achieving saturation in its test does not prove the existence of general artificial intelligence: its closed, deterministic puzzles do not replicate the open-ended complexity of the real world.
‘Benchmaxxing’, the new religion of tech gurus
This controversy lends particular relevance to Stella Wohnig’s work. The postdoctoral researcher at the University of Luxembourg co-authored the paper with 15 other researchers «Benchmarking is Broken – Don’t Let AI be its Own Judge», presented at the NeurIPS 2025 conference. She currently works in the Cryptographic Protocols group at the Faculty of Science, Technology and Medicine.
The text does not deal with GPT-6 Astra, which was launched nearly a year later. It does, however, criticise the central mechanism used to make it a historic event: the publication of record scores by the company that develops the model itself. The authors describe a ‘Wild West’ of evaluation. Public question sets may have been incorporated into the training data, enabling the model to reproduce answers it has already encountered rather than solving a new problem. Developers can select the most favourable tests, adjust the instructions, increase the number of attempts or devote more computing power to an evaluation until they achieve the best possible result.
This practice now has a name in the industry: ‘benchmaxxing’. It involves modifying the prompts, the software environment or the resources allocated to the model in order to achieve peak performance. The score obtained therefore provides an indication of what the system can achieve under optimised laboratory conditions, but says far less about its immediate reliability for a user or a business.
The authors also highlight the lack of common standards. Each benchmark may use its own data, scoring methods and execution software. Some tests become so outdated that they are predictable; others remain proprietary and are therefore impossible to audit. Results produced under these conditions do not always allow for a comparison between two models or for the claims made by their designers to be verified.
The gap is already apparent in Astra’s assessments. According to results compiled by Artificial Analysis, the new model scored 61 points on its general intelligence index, the same level as GPT-5.6 Sol. It also fell in the relative ranking of a test covering professional tasks carried out in 44 different occupations. Setbacks were noted in customer service, scientific programming in Python and the processing of long documents.
The PeerBench project is more reliable
That is not to say, however, that Astra has made no progress. External evaluations confirm, in particular, a substantial improvement in its performance in certain programming tasks. But these sector-specific advances do not demonstrate that general intelligence comparable to that of a human has been achieved.
The study co-authored by Stella Wohnig proposes replacing the accumulation of rankings with a continuous, monitored and collectively governed assessment. Her PeerBench project involves tests that are kept confidential whilst in use, updated regularly, carried out in a closed environment and then made public to allow for auditing. The results would thus be less susceptible to memorisation, opportunistic adjustments and the selection of only favourable scores.
The authors compare this system to high-stakes human examinations, where the questions are secure, the sessions are supervised and the procedures are strictly regulated. Their question is simple: why impose stricter assessment rules on students or professionals than on artificial intelligence systems designed to be used in the fields of law, finance, medicine or cyber security?
The research carried out by the researcher at the University of Luxembourg therefore does not allow us to conclude that GPT-6 Astra is not a form of general artificial intelligence. It shows that the findings put forward by OpenAI do not, either, allow us to conclude that it is one. Before determining whether AGI has arrived, the industry must first agree on a credible way of measuring it.



