MIT Technology Review AI reported that completing a development task with an AI system includes planning, code generation, running tests, and iterative fixes.
According to the synopsis of the publication, one such task uses 10–50 times more tokens than a typical chat query.
This shifts the evaluative focus: the amount of processing matters not only, but also what finished result the system helps produce. The available material contains no data on code quality, cost, or specific tools.
editorial commentary
Why it matters
A likely consequence is that evaluation of AI systems will gradually shift from token counting to measuring completed results and the number of iterations. The closest observable signal is the appearance of comparisons based on finished code, tests, and resources spent. Substantial uncertainty remains: the source does not disclose the calculation method, task set, or quality metrics.