Maybe AI agents can be lawyers after all

Last month, I wrote about a new benchmark from Mercor that measures how well AI agents perform professional tasks such as legal work and corporate analysis. At the time, the results were underwhelming, with every central AI lab scoring below 25%. Based on those numbers, it seemed reasonable to conclude that lawyers were safe from AI-driven displacement — at least for the foreseeable future.

But AI capabilities can evolve rapidly, even within weeks.

This week, the release of Anthropic's Opus 4.6 significantly reshuffled the rankings. Anthropic's latest model achieved just under 30% accuracy in one-shot trials and averaged around 45% accuracy after multiple attempts on the same tasks. Notably, the update also introduced a range of new agentic capabilities, including so-called "agent swarms," which may have contributed to improved performance on complex, multi-step reasoning problems.

Regardless of the exact cause, the results represent a substantial leap over the previous state of the art and signal that progress in foundation models is continuing at a rapid pace. Mercor CEO Brendan Foody expressed particular surprise at the improvement, saying that "jumping from 18.4% to 29.8% in a few months is insane."

While a 30% score is still far from perfect and nowhere near the level required to replace human lawyers fully, it suggests that confidence in long-term job security should be tempered. Lawyers may not need to worry about being replaced by AI next week — but they may want to feel a bit less comfortable than they did just a month ago.