You're Still Paying Full Price to Build the One Skill AI Just Made Worthless
Sixteen professors. Fourteen schools. Three thousand questions. The machine won the half of lawyering we already knew how to automate, and the half your clients pay a premium for was never on the test
What Law Schools Just Learned About AI, and Why It’s a Warning for Law Firms
TL;DR: A Stanford study had law professors blindly grade answers to student questions, and they picked the AI over their own colleagues about 75% of the time. Read it as a tutoring story and it’s a curiosity. Read it as a story about how lawyers actually get made and it turns into a warning, because the two things that have always built a lawyer, the grind work and the hours spent next to someone who knows more than you, are getting automated in the same stroke. Skip the part where you replace them and you’ll be running a firm full of associates who are quick, fluent, and can’t tell when they’re wrong.
How lawyers actually get made
Think back to how you became good at this.
Odds are it didn’t happen in a classroom. It happened at 2 a.m., in a document review, the night you finally saw the pattern nobody had bothered to explain. It happened when your first real memo came back from a partner so marked up you could barely find your own words underneath. It happened in a hallway, when someone with twenty years on you said one sentence that made a doctrine click in a way three years of school never had.
That’s the apprenticeship. Grind plus proximity. You did an enormous volume of junior work under supervision, and you absorbed judgment by standing close to people who already had it. Every lawyer reading this was built that way.
A new study out of Stanford is being passed around as a story about law students and AI tutors. It’s getting the easy headline. But read it as a measurement of that apprenticeship, and it says something most firms have not noticed: the ground the next generation trains on is washing out from under them, and it’s going from two directions at once.
What they actually did
The setup is worth a minute. Sixteen contracts professors, fourteen law schools. They wrote the questions a 1L actually asks after class, answered them in their own words, then turned around and judged the answers blind. Roughly 3,000 head-to-head pairs. Some from other professors, some from AI, nobody told which was which.
The AI won about 75% of the matchups. It held its own with the best human in the room, and the weaker humans got buried. The number that stopped me wasn’t the win rate, though. It was harm. When a judge marked an answer as the kind that sets a student back, the human answers drew that flag more than three times as often as the AI’s. 12% against 3.5%.
And keep in mind what the task was: pick the answer you’d actually rather hand a student, all things considered, not the one that scores best on a rubric. On settled doctrine, the stuff with a known shape, the one they kept reaching for was the machine’s.
The training ground is disappearing
Those two pathways, grind and proximity, are both being automated. And this study is the proof of the second one.
Start with the grind. The junior work that taught you, document review, cite-checking, first drafts, research memos, due diligence, is the exact work these tools now do in minutes. Firms are already shrinking first-year classes and arguing about the shape of the pyramid. Whatever you think about the economics, watch what it does to training. Partners romanticize this part, the war stories, the all-nighters, and I roll my eyes at most of it. But strip out the nostalgia and the point survives: the work that made lawyers was the work nobody wanted to do, and it’s the first thing to go.
Picture the change at the individual level. A first-year used to lose a weekend to a research memo, and the losing was the point. The dead ends, the case that looked perfect until the last paragraph, the slow build of a feel for where the law actually lives. Now that memo takes twenty minutes and a decent prompt. The work gets done. The learning that used to come with it does not.
Now the proximity. For decades, the answer to “I don’t understand mutual assent” or “remind me how the parol evidence rule works” came from a person, a professor in office hours, a senior associate down the hall. That’s the clarification layer. The Stanford study just showed that for those questions, a frontier model gives an answer experts rate higher than what a rushed human produces, and is less likely to mislead.
So both rungs of the ladder are being sawed off. The boring reps that built pattern recognition. The on-demand explanation that filled the gaps. A 25-year-old with a JD used to climb from one to the other. Take both away and put nothing in their place, and you don’t get a faster lawyer. You get a fluent one who never learned to judge.
Why the machine won
This is where I’d slow down, because the reason the AI won is the actual warning.
The professors lost at recall-and-explain. Asked to perform in three minutes, with no time to research, they did the same thing the model does: pulled up a known rule and explained it cleanly. It’s very good at that, and it doesn’t get tired, annoyed, or pulled into a partner meeting.
But recall-and-explain is the cheapest kind of legal knowledge to acquire and the least valuable to own, precisely because AI now owns it too. And it’s the thing junior training still rewards most. We hire for it. The bar exam is mostly a test of it. And years in, we still light up at the associate who can recite the standard cold.
Sit with what that means. We’re spending enormous effort turning young lawyers into excellent versions of the one skill that just got automated. The recall-and-explain associate is a depreciating asset. The machine reset the price of it to roughly zero, and plenty of firms are still paying full freight to build it.
The part that wasn’t on the test
Now ask what the study didn’t measure, because that’s the part clients still pay a premium for.
It didn’t test knowing a settlement number is wrong when the spreadsheet swears it’s fine. Or reading a general counsel well enough to know she wants the careful answer this quarter, not the clever one. Nothing in it touched the instinct to run a conflict check before anyone asks, or the feel of a deposition when a witness shifts in his chair and you know the case just turned.
That’s judgment. It gets taught by reps on real stakes and by proximity to people who have it, the two things AI is now eating. And the machine that explains the parol evidence rule beautifully will still invent a case citation with total confidence. I’ve written about that verification gap before, and it has not closed. So the human in the loop isn’t going anywhere. They are moving up, from the person who knows the rule to the person who knows when to break it and is willing to put a name on the call.
A gap is opening between those two roles. I’ve written about the verification gap; this is a different one. Call it the apprenticeship gap: the distance between automating the work that used to train lawyers and building anything to replace it.
The objections, and where they hold up
Some of the pushback is fair, so let me take it head on.
The humans were rushed. Completely true. They had three minutes and no research, and a partner who sits down and thinks would beat the model on most of these. But that cuts the wrong way for comfort. The rushed expert is exactly the version a junior usually gets, and the machine is the version that’s always there and never irritated. No one here showed that AI can practice law. What they showed is smaller and more uncomfortable: on settled doctrine, an always-available machine out-explains a distracted human, and the distracted human is the one most juniors actually reach. That is a real hole in how you train them.
We’ve seen tech panics before, and people who cry wolf about automation tend to be wrong. I make my living on this technology, so weigh the next sentence accordingly. The mechanism here is not spreadsheets replacing paper ledgers. It’s the training itself going away, which is a different kind of problem, and a quieter one, because nobody files a memo the day an apprenticeship stops working.
And yes, this is one model family, one casebook, contracts, short answers. It is not proof that AI can run a matter. It’s a clean look at one rung of how lawyers learn. Worth noting, too: the strongest models on the market now are a step past what the study even tested. Its top performer was Claude Opus 4.7. Today’s leaders, Claude Opus 4.8 and GPT-5.5, are newer still, and the gap to human instructors in this kind of task has only widened.
What to do Monday
None of this is “form a committee.” It’s three habits, and they’re harder than they sound.
The first is just honesty about your own training reps. Walk through what your juniors actually do to learn, and mark the ones the machine now does for them. If AI writes the first research memo, nobody learned anything by writing it, and pretending otherwise gives you an associate who believes they’ve done the work. Find the reps that still build judgment and guard them like they cost something, because they do.
The second is to move your senior people off explaining and onto judging. Partners answering “what’s the rule” is a waste now; AI has that cold. What the machine can’t do is sit on a live matter and say out loud, with a junior listening, here’s why I’m going to ignore the rule this time, for this client, this week. That sentence is the entire job. Spend the expensive hours there.
And the third, the one almost nobody does, is to say what you’re training people to become. Out loud, in plain words. You’re building judgment and the nerve to own a call. Then go make your reviews and your raises line up with that, which is the uncomfortable part, because it stays a lot easier to reward the associate who can recite the standard cold.
So here’s where I land. The machine didn’t beat the professors at being lawyers. It beat them at the one slice of being a lawyer we already knew how to automate. The firms still standing in five years will be the ones that quit paying people to be good at that slice and start spending real money, and real partner time, on the part that was never on the test. I keep having this conversation, and the leaders who get it have stopped asking how to make their associates faster. They’ve started asking what their associates are supposed to become.
If you enjoyed this, please share it with three friends. If you want to chat further on it, reach out to me at steve@intelligencebyintent.com. If there’s a topic you’d like me to cover, let me know.