AI Can Pass the Bar—But It Still Needs Supervision

What This Article Is About

Legal AI has arrived in force, promising to transform everything from document review to motion drafting. Some firms are achieving remarkable efficiency gains. Others are learning hard lessons about the limits of automation. As AI tools flood the legal market, this piece cuts through the hype to deliver what matters: a practical framework for integrating AI into legal practice without compromising professional standards or client outcomes.


The Great Legal AI Gold Rush

The legal tech landscape is experiencing its own version of the California Gold Rush. Dozens of companies now offer “legal agents”—AI tools that promise to automate core legal functions at superhuman speed. The pitch is seductive: first-year associate work completed in minutes, not days. Perfect recall of case law. Flawless document generation.

But here’s what the sales decks don’t emphasize: speed without judgment is just fast failure.

Consider what happened this week at Butler Snow. Three attorneys were disqualified by a federal judge after submitting AI-generated filings containing fabricated case law—classic AI hallucinations dressed up as legal precedent. The attorneys admitted they hadn’t adequately reviewed the AI’s output before filing. That’s not just embarrassing. It’s potentially career-ending.


Why AI Passing the Bar Doesn’t Mean What You Think

Yes, GPT-4 scored in the 90th percentile on the bar exam. Stanford researchers confirmed it. Impressive? Absolutely. But let’s be clear about what this means—and what it doesn’t.

The bar exam tests knowledge recall and analytical reasoning within defined parameters. It doesn’t test for:

  • Client empathy when delivering difficult news
  • Strategic intuition about which arguments will resonate with a particular judge
  • Professional judgment about when to push forward or pull back
  • Ethical navigation through complex conflicts of interest
  • Pattern recognition built from years of practice

As one practicing attorney put it bluntly on X: “AI routinely hallucinates. It makes stuff up. It simply cannot be trusted without careful scrutiny.”

The Human-AI Partnership Model

Here’s where we need to reframe the conversation. The question isn’t whether AI will replace lawyers. It’s how lawyers can leverage AI while maintaining the professional standards that protect clients and preserve the integrity of our legal system.

What AI Does Well:

  • Rapid document analysis and pattern identification
  • First-draft generation for routine filings
  • Research acceleration across vast case databases
  • Standardized contract review and redlining
  • Timeline construction and fact organization

What Humans Must Still Do:

  • Verify every citation and legal principle
  • Apply strategic judgment to case theory
  • Navigate nuanced ethical considerations
  • Build trust and rapport with clients
  • Exercise the intuition that comes from experience

The Compliance Imperative: Courts Are Watching

The legal system is responding swiftly to AI’s emergence. In a recent example garnering news attention, Judge Craig L. Schwall Sr. of the Fulton County Superior Court (Atlanta Judicial Circuit) has issued a standing order requiring mandatory disclosure of AI use in legal filings. The order requires:

  • Disclosure of any AI use in legal filings
  • Identification of specific AI tools employed
  • Certification of accuracy after human review
  • Documentation in footnotes or separate statements

This isn’t isolated. Over a dozen courts now mandate AI disclosure. The message is clear: AI use in legal practice is now a compliance issue, not just a technology choice.

Lawyers Have to Face It: AI Will Be in the Federal Rules of Evidence

As AI-generated evidence becomes more common in litigation, the courts are beginning to codify how it should be treated. A newly proposed Federal Rule of Evidence 707 would require AI-generated output—like summaries, images, or analytical conclusions—to meet the same reliability standards as expert testimony under Rule 702 (Daubert). In other words, if a party presents AI-derived material without a human expert, the burden will be on them to prove the system’s methods are trustworthy. This marks a turning point: AI is no longer just a tech issue—it’s a matter of admissibility.

The Junior Associate Paradox

Andrew Yang recently posted on X that AI generates motions “in an hour that might take an associate a week. And the work is better.”

Let’s unpack this. Yes, AI can generate text faster. But “better” is doing heavy lifting here. As someone who spent years as a junior associate, I remember the countless drafts that came back covered in red ink. A partner coming into my office asking me if I had reviewed X case. In many cases, I had, but didn’t see the relevance the same way the partner with 30 (or sometimes 40) more years of experience on me had. Or, maybe I didn’t see the case because [big legal research engine] didn’t pull it up in the keywords. But that partner had an instinct that it was “out there”. And sometimes, that would be all he said. “There’s some case out there about a 1031 exchange involving these crazy facts in 1971 – go find it.” I clerked for a federal judge, and it was the same vibe.

Those weren’t just corrections—they were lessons. Each revision taught me to think more precisely, argue more persuasively, and anticipate counterarguments more effectively. And what I learned also? No matter how hard I looked, there is always some other rabbit hole to go down. In many instances, the skill involves stepping back and listening to your gut. Your intuition When I use AI in my day-to-day legal research, I have that partner’s voice in my head. And my intuitive voice is on high volume. But I’m not a junior associate anymore. And we (the more senior lawyers) have to recognize that the challenges are really different if you graduated law school since 2021. It’s a different world.

AI can be fine-tuned with more data and shaped by reinforcement learning—but it doesn’t learn from experience the way lawyers do. It doesn’t absorb redline comments, internalize strategic nuance, or build courtroom instincts. The kind of feedback that turns a junior associate into a trusted counselor—real-time correction, professional stakes, pattern recognition across messy cases—is still uniquely human.

A Practical Framework for Legal AI Integration

Based on our work with legal teams navigating this transition, here’s a tested approach:

1. Start With Parallel Workflows

Run AI tools alongside traditional processes before going live. Compare outputs. Document discrepancies. Build confidence through verification, not faith.

2. Implement Rigorous Review Protocols

  • Every AI output requires senior review
  • Maintain audit logs of all AI interactions
  • Version control must track human modifications
  • Create “hallucination checklists” for common errors

3. Ask Vendors the Hard Questions

  • What training data underlies their model?
  • How is client data stored and protected?
  • What is their RAG framework?
  • What are documented error rates?
  • Where does liability fall when errors occur?

4. Train Everyone—Not Just Associates

Partners need to understand AI capabilities and limits. Paralegals need to spot hallucinations. Clients need transparency about how their matters are handled.

5. Make AI Training in Law School Mandatory

Law schools should implement comprehensive AI training to help future lawyers understand how to spot hallucinations, evaluate AI-generated work, and apply legal judgment. While some schools are making progress—Case Western now requires AI literacy for all 1Ls. WashU Law is actively integrating AI into its curriculum, starting with the Class of 2025. Their approach embeds AI into the first-year Legal Research program, teaching students both traditional and AI-assisted research methods to prepare them for a tech-driven legal future., and Harvard, Berkeley, and Penn have launched formal tracks—only about 62 U.S. law schools offer AI-related courses. That’s barely a third of all law schools. The gap between tech development and legal education is still wide—and closing it is essential.


The Bottom Line: Augmentation, Not Automation

The future of legal practice isn’t human versus machine. It’s human with machine—each doing what they do best. AI can be your tireless research assistant, your first-draft generator, your pattern spotter. But it cannot be your judgment, your ethics, or your intuition.

Remember: Jobs aren’t lost to machines—they’re lost to people who know how to use machines effectively.

The firms thriving with AI aren’t the ones treating it as a magic solution. They’re the ones building thoughtful frameworks that enhance human capability while maintaining the professional standards our clients deserve.

What’s Your Move?

As AI tools proliferate and courts tighten oversight, every legal professional faces a choice: resist the technology and risk obsolescence, or embrace it recklessly and risk malpractice.

There’s a third way: thoughtful integration that respects both the power of AI and the irreplaceable value of human judgment.

At Anant, we help legal teams navigate this transition with frameworks that work in practice, not just in theory. Because in law, as in life, the difference between success and failure often comes down to knowing not just what you can do, but what you should do.

Ready to build your AI integration strategy? Let’s talk about turning this framework into your firm’s competitive advantage.

About the Author: Lili Kazemi is General Counsel and AI Policy Leader at Anant Corporation. She advises on the legal, regulatory, and practical challenges of AI integration in professional services. Connect with her on LinkedIn for more insights on the intersection of law, technology, and human judgment.

🔍 Discover What We’re All About

At Anant, we help forward-thinking teams unlock the power of AI—safely, strategically, and at scale.
From legal to finance, our experts guide you in building workflows that act, automate, and aggregate—without losing the human edge.
Let’s turn emerging tech into your next competitive advantage.

👇 Click on the image to subscribe to our weekly AI newsletter: