The No. 1 AI Risk Is Losing Human Judgment: 8 Principles for an Ethical AI Future

AI’s greatest danger is not only that machines may be wrong. It is that people and institutions may stop questioning how they know. Building an ethical AI future requires reliability, meaningful transparency, human capability, and duties that follow power.

The Trolley Problem, With an AI Twist

The classic trolley problem asks whether you should pull a lever to divert a runaway trolley from five people onto another track, where it will kill one.

Now give the dilemma an AI twist. The lever is no longer visible. It has been embedded upstream in a model’s objective, training data, confidence threshold, ranking rule, or default setting. The system produces an answer that appears neutral, and the human accepts it because the output arrives with a score, a recommendation, or the authority of mathematics.

No one feels as though they pulled the lever. But a value judgment was made.

That is why I believe the number-one AI risk is epistemic. The danger is not only that a model may produce false information. It is that people and institutions may outsource the habits by which they question sources, weigh context, recognize uncertainty, challenge assumptions, and take responsibility for what they believe to be true. Cognitive offloading at institutional scale makes every other AI risk harder to see, contest, and govern.

In 8 Principles for Getting AI Nativity Right, I argued that AI cannot be considered only from a technological perspective. Its rapid ascent has consequences for ethics, equity, accessibility, power, and human agency. This article is the next step: how can an organization become AI-native without weakening the judgment that makes responsible adoption possible?

Manifestos Manifest Underlying Mistrust of AI’s Ethical Future

Mark Zuckerberg’s 2026 manifesto, The Future of AI Is for Everyone, argues that concentrating advanced AI in a few institutions is itself a safety risk. His emphasis on individual empowerment, pluralism, and balance of power matters. But distribution alone is not empowerment. If more people gain access to AI while losing the capacity to interrogate it, we have distributed dependence, not distributed power.

Sometimes, the messenger impacts the message. As TechCrunch points out, Zuckerberg’s past comments about social media being for “everyone” lower his credibility when he talks about AI, especially since social media has caused significant harm. But does that mean his message is incorrect? Should we overlook the positive aspects and the people using these platforms for good just because social media has its downsides?

That promise of distributed power must be tested against Meta’s record as a social-media gatekeeper. New Mexico has already secured a major youth-safety judgment, while California is helping lead a federal trial involving 29 states over alleged addictive design, misleading safety claims, and children’s data; related state and individual cases are also proceeding nationwide, and European regulators have preliminarily challenged Instagram and Facebook under the Digital Services Act. Meta disputes the claims and is appealing the New Mexico rulings, but access is not empowerment unless it is matched by enforceable safeguards, transparency, and meaningful redress.

Philosophical Problems for Ethics Can Be Solutions for Ethical AI

As I complete the my certification in Ethical AI from the London School of Economics and Political Science, I am working through Debbie Sue Jancis’s AI Ethics: Status of the Present, Ethical Dilemmas, and Frameworks for the Practical Mind. I was inspired to do this after reading that big AI companies are now actively hiring Philosophy Ph.D.s. For a deep dive on this, consult the linked article in the Atlantic.

The practical role of philosophy is clear. Ethics is one part of a broader responsible AI framework that also includes reliability, privacy, security, data governance, oversight, documentation, accountability, and redress. Ethics gives that framework direction.

1. Protect the Human Capacity to Know

AI should extend human thought, not replace the process by which judgment forms.

AI can summarize a report, draft an argument, rank a candidate, identify a face, estimate a medical risk, or recommend a course of action. The epistemic risk begins when the output becomes the end of the inquiry rather than the beginning.

Cognitive offloading is not inherently harmful. Humans have always used instruments to extend memory and calculation. The ethical question is whether the instrument expands human capability or quietly replaces it.

A 2025 Microsoft Research study surveyed 319 knowledge workers about 936 examples of using generative AI at work. Higher confidence in AI was associated with less critical thinking, while greater confidence in one’s own ability was associated with more. The study was self-reported and does not establish that AI causes cognitive decline. It does identify a governance challenge: employees asked to verify and supervise AI outputs need enough knowledge and confidence to do so.

2. Treat Reliability as an Ethical Obligation

Reliability is contextual: error rates, affected populations, and consequences all matter.

A system is not reliable simply because it performs well on average. Leaders must ask: reliable for whom, under what conditions, against which population, with what type of error, and at what human cost when it fails?

The ACLU of Minnesota says it sued on behalf of Kylese Perryman, whom it describes as falsely arrested and detained based solely on an incorrect facial identification. NIST testing has documented demographic differentials in face-recognition error rates, while also showing that results vary by algorithm, image quality, and use case. The lesson is not that every system performs identically. It is that reliability is contextual, and the consequence of an error matters.

Reliability therefore requires subgroup testing, context-specific validation, documented limitations, drift monitoring, incident escalation, and revalidation when the model, data, or use changes.

3. Treat Data, Privacy, and Security as Ethical Infrastructure

AI governance is impossible without governed, protected, and traceable data.

AI governance is impossible without data governance. Every system depends on choices about what data enters, what leaves, where it travels, who may access it, how long it is retained, and whether a vendor may reuse it.

A 2026 review in Frontiers in Digital Health examined AI models used to predict adverse pregnancy outcomes. Reported performance varied considerably, with AUROC values across the reviewed studies ranging from approximately 0.73 to 0.97. The authors identified eight forms of bias and noted limited external validation. They emphasized inclusive datasets, multisite validation, human oversight, transparency, accountability, and workforce capacity.

In maternal health, underrepresentation can contribute to missed interventions, false reassurance, unnecessary surveillance, anxiety, and deeper disparities. Privacy and consent also matter when sensitive data may be repurposed or retained beyond what a patient reasonably expects.

A model can be mathematically impressive and ethically weak.

4. Make Transparency Useful Enough to Support Redress

Transparency matters when it enables people to understand, challenge, correct, and appeal a decision.

Transparency should not mean burying users in technical documents or adding a generic statement that AI may be used. Its ethical purpose is to preserve agency.

A person materially affected by an AI-assisted decision should understand enough to act, including how to correct data and request human review or appeal.

A 2026 Stanford study followed 3.4 million people who submitted 4 million applications to 1,700 job postings across 150 employers, all screened by one vendor’s hiring tool. At the position level, 26 percent of Black applicants and 15 percent of Asian applicants applied to roles where recommendations showed adverse impact under the EEOC’s four-fifths rule. Pooling the results across positions would have hidden those disparities.

The four-fifths rule is a screening measure, not a final legal conclusion, and the study concerns one vendor rather than every hiring system. Its broader warning remains: the level at which performance is measured determines whether harm stays visible. The researchers also found that shared dependence on one screening architecture could repeatedly reject the same applicants.

5. Use Philosophy When Optimization Cannot Choose Between Values

Data can predict what may happen. Philosophy helps decide what ought to happen.

AI can optimize for an objective. It cannot decide whether the objective is morally justified.

Jancis applies several traditional theories to AI. Utilitarianism asks which choice produces the greatest overall benefit or least harm. Deontological ethics asks which duties and rights must be honored even when another result looks more efficient. Virtue ethics asks what a prudent, fair, or compassionate decision-maker would do. Pragmatic ethics tests principles against context and lived consequences.

The enterprise trolley lever may be a fraud threshold, candidate-ranking rule, medical-risk cutoff, content-removal standard, facial-recognition confidence level, or efficiency target that reduces human review.

AI does not acquire moral character through lived experience. It can be programmed to prioritize a value, but it cannot become honest, compassionate, or prudent in the human sense. Nor can it feel remorse, accept legal duty, or answer to someone it has harmed.

6. Put the Earliest Duty on the Actors With the Most Knowledge and Control

Accountability should follow knowledge, control, and the practical ability to prevent harm.

Who is better positioned to govern AI, private companies or governments? The answer is not either-or.

Governments establish enforceable rights, market-wide rules, remedies, and democratic accountability. Private companies often possess the earliest knowledge of capabilities, limitations, product architecture, security weaknesses, and deployment failures. They also control design, procurement, integration, testing, staffing, and release.

Regulation necessarily moves on a different clock from product development. The absence of a final rule cannot become ethical permission to do nothing. When an actor can prevent foreseeable harm at a reasonable and proportionate cost, its capacity to act can create a duty to act.

The Business Software Alliance has advanced a role-based approach across the AI value chain. Developers, integrators, and deployers have different information and different abilities to address risk. Responsibilities should fit those roles, but no company should be able to outsource ethics by contract.

7. Treat Human Upskilling as Governance and Competitive Advantage

Upskilling is not a perk outside governance. It is what makes meaningful human oversight possible.

“Human in the loop” is not a magic phrase. Sometimes the human is simply the final click.

Meaningful oversight requires authority, information, time, training, and a realistic ability to disagree. A reviewer who cannot inspect the evidence, does not understand the model’s limits, is penalized for overriding it, or must process hundreds of decisions per hour is not exercising independent judgment.

Upskilling is therefore not a perk outside governance. It is a control.

The Business Software Alliance has described AI skills as a cornerstone of widespread adoption and has placed workforce training among the pillars of enterprise AI adoption. Enterprise customers can distinguish between vendors that sell automation and those that also help organizations preserve expertise, document decisions, protect information, and manage risk.

Trustworthy deployment can become a unique selling proposition.

8. Build Sustainable AI That Sustains More Than Compute

Sustainable AI must preserve resources, human capability, knowledge practices, and institutional trust.

AI ethics and climate governance are not the same issue, but the institutional lesson of the sustainability and net-zero era is useful.

Aspirations became governable when organizations translated them into ownership, strategy, risk management, baselines, metrics, targets, controls, and disclosures. The structure reflected in IFRS sustainability standards offers a familiar analogy. A commitment without an owner, measurement method, evidence trail, and accountability mechanism is difficult to evaluate and easy to market.

The AI equivalent of a net-zero pledge without an emissions inventory is an AI principles page without an AI inventory, named owners, impact assessments, testing records, monitoring logs, incident files, or an appeal process.

That is ethics-washing.

Customers, boards, and buyers can demand evidence. They can ask how data is protected, how the system is evaluated, how employees are trained, how changes are documented, and what an affected person can do when the system is wrong.

Sustainable AI must also mean more than energy, water, hardware, and carbon. It must be environmental, human, epistemic, and institutional. It should preserve natural resources, human capability, the ability to question and verify, and the trust and due process on which institutions depend.

Conclusion: An Ethical AI Future Must Preserve Human Judgment

The goal of ethical AI is not to make machines virtuous.

Machines do not acquire wisdom through experience, accept responsibility, or understand human dignity as human beings do. The task is to build organizations capable of exercising those qualities while using powerful technology.

That requires reliable systems, protected data, useful transparency, meaningful oversight, role-based accountability, contestability, redress, and continuous investment in human knowledge.

The number-one AI risk is not simply that a machine will make the wrong decision. It is that humans will stop recognizing that a decision was made, stop asking what values shaped it, and stop believing they remain responsible for the result.

An ethical AI future is not one in which machines become moral. It is one in which human institutions remain capable of moral judgment.

Read More From Anant

• 8 Principles for Getting AI Nativity Right. A human-centered architecture for AI-native organizations.
• AI Safety: 10 Reasons It Must Be the Operating Priority of the Agentic Era. Why AI safety is becoming an operating priority for governance and engineering.
• AI Compliance Has a Clock. Most Companies Are Pretending It Doesn’t. Why evidence, ownership, and implementation timelines matter.
• How to Talk to AI Vendors Without Being an AI Tourist. The questions enterprise buyers should ask before deployment.

About Lili Kazemi

Lili Kazemi is General Counsel and AI Policy Leader at Anant Corporation, where she advises on AI governance, risk, compliance, contracts, and policy. She brings more than 20 years of experience across Big Law, Big Four, and federal government roles, with a background in international tax, regulatory strategy, and cross-border legal frameworks. She is completing her certification the London School of Economics and Political Science’s Ethics of AI Masterclass and writes The Human Edge of AI, a LinkedIn newsletter examining AI at the intersection of law, policy, work, and everyday life.

You can now also follow Lili on substack at https://substack.com/@lilikazemi
Subscribe – The Human Edge of AI

About Anant

Anant helps forward-thinking teams unlock the power of AI safely, strategically, and at scale. From legal and policy to data, knowledge, security, and enterprise systems, Anant helps organizations build workflows that act, automate, and aggregate without losing the human edge.

Disclaimer

This article is provided for general informational purposes and does not constitute legal, medical, or investment advice. It reflects the author’s independent analysis and should not be attributed to, or treated as an endorsement by, the London School of Economics and Political Science or any organization cited.