AI Can Recommend. Humans Still Have to Answer
In the previous post, I argued that graduates need to make AI use visible in their professional work. They should be able to explain what the system contributed, how they evaluated that contribution, and what decisions remained their own.
But transparency is only the beginning. Once AI’s role is visible, a more difficult question follows: Who is responsible for what happens next?
AI systems are increasingly capable of generating recommendations, ranking alternatives, identifying risks, drafting decisions, and initiating actions. In many workplaces, employees will not simply use AI to produce content. They will use it to inform choices that affect other people. A hiring system might recommend which applicants deserve interviews. A healthcare system might identify a patient as high risk. A financial system might flag a transaction as suspicious. An advising system might recommend which students need intervention. A cybersecurity system might classify activity as a threat. An AI agent might decide which messages require a response, which requests should be escalated, or which steps to take within a larger workflow.
These systems may help professionals process information more quickly and identify patterns that would otherwise be difficult to recognize. But they also create a temptation to separate the recommendation from the responsibility for using it. “The AI identified the applicant.” “The system assigned the risk score.” “The model recommended the intervention.” “The agent completed the task.”
These statements may accurately describe part of the process. But they can also make a decision sound as if it simply emerged from the technology rather than from a system designed, selected, configured, and used by people.
AI does not enter a workplace on its own. Someone chooses the tool. Someone decides what data it receives. Someone determines what it is allowed to do. Someone defines how its output will influence a decision. And someone chooses whether to accept, reject, question, or act on its recommendation. Responsibility may be distributed across many people and parts of an organization, but it does not disappear simply because AI participated.
This is why I believe one of the most important forms of career preparation in an AI-rich world will be teaching students to take responsibility for AI-assisted decisions. That sounds straightforward. We might simply tell students that humans are responsible for checking AI outputs. But checking is more complicated than it appears.
When an AI recommendation is clear, polished, and presented with confidence, it can influence how a person sees the problem before they begin their own analysis. Instead of independently evaluating the situation, the human may start from the AI’s conclusion and look for reasons to confirm it.
This tendency is often described as automation bias: the inclination to over-rely on suggestions or decisions produced by automated systems. Automation bias does not necessarily occur because people are careless. It can emerge because automation is introduced precisely where the work is difficult, repetitive, time-sensitive, or too large in scale for a person to manage easily.
Imagine being asked to review one AI-generated recommendation. You might approach it carefully. Now imagine reviewing hundreds of recommendations while also managing deadlines, messages, meetings, and other responsibilities. If the system is usually correct, accepting its recommendation can quickly become the normal path. Over time, review may become approval.
The human remains “in the loop,” but their presence may offer little meaningful protection.
Research on human oversight warns against assuming that adding a person to an automated process automatically creates accountability. A multidisciplinary analysis of effective human oversight concluded that oversight depends on several conditions, including whether people have sufficient information, competence, authority, time, and practical ability to intervene. A person cannot provide meaningful oversight merely because an approval button appears in front of them. (Sterz et al., 2024)
This distinction matters because organizations often treat human review as a solution to the risks created by AI. The system makes a recommendation, but a human makes the final decision.
On paper, that appears to preserve human control. In practice, we need to ask what making the “final decision” actually means. Did the person independently examine the evidence? Do they understand how the recommendation was produced? Can they recognize when the system is operating outside its area of competence? Do they have the expertise necessary to disagree? Are they allowed to override the recommendation? Do they have enough time to review it? Will they be expected to justify disagreement with the system while acceptance requires no explanation?
If those conditions are absent, the human may function less as a decision-maker and more as a rubber stamp. That creates an uncomfortable arrangement. The organization receives the speed and scale of automated decision-making while the individual employee appears to carry responsibility for each outcome.
Researcher Madeleine Clare Elish describes a related problem as the “moral crumple zone.” Like the physical crumple zone in a car, which absorbs the force of a collision, a person operating within a complex automated system may absorb the blame when that system fails—even when the person had limited understanding or control over its operation. (Elish, 2019)
This concept complicates the familiar statement that a human must always be responsible. Yes, AI systems should not become an excuse for avoiding human accountability. A professional cannot blindly accept a recommendation and then claim that the technology made the decision. But accountability must be connected to actual agency.
It is not reasonable to place full responsibility on an employee if the organization gave them an opaque system, insufficient training, unrealistic review expectations, and no meaningful ability to challenge its outputs. Responsibility for AI-assisted decisions exists at multiple levels. The person using the system has a responsibility to apply professional judgment. The organization has a responsibility to choose appropriate systems, establish policies, provide training, and create realistic conditions for oversight. Developers have a responsibility to test systems, communicate limitations, and design interfaces that support rather than manipulate human judgment. Leaders have a responsibility to determine where AI should and should not be used. Accountability cannot rest only with the person closest to the final decision.
The National Institute of Standards and Technology’s AI Risk Management Framework reflects this broader view. It recommends that organizations clearly define and differentiate roles and responsibilities for human–AI configurations. It also emphasizes cultivating a critical-thinking and safety-focused culture rather than treating risk management as the responsibility of a single user at the end of the process. (NIST AI Risk Management Framework)
This is important preparation for students because many will enter organizations where these roles are still unclear. They may be told to use AI without being told what they remain responsible for. They may be expected to review outputs without being trained to evaluate them. They may be asked to approve recommendations produced through processes they cannot inspect. They may discover that challenging the system creates more work than accepting it. They may even be held responsible for outcomes they had little power to prevent.
Preparing students for this environment means helping them understand both sides of accountability. They must learn not to hide behind AI. But they must also learn to recognize when responsibility is being assigned without sufficient authority, information, or control.
That requires more than technical skill. It requires professional courage. A prepared graduate should be willing to pause a process when the recommendation does not make sense. They should be able to ask what evidence supports it, what information may be missing, and who could be affected if it is wrong. They should know when a decision exceeds their expertise. They should know when to seek a second opinion. They should be able to document why they accepted or rejected a recommendation. And they should recognize when the safest decision is not to use AI for the task at all.
These habits are difficult to develop if students encounter AI only as a tool that produces answers. They need opportunities to practice making consequential decisions with AI-generated information.
For example, students could receive an AI-generated recommendation related to their future profession and be asked to decide whether to act on it. The recommendation should not be obviously correct or obviously flawed. It should include incomplete information, plausible assumptions, and trade-offs requiring professional judgment. The purpose would not be merely to find the AI’s mistake.
Students would need to identify what they would verify, determine who might be affected, explain the limitations of their own expertise, and decide what additional information they would need before acting. Most importantly, they would need to take a position. Would they accept the recommendation? Modify it? Reject it? Escalate it? Delay the decision? What responsibility would they accept for that choice?
Activities like this make accountability part of the learning process. They show students that evaluating AI is not only about determining whether an answer is factually correct. A recommendation can be accurate in a narrow sense and still be inappropriate, unfair, unsafe, or poorly suited to the context.
Professional responsibility begins where simple verification ends. It requires considering consequences. Who benefits from this decision? Who carries the risk? What happens if the system is wrong? Can the decision be reversed? Will the affected person have an opportunity to question it? Is the use of AI appropriate given the stakes?
These questions become especially important when AI recommendations affect people who cannot see or challenge the system influencing the decision. An applicant may never know that AI helped screen their résumé. A patient may not know that an automated system influenced how their risk was assessed. A student may not know that an algorithm helped determine whether they received additional support. A customer may not know why a request was denied or escalated.
The professionals using these systems may become the only people positioned to notice when the process is producing questionable outcomes. That makes responsibility more than an individual obligation. It becomes a form of protection for the people affected by AI-assisted decisions.
This is why preparing students for an AI world cannot stop at teaching them how to use the tools efficiently. They need to understand that AI can participate in a decision without bearing its consequences. The system does not face the rejected applicant. It does not explain the recommendation to the patient. It does not repair the relationship with the customer. It does not accept professional discipline when the decision causes harm. People do.
AI can generate possibilities, predictions, and recommendations. But humans and organizations remain responsible for deciding how much authority those outputs should receive. Our graduates will need to understand that responsibility does not begin after an AI system fails. It begins when someone decides to place the system inside the decision-making process.
And before students can responsibly use AI in any professional process, they must also understand what information should never be placed into it.
That is where the next post will turn.
Continuing the Conversation
Series 1: AI Is Exposing Existing Problems✓ Completed
Series 2: What We Do About It✓ Completed
Series 3: Cultivating Human Thinking in an AI World✓ Completed
Series 4: Learning Alongside AI✓ Completed
Series 5: Preparing Students for an AI World
Current Post (3 of 8): AI Can Recommend. Humans Still Have to Answer.
Next Up: Protecting Data in an AI-Enabled Workplace