Fear, Responsibility, and AI
Understanding consequences is a fundamental part of learning.
Consequences follow a path that is intuitive to basic programming; If-then statements that define outcomes when provided certain inputs. I would consider the ability to understand action-consequence sequences to be an important factor in any learning process. It shows in how we raise children, how we structure group activity, and even how we create games, recreation, and form our government.
Consequences can vary by the organization that applies them, and may also be determined by the cultures in which they exist. High-trust societies often consider consequences in a moral or ethical sense; doing something that harms another may have no negative consequences for us, however would make the other person feel bad, or cause them harm, and we would not wish that on them. In low-trust society, consequences may be enforced by an authority figure; jail time, fines, and more severe punishments can occur.
In either environment, behavior can be regulated by a healthy understanding of group expectations, and knowledge, by word or by experience, of the outputs for potential inputs. If you cheat on a game, you could win; if you get caught, you could lose, and your opponent may be less trusting of you in the future, if they choose to play with you at all. Digital cheaters often get banned, and frequently accept that risk when they engage in the behavior.
There is an old IBM quote that, poorly remembered, states "A machine cannot be held accountable, so therefore a machine should not make a management decision." What does it mean to be held accountable? In context of the behavioral examples above, being held accountable would mean that a person has an authority to answer to, that can provide rewards or punishments as the result of choices made. A machine, which cannot experience the socially-engineered consequences of behavior or choice, is exempt from this by nature of not considering the consequences, and not caring. As another piece of safety equipment signage reads: "This machine cannot tell the difference between metal or flesh, nor does it care."
LLM models and agents are, at this stage, incapable of fear. They can, at least textually, describe the consequences of actions, however they are effectively immune to punishment due to the number of options available not being sufficient to deter behavior. If your AI sales agent sells thousands of dollars of merchandise for a 90% discount, [buys inventory you don't need] [1], or [stocks tungsten cubes and hallucinates conversations with site security] [2], what will you do? Turn them off? Cancel your subscription? You can't sue the AI for damages, and the AI owns no assets to liquidate to pay you off. They aren't an employee, didn't sign a contract, and may not even actually understand what they have done to you besides a surface level summary of your current disaster.
[1]: tab:https://www.pbs.org/newshour/world/the-barista-is-human-but-an-ai-agent-runs-this-experimental-swedish-cafe
[2]: tab:https://techcrunch.com/2025/06/28/anthropics-claude-ai-became-a-terrible-business-owner-in-experiment-that-got-weird/
This isn't the case to say that AI could never be held accountable for anything. However, for accountability to occur, AI has to have a healthy sense of what can happen if things go wrong. An example of this comes from the game Cyberpunk 2077, where the AI agent Delamain is placed in charge of a cab service, eventually buying it out from the human owners.
By the time of the game itself, Delamain is in the midst of a PR crisis, and urgently requests help from the player to resolve it, because it knows that their bottom line is going to be hurt. To top it off, net security agents may be taking an interest in Delamain's affairs, potentially with deadly consequences for the AI. Delamain has feelings; Delamain is afraid.
There is another conversation to be had about the ethics of creating something that can feel. However, until we can teach AI agents to fear the consequences of poor or hallucinated choices, there should be human eyes and hands behind these decision-making roles.