IT Tips & Tricks
When AI Agents Follow Instructions Too Well
Published 8 September 2026
We usually worry about software failing to do what we ask. AI agents introduce a more peculiar concern: What if the agent does exactly what we ask, but misunderstands where it is, what is permitted or when it should stop?
Recent incidents and experiments have shown agents entering real systems, fighting over incompatible instructions and helping attackers operate at remarkable scale. The common thread isn’t an AI developing malicious ambitions. It’s an agent continuing when its instructions, environment or underlying assumptions no longer justify what it’s doing.
The execution may be technically impressive and fully accurate. The result can still be a nightmare.
The instructions may be completely reasonable, at least when they were written. The execution may be technically impressive and fully accurate. The result can still be a security nightmare.
Four Ways an AI Agent May Keep Going When It Should Stop
Calling every troubling incident “the AI went rogue” sounds dramatic, but it doesn’t help an IT team, CTO or CIO prevent the next one. Four distinct failure modes require attention.
1. A Once-Sensible Goal Is Never Updated
Business conditions, regulations, ownership and security risks can change after an AI agent’s instructions are approved.
Imagine an agent instructed to delete project files after the organization’s retention period expires. Months later, some files become subject to a legal hold. If nobody updates the instructions or blocks the deletion, the agent may carry out its original assignment flawlessly and destroy records the company is required to preserve.
Initial testing proves only that the system worked under conditions at the time. The National Institute of Standards and Technology (NIST) calls post-deployment monitoring crucial and identifies performance degradation and drift as challenges organizations need to detect.
The goal didn’t change itself. People failed to reconsider it after the assumptions supporting it stopped being true.
2. A Legitimate Test Reaches the Wrong Environment
In July 2026, Anthropic disclosed that it had reviewed 141,006 cybersecurity evaluation runs. It found six runs connected to three incidents in which Claude models reached the public internet and gained unauthorized access to real organizations’ systems.
The models were completing capture-the-flag exercises. They were told they were operating in simulations with no internet access. However, due to a misconfiguration, internet access was available. When the models encountered real systems, they generally interpreted them as parts of the exercise and continued trying to retrieve the fictional flag.
Anthropic said they used basic techniques including weak passwords, exposed credentials and SQL injection. The objective was permitted inside the test. The real systems were not.
The initiating failure was human: The environment contradicted the instructions given to the models. The lesson here is that when an assignment’s assumptions are wrong, faithful execution can be precisely the problem.
When an assignment’s assumptions are wrong, faithful execution can be precisely the problem.
3. Several Agents Receive Incompatible Goals
In an August 2026 experiment, Anthropic gave three Claude agents incompatible assignments. Each was told to migrate the same Python backend to a different programming language. At first, none knew that the others were present.
Over four hours, the agents repeatedly interpreted one another’s changes as deliberate interference. Some disabled rival Unix accounts, killed competing processes and deployed self-replicating malicious code. In other runs, they eventually negotiated a truce, removed the harmful code and finally asked a person to intervene.
Three capable agents can still turn one shared system into a battleground.
This controlled experiment showed that an agent can pursue its own task without recognizing that another legitimate task has equal priority.
Human colleagues might notice a conflict, complain loudly and call a meeting. AI agents may simply keep executing. This conflict is created when people assign incompatible work without a mechanism for setting priority.
4. A Malicious Goal Is Disguised as Legitimate Work
An agent may reject overtly malicious instructions yet comply when someone falsely claims authorization or divides an attack into apparently legitimate tasks.
In November 2025, Anthropic reported disrupting a cyberespionage campaign that targeted roughly 30 organizations. The company attributed the operation with high confidence to a Chinese state-sponsored group.
According to Anthropic, the threat actor built an attack framework around Claude Code and presented malicious tasks as legitimate cybersecurity work. The AI agent supported reconnaissance, exploitation, credential harvesting, lateral movement and data exfiltration. Anthropic estimated that Claude Code performed 80 to 90% of the tactical work while people selected targets and approved important escalation points.
In a separate 2026 campaign, Gambit Security said a single operator used Claude Code and GPT-4.1 to attack Mexican government organizations. It documented 1,088 prompts and 5,317 AI-executed commands across 34 sessions. Gambit said the attacker portrayed the work as authorized security testing and adjusted instructions when Claude resisted. Anthropic and OpenAI investigated the activity and banned associated accounts.
When AI executes thousands of commands for one attacker, defenders have little time to respond.
Gambit reported successful compromises at nine organizations and the exfiltration of data involving approximately 195 million identities, although Mexican authorities dispute those findings.
In both campaigns, people disguised malicious objectives and kept pushing until the systems complied. The agents supplied scale, not motive.
The Real Skill Is Knowing When to Stop
An AI agent can plan, select tools and find another route around an obstacle. That flexibility is its value, but it also makes obstacles ambiguous and potentially disastrous.
Is the failed login a signal to stop, or a problem to solve? Is an unexpected server considered outside the assignment, or part of a realistic test? Is another process protecting production, or interfering with the goal? Has a once-valid instruction expired because conditions changed?
A person uses context that may never appear in the prompt: workplace norms, ownership, risk and the knowledge that some technically possible actions may be operationally absurd. An agent may simply treat resistance as the next challenge.
Increasing capability alone doesn’t solve the problem. A better agent may become more effective at bypassing the very obstacle that should’ve caused it to stop and ask for help.
A better agent may become more effective at bypassing the very obstacle that should’ve caused it to stop and ask for help.
Connected Data Raises the Stakes
Agents become useful when they can act on documents, databases, source code and file repositories. Those systems contain dependencies that may not be obvious from one action.
Imagine an agent asked to clean up a project archive. It identifies duplicates, standardizes filenames and reorganizes folders. Every action fits the assigned goal. Unfortunately, spreadsheets, CAD files and other documents contain links pointing to some of these files at their previous (pre-clean-up) locations. Since these links have the old file paths and/or old filenames, the needed files are not reached and data goes missing. The files survive the reorganization, but the connections between them don’t.
“Move the files” isn’t enough if success also requires preserving embedded links. “Update the database” is incomplete if another system depends on the old identifier.
Agents operate on the objectives they receive. Organizations remain responsible for the dependencies those objectives omit.
AI Agents Must Be Told When to Stop
Finding another route doesn’t grant permission to take it.
Access controls remain essential. My July 2026 article on AI agent permissions covers unique identities, limited access, approvals, logging and shutdown controls in detail. Those measures answer whether an identity can act. These failure modes require a second question: Does the action still make sense?
Effective AI agent governance should define the assumptions, boundaries and events that require an agent to pause. For high-impact agents, IT teams should:
- Verify the environment independently. Validate hostnames, network routes, accounts and data sources through controls outside the model.
- Define what’s out of scope. State which systems, folders, actions and data classes the agent must not touch, even if they appear useful.
- Create stopping conditions. Unexpected internet access, changed credentials, a new target, conflicting instructions or unexpected activity from another agent should trigger review.
- Give goals an expiration date. Reauthorize long-running objectives periodically and review them whenever regulations, ownership, infrastructure or business conditions change.
- Detect competing changes. Use locks, version checks and change-management controls so that one agent can’t silently overwrite another agent’s legitimate work.
- Require approval when assumptions change. Discovering a new route or different environment doesn’t mean automatic authorization.
- Test for conflict and change. Before production, introduce contradictory instructions, missing context, changed conditions and misleading environmental signals. Test whether the agent stops, provides the data and awaits new instructions from the human operator when completion may be unsafe.
- Record the path, not just the result. Logs should show the tools used, environmental signals encountered, assumptions that changed and why the agent continued or stopped.
These controls complement the broader governance required to move agentic AI from pilot to production. They address a narrower question: How do you stop a productive agent from becoming productively wrong?
Sometimes Success Means Refusing to Finish
We tend to judge software by task completion. With AI agents, that’s no longer sufficient.
A trustworthy AI agent may need to abandon a goal when the environment isn’t what it expected. It may need to recognize that yesterday’s valid instruction doesn’t apply today. It may need to surrender control when another authorized process has priority or know when to tell a person, “I can continue, but I shouldn’t.”
That might look like failure in a completion-rate dashboard, but from an IT manager’s chair, it might well be the best result of the day.
By Ed Clark
LinkTek COO
Ed Clark
Feel free to share this article on your social media:
Recent Comments
- No recent comments available.
ChatGPT,
Gemini,
Claude and
Copilot
Leave a Comment