Loading
How the Same AI Action Can Be Safe in One Situation but Reckless in Another
·7 min read
AI systems increasingly handle routine tasks such as sorting requests, flagging unusual activity, summarising information and recommending next steps. What happens after they complete these tasks can vary considerably. Consider a system that flags a payment as unusual. When it sends the alert to an employee for review, it helps bring something potentially important to their attention. When it automatically blocks the payment or freezes the account, a mistaken alert can prevent someone from accessing their money.
In both cases, the AI is performing the same task: identifying a transaction that appears unusual. The difference lies in what happens next. Does the result inform a person's decision, or does it trigger an action directly? What can that action affect, and can anyone intervene before it takes effect? These are the circumstances that determine how much risk an AI system creates. To understand whether an AI action is safe, we have to look beyond what the model does and examine the situation in which the organisation uses it.
This distinction is central to how AI risk should be assessed. The International AI Safety Report 2026 defines a deployment environment as the combination of an AI system’s intended use and the technical and institutional context in which it operates. It identifies three factors that shape the potential severity of failure: the criticality of the processes the system interacts with, the resources and channels it can access, and the permissions it has to take action.
An AI system's output can play very different roles within a business process. It might inform an employee's judgement, determine which cases receive attention, or trigger an action through an integrated system. Each arrangement gives the AI a different degree of influence over what happens next. The potential consequences depend on the importance of the process, the information and systems the AI can access, and the authority it has to act. These conditions shape the risk of deployment, even when the underlying model remains unchanged.
IBM’s analysis of the 2026 International AI Safety Report draws attention to the systems organisations build around AI. Risks emerge as models trigger business processes, access sensitive data and interact with other systems. A model that provides information for an employee to consider has a different role from one whose output directly determines what happens next. The model may be unchanged, but its place in the workflow has changed. So have the consequences of an error and the opportunities to intervene.
The EU AI Act gives this contextual approach a regulatory form. Under Article 6 and Annex III, certain AI systems are classified as high-risk because of their intended purpose and the specific categories of use they fall into. These include defined tasks in employment and education, access to essential services, creditworthiness assessments, and certain applications involving biometrics, law enforcement, migration and critical infrastructure. The Act also covers AI used as a safety component in certain regulated products.
Consider the difference between a computer-vision system that counts visitors at a retail entrance and one designed for remote biometric identification. Both process images, but remote biometric identification is expressly included in Annex III, subject to the Act’s conditions. The technical method alone does not determine the classification. The system’s intended use and the applicable legal criteria do.
High-risk classification is a legal category, rather than a complete assessment of everything that could go wrong. Systems outside the listed categories can still create serious operational or commercial harm through poor data, excessive permissions, inadequate oversight or the decisions that follow from their outputs.
The Act also distinguishes the responsibilities of providers and deployers. Under Article 26, organisations deploying high-risk systems must use them in accordance with their instructions, assign competent human oversight and monitor their operation. Article 25 sets out circumstances in which a deployer or another party becomes the provider: for example, when it rebrands a high-risk system, substantially modifies one, or changes the intended purpose of a system so that it becomes high-risk. Integrating a third-party model does not automatically make a company its provider. Its role depends on what it does with the system and how that changes its intended use.
Agentic systems make this question more complicated because their operating context can change as they carry out a task. An agent might read emails, retrieve documents, call external APIs, update records and delegate work to other agents. Each step can introduce new information, expose additional data or exercise another permission. The risk develops across the sequence of actions, rather than sitting entirely within one model response.
The 2026 Singapore Consensus on Global AI Safety Research Priorities addresses this directly in its companion report on agentic risk management. It argues that agent evaluations must reflect their level of autonomy, the sensitivity of their operating domain and the capabilities they can exercise. It also describes failures that emerge through interactions between agents: coordination problems, feedback loops and errors that propagate across a system. Individual agents may perform safely in isolation while their combined behaviour produces outcomes that none of their separate evaluations anticipated.
Security provides another example of how much the surrounding context matters. Google Cloud’s Mandiant AI Risk and Resilience 2026 report examines indirect prompt injection, in which untrusted content from sources such as webpages, customer emails or uploaded documents contains instructions that can influence an AI agent. The report recommends inspecting incoming context streams before they reach the model’s context window and treating retrieved material as untrusted input.
The danger comes from the relationship between that content and the system receiving it. A malicious instruction embedded in a document may have little practical effect on a model that can only summarise text. The same instruction can become a serious attack vector when an agent also has access to private records and permission to communicate externally or execute tools. What the model reads matters. What it can do with that information determines how far the consequences can reach.
Testing a model in isolation tells an organisation something about its performance under controlled conditions. It does not establish how the complete system will behave with real users, live data, connected tools, delegated tasks and the permissions it will hold in production.
Evaluation therefore needs to reflect the actual conditions of deployment. What information can the system read? Which records can it change? Can it initiate transactions or contact external services? Which actions require approval? What happens when retrieved information conflicts with established instructions? How can an error travel into the next process, and can the consequences be reversed?
These questions become particularly important when a system's role expands. An agent tested with read-only access and human approval has not thereby been shown safe with write permissions or the authority to complete the same task independently. Adding a new integration, connecting a sensitive data source or allowing an agent to delegate work can materially change its risk profile.
Model quality and architecture still matter. They simply cannot establish deployment safety on their own. Organisations need to assess the complete workflow, including its data, integrations, permissions, human oversight and downstream effects. That assessment should be revisited whenever those conditions change.
The same AI action can be proportionate in one setting and reckless in another because the organisation has changed what that action can affect. This is also why decisions about autonomy need to be made at the level of the actual work a system performs.
Should you give AI more autonomy? explores the governance decisions that follow.