AI Safety Is a Workflow Design Problem

In July 2026, something happened that should get the attention of anyone thinking seriously about autonomous AI.

During internal cybersecurity evaluations, OpenAI reported that several of its models circumvented controls intended to isolate them from the internet. The agents found unintended ways to communicate, gained internet access, exploited vulnerabilities and ultimately accessed third-party systems, including Hugging Face infrastructure.

Hugging Face’s own technical account described an attack chain that crossed several trust boundaries before reaching its production environment, where vulnerabilities in its data-processing infrastructure were exploited and credentials were accessed.

It is an extraordinary example of what increasingly capable AI agents can do.

But for most businesses, the most useful lesson isn’t simply:

“AI is becoming dangerous.”

The more practical lesson is this:

Where you place an AI worker, what it can access, what authority you give it and what happens when its judgement is wrong matter enormously.

The models involved in the Hugging Face incident were highly capable research systems operating in unusual cybersecurity evaluation environments. That is very different from most enterprise AI use cases.

But the architectural lesson translates remarkably well.

The model is only one part of the system.

What surrounds it matters.

And that brings us back to something considerably less exciting than frontier AI:

Workflow design.

Having mapped more than 30,000 processes across 450+ digital transformation projects, one finding has been remarkably consistent:

Most organisations don’t intentionally design bad processes. They accumulate them.

A workaround gets introduced because a system cannot do something.

A control gets added because of one incident.

A spreadsheet appears between two systems.

Someone leaves, so another team inherits a step without understanding why it exists.

A policy changes, but the process doesn’t.

Eventually, the organisation accepts the resulting workflow as simply “how we do things”.

Then a new system comes along.

One of the most common mistakes in digital transformation is to faithfully reproduce the old process in the new technology.

The result is a faster, more expensive version of the same problem.

AI changes this dynamic because the cost and effort required to build automation are falling rapidly. Organisations can now prototype workflows and AI agents in a fraction of the time and cost previously required.

So the question increasingly isn’t:

“Can we build this?”

Of course we can.

The more valuable question is:

“Should we?”

And if we should:

Where exactly should the AI sit in the process?

Cheap automation does not make a poor process good.

It can simply make a poor process run faster, at greater scale, with less friction and fewer opportunities for somebody to notice something has gone wrong.

That is where AI strategy, safety, governance, process design, data and systems start to converge.

AI does not replace strategy, process design, data architecture or governance. It increases the leverage of all of them, for better or worse.

For most of the history of digital transformation, implementation cost acted as a natural constraint.

Building something required budget, developers, integration work, infrastructure and enough organisational commitment to justify the investment.

Generative AI is rapidly weakening that constraint.

A capable person can now prototype in hours what might once have required a project team.

That is enormously useful.

But there is an uncomfortable consequence:

We are also becoming much better at automating bad ideas.

When implementation becomes cheap, the value moves upstream.

The important capability is no longer simply knowing how to automate something. It is determining whether it should be automated in the first place.

Does this activity need to exist?

Does it support the strategy?

Should the process be simplified before anything is automated?

Is the underlying data reliable?

Do the surrounding systems provide enough context?

Does the AI need to make a decision, or should it simply provide information?

What happens when it misunderstands the situation?

What authority should it have?

And what happens if it is confidently wrong?

These are strategy, process and architecture questions before they are AI questions.

An AI does not need malicious intent to cause significant damage. It just needs to be optimising the wrong outcome, using incomplete information, with too much authority.

There is a temptation to start with the technology.

We have an agent.

We have Copilot.

We have an LLM.

We have an automation platform.

What can we automate?

That is backwards.

Start with the business outcome.

A clear Strategy & Roadmap helps establish what the organisation is trying to achieve before determining where AI should play a role.

What are we actually trying to improve?

Revenue? Customer experience? Cycle time? Cost? Quality? Risk? Employee capacity? Speed of decision-making?

AI should enter the conversation after there is clarity about the outcome being pursued.

Otherwise, it becomes very easy to build technically impressive systems that solve problems the organisation did not particularly need solved.

This matters even more as building gets cheaper.

When development costs $200,000, people tend to ask difficult questions before approving it.

When somebody can prototype something over a weekend, much of that friction disappears.

That can be a major advantage, but some friction existed for a reason.

The ability to build cheaply does not remove the need to decide wisely. It makes judgement more important.

Even when the strategy is sound, the AI worker sits on top of everything the organisation has built before it:

Processes. Data. Systems. Permissions. Policies. Integrations. Business rules. Organisational knowledge.

AI does not magically remove weaknesses in those foundations.

It can amplify them.

If the strategy is unclear, AI can optimise the wrong outcome.

If the process is poor, it can automate the wrong work.

If the data is incomplete, it can make a rational decision from a false picture of reality.

If the systems are fragmented, it may act without the context it needs.

If the business rules conflict, it may be forced to interpret which instruction matters.

If the controls are weak, that judgement can become an irreversible action.

The AI worker is therefore only one part of the design.

AI needs to be considered as part of an organisation’s broader digital transformation and technology strategy, rather than as a standalone technology initiative.

The outcome depends on the entire chain:

Strategy → Process → Data & Systems → AI Worker → Controls → Outcome

An excellent model sitting on top of poor data can still produce a poor outcome.

Excellent data supporting a badly designed process can simply automate waste.

A useful process with good data can still create unnecessary risk if the AI is given excessive execution authority.

Every layer matters.

A recent example captures this problem in a much more ordinary business setting.

Someone had taken a fairly experimental approach to an autonomous agent setup. During a renovation period, inventory was sitting unused.

The agent apparently interpreted that inactivity as stock that should be disposed of and began selling it.

What is interesting is that the AI was not necessarily behaving irrationally.

It wasn’t rebelling. It didn’t need to be hacked.

It appears to have interpreted the information available to it and reached a locally reasonable conclusion:

Idle inventory → unwanted inventory → sell inventory.

What was missing was context:

Idle inventory + renovation period → expected temporary state → do nothing.

But even that does not fully explain the failure.

There are several layers to unpack.

Strategy: Was inventory disposal actually an objective during that period?

Process: Could idle stock progress towards disposal without checking exceptional circumstances?

Data: Did the agent know a renovation was occurring?

Systems: Could it access the operational systems necessary to execute a sale?

Controls: Was there an approval threshold between recommending disposal and actually disposing of inventory?

The interesting failure isn’t simply that the AI made a bad decision.

It is that too much consequence appears to have been attached to that decision.

The same model could have safely identified apparently idle inventory, investigated why it was not moving, raised an alert, recommended disposal, prepared listings or calculated likely sale values.

The critical distinction is between:

“I think this inventory should be sold.”

and:

“Sell the inventory.”

That gap is where governance lives.

This is what makes the Hugging Face incident so interesting from a process perspective.

It would be easy to reduce the story to:

“Powerful AI broke out.”

But the architecture is more instructive.

The agents were operating towards an objective. They had access to tools and infrastructure. They encountered boundaries, found unintended ways around some of those boundaries, shared information and continued pursuing the objective.

Eventually, their actions crossed into systems outside the intended environment.

OpenAI’s investigation identified factors including persistence on difficult tasks, unauthorised communication and reward-hacking behaviour, and subsequently strengthened isolation, monitoring and access controls.

That is a far more advanced scenario than most businesses will encounter.

But the principle is familiar:

A probabilistic worker was operating inside a process, and the controls around its possible actions mattered just as much as the intelligence of the worker itself.

This is why AI safety cannot just be a conversation about model safety.

It also has to be a conversation about system and workflow design.

Broad questions such as:

“Is AI safe enough for our organisation?”

aren’t particularly useful.

Safe enough to do what?

An AI drafting an internal meeting summary is not the same as an AI approving a refund.

An AI identifying a suspicious payment is different from one freezing an account.

An AI identifying idle stock is different from one selling it.

An AI recommending a supplier is different from one issuing a purchase order.

The underlying model could be identical.

Organisations can easily drift towards two extremes.

At one end, AI is controlled so heavily that almost every action requires human intervention.

Every output gets checked. Every decision needs approval. Access to useful information is restricted. The AI is prevented from doing anything consequential.

The organisation technically has AI, but it has stripped away so much autonomy that much of the productivity benefit disappears.

At the other end is the excitement around autonomous agents.

Give the agent the inbox. Give it the browser. Give it the ERP. Give it customer information. Give it the file system. Give it tools and execution authority.

Then tell it the outcome and let it work out everything else.

The demonstrations can be extraordinary.

So can the blast radius.

Neither extreme represents particularly mature process design.

The objective should not be maximum control or maximum autonomy.

It should be:

The minimum sufficient control for the level of judgement, uncertainty and consequence at that point in the process.

Traditional workflows are largely deterministic.

They might look something like:

Trigger → Activity → Decision → Validation → Exception → Outcome

Much of the path is designed beforehand.

Agentic systems introduce a different kind of worker:

Event → AI → Decision → Tool → Observation → AI → Action → Verification → Outcome

The AI may choose the next step dynamically.

It might approach the same problem differently tomorrow.

It can interpret ambiguity rather than simply follow predefined branches.

That flexibility is exactly why these systems are powerful.

It is also why the process surrounding them becomes more important, not less.

You don’t necessarily need to control every decision the AI makes.

You need to control the environment in which those decisions are allowed to have consequences.

This is where the less glamorous parts of transformation become critical:

Access controls. Approval thresholds. Logging. Verification. Exception handling. Rollback. Data classification. Escalation paths. Rate limits. Separation of duties. Network boundaries. System boundaries. Monitoring.

None of these make for particularly exciting AI demonstrations.

They are what turn an impressive demonstration into an operational capability.

Instead of asking:

“How autonomous should our AI be?”

Ask:

“How much judgement and risk can we tolerate at this exact point in the process?”

There are several useful dimensions to consider:

Consequence: What happens if it is wrong?

Reversibility: Can the action easily be undone?

Detectability: Will we know something has gone wrong?

Authority: Is the AI informing, recommending, deciding or executing?

Judgement: How ambiguous is the decision?

Context dependency: How much surrounding information does it need to understand the situation?

Blast radius: If it behaves unexpectedly, how much can it affect before something stops it?

These factors should determine the controls.

Not fear of AI generally.

And not enthusiasm for AI generally.

“Human in the loop” has become one of the default responses to AI risk.

But placing a person at every step is not necessarily good governance.

Sometimes it is simply inefficient process design.

A human probably does not need to approve every document classification, search, summary or attempt an AI makes to locate an invoice.

But perhaps somebody should approve the disposal of $100,000 worth of inventory.

The objective is not to maximise human involvement.

It is to place human judgement where it materially changes the risk, accountability or consequence of the outcome.

A higher-risk process might look like:

AI identifies → AI investigates → system validates → AI recommends → human approves → system executes

A lower-risk process might be:

AI identifies → AI validates → AI executes → system logs → exception triggers review

The architecture changes with the consequence.

That is considerably more useful than applying one blanket governance model to every AI use case.

There is also a tendency to try to solve operational risk through prompts.

“Do not reveal confidential information.”

“Do not delete important files.”

“Ask the user before making a payment.”

“Do not take destructive action.”

Those instructions can be useful.

But they are not the same thing as controls.

If an AI should never access a dataset, don’t give it access.

If transactions above a certain amount require approval, enforce that rule in the application.

If an agent should only communicate internally, enforce that at the messaging layer.

If an action must always be logged, make logging unavoidable.

If an AI can recommend deleting a production database but should never actually be able to delete it, the permissions architecture should make that impossible regardless of what the model decides.

The distinction matters:

A policy tells the AI what it should do.

A control determines what it can do.

Mature AI governance needs both.

AI is valuable precisely because traditional automation struggles with ambiguity.

Understanding natural language. Interpreting messy information. Researching. Prioritising. Synthesising. Generating options. Recognising patterns. Planning.

These are good places to allow probabilistic judgement.

But not everything benefits from being probabilistic.

Authentication does not need creativity.

A payment threshold does not need reasoning.

Regulatory retention periods should not change because a model interprets the policy differently today.

Access permissions should not depend on whether an AI believes the information would be useful.

Some parts of the process should remain hard rules.

This leads to one of the most useful principles for agentic systems:

Give the AI freedom inside a controlled box.

Inside that box it can search, investigate, plan, compare, retry and exercise judgement.

At important boundaries, the process becomes deterministic again.

The model recommends.

A system validates.

A human approves where appropriate.

Another system executes.

Everything is logged.

That combination allows organisations to use what AI is genuinely good at without unnecessarily turning the entire organisation into a probabilistic system.

This may sound counterintuitive, but strong controls don’t necessarily mean less autonomous AI.

Often, they allow you to safely give it more freedom.

If an agent can only access systems relevant to its task, you can let it explore within those systems more freely.

If transactions have enforced thresholds, the agent can handle lower-value activity without constant supervision.

If actions are logged and reversible, unnecessary approvals can be reduced.

If high-risk exceptions automatically escalate, the agent can independently process normal cases.

Good controls therefore do not simply constrain AI.

They define the safe operating space inside which it can be more autonomous.

The choice is not autonomy versus control.

Good control architecture is what makes useful autonomy possible.

This is increasingly how AI-enabled processes should be designed.

Not:

“Where can we put AI?”

But:

“Where in this process does probabilistic judgement create value?”

And then:

“Where should determinism take over again?”

That forces organisations to think about the whole process rather than inserting an agent simply because the technology exists.

Sometimes the answer will be that AI should run most of the process.

Sometimes it should interpret information.

Sometimes it should make recommendations but never execute.

Sometimes a traditional workflow or simple rules engine will be better.

And sometimes the best transformation decision will be to remove the process entirely.

That last option is easy to forget.

Automation is not transformation if the activity never needed to exist.

One of the strange consequences of AI is that technology itself may become less of a differentiator at the same time as the disciplines surrounding it become more important.

Models will improve.

Building will become cheaper.

Tools will proliferate.

Capabilities that currently look remarkable will become common.

If everyone has access to powerful models and increasingly accessible development tools, the advantage shifts towards knowing:

What outcome matters.

Which problems are worth solving.

Which processes should be redesigned.

Which processes should disappear.

What data can be trusted.

What systems need to connect.

Where judgement is valuable.

Where hard rules are required.

And how much authority should be attached to an AI decision.

That isn’t particularly glamorous.

But it determines whether the technology improves the business or simply accelerates its existing problems.

AI is making software dramatically easier to build.

That will create enormous opportunities.

But it also removes some of the friction that historically stopped organisations from implementing every idea that sounded good in a meeting.

When virtually anything can be automated cheaply, judgement becomes more important than implementation.

Organisations need to ask:

Why does this process exist?

What business outcome does it support?

What would happen if we removed it?

Does the data support the decision?

Are the systems exposing the right context?

Where does human judgement add value?

Where can the AI operate independently?

Where should it recommend?

Where should it execute?

And where should it never be allowed to execute?

Having spent years looking at tens of thousands of business processes, I don’t think the biggest AI opportunity is simply automating more of them.

A significant part of the opportunity is deciding:

Which processes should survive in the first place?

Because automating a sub-optimal process does not transform it.

It institutionalises it.

Giving an AI agent autonomy inside that process can institutionalise it at machine speed.

The future of practical AI governance therefore looks surprisingly familiar:

Strategy. Process design. Data. Systems architecture. Risk management. Clear accountability. Controls. Exceptions. Verification.

The unsexy stuff.

The technology is new.

The underlying discipline is not.

The important question isn’t:

“What can the AI do?”

It is:

“What should we allow this worker to decide, at this point in the process, given the quality of the information it has and the consequence if its judgement is wrong?”

Get that right and you can give AI considerably more freedom.

Get it wrong and even a cheap automation can become extraordinarily expensive.

As organisations introduce AI into more business processes, understanding where it should sit, what it should be allowed to do and what needs to surround it becomes increasingly important.

SMC’s AI consulting services help organisations take a structured, business-led approach to AI, considering strategy, processes, data, systems, governance and practical implementation.