Don’t Just Govern AI. Design What It Is Allowed to Do.

By Ashok Kanagarajah · 17 August 2026

For many organisations, AI adoption began innocently enough. An employee used generative AI to summarise a long document. Another used it to draft an email. A marketing team experimented with content creation. Someone discovered that AI could analyse a spreadsheet faster than they could.

The governance questions seemed relatively straightforward. Can employees use these tools? What information can they put into them? Is the output accurate? Has someone checked it before it is used?

Those questions still matter. But they are no longer enough.

Imagine what happens next. A team connects an AI assistant to the organisation’s customer relationship management system so it can retrieve information automatically. Another allows AI to prepare customer responses using account history. A service team wants AI to prioritise incoming requests. Someone proposes an agent that can inspect a customer record, determine the next step, send a communication, update the system and trigger follow-up — without waiting for a person at every stage.

The organisation is still “using AI.” But something fundamental has changed. AI has moved from producing an output for a person towards participating in the work itself.

The governance question therefore shifts from:

Is the AI-generated answer accurate?

towards:

What is this AI allowed to see, decide and do?

That distinction matters because AI capability is moving along a continuum: from generating information, to recommending actions, influencing decisions, taking actions and increasingly orchestrating sequences of work across tools and systems. As authority expands, so does potential value. But so does the consequence of failure.

This shift is becoming visible in formal guidance. In May 2026, the Australian Signals Directorate’s Australian Cyber Security Centre, together with cyber-security agencies in the United States, Canada, New Zealand and the United Kingdom, published joint guidance on the careful adoption of agentic AI. The guidance distinguishes these systems from conventional generative AI because they can combine AI models with external tools, data, memory and planning processes to take actions with less continuous human intervention. It recommends tightly controlled access, ongoing monitoring, explicit accountability and cautious deployment, particularly where systems interact with sensitive data or critical systems.

The systems problem becomes even more significant when agents begin interacting with one another. A study prepared by Gradient Institute for Australia’s AI Safety Institute in August 2026 argues that individually reliable agents do not necessarily create a reliable multi-agent system. New failure modes can emerge from interactions, including erroneous hand-offs, cascading mistakes and behaviours that no single organisation completely controls. As agents cross organisational boundaries, visibility and the ability to intervene may decline precisely as consequences become more distributed.

This is why the governance challenge can no longer be framed only as: Is this model safe? or Is this AI tool approved? The more useful question is increasingly: what authority does this AI have inside this piece of work?

Yet much organisational AI governance remains designed for an earlier phase of adoption. The familiar response has been to create an acceptable-use policy, establish an AI committee, identify prohibited uses, define approval requirements and ask employees to comply. These mechanisms remain necessary — organisations still need privacy protections, security controls, accountability, regulatory compliance and clear standards for unacceptable use. But policies cannot answer every operational question that appears when AI enters a real workflow. Can an agent read every customer record or only the one associated with its current task? Can it draft a response but not send it? Can it recommend changing an account status? Can it make that change after human approval? Could it eventually make certain changes automatically within predefined limits? When must it stop? Who receives an escalation? What evidence would justify giving it more authority tomorrow than it has today?

These are not merely technology-policy questions. They are questions about how work is designed and how organisational authority is allocated.

That also changes the familiar debate between innovation and governance. At one extreme, minimal governance can create speed, at least temporarily — teams experiment, employees discover useful applications, valuable learning occurs. But as experimentation moves towards consequential work, uncertainty accumulates. Who authorised the use? What data can AI access? Who owns the consequence? Eventually, an organisation can reach a paradox: many AI experiments, but little confidence in its ability to scale them.

The opposite response can be equally damaging. If every AI use must pass through a lengthy central approval process, governance becomes a source of friction rather than clarity. Some activity may move outside formal processes because the organisational effort required to test an idea exceeds its likely benefit.

The choice is therefore not unrestricted experimentation versus blanket control. A better objective is enabling governance: boundaries clear enough that people know where they can move quickly, where stronger evidence is required and where human authority must remain.

Good governance should make the safe path the fast path.

From generate to orchestrate

A useful way to see why the governance question changes is through five levels of AI participation:

At Generate, AI produces something for a person: a draft, summary, analysis or report. A recognisable human decision point usually remains between the output and an organisational action.

At Recommend, AI begins shaping what should happen. It might suggest how a customer enquiry should be handled, identify a transaction for review or propose the next action.

At Decide, AI begins classifying, prioritising or routing work. Its judgement may determine which path a case follows, who receives attention and which information reaches a human.

At Act, AI can send the response, update the record, book the appointment, initiate the refund or trigger another stage of the workflow. The organisation is no longer merely consuming an AI output. The organisation is acting through the AI system.

At Orchestrate, AI coordinates several steps across tools, data and systems. An agent might retrieve a customer history, analyse the request, determine an appropriate response, check another system, update a record, send the communication, schedule follow-up and escalate an exception. Each step may appear manageable in isolation. Together they create something qualitatively different: a workflow capable of progressing towards an outcome without requiring a human decision at every stage.

Figure 1: From AI Output to AI Action. As AI authority and organisational consequence increase, governance must increasingly address what AI is allowed to do, not merely what it produces.

The management implication is simple but profound. The same underlying AI capability can represent very different organisational consequences depending on the authority surrounding it. There is a significant difference between allowing AI to:

• draft a refund response;
• recommend that a refund be approved;
• determine that a refund satisfies policy;
• issue the refund; or
• manage the entire customer-resolution process.

The technology may look similar. The authority is not.

Instead of asking only, “Can this AI do the task?”, leaders increasingly need to ask: what level of authority should we give it to perform this task? Once authority becomes the object of governance, the unit of analysis also needs to change. We have to move from the AI tool towards the work itself.

Start with the work, not the AI policy

When organisations first recognise that AI introduces new risks, the instinctive response is often to govern the technology. A committee is formed. An acceptable-use policy is drafted. Approval processes are established. Those measures can be necessary. But they are rarely sufficient.

Policies operate at the level of the organisation. The consequences of AI emerge inside particular uses and workflows. A policy can state that confidential information must be protected and that people remain accountable for important decisions. It still does not tell a manager what should happen inside a specific piece of work. What exactly is AI doing? What information does it need? What decision is it influencing? What action can it initiate? Where does human authority begin and end? What happens when AI is wrong?

Australia’s current Guidance for AI Adoption makes this distinction explicit. Some governance practices apply organisation-wide, while other controls need to be designed around individual AI systems and particular uses. The guidance recognises that the same tool may create very different risks depending on how it is used, and recommends reviewing each use in its own context.

Consider a customer-service workflow. Using AI to summarise previous interactions may be relatively straightforward. Using it to draft a response introduces a different responsibility. Allowing it to recommend a resolution changes the role again. Permitting it to send the response, update the customer record or initiate a refund moves the organisation into different governance territory. The underlying technology may be identical. Its role in the work is not.

Govern the use, not merely the technology.

This connects governance directly to workflow redesign. Before automating work, leaders should understand what outcome matters, where friction occurs, what work can be eliminated or simplified and where AI can genuinely improve the system. The principle is:

Don’t automate the mess.

Governance is the next layer of the same design problem. Workflow redesign asks: how should the work happen? Governance asks: who — human or AI — is authorised to do what within it? The questions are inseparable. A useful sequence is:

A poorly designed workflow can be automated badly. But even a well-designed workflow can become risky if access is excessive, authority is vague, escalation is unclear or nobody knows who remains accountable. Policy sets the organisational rules. Work-level governance defines how those rules operate in practice.

This is not a new debate about automation

The question of how much authority to allocate to machines has a long intellectual history. Human-factors researchers have studied levels and types of automation for decades. Parasuraman, Sheridan and Wickens argued in 2000 that the relevant design challenge was not simply whether automation was technically possible, but which functions should be automated and to what degree — and that automation does not merely remove human activity, it changes what humans must do and creates new coordination demands.

Contemporary regulation builds on related principles. For high-risk systems, the European Union’s AI Act requires human oversight measures to be proportionate to the risks, the system’s autonomy and its context of use, and explicitly recognises the danger that humans may automatically over-rely on AI recommendations.

The contribution here is therefore not another claim that automation exists at different levels, or that AI needs human oversight and proportional risk management. Those ideas have substantial intellectual and regulatory foundations. The more useful synthesis is to bring them together around a practical management problem:

What authority should AI have inside this work, and what evidence justifies that authority?

I describe this broader discipline as AI Authority Design. It treats governance as the deliberate allocation and continuing adjustment of authority within a human–AI work system, connecting workflow design, access, decision rights, consequence, evidence, human responsibility, containment and organisational learning.

Design the boundary: the Six Questions of Practical AI Governance™

AI governance often becomes abstract very quickly — responsible AI, human oversight, acceptable risk, trust, accountability. These concepts matter. But a manager designing a real AI-enabled workflow needs to convert them into operating decisions.

For every meaningful AI use case, leaders should be able to answer six questions:

1. What is AI here to achieve?
2. What may it access?
3. What authority does it have?
4. Where does the human remain involved?
5. What evidence justifies that role?
6. What happens when the system reaches its limits?

1. Purpose — what outcome is AI here to help create? Begin with the outcome, more specific than “improve productivity.” Purpose also defines what AI is not there to do. Capability should not determine purpose. Purpose should determine capability.

2. Access — what information, systems, tools and context may it use? Broader access increases usefulness — it also increases exposure. For agentic systems this matters more, because access may include execution privileges, not merely visibility. The governing principle: minimum necessary access. Access should follow purpose.

3. Authority — what may AI draft, recommend, decide or do? Access defines what AI can see. Authority defines what it is permitted to do with what it sees. The relevant question is not “Can AI do this?” It is “Should AI be authorised to do this in this workflow, under these conditions?” A useful illustration comes from Moffatt v Air Canada, where an automated chatbot gave a customer inaccurate information about a bereavement-fare policy. The British Columbia Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation and rejected the argument that the chatbot could somehow be treated as a separate entity from the company’s website. The managerial lesson: when AI is allowed to speak or act for an organisation, authority and accountability cannot be separated simply because the intermediary is software.

4. Human Role — who reviews, approves, supervises or intervenes? Many governance discussions stop with “there will be a human in the loop.” That statement tells us almost nothing. A useful design specifies who the human is, what they review, when they enter the workflow, what information they receive, and whether they can override or stop the system.

5. Evidence — what evidence justifies the authority we are giving it? AI can appear impressive quickly. That does not mean it has earned consequential authority. Different tasks require different evidence thresholds.

Authority should expand only as evidence grows.

6. Containment — what happens when AI is wrong, uncertain or outside its scope? No serious governance model should assume AI will always perform correctly. Depending on the workflow, the system might stop, request clarification, escalate to a human, prevent or reverse an action, or suspend operation while the issue is investigated. The system should not be rewarded simply for continuing.

Failure must be detectable — and containable.

Taken together, the Six Questions form one boundary. A system with narrow access and limited authority may justify relatively light controls. A system with broad access, material decision rights and the ability to act autonomously requires stronger evidence, clearer human responsibilities and more reliable containment.

From intellectual architecture to practical application

Understanding the Six Questions is useful. Applying them to a real workflow is more valuable. I have brought the core ideas in this article together in the Trusted AI Boundary Canvas™: a one-page practical instrument for defining the purpose of an AI use, its consequence level, access boundary, authority boundary, human decision rights, evidence requirements, tolerance boundaries, escalation conditions and next experiment.

The Canvas is deliberately not another framework. It is the practical instrument through which the frameworks in this article come together around one question:

What level of AI authority is justified by the evidence we have today?

Design human–AI decision rights

One of the most common responses to AI risk is reassuringly simple: “a human will always be in the loop.” The phrase sounds responsible. The problem is that it describes very little.

Research on automation has warned for decades that people can become overly dependent on automated advice. In experimental settings, automation bias has led people to accept incorrect recommendations and overlook information that should have caused them to challenge the system. Article 14 of the EU AI Act explicitly requires those overseeing high-risk systems to remain aware of the tendency to automatically or excessively rely on AI output.

Recent healthcare research goes further. Van de Sande, Economou-Zavlanos and van Genderen argue that merely having a clinician present does not make oversight meaningful. Their framework identifies four conditions that matter: whether the person has the knowledge to understand the system, cognitive space to exercise judgement, authority to make a decision and practical ability to intervene.

For managers, this translates into seven practical questions:

• Which human? Responsibility should be assigned to an appropriate role, not to an abstract “human.” The correct human depends on the work.
• Reviewing what? A person might inspect every output, approve only proposed actions, review exceptions, or monitor aggregate patterns. These are not the same form of control.
• At what point? Review before an action may prevent an error. Review afterwards may only identify a problem after the consequence has already occurred.
• With what information? A reviewer cannot exercise meaningful judgement if they see only the AI’s conclusion. Approval without adequate context can create the appearance of oversight without its substance.
• With what authority? Even a knowledgeable reviewer has little value if they cannot intervene, reject, override, pause or escalate.
• With what competence? The reviewer needs both domain competence and sufficient AI literacy.
• Within what timeframe? If AI generates 500 recommendations a day and a manager is expected to approve every one, approval becomes habitual and exceptions become harder to see. The human remains “in the loop.” But the loop has become a rubber stamp.

Design the human role as carefully as the AI role.

The Human–AI Decision-Rights Ladder™

Level 1 — Human decides; AI informs. The human retains the decision. AI may retrieve information, summarise evidence or identify patterns, but does not determine what should happen.

Level 2 — AI assists; human directs. The person sets the objective; AI accelerates the process, drafting components or investigating an issue.

Level 3 — AI recommends; human approves. AI proposes what should happen. The formal decision remains human — but if recommendations are correct most of the time, approval can gradually become less thoughtful. Governance needs to ask whether approval remains meaningful, not simply whether it exists.

Level 4 — AI acts with explicit approval. AI prepares the action but cannot execute it until an authorised human approves. This removes considerable administrative work while preserving human authority at the point of consequence.

Level 5 — AI acts within predefined boundaries. AI may act without individual approval, but only within defined limits — transaction value, case category, permitted actions, exception conditions, stopping rules. Human oversight shifts towards designing boundaries, monitoring performance and handling exceptions.

Level 6 — AI operates within demonstrated competence. This should not be interpreted as unlimited autonomy. It means AI may operate independently inside a domain in which the organisation has sufficient evidence that performance is reliable and failures can be detected and contained. Authority should increase because evidence supports it — not because a model is newer or marketed as more capable.

Morgan Stanley’s AI @ Morgan Stanley Debrief provides a useful illustration. With client consent, the system generates meeting notes, summarises key points, drafts an email for the financial adviser to edit and send at their discretion, and saves a meeting note into Salesforce. Notice the different allocations of authority within one workflow: AI performs transcription and summarisation, drafts an external communication, but the adviser retains authority over whether that communication is sent — while the system can save an internal record without approval.

That is why the ladder is not a maturity model. Different tasks can occupy different levels inside the same workflow. The foundational distinction remains: what can AI do? versus what should AI be authorised to do? Technology answers the first. Management and governance answer the second.

Govern according to consequence, not fear

An organisation can easily make one of two mistakes: applying the same heavy governance process to every AI use, or creating controls light enough for routine applications and then applying those same controls to consequential work. Neither is proportional.

Controls should escalate with consequence, not with organisational anxiety.

A practical model uses four broad consequence levels:

Low consequence — generating ideas, organising notes, drafting internal material, summarising documents. These uses still require sensible controls around privacy and review, but should generally be easy to explore.

Moderate consequence — AI begins influencing people outside the organisation or materially shaping operational activity. External communications are a useful example: the system may prepare them, but approval, sampling or auditing may be appropriate.

High consequence — AI influences material operational or financial decisions. There should be a named owner, explicit thresholds, and designed monitoring and escalation. A system that is “98 per cent accurate” tells leaders very little until they understand what happens in the remaining 2 per cent.

Critical consequence — decisions involving health, safety, legal rights or significant financial interests. AI may still contribute substantial value by analysing information or preparing options. But technical capability does not automatically justify transferring the final decision right.

The consequence category should not be treated as permanent — an application may begin by drafting internal material and later be allowed to communicate externally. Nor should entire AI products be classified as simply “low risk” or “high risk.” The same platform might summarise meeting notes, prepare customer communications and support a clinical decision. The tool has not changed. Its authority inside the work has.

Do not ask, “How risky is this AI?” Ask: “How consequential is what we are allowing this AI to do?”

Let evidence determine AI authority

Once governance has been matched to consequence, organisations face a harder question: when should AI be allowed to do more? The instinctive answer is often expressed in terms of trust. But trust questions are not sufficient for allocating consequential authority. A more disciplined question is:

What evidence justifies the authority we are giving it?

NIST’s AI Risk Management Framework reflects a related lifecycle logic — understanding context, measuring AI performance and risks, then managing those risks against organisational priorities, as an adaptive framework rather than a one-time approval test. Australia’s implementation guidance similarly emphasises testing before deployment, monitoring after deployment and continuing review.

For practical authority decisions, three conditions deserve particular attention:

Predictability — does it perform reliably enough within the defined work? Not whether AI can succeed once, but whether the organisation can reasonably expect acceptable performance across the situations it is authorised to handle. Evaluate AI capability within the operating boundary you intend to authorise, not only in aggregate — and compare it against the current human-led baseline.

Detectability — can we recognise when AI is wrong or outside its competence? AI can fail visibly, or it can fail convincingly. The harder failures are plausible but wrong. A system that fails occasionally but transparently may sometimes be safer than one that fails less frequently but invisibly.

Containment — can failure be stopped before it spreads? An incorrect recommendation may be caught before anything occurs. An agent that sends a communication, changes a record and triggers several further steps may propagate an error before anybody notices.

A system should not be given more authority than the organisation can reliably contain when it fails.

The conditions interact. A less predictable system may still be useful if uncertainty is readily detected and difficult cases are reliably transferred to humans. A system with a very low failure rate may still be unsuitable for greater autonomy if failures are difficult to recognise and hard to reverse.

Autonomy expands only inside demonstrated tolerance boundaries.

Critically, movement must be possible in both directions. If evidence improves, authority may expand. If performance deteriorates, authority may contract.

Autonomy is a managed operating condition, not a one-time entitlement.

Klarna: evidence can change the operating design

Klarna provides a useful illustration of why authority and human involvement should remain adjustable. In February 2024, Klarna reported that its AI customer-service assistant handled 2.3 million conversations in its first month — approximately two-thirds of its customer-service chats — with repeat enquiries falling 25 per cent and average resolution time falling from 11 minutes to under two minutes, with customer satisfaction comparable to human agents.

The story then evolved. By September 2025, CEO Sebastian Siemiatkowski told Reuters that Klarna had moved too aggressively towards AI-enabled cost reduction and had been course-correcting, with greater emphasis on products, service quality and growth — again hiring people while continuing to invest in AI.

This does not prove the initiative failed, nor that customer service should remain human-led. The more useful lesson is that early efficiency evidence does not settle the operating design permanently. A mature governance system needs to respond to new evidence as it emerges. The options should never be only automate or do not automate. They should include:

This also clarifies what we mean by trust. The relevant unit is not the AI model alone. It is the complete human–AI system:

AI + data + workflow + people + decision rights + controls + monitoring + escalation + organisational accountability

A capable model inside a poorly designed system can still produce poor outcomes. The governance question should therefore evolve from “Do we trust the AI?” to “What level of authority has this human–AI system demonstrated that it can handle?”

Govern one workflow — then learn

A common governance model assumes the important decision occurs before deployment: a team proposes a use, risk and compliance review it, security signs off, the system is deployed, governance moves on. AI-enabled work is less static than that. Users discover new uses, workflows change, models are updated, and as confidence increases, teams naturally ask whether AI can do more.

Governance should be an operating rhythm, not an approval gate.

The practical starting point is smaller than an enterprise-wide transformation programme:

1. Pick one workflow. Start with the work, not the technology. Choose something meaningful but bounded — customer enquiry handling, referral processing, invoice follow-up. Map the current system and establish a baseline where possible. Without a baseline, an AI pilot may look impressive without proving that anything important improved.

2. Classify the consequence. Use the four levels — low, moderate, high, critical — to force consequence into the conversation before autonomy.

3. Define the boundary. Apply the Six Questions of Practical AI Governance™ to produce something understandable by the people doing the work: this is what AI is here to do, what it may access, what it may do, where people remain responsible, how we will judge performance, and what happens when it reaches its boundary. This is where the Trusted AI Boundary Canvas™ becomes useful.

4. Run a limited experiment. The experiment should be time-boxed, observable and reversible. It should begin with the minimum authority required to test the hypothesis — AI might initially draft responses for human review, then progress to recommending an action, then perhaps send a tightly defined category of routine responses without individual approval. The organisation learns progressively rather than granting the final level of autonomy on day one.

The same logic matters even more in high-consequence environments. Consider perioperative healthcare. An AI-enabled system might retrieve and organise information, identify missing administrative details and send a predefined request for missing information inside a bounded workflow — with no independent authority to diagnose, change treatment or determine whether a patient is fit for surgery. The relevant question is not “Can AI manage this patient journey?” It is “What level of AI authority is justified at each point in the journey?”

5. Review the evidence. Return to the outcome and evidence defined at the beginning. Did the workflow improve? Did AI remain inside its authorised role? Could errors be detected? Did containment work? Most importantly: has the evidence justified the level of authority AI currently has?

The answer should lead to one of four decisions — Expand, where evidence supports greater authority; Maintain, where the current boundary remains appropriate (not a failure to progress); Restrict, where AI remains useful but scope or human involvement should be strengthened; or Stop, where the expected benefit has not materialised.

Repeated across workflows, this builds something more valuable than a collection of AI pilots. It builds organisational capability. There is another benefit: designing AI authority often exposes weaknesses that existed before AI. Ask who receives this escalation? and you may discover ownership was already unclear. Ask what is the baseline? and you may discover performance has never been measured. In that sense, AI governance can become a forcing function for better organisational design.

The goal is trusted capability

AI capability will continue to advance. The important question for leaders is not whether AI will become capable of doing more — it almost certainly will. The harder question is: as AI becomes capable of doing more, how much more should the organisation allow it to do?

Policies remain necessary, but they cannot anticipate every combination of workflow, data, context, authority and consequence that increasingly capable AI systems will create. Nor should authority expand automatically because a new model can perform a task that previously required a person. Technical capability is not the same as organisational permission.

The alternative is not permanent human approval of every AI action. It is more deliberate design: begin with the work, define the boundary, make human and AI decision rights explicit, match governance to consequence, require stronger evidence as authority increases, ensure failure can be detected and contained, then learn from real use.

This argument builds on established work in automation, human factors, responsible AI and risk-based governance. Its distinctive proposition lies in the synthesis: AI governance increasingly becomes the deliberate design, evidence and adaptation of authority within human–AI work systems. That is AI Authority Design.

Trust should emerge from the performance of the complete system — not the model alone. A capable model inside a poorly designed system can still produce poor outcomes. Governance therefore becomes more than a control function. It becomes an organisational capability: knowing how to distinguish low-consequence experimentation from consequential action, how to allocate decision rights between humans and AI, what evidence is required before authority changes, and how to move authority in either direction.

The goal is not less innovation. Nor is it zero risk. The goal is trusted capability.

Seen this way, governance becomes the design discipline that allows autonomy to expand responsibly.

Put the idea into practice

Choose one real workflow in your organisation and work through the Trusted AI Boundary Canvas™: Purpose, Outcome & Use Case; Consequence Level; Access Boundary; Authority Boundary; Human Decision Rights; Evidence & Tolerances; Escalation & Containment; and Next Experiment / Authority Decision.

Download the Trusted AI Boundary Canvas™ — use it to make the boundaries around one piece of work explicit, then decide whether the evidence justifies expanding, maintaining, restricting or stopping.

This article is the first of a two-part companion set. The supplementary worked example, Designing Trusted AI Around the Perioperative Patient Journey, tests this same architecture against a high-consequence healthcare setting — following one patient through preoperative assessment, surgery, recovery, discharge and follow-up to ask where AI may retrieve, flag, recommend or act, and where clinical decision rights must remain human.