Designing Trusted AI Around the Perioperative Patient Journey

By Ashok Kanagarajah · 17 August 2026

Supplementary article to: Don’t Just Govern AI. Design What It Is Allowed to Do.

As AI moves from retrieving and summarising information towards recommending, coordinating and acting, healthcare organisations face a more difficult question than whether AI should be used:

What should AI actually be authorised to do within patient care?

The answer cannot be the same for every part of a clinical journey. Retrieving information, coordinating appointments, identifying missing records, recommending clinical review and changing treatment carry very different consequences. AI authority should therefore be designed task by task, around the patient journey. Some work may eventually support relatively high levels of bounded AI action. Other work should retain strong human clinical decision rights.

Using the perioperative journey of one patient as a worked example, this article applies the ideas developed in Don’t Just Govern AI. Design What It Is Allowed to Do. It asks six practical questions about Purpose, Access, Authority, Human Role, Evidence and Containment, then explores how AI authority might change from preoperative preparation through surgery, postoperative care, discharge and follow-up.

The aim is not to design autonomous healthcare. It is to explore how AI might become a trusted participant in a deliberately designed human–AI care system.

The examples below are intentionally exploratory rather than prescriptive: they are not a clinical protocol, implementation blueprint or recommendation for any particular AI system. They are offered to stimulate frontier thinking, surface potential benefits and risks, and encourage discussion about how healthcare might be redesigned to deliver safer, more coordinated and better patient care.

Start with the patient outcome, not the AI agent

Imagine a patient admitted for major elective surgery. At a high level, the journey might look like this:

A technology-led conversation might begin by asking: could we build an AI agent to coordinate this entire journey? That is probably the wrong first question. A better starting point is: how can we deliver a safe, coordinated and timely perioperative journey in which the right information, decisions and actions reach the right clinician or team at the right time? Only then should we ask where AI might contribute.

This distinction matters because patient care is not one homogeneous activity. Some work is administrative. Some involves information retrieval. Some involves coordination. Some involves clinical interpretation. Some involves recommendations. And some involves decisions where an error could significantly affect patient safety.

The Australian Commission on Safety and Quality in Health Care’s 2026 National Model for Clinical Governance explicitly incorporates digitally enabled care and AI-supported decision-making within clinical governance. It recognises that governance structures need to adapt as digitally enabled models change clinical workflows and technical requirements. AI in healthcare cannot simply be treated as another Information and Communications Technology (ICT) implementation. Once AI begins participating in clinical and operational workflows, it becomes part of the system through which care is delivered.

One patient journey, many different AI roles

Imagine a 76-year-old patient — we will call her Mrs A — admitted for major abdominal surgery. She has several chronic conditions and takes multiple medications. Her relevant information is distributed across referral documents, previous records, medication lists, pathology, imaging, anaesthetic assessment, surgical planning and nursing documentation.

AI could potentially contribute at many points. But its authority should not necessarily be the same at every point. Consider six representative examples:

These allocations are illustrative, not prescriptions for every hospital. The important point is: AI authority can legitimately vary within the same patient journey.

Apply the Six Questions of Practical AI Governance™

Now imagine that the hospital is considering a perioperative care coordination agent, purposed to help coordinate information and workflow from preoperative preparation through discharge. Rather than asking whether the agent should simply be “approved,” we can work through the six questions.

1. Purpose — what outcome is AI here to help create? A possible purpose: reduce information gaps, unnecessary waiting, coordination failures and missed follow-up across the perioperative journey while helping clinicians access relevant information at the right time. That statement should also define what lies outside the agent’s purpose — for example, that it is not authorised to independently diagnose, prescribe, change treatment, determine surgical fitness or make other consequential clinical decisions.

This distinction becomes particularly important in healthcare because intended purpose can affect regulatory status. The Therapeutic Goods Administration explains that software or AI intended for functions including diagnosis, monitoring, prediction, prognosis or treatment may meet the definition of a medical device and be subject to medical-device regulation. Capability should therefore not silently become purpose. Just because an AI system can produce a clinical recommendation does not mean that recommendation is part of its authorised role.

2. Access — what may the AI see? A perioperative coordination system could potentially access a large amount of information:

The temptation may be to give the agent broad access because broader access makes coordination easier. But the appropriate principle remains minimum necessary access for the authorised purpose. A scheduling function does not automatically need access to the entire clinical record. Read access should not automatically imply write access, and the ability to retrieve information should not automatically imply authority to modify it. As AI becomes increasingly agentic, these distinctions become central to governance.

3. Authority — what may the AI actually do? Across the perioperative journey, AI might potentially be authorised to:

That does not mean it should automatically be authorised to:

The underlying AI may technically be capable of both. Governance determines which activities it is authorised to perform.

Consider medication reconciliation. An AI system might compare medication lists from several sources and identify an inconsistency. That could be extremely useful. But there is a significant difference between “These medication lists appear inconsistent — clinical review required” and “Medication X should be stopped.” The first is information synthesis and escalation. The second moves towards clinical decision-making. Technical capability does not determine clinical authority.

4. Human Role — who reviews, approves, intervenes and remains accountable? In perioperative care, this cannot be answered with a generic statement such as “a clinician will remain in the loop.” The human role needs to be specific to the task. If AI identifies missing preoperative administrative information, an administrative team member may be the appropriate person to resolve it. If it detects a medication discrepancy, a pharmacist or treating clinician may need to review it. If it flags a pattern suggesting postoperative deterioration, the escalation may need to reach a nurse, resident medical officer, physician, anaesthetist or surgeon depending on the clinical context and local escalation pathway.

For Mrs A’s medication discrepancy, the workflow might specify: discrepancy detected → AI flags and stops → pharmacist or responsible clinician reviews → reconciliation decision remains human-led. The important question is not simply whether the system generated an alert. It is whether that alert reaches the right person, with the right information, at the right time, and with the authority to act.

Design the human role as carefully as the AI role.

5. Evidence — what evidence would justify the authority we are giving it? In healthcare, evidence should be tied to the actual workflow and the outcome being improved. Evidence should include more than model accuracy: completeness, correction rates, missed information, false alerts, time saved, rework, clinician acceptance, patient-flow impact, escalation performance, and safety incidents or near misses — measured against the existing human-led baseline. The relevant question is not whether AI is perfect. It is whether the combined human–AI system performs better, more consistently or more safely than the current way of working.

AI authority should expand only when the evidence justifies it.

6. Containment — what happens when AI is wrong, uncertain or outside its scope? In a perioperative workflow, containment might include stopping automated action, escalating to a named clinician or team, preventing AI from modifying a record, requiring approval before communication is sent, reverting to the existing workflow, or suspending the AI function while an issue is investigated. If AI detects a medication discrepancy, it should not resolve it autonomously simply because it can generate a plausible answer: discrepancy detected → AI flags and stops → pharmacist or clinician reconciles. Similarly, if the postoperative system encounters missing observations, conflicting data or a pattern outside its approved operating scope: stop → escalate → human clinical assessment.

A safe AI system must know not only how to act, but when to stop acting.

How AI authority might change across the patient journey

The useful insight is that AI authority need not move in one direction. It may rise and fall depending on the nature and consequence of the work.

During preoperative preparation, AI might retrieve relevant information, structure it and identify missing elements — AI assists, then AI recommends. If routine administrative information is missing, the system might also be permitted to send a predefined request automatically. Interpretation of clinically significant information remains with the appropriate clinician.

On the day of surgery, AI might check whether defined workflow prerequisites appear to have been completed — documentation present, results available, teams notified. But the system should not silently convert workflow completeness into a clinical conclusion such as “patient is fit for surgery.” A more appropriate output: “the defined workflow requirements appear complete; the following item requires clinical review.” The final clinical decision remains where it belongs.

During postoperative ward care, the consequence of AI use can rise sharply. A suitably governed system might synthesise observations, laboratory trends, medication administration and nursing documentation to identify patterns or flag a patient for review. But there is an important distinction between “this pattern meets the criteria for clinical review” and “the patient has condition X and treatment Y should be commenced.” Where AI is intended for monitoring, prediction, diagnosis or treatment, regulatory status and clinical assurance requirements become particularly important — the TGA’s guidance on AI and medical device software regulation is relevant here.

During discharge and follow-up, the authority profile changes again. AI might prepare a draft discharge summary, assemble approved patient information, check that follow-up requirements are documented, prepare communication for the GP, or schedule an approved appointment. Clinical content may still require appropriate verification, while routine, reversible coordination actions might eventually operate with greater bounded autonomy.

AI authority should follow the consequence of the task — not the sophistication of the technology.

“Human in the loop” needs to become specific

Healthcare often uses a reassuring phrase: “there will always be a clinician in the loop.” That sounds responsible. But it does not tell us whether meaningful oversight actually exists. We need to ask: which clinician? At what point? Reviewing what? With what information? With what authority? With what competence? Within what timeframe?

If an AI system identifies a concerning postoperative pattern, meaningful human oversight requires more than sending an alert somewhere. Who receives it? How quickly? What happens if nobody responds? Can AI continue taking other actions while the escalation is unresolved? A nominal human checkpoint may offer little protection if the person sees the issue too late, lacks sufficient information, cannot recognise the AI’s error, lacks authority to intervene, or is overwhelmed by the volume of AI-generated alerts.

The goal is therefore not to place a human somewhere in the workflow. It is to make sure human judgement is present where it can still change the outcome.

A more consequential test: the postoperative deterioration signal

To test the framework more seriously, consider a more consequential use. Mrs A is now on the postoperative ward. Her observations, pathology and nursing documentation begin to change. An AI-enabled system identifies a pattern associated with possible deterioration. Before granting that system meaningful authority, apply the three conditions for trusted AI use.

Predictability. Not simply was the algorithm accurate in a research study? but does the complete system perform acceptably in this hospital, with these patients, workflows and data? What kinds of patients does it handle well? Where does performance deteriorate? What happens when observations are missing, or a patient falls outside the circumstances for which the system has been evaluated? How does the human–AI system perform compared with the existing care process?

Detectability. Can we recognise when the AI may be wrong? Can the system recognise incomplete observations, conflicting records, uncertainty, unusual clinical circumstances or cases outside its intended scope? Can clinicians see why a case has been escalated? A system that fails occasionally but transparently may sometimes be safer than one that fails less often but invisibly.

Containment. If the system fails, what prevents that failure from spreading? If AI misses a deterioration signal, what other clinical safeguards remain? Can a clinician override an inappropriate alert? Can AI authority be reduced quickly, or the function suspended? Does care reliably return to the established clinical pathway?

For consequential healthcare applications, trusted AI use therefore depends not merely on whether AI performs well most of the time. It depends on whether performance is sufficiently predictable, failure sufficiently detectable and consequences sufficiently containable.

Clinical AI authority should expand only inside demonstrated tolerance boundaries.

When AI reaches its clinical boundary

Consider a routine postoperative follow-up agent. It may be authorised to retrieve approved information, send routine instructions, schedule an appointment, and confirm the patient received the communication. But it must stop when the patient reports unexpected symptoms, information is inconsistent, a medication issue arises, a threshold is exceeded, the requested action lies outside approved rules, a new clinical concern emerges, or the patient requests advice outside the agent’s authorised scope. At that point:

That hand-back is not evidence that AI has failed. It may be evidence that the governance design is working correctly. A capable AI system should not merely know how to act. It should operate inside a system that knows when it should stop acting.

Govern the complete human–AI care system

The unit of governance should not be the AI model alone. An AI model can perform well while the patient-care system still performs badly. AI correctly identifies that Mrs A requires review — but the alert goes to the wrong person. The recommendation is correct — but the clinician receives so many alerts that it is overlooked. AI appropriately identifies uncertainty — but the escalation pathway is too slow.

The relevant unit of governance is therefore the complete system:

AI + data + workflow + clinicians + decision rights + monitoring + escalation + organisational accountability

Trust should emerge from the performance of that complete human–AI care system. The 2026 National Model for Clinical Governance similarly places digitally enabled care within clinical governance rather than treating it as a separate technical initiative. The organisational question therefore becomes larger than “does the AI work?” We need to ask: “does the complete human–AI care system work?”

Start with one bounded experiment

How might a hospital begin? Probably not by deploying an autonomous perioperative agent from admission to discharge. A better starting point is one clearly bounded workflow: improve the completeness of information available before preoperative clinical review.

The experiment could be designed explicitly:

• Outcome: reduce missing information and time spent manually locating it.
• Consequence: moderate, provided AI does not make clinical decisions.
• Access: only the records required for the defined task.
• Authority: retrieve, organise, summarise and flag missing information.
• Human decision right: clinicians retain interpretation and all consequential clinical decisions.
• Evidence: completeness, errors, missed information, time saved, clinician correction rates, exceptions, impact on workflow.
• Containment: uncertain or conflicting information is flagged rather than resolved autonomously.
• Experiment: run the system with a limited patient cohort while existing clinical safeguards remain in place.

Then review the evidence and decide:

This is much more useful than asking whether the hospital is generally “ready for AI.” It allows the organisation to learn which level of AI authority is justified for which work.

Governance may expose problems that existed before AI

Perhaps one of the most interesting outcomes of this exercise is that it may reveal problems that have little to do with AI. Ask who receives this escalation? and you may discover ownership was already unclear. Ask what is the acceptable baseline? and you may realise current performance has never been measured. Ask who has the decision right? and you may discover ambiguity across teams. Ask what happens when the system fails? and you may find the existing fallback process is poorly defined.

Designing AI authority can force us to clarify how the human care system itself is supposed to work.

This connects directly with the earlier proposition: don’t automate the mess. First design the patient journey. Then define human and AI roles. Then define decision rights. Then define the boundaries. Then test. Then allow authority to expand only when the evidence supports it.

From trusted AI to trusted perioperative care

The aim of this worked example is not to argue that AI should become autonomous across perioperative care. Nor is it to argue that humans should permanently approve every action AI performs. The more useful proposition lies between those extremes.

AI may receive different levels of authority for different kinds of work. Routine, reversible and well-understood coordination tasks may eventually support relatively high levels of bounded AI action. Activities involving interpretation, uncertainty or important patient consequences may require stronger human decision rights. And the boundary should change only when evidence justifies it.

This gives us a much more useful question than “can AI manage this patient journey?” Instead ask:

Across this patient journey, where can AI assist, where can it recommend, where can it act, where must it escalate — and where must the decision remain human?

That is a richer governance conversation — worth having with clinicians, nurses, pharmacists, administrators, hospital leaders, technologists and patients. Each sees different parts of the journey, and each may see different consequences when the system fails.

An invitation to discuss

Perioperative care is only one worked example. The same approach could be applied to emergency care, chronic disease management, hospital-at-home, referral pathways, outpatient specialist care, medication management, diagnostic services, and discharge and transition-of-care pathways.

The purpose is not to provide the final answer. It is to make the governance question tangible enough that healthcare teams can debate it. Take one real patient journey. Map the work. Then ask:

• What outcome matters?
• What may AI see?
• What may AI recommend?
• What may AI actually do?
• Where must human decision rights remain?
• What evidence would justify greater authority?
• How will failure be detected?
• How will failure be contained?
• When must AI stop and return control to a human?

And perhaps the most important question of all:

What level of AI authority has the complete human–AI care system demonstrated that it can safely handle today?

That is where AI governance becomes more than restriction. It becomes a discipline for progressively designing trusted capability around the patient journey. And that leads to the next question for healthcare organisations: do we have the leadership, workforce capability, workflow discipline, governance, data, measurement and learning systems required to do this repeatedly — and at scale? That is the conversation that takes us beyond deploying AI tools and towards building an AI-ready healthcare organisation.

Put the idea into practice

The same instrument used in the companion article applies directly here: choose one real workflow along your own patient journey and work through the Trusted AI Boundary Canvas™.

Download the Trusted AI Boundary Canvas™ — use it to define purpose, consequence, access, authority, human decision rights, evidence, containment and the next experiment, for one bounded piece of clinical or administrative work.

For the underlying framework this worked example applies, see the companion article: Don’t Just Govern AI. Design What It Is Allowed to Do.