AI governance: auditing what is not deterministic

Most organisations already using AI in something that matters can answer whether they use it. Fewer can say which decisions it touches, who approved go-live, which model version ran at the time, and where the context data came from. That is not a technology problem but a governance one, and it looks the way information security did fifteen years ago: practice that works, without traceability that survives questioning. What follows is for those past the opening stage: models in production, agreements in place, someone asking how it is governed.

What ISO/IEC 42001 adds

ISO/IEC 42001 is a management system for artificial intelligence, built on the harmonised structure it shares with ISO 27001 and ISO 9001, and certifiable in the same way. The scaffolding is familiar. What is new is what it asks you to govern.

It requires an AI policy, defined roles across the life cycle of an AI system, governance of the data the system rests on, a statement of applicability for the Annex A controls, and above all an AI system impact assessment. The difference between that and an ordinary risk assessment is not academic. A 27001 risk assessment asks what could harm the business. A 42001 impact assessment also asks what the system could do to the individuals, groups and societies affected by its decisions. That question cannot be inherited from the ISMS.

The standard does not stand alone. ISO/IEC 22989 supplies the vocabulary, ISO/IEC 23894 the guidance on AI risk management, and ISO/IEC 42005 the impact assessment itself. Berigo is an authorised PECB partner for ISO/IEC 42001.

The relationship to ISO 27001

If you have a working ISMS, do not build a parallel one beside it. Scope, risk methodology, nonconformity handling, internal audit and management review can be shared. What has to be added is a risk taxonomy specific to AI and its own set of controls. Note that scope itself behaves differently: an ISMS is usually drawn around the organisation and its services, whereas an AI management system has to be drawn around the AI systems you actually have and the roles you hold in them: provider, deployer, or both.

27001 protects confidentiality, integrity and availability. 42001 adds properties that do not measure the same way: robustness, explainability, fairness and the ability to intervene while the system is running. A system can be impeccably secured and still produce systematically skewed outcomes. That is no security nonconformity in 27001 terms, but it is a failure of AI governance, and catching it is what 42001 is for.

The AI Act: your role determines your duties

The AI Act, Regulation (EU) 2024/1689, classifies by risk: prohibited practices, high risk, uses carrying transparency obligations, and everything else. Alongside that runs a separate regime for general-purpose AI models, aimed mainly at whoever builds the model. But the practical starting point is seldom the risk class. It is the role: most organisations are deployers rather than providers, and the obligations differ materially.

A deployer's duties for a high-risk system are concrete and less widely known than they should be. The system is to be used in line with the provider's instructions for use. Human oversight must be assigned to named people with the competence, training and authority to overrule it, and the support they need to actually do so. Where you control the input data, it must be relevant and sufficiently representative for the intended purpose. Operation must be monitored, with the provider and the authorities told of risks and serious incidents. Logs the system generates automatically, so far as they are under your control, must be retained, as a rule for at least six months. Where it is used in the workplace, employees and their representatives must be told beforehand. And people who are the subject of decisions taken with the system are entitled to be told; where the decision carries legal or similarly significant effects, they may ask what part the system played in it.

The role changes more easily than people expect. Put your own name or trademark on a high-risk system, change its intended purpose so that it becomes high risk, or modify it substantially, and you may be treated as the provider, with a provider's full documentation, conformity and registration duties. An internal “small adjustment” to a bought-in model need not be small in law.

Two clarifications belong here. Some deployers must also carry out a fundamental rights impact assessment before putting a high-risk system into use: public bodies, private entities providing public services, and those using such systems to assess the creditworthiness of individuals or to price and assess risk in life and health insurance. And the regulation is marked EEA-relevant. Whenever it enters Norwegian law, the requirements already reach Norwegian businesses through customers and owners in the EU, and through the Act's reach over use outside the Union whose output is used inside it. Requirements phase in over time and the timelines have been adjusted politically along the way: plan against the substance, not the dates.

Three assessments that are not the same thing

Three assessments are routinely conflated, although they rest on different legal bases, ask different questions and are triggered by different things. A data protection impact assessment, the DPIA under GDPR Article 35, is triggered where processing is likely to result in a high risk to the rights and freedoms of data subjects, and concerns personal data. The ISO/IEC 42001 impact assessment covers a broader set of effects of the AI system, including cases with no personal data in them at all. The fundamental rights impact assessment under the AI Act applies only to particular deployers and particular high-risk systems.

They overlap, and the work can be reused, but none of them substitutes for another. In practice it pays to keep one assessment template with modules, so that the same body of fact serves all three and the record shows which duty each part discharges. The alternative is three documents saying almost the same thing, written at different times, reaching different conclusions.

Data governance: where did the data come from, and who has seen it

This is where most programmes come apart under examination: the questions are easy to ask and hard to answer afterwards. Which data was used for training or fine-tuning, on what lawful basis and under what licence? What is pulled in as context at run time, from which sources? Do prompts leave the organisation for a supplier, are they used to train that supplier's models, how long are they retained and where? And where data gathered for one purpose becomes training material for another, purpose limitation is a question in its own right, not merely a security question.

The most common failure sits in access control on the retrieval index. When an assistant fetches documents to answer, the answer inherits the index's permissions, not the user's, unless access is enforced at query time. The result is a model faithfully reproducing content the user should never have seen, with nothing in the logs resembling a breach. Insist on document-level access control at run time, not only at indexing.

Data governance in the 42001 sense is also about more than quality. It is about provenance: where the dataset came from, who approved its use, and which version underpinned the model that was running in March. Without a data catalogue carrying version and named ownership, these are questions nobody can answer six months later.

Model and supplier risk

An AI system is not one product but a chain: base model, adaptation layer, system prompt, knowledge source, filters, application. Every link can change without anyone asking you. A supplier who upgrades the model, adjusts the safety filters or retires a version has changed a system you validated.

So treat model version, prompt version, index version and filter configuration as configuration items under change control. And put in the contract what you need in order to govern: notice of material model changes, a defined period before a version is retired, the right to test before changes reach production, access to logs, a processing agreement listing sub-processors and the transfer basis where data leaves the EEA, and an exit clause that can actually be executed.

Human oversight as a control, not as a phrase

“Human in the loop” is among the cheapest sentences to write and the costliest to honour. The control is real only when the person has time, independent information and an incentive to disagree. Without that you get automation bias: an approver who confirms.

There is a separate legal requirement here too. GDPR Article 22 gives individuals the right not to be subject to a decision based solely on automated processing where it produces legal effects or similarly significantly affects them. The exemptions come with conditions, among them a right to human intervention. Someone who only presses approve does not make the processing any less automated in law.

So measure it. Override rate is among the most informative indicators you can have, and if it sits at zero over time the control is probably ceremonial. Log which cases were overridden, why and to what outcome. That both evidences the control and improves the system.

Auditing something that is not deterministic

An auditor cannot obtain assurance that the system always does X, because it does not. So the object of the audit has to move: not the individual answer, but the boundaries, the evidence and the process around it. In practice an auditor tests two things: that the control is designed so that it can work, and that it has in fact worked over time. Neither requires the model to answer identically twice.

Concretely: a frozen evaluation set with defined acceptance criteria, re-run on every change; structured logs tying each output to model version, prompt, source documents and user; sampled human review against a written rubric; monitoring for drift in performance; and thresholds that trigger rollback. Full reproducibility is rarely achievable. Being able to account for what happened is, provided you decided to keep the evidence before you needed it.

From ad hoc use to a governed system

Sequence matters more than ambition. Begin with an inventory: what is actually in use, including the features bundled into tools you already had. Then triage by consequence: most use cases need very little, a few need everything. Then gates for the few: impact assessment, named owner, defined human oversight, evaluation before production, logging. Finally evidence, in a form that survives being asked for six months later.

Berigo treats AI governance as part of the management system rather than a document project alongside it, and trains the people who will own it internally.

This is not a larger programme than an ISMS. It is usually the same programme with a new chapter, and the chapter is easier to write while you still remember why the decisions were made.