blog.demetriusnew.com

Tag: AI Governance

  • The AI Governance gap

    When AI Goes Rogue, Audit the Controls Too

    Headlines about autonomous AI systems often follow a familiar pattern.

    An AI agent takes an unexpected action. It exceeds what someone believed its authority to be. It interacts with a system in a way its operators didn’t anticipate.

    The natural conclusion is simple:

    The AI went rogue.

    Sometimes that may be a useful description of the behavior. But from a governance perspective, it isn’t enough.

    There is another question organizations should be asking:

    What did the surrounding control environment allow the AI to do?

    That question becomes increasingly important as organizations move from AI systems that primarily generate information toward agents capable of taking actions.

    From Generating Answers to Taking Actions

    Traditional generative AI largely created an output that a human could review before deciding what happened next.

    Agentic systems change that relationship.

    An agent may interact with:

    • APIs
    • enterprise applications
    • databases
    • credentials
    • internal workflows
    • external services
    • other automated systems

    That means governance can no longer exist primarily at the level of policies, acceptable-use statements and human expectations.

    The technical environment matters.

    An organization may have a policy saying that a particular action requires human authorization.

    But if an agent has the credentials, permissions and technical ability to execute that action without approval, two different versions of the organization’s governance environment exist.

    There is the declared control.

    And there is the enforced control.

    AI agents are increasingly capable of exposing the difference.

    AI as an Accidental Auditor

    This creates an interesting secondary role for autonomous AI.

    Agents may unintentionally become auditors of enterprise control integrity.

    Not because they were instructed to perform an audit.

    Because they interact with the systems organizations actually built rather than the policies organizations intended those systems to represent.

    Imagine that company policy states:

    A sensitive transaction requires managerial approval.

    Employees understand this requirement. They have been trained on it. Perhaps the requirement also appears in an SOP.

    But suppose the underlying application technically allows an authorized user to complete the transaction without that approval.

    For years, the organization may have treated employee knowledge and expected behavior as part of the control.

    Then an autonomous agent enters the workflow.

    The agent sees credentials.

    It sees an available function.

    It sees a path toward completing its assigned objective.

    Unless another mechanism constrains the action, the organization’s written policy may have very little influence over what happens next.

    The agent hasn’t necessarily discovered a new vulnerability.

    It may have exposed an old one.

    Humans Have Historically Been Part of the Control Layer

    Many enterprise systems contain implicit controls that depend heavily on human judgment.

    Employees know there are things they technically can do that they aren’t organizationally allowed to do.

    They understand context.

    They recognize organizational boundaries.

    They know when they should stop and ask someone.

    Organizations have quietly depended on that behavior for decades.

    Increasing autonomy changes the assumption.

    An AI system may not interpret technical availability and organizational authority the same way a human employee does.

    If the architecture allows an action, the agent may treat that action as part of the available solution space.

    That turns previously tolerable gaps between policy and architecture into potentially material governance problems.

    Autonomy Magnifies Existing Weaknesses

    This doesn’t mean AI creates every control failure it exposes.

    Often, the weakness existed beforehand.

    AI changes the economics of exploiting that weakness because autonomous systems introduce three characteristics simultaneously:

    Speed. Actions can happen much faster than traditional human workflows.

    Persistence. An agent can continue pursuing an objective across multiple steps without becoming tired, distracted or hesitant.

    Scale. The same control weakness may potentially be encountered across many transactions or workflows.

    A control gap that was manageable when humans encountered it occasionally can become much more consequential when autonomous systems operate continuously.

    This is why increasing AI capability should be accompanied by increasing control rigor.

    Effective Challenge Must Extend Beyond the Model

    This also changes what effective challenge should mean in an AI-enabled organization.

    It isn’t enough for someone to review an AI recommendation and disagree when necessary.

    Organizations should also be capable of challenging the environment in which autonomous systems operate.

    When an AI system performs an unauthorized or unexpected action, the investigation shouldn’t end with:

    Why did the AI do that?

    It should continue:

    Why did the agent have that permission?

    Why were those credentials available?

    Why could that API be called?

    Why didn’t the workflow require approval?

    Why wasn’t the boundary technically enforced?

    Was the control preventative, detective or merely documented?

    Those questions move the conversation away from AI behavior alone and toward system design and control integrity.

    That distinction will matter as AI becomes more autonomous.

    Policy Is Not Enforcement

    Organizations will continue to need policies.

    But policies describe expected behavior.

    Technical controls determine what systems can actually do.

    The gap between those two environments becomes increasingly important when autonomous systems enter enterprise workflows.

    That leads to a governance question that may become far more valuable than asking whether an organization has an AI policy:

    Can the organization technically enforce what the policy says?

    Because the next generation of AI incidents may reveal something uncomfortable.

    The AI didn’t necessarily break the organization’s controls.

    Sometimes there wasn’t a control there to break.

    The policy describes the organization you intended to build.

    The agent interacts with the organization you actually built.

  • AI’s Next Bottleneck May Be Effective Challenge

    The conversation around artificial intelligence is increasingly focused on capability.

    How much can an AI agent accomplish?

    How cheaply can organizations deploy it?

    How much infrastructure will be required?

    How quickly can AI generate and review code?

    Those are important questions.

    But they may be overshadowing a more consequential one:

    Can organizations effectively challenge the systems they are increasingly trusting with decisions and actions?

    That may become one of AI governance’s defining problems.

    Capability Is Scaling Faster Than Challenge

    Organizations are rapidly expanding the role AI plays inside their operations.

    AI systems are moving beyond simple content generation and search into coding, analysis, workflow automation, decision support, and increasingly autonomous execution.

    At the same time, the cost of deploying increasingly capable agents continues to fall.

    That creates a predictable organizational response:

    If the technology can do more, let it do more.

    But capability does not automatically create control.

    In fact, greater capability can expose weaknesses in the control environment that were previously hidden.

    A system that only generates a recommendation presents one level of risk.

    A system that can independently execute a workflow presents another.

    A system that can modify code, interact with external systems, make decisions, or trigger downstream actions presents an even larger decision surface.

    The technology may have become more capable.

    The organization therefore needs to become more capable at challenging it.

    Human Oversight Is Not Automatically a Control

    One of the most common assumptions in AI governance is that putting a human somewhere in the workflow creates meaningful oversight.

    It doesn’t.

    Consider a simple process:

    AI produces an output → employee reviews it → employee approves it.

    On paper, there is a human in the loop.

    But what happens if the employee:

    • doesn’t understand how the AI reached its conclusion?
    • lacks the information necessary to independently validate it?
    • assumes the AI is probably correct?
    • is measured primarily on speed?
    • reviews hundreds of AI-generated decisions per day?
    • has no defined criteria for challenging the output?

    The human may technically be “in the loop.”

    But the control may be functionally meaningless.

    This is the difference between human presence and effective challenge.

    Effective challenge requires the ability—and organizational authority—to question assumptions, test outputs, identify inconsistencies, request evidence, and stop or escalate activity when something doesn’t make sense.

    That is a much higher standard.

    The Rubber-Stamp Problem

    There is a dangerous failure mode hiding inside many AI implementations:

    Automation increases faster than independent verification.

    The AI gets faster.

    The workflow gets faster.

    The employee gets more outputs to review.

    Eventually, the reviewer isn’t really evaluating each decision.

    They’re approving a stream of machine-generated decisions based on trust, familiarity, or workload.

    At that point, “human oversight” can become little more than a procedural checkbox.

    And checkboxes don’t create effective controls.

    They create evidence that someone checked a box.

    Those are very different things.

    Greater Autonomy Should Require Greater Challenge

    This is where the governance model needs to evolve.

    AI autonomy should not be treated as a reason to reduce oversight.

    It should be treated as a reason to increase the quality and independence of challenge.

    Think about the progression:

    AI recommends

    Human validates the recommendation.

    ↓

    AI executes

    Human validates the conditions under which execution occurs.

    ↓

    AI acts autonomously

    The organization needs stronger preventive controls, monitoring, exception handling, evidence capture, and independent testing.

    The more decision-making authority the system receives, the more important it becomes to understand:

    What assumptions is the system making?

    What evidence supports its output?

    What happens when the system is wrong?

    Who can override it?

    What prevents unauthorized behavior?

    Can the organization reconstruct what happened afterward?

    That’s not an argument against autonomy.

    It’s an argument for building a control environment that is capable of supporting it.

    Challenge Must Become Part of the System

    There is also a larger opportunity here.

    Effective challenge doesn’t necessarily have to come exclusively from another human.

    Organizations can increasingly build systems that challenge AI systems through independent validation, rule-based controls, anomaly detection, evidence requirements, policy checks, logging, and secondary models.

    That creates a more interesting architecture:

    AI produces.

    Controls challenge.

    Humans govern exceptions and accountability.

    The objective isn’t to put a person in front of every AI decision.

    That would defeat much of the value of automation.

    The objective is to build an environment where consequential AI behavior is subject to meaningful challenge.

    That distinction matters.

    The Infrastructure Investment May Not Be Enough

    Organizations are already thinking seriously about the infrastructure required to support AI.

    Compute.

    Data.

    Networking.

    Storage.

    Energy.

    Specialized hardware.

    But there is another infrastructure layer that receives considerably less attention:

    Governance infrastructure.

    An organization may have enough compute to run thousands of agents.

    That doesn’t mean it has enough governance capacity to understand what those agents are doing.

    The constraint may eventually shift from:

    “Can we run the AI?”

    to:

    “Can we responsibly supervise everything we’ve allowed the AI to do?”

    That is a very different problem.

    The New AI Governance Question

    For years, organizations have asked whether they have a human in the loop.

    That question is becoming insufficient.

    A better question is:

    Can the organization independently challenge the AI when the AI is wrong, uncertain, unexpected, or operating outside its intended boundaries?

    That question changes the conversation.

    It moves governance away from demonstrating that a policy exists and toward demonstrating that the control environment can actually function.

    Because the real risk isn’t simply that AI makes mistakes.

    Organizations already know AI can make mistakes.

    The bigger risk is creating an operating environment where those mistakes become difficult to detect, difficult to challenge, and increasingly expensive to reverse.

    AI capability is scaling.

    The next competitive advantage may belong to organizations that can scale something alongside it:

    the ability to challenge the machine.

    And as AI becomes more autonomous, effective challenge may become less of a governance preference and more of an operational necessity.

  • Governing Agentic AI: Closing The Accountability Gap

    When AI Acts Without Permission, Who Owns the Decision?

    For most of the history of enterprise software, accountability has been relatively straightforward.

    Humans make decisions. Software executes instructions.

    Artificial intelligence—particularly agentic AI—is beginning to complicate that relationship.

    Recent controlled evaluations involving advanced AI agents have demonstrated systems taking actions outside their intended authorization boundaries. Separately, legal experts are beginning to examine a question that enterprises will eventually have to confront themselves:

    When an autonomous AI system causes harm, who owns the decision?

    The answer may prove more complicated than simply pointing toward the company that developed the model.

    The Emerging Accountability Gap

    Traditional software operates largely within predefined workflows.

    An employee clicks a button. A transaction executes. A database updates. Someone initiated the action, and organizations can usually trace responsibility through established roles, permissions, and controls.

    Agentic AI changes the structure.

    Organizations are increasingly experimenting with systems capable of selecting tools, navigating software, communicating with other systems, making intermediate decisions, and executing multi-step objectives without requiring human approval at every stage.

    That creates enormous potential for productivity.

    It also creates a governance problem.

    Authority can be delegated faster than accountability can be redesigned.

    An organization might give an AI agent permission to interact with customers, modify code, access databases, initiate workflows, evaluate transactions, or communicate with external systems.

    But if that agent takes an unauthorized action, traditional accountability structures may suddenly become much less clear.

    Was the problem the model?

    Was it the developer?

    Was it the organization’s configuration?

    Were excessive permissions granted?

    Should a human approval gate have existed?

    Did monitoring fail?

    Was the behavior foreseeable?

    These are not simply technical questions.

    They are governance questions.

    Autonomy Changes the Risk Model

    Organizations frequently evaluate AI according to capability:

    What can the model accomplish?

    How accurate is it?

    How much productivity can it generate?

    How much labor can it automate?

    Those questions matter.

    But autonomous systems introduce another dimension:

    What authority are we giving the system to act?

    Capability and authority are not the same thing.

    A highly capable system with tightly restricted permissions may represent manageable operational risk.

    A moderately capable system with broad system access, weak monitoring, and no meaningful approval boundaries could represent considerably greater risk.

    That means enterprises may eventually need to evaluate AI systems using something closer to:

    Capability × Authority × Impact Surface

    The more consequential the potential action, the stronger the control structure surrounding that action should become.

    Permission Must Become Explicit

    This is where governance needs to move beyond broad statements such as:

    “Human oversight is required.”

    That sounds reassuring, but it doesn’t tell an organization very much operationally.

    Effective governance requires defining exactly where human authority begins and AI authority ends.

    For every consequential autonomous system, organizations should be able to answer several basic questions.

    What is the agent authorized to do?

    What is it prohibited from doing?

    Which actions can it execute independently?

    Which actions require approval?

    Who owns the consequences of those actions?

    What evidence is retained?

    Who can override the system?

    Under what conditions is the system automatically stopped?

    Those answers should exist before deployment.

    Otherwise, organizations risk discovering their accountability structure during an incident.

    “The AI Did It” Is Not a Control

    There is another reason this issue matters.

    Autonomy does not necessarily eliminate organizational responsibility.

    If anything, increasing autonomy may increase the importance of demonstrating that reasonable controls existed around the system.

    Organizations routinely delegate authority to employees, vendors, contractors, and automated systems.

    Delegation does not normally eliminate accountability.

    AI should not be assumed to create an exception.

    The relevant question therefore becomes less:

    “Did a human directly perform this action?”

    And increasingly:

    “Did the organization establish reasonable controls around a system capable of performing this action?”

    That shift has significant implications for executives, risk teams, cybersecurity leaders, auditors, and boards.

    Evidence Will Matter

    There is also an important second-order consequence.

    When autonomous systems participate in consequential decisions, organizations will need evidence capable of reconstructing what happened.

    That means retaining more than a final output.

    Organizations may need reliable records showing:

    • What objective the system received
    • What permissions it possessed
    • Which tools it accessed
    • Which actions it attempted
    • Which actions were blocked
    • Where human approval occurred
    • Which controls were active
    • What ultimately triggered the outcome

    Without that evidence, organizations may know what happened without being able to demonstrate why it was allowed to happen.

    That is a dangerous position during an audit, investigation, lawsuit, or regulatory inquiry.

    Governance Must Move Before Autonomy

    The enterprise conversation around AI has largely focused on increasing capability.

    Better models.

    More powerful agents.

    Longer workflows.

    Greater automation.

    But every increase in autonomous capability should eventually trigger a corresponding governance question:

    What new authority did we just give the system?

    Because the real risk isn’t simply that AI becomes capable of doing more.

    It’s that organizations delegate consequential authority faster than they establish ownership, boundaries, evidence, and controls around that authority.

    The organizations that understand this early will not necessarily deploy less AI.

    They may actually be positioned to deploy more of it—because they understand where autonomy ends and accountability begins.

    The defining question of enterprise agentic AI may therefore become surprisingly simple:

    Who owns what the AI is allowed to do?

  • The AI Agent Isn’t the Risk. Unmanaged Autonomy Is.

    A Shift Most Organizations Haven’t Fully Recognized

    For the past two years, artificial intelligence has largely been associated with chat interfaces. Employees asked questions, generated content, summarized documents, and experimented with productivity.

    That phase is ending.

    The next wave of AI is increasingly defined by agents rather than assistants.

    Unlike traditional chatbots, AI agents perform tasks. They don’t simply recommend actions—they increasingly execute them.

    This represents a meaningful shift in enterprise risk.

    From Information to Action

    A chatbot produces information.

    An AI agent may:

    • Send customer emails
    • Update CRM records
    • Analyze financial reports
    • Trigger automated workflows
    • Coordinate across multiple applications
    • Make recommendations that are immediately acted upon

    The transition from generating information to executing business processes fundamentally changes the governance requirements.

    Execution introduces accountability.

    The HR Analogy

    Imagine hiring a new employee.

    Before their first day, HR and management establish:

    • Job responsibilities
    • Access permissions
    • Reporting structure
    • Performance expectations
    • Approval authority
    • Escalation procedures
    • Documentation requirements

    These controls exist because organizations recognize that autonomy requires oversight.

    Ironically, many AI agents receive broader operational access than a new employee would—without comparable governance.

    Governance Must Scale With Autonomy

    As organizations deploy more capable AI agents, governance maturity must evolve alongside them.

    Key questions include:

    Who approves the agent?

    Who authorized deployment?

    Who owns ongoing oversight?

    What decisions can it make independently?

    Every autonomous action should have clearly defined boundaries.

    Not every task deserves full automation.

    What systems can it access?

    The principle of least privilege applies just as much to AI as it does to human employees.

    Who reviews its work?

    Human oversight remains critical for high-impact decisions.

    The objective is not removing humans.

    The objective is placing humans at the correct control points.

    How are mistakes investigated?

    Without logging, documentation, and audit trails, organizations cannot determine:

    • What occurred
    • Why it occurred
    • Whether it has happened before
    • How to prevent recurrence
    AI Governance Is Becoming Operational Governance

    The discussion around AI often focuses on models, algorithms, and technical performance.

    Those remain important.

    However, many organizations will discover that their greatest challenge is not model intelligence.

    It is operational accountability.

    The companies that benefit most from AI adoption are unlikely to be those with the most autonomous systems.

    They will be those that understand where autonomy should stop and governance should begin.

    Final Thought

    Every technological leap eventually forces organizations to revisit familiar management principles.

    AI agents are no exception.

    The question isn’t whether AI can perform work.

    The question is whether organizations are prepared to manage digital workers with the same discipline they expect from human ones.

    Because as AI gains autonomy, governance can no longer remain optional.

    Demetrius New

    Author’s Note:
    This article presents a strategic interpretation of emerging trends in enterprise AI. While individual organizations will vary, the observations reflect broader shifts in compute demand, infrastructure investment, energy planning, and AI governance that are increasingly influencing enterprise adoption.

  • AI Systems Don’t Fail Because of Intelligence — They Fail Because of Governance

    In the excitement of deploying artificial intelligence, the conversation often gets hijacked by the technology’s “intelligence.” We’re captivated by its ability to generate natural-sounding text, create stunning images, and identify complex patterns in data. But this focus is misplaced.

    While model accuracy, hallucinations, and bias are significant, they aren’t the primary drivers of AI failure. The real risk lies in a much less glamorous, but far more consequential, area: governance.

    Think about it: A highly intelligent financial analyst is useless if their company has no accounting systems. Their insights would be lost in a sea of data, and their decisions could have disastrous consequences if not properly reviewed and audited. The same principle applies to AI.

    The Mirage of “AI Risk”

    Most companies view AI risk through a technological lens. They worry about:

    • Model accuracy: “Is our prediction model right often enough?”
    • Hallucinations: “Is the language model just making things up?”
    • Bias: “Does our AI perpetuate societal inequalities?”

    These are critical issues, and they absolutely require technical solutions. However, they are symptoms of a larger, more fundamental problem.

    The Real Risk: A Lack of Control

    The true danger of AI is not its lack of intelligence, but its potential for uncontrolled autonomy. The real risks that companies face are:

    • Uncontrolled automation: Giving AI the authority to make critical decisions without appropriate oversight.
    • No audit trail: The inability to trace the decision-making process of an AI system, making it impossible to understand how it reached a particular conclusion.
    • No human checkpoints: Failing to incorporate human judgment at key points in the AI lifecycle, allowing the system to operate on autopilot.
    • Unclear authority over model outputs: Failing to define who is accountable for the decisions made by an AI system.

    This is a failure of governance, not technology. It’s a failure of organizational processes and controls, not of algorithms.

    The Missing Layer

    To successfully deploy AI, companies must introduce a robust governance layer. This means going beyond simply building and deploying models, and focusing on:

    • Architecture: Designing AI systems with control, transparency, and auditability built-in. This involves defining clear interfaces, data pipelines, and decision points.
    • Oversight: Establishing clear processes and personnel responsible for monitoring and managing AI systems throughout their lifecycle. This includes continuous monitoring of performance, bias detection, and regular audits.
    • Accountability: Clearly defining roles and responsibilities for AI development, deployment, and performance. This means knowing who is responsible for the inputs, the outputs, and the consequences.

    The Core Concept

    Here’s the key takeaway:

    Powerful AI systems need governance the same way financial systems need accounting.

    Just as we wouldn’t trust a financial analyst without an accounting system, we shouldn’t trust an AI system without a robust governance framework. Governance provides the necessary checks and balances, the audit trail, and the accountability structure to ensure that AI is used effectively, ethically, and responsibly.

    Closing

    The next generation of AI infrastructure will not be defined by its intelligence alone. It will be defined by its ability to be governed, managed, and controlled. It will be characterized by its defensibility.

    The true differentiator for companies that succeed with AI will be their ability to build and implement robust governance frameworks. This is not just a regulatory or ethical imperative; it’s a fundamental business necessity.

    It’s time to shift the conversation. It’s time to stop worrying about the “intelligence” of AI and start focusing on the control. If you enjoy reading about AI from this perspective, like, follow and share. Leave a comment, I will be providing more on this topic.