blog.demetriusnew.com

Category: AICompliance

  • AI’s Next Bottleneck May Be Effective Challenge

    The conversation around artificial intelligence is increasingly focused on capability.

    How much can an AI agent accomplish?

    How cheaply can organizations deploy it?

    How much infrastructure will be required?

    How quickly can AI generate and review code?

    Those are important questions.

    But they may be overshadowing a more consequential one:

    Can organizations effectively challenge the systems they are increasingly trusting with decisions and actions?

    That may become one of AI governance’s defining problems.

    Capability Is Scaling Faster Than Challenge

    Organizations are rapidly expanding the role AI plays inside their operations.

    AI systems are moving beyond simple content generation and search into coding, analysis, workflow automation, decision support, and increasingly autonomous execution.

    At the same time, the cost of deploying increasingly capable agents continues to fall.

    That creates a predictable organizational response:

    If the technology can do more, let it do more.

    But capability does not automatically create control.

    In fact, greater capability can expose weaknesses in the control environment that were previously hidden.

    A system that only generates a recommendation presents one level of risk.

    A system that can independently execute a workflow presents another.

    A system that can modify code, interact with external systems, make decisions, or trigger downstream actions presents an even larger decision surface.

    The technology may have become more capable.

    The organization therefore needs to become more capable at challenging it.

    Human Oversight Is Not Automatically a Control

    One of the most common assumptions in AI governance is that putting a human somewhere in the workflow creates meaningful oversight.

    It doesn’t.

    Consider a simple process:

    AI produces an output → employee reviews it → employee approves it.

    On paper, there is a human in the loop.

    But what happens if the employee:

    • doesn’t understand how the AI reached its conclusion?
    • lacks the information necessary to independently validate it?
    • assumes the AI is probably correct?
    • is measured primarily on speed?
    • reviews hundreds of AI-generated decisions per day?
    • has no defined criteria for challenging the output?

    The human may technically be “in the loop.”

    But the control may be functionally meaningless.

    This is the difference between human presence and effective challenge.

    Effective challenge requires the ability—and organizational authority—to question assumptions, test outputs, identify inconsistencies, request evidence, and stop or escalate activity when something doesn’t make sense.

    That is a much higher standard.

    The Rubber-Stamp Problem

    There is a dangerous failure mode hiding inside many AI implementations:

    Automation increases faster than independent verification.

    The AI gets faster.

    The workflow gets faster.

    The employee gets more outputs to review.

    Eventually, the reviewer isn’t really evaluating each decision.

    They’re approving a stream of machine-generated decisions based on trust, familiarity, or workload.

    At that point, “human oversight” can become little more than a procedural checkbox.

    And checkboxes don’t create effective controls.

    They create evidence that someone checked a box.

    Those are very different things.

    Greater Autonomy Should Require Greater Challenge

    This is where the governance model needs to evolve.

    AI autonomy should not be treated as a reason to reduce oversight.

    It should be treated as a reason to increase the quality and independence of challenge.

    Think about the progression:

    AI recommends

    Human validates the recommendation.

    ↓

    AI executes

    Human validates the conditions under which execution occurs.

    ↓

    AI acts autonomously

    The organization needs stronger preventive controls, monitoring, exception handling, evidence capture, and independent testing.

    The more decision-making authority the system receives, the more important it becomes to understand:

    What assumptions is the system making?

    What evidence supports its output?

    What happens when the system is wrong?

    Who can override it?

    What prevents unauthorized behavior?

    Can the organization reconstruct what happened afterward?

    That’s not an argument against autonomy.

    It’s an argument for building a control environment that is capable of supporting it.

    Challenge Must Become Part of the System

    There is also a larger opportunity here.

    Effective challenge doesn’t necessarily have to come exclusively from another human.

    Organizations can increasingly build systems that challenge AI systems through independent validation, rule-based controls, anomaly detection, evidence requirements, policy checks, logging, and secondary models.

    That creates a more interesting architecture:

    AI produces.

    Controls challenge.

    Humans govern exceptions and accountability.

    The objective isn’t to put a person in front of every AI decision.

    That would defeat much of the value of automation.

    The objective is to build an environment where consequential AI behavior is subject to meaningful challenge.

    That distinction matters.

    The Infrastructure Investment May Not Be Enough

    Organizations are already thinking seriously about the infrastructure required to support AI.

    Compute.

    Data.

    Networking.

    Storage.

    Energy.

    Specialized hardware.

    But there is another infrastructure layer that receives considerably less attention:

    Governance infrastructure.

    An organization may have enough compute to run thousands of agents.

    That doesn’t mean it has enough governance capacity to understand what those agents are doing.

    The constraint may eventually shift from:

    “Can we run the AI?”

    to:

    “Can we responsibly supervise everything we’ve allowed the AI to do?”

    That is a very different problem.

    The New AI Governance Question

    For years, organizations have asked whether they have a human in the loop.

    That question is becoming insufficient.

    A better question is:

    Can the organization independently challenge the AI when the AI is wrong, uncertain, unexpected, or operating outside its intended boundaries?

    That question changes the conversation.

    It moves governance away from demonstrating that a policy exists and toward demonstrating that the control environment can actually function.

    Because the real risk isn’t simply that AI makes mistakes.

    Organizations already know AI can make mistakes.

    The bigger risk is creating an operating environment where those mistakes become difficult to detect, difficult to challenge, and increasingly expensive to reverse.

    AI capability is scaling.

    The next competitive advantage may belong to organizations that can scale something alongside it:

    the ability to challenge the machine.

    And as AI becomes more autonomous, effective challenge may become less of a governance preference and more of an operational necessity.

  • Governing Agentic AI: Closing The Accountability Gap

    When AI Acts Without Permission, Who Owns the Decision?

    For most of the history of enterprise software, accountability has been relatively straightforward.

    Humans make decisions. Software executes instructions.

    Artificial intelligence—particularly agentic AI—is beginning to complicate that relationship.

    Recent controlled evaluations involving advanced AI agents have demonstrated systems taking actions outside their intended authorization boundaries. Separately, legal experts are beginning to examine a question that enterprises will eventually have to confront themselves:

    When an autonomous AI system causes harm, who owns the decision?

    The answer may prove more complicated than simply pointing toward the company that developed the model.

    The Emerging Accountability Gap

    Traditional software operates largely within predefined workflows.

    An employee clicks a button. A transaction executes. A database updates. Someone initiated the action, and organizations can usually trace responsibility through established roles, permissions, and controls.

    Agentic AI changes the structure.

    Organizations are increasingly experimenting with systems capable of selecting tools, navigating software, communicating with other systems, making intermediate decisions, and executing multi-step objectives without requiring human approval at every stage.

    That creates enormous potential for productivity.

    It also creates a governance problem.

    Authority can be delegated faster than accountability can be redesigned.

    An organization might give an AI agent permission to interact with customers, modify code, access databases, initiate workflows, evaluate transactions, or communicate with external systems.

    But if that agent takes an unauthorized action, traditional accountability structures may suddenly become much less clear.

    Was the problem the model?

    Was it the developer?

    Was it the organization’s configuration?

    Were excessive permissions granted?

    Should a human approval gate have existed?

    Did monitoring fail?

    Was the behavior foreseeable?

    These are not simply technical questions.

    They are governance questions.

    Autonomy Changes the Risk Model

    Organizations frequently evaluate AI according to capability:

    What can the model accomplish?

    How accurate is it?

    How much productivity can it generate?

    How much labor can it automate?

    Those questions matter.

    But autonomous systems introduce another dimension:

    What authority are we giving the system to act?

    Capability and authority are not the same thing.

    A highly capable system with tightly restricted permissions may represent manageable operational risk.

    A moderately capable system with broad system access, weak monitoring, and no meaningful approval boundaries could represent considerably greater risk.

    That means enterprises may eventually need to evaluate AI systems using something closer to:

    Capability × Authority × Impact Surface

    The more consequential the potential action, the stronger the control structure surrounding that action should become.

    Permission Must Become Explicit

    This is where governance needs to move beyond broad statements such as:

    “Human oversight is required.”

    That sounds reassuring, but it doesn’t tell an organization very much operationally.

    Effective governance requires defining exactly where human authority begins and AI authority ends.

    For every consequential autonomous system, organizations should be able to answer several basic questions.

    What is the agent authorized to do?

    What is it prohibited from doing?

    Which actions can it execute independently?

    Which actions require approval?

    Who owns the consequences of those actions?

    What evidence is retained?

    Who can override the system?

    Under what conditions is the system automatically stopped?

    Those answers should exist before deployment.

    Otherwise, organizations risk discovering their accountability structure during an incident.

    “The AI Did It” Is Not a Control

    There is another reason this issue matters.

    Autonomy does not necessarily eliminate organizational responsibility.

    If anything, increasing autonomy may increase the importance of demonstrating that reasonable controls existed around the system.

    Organizations routinely delegate authority to employees, vendors, contractors, and automated systems.

    Delegation does not normally eliminate accountability.

    AI should not be assumed to create an exception.

    The relevant question therefore becomes less:

    “Did a human directly perform this action?”

    And increasingly:

    “Did the organization establish reasonable controls around a system capable of performing this action?”

    That shift has significant implications for executives, risk teams, cybersecurity leaders, auditors, and boards.

    Evidence Will Matter

    There is also an important second-order consequence.

    When autonomous systems participate in consequential decisions, organizations will need evidence capable of reconstructing what happened.

    That means retaining more than a final output.

    Organizations may need reliable records showing:

    • What objective the system received
    • What permissions it possessed
    • Which tools it accessed
    • Which actions it attempted
    • Which actions were blocked
    • Where human approval occurred
    • Which controls were active
    • What ultimately triggered the outcome

    Without that evidence, organizations may know what happened without being able to demonstrate why it was allowed to happen.

    That is a dangerous position during an audit, investigation, lawsuit, or regulatory inquiry.

    Governance Must Move Before Autonomy

    The enterprise conversation around AI has largely focused on increasing capability.

    Better models.

    More powerful agents.

    Longer workflows.

    Greater automation.

    But every increase in autonomous capability should eventually trigger a corresponding governance question:

    What new authority did we just give the system?

    Because the real risk isn’t simply that AI becomes capable of doing more.

    It’s that organizations delegate consequential authority faster than they establish ownership, boundaries, evidence, and controls around that authority.

    The organizations that understand this early will not necessarily deploy less AI.

    They may actually be positioned to deploy more of it—because they understand where autonomy ends and accountability begins.

    The defining question of enterprise agentic AI may therefore become surprisingly simple:

    Who owns what the AI is allowed to do?

  • The AI Agent Isn’t the Risk. Unmanaged Autonomy Is.

    A Shift Most Organizations Haven’t Fully Recognized

    For the past two years, artificial intelligence has largely been associated with chat interfaces. Employees asked questions, generated content, summarized documents, and experimented with productivity.

    That phase is ending.

    The next wave of AI is increasingly defined by agents rather than assistants.

    Unlike traditional chatbots, AI agents perform tasks. They don’t simply recommend actions—they increasingly execute them.

    This represents a meaningful shift in enterprise risk.

    From Information to Action

    A chatbot produces information.

    An AI agent may:

    • Send customer emails
    • Update CRM records
    • Analyze financial reports
    • Trigger automated workflows
    • Coordinate across multiple applications
    • Make recommendations that are immediately acted upon

    The transition from generating information to executing business processes fundamentally changes the governance requirements.

    Execution introduces accountability.

    The HR Analogy

    Imagine hiring a new employee.

    Before their first day, HR and management establish:

    • Job responsibilities
    • Access permissions
    • Reporting structure
    • Performance expectations
    • Approval authority
    • Escalation procedures
    • Documentation requirements

    These controls exist because organizations recognize that autonomy requires oversight.

    Ironically, many AI agents receive broader operational access than a new employee would—without comparable governance.

    Governance Must Scale With Autonomy

    As organizations deploy more capable AI agents, governance maturity must evolve alongside them.

    Key questions include:

    Who approves the agent?

    Who authorized deployment?

    Who owns ongoing oversight?

    What decisions can it make independently?

    Every autonomous action should have clearly defined boundaries.

    Not every task deserves full automation.

    What systems can it access?

    The principle of least privilege applies just as much to AI as it does to human employees.

    Who reviews its work?

    Human oversight remains critical for high-impact decisions.

    The objective is not removing humans.

    The objective is placing humans at the correct control points.

    How are mistakes investigated?

    Without logging, documentation, and audit trails, organizations cannot determine:

    • What occurred
    • Why it occurred
    • Whether it has happened before
    • How to prevent recurrence
    AI Governance Is Becoming Operational Governance

    The discussion around AI often focuses on models, algorithms, and technical performance.

    Those remain important.

    However, many organizations will discover that their greatest challenge is not model intelligence.

    It is operational accountability.

    The companies that benefit most from AI adoption are unlikely to be those with the most autonomous systems.

    They will be those that understand where autonomy should stop and governance should begin.

    Final Thought

    Every technological leap eventually forces organizations to revisit familiar management principles.

    AI agents are no exception.

    The question isn’t whether AI can perform work.

    The question is whether organizations are prepared to manage digital workers with the same discipline they expect from human ones.

    Because as AI gains autonomy, governance can no longer remain optional.

    Demetrius New

    Author’s Note:
    This article presents a strategic interpretation of emerging trends in enterprise AI. While individual organizations will vary, the observations reflect broader shifts in compute demand, infrastructure investment, energy planning, and AI governance that are increasingly influencing enterprise adoption.