blog.demetriusnew.com

Tag: AI oversight

  • AI’s Next Bottleneck May Be Effective Challenge

    The conversation around artificial intelligence is increasingly focused on capability.

    How much can an AI agent accomplish?

    How cheaply can organizations deploy it?

    How much infrastructure will be required?

    How quickly can AI generate and review code?

    Those are important questions.

    But they may be overshadowing a more consequential one:

    Can organizations effectively challenge the systems they are increasingly trusting with decisions and actions?

    That may become one of AI governance’s defining problems.

    Capability Is Scaling Faster Than Challenge

    Organizations are rapidly expanding the role AI plays inside their operations.

    AI systems are moving beyond simple content generation and search into coding, analysis, workflow automation, decision support, and increasingly autonomous execution.

    At the same time, the cost of deploying increasingly capable agents continues to fall.

    That creates a predictable organizational response:

    If the technology can do more, let it do more.

    But capability does not automatically create control.

    In fact, greater capability can expose weaknesses in the control environment that were previously hidden.

    A system that only generates a recommendation presents one level of risk.

    A system that can independently execute a workflow presents another.

    A system that can modify code, interact with external systems, make decisions, or trigger downstream actions presents an even larger decision surface.

    The technology may have become more capable.

    The organization therefore needs to become more capable at challenging it.

    Human Oversight Is Not Automatically a Control

    One of the most common assumptions in AI governance is that putting a human somewhere in the workflow creates meaningful oversight.

    It doesn’t.

    Consider a simple process:

    AI produces an output → employee reviews it → employee approves it.

    On paper, there is a human in the loop.

    But what happens if the employee:

    • doesn’t understand how the AI reached its conclusion?
    • lacks the information necessary to independently validate it?
    • assumes the AI is probably correct?
    • is measured primarily on speed?
    • reviews hundreds of AI-generated decisions per day?
    • has no defined criteria for challenging the output?

    The human may technically be “in the loop.”

    But the control may be functionally meaningless.

    This is the difference between human presence and effective challenge.

    Effective challenge requires the ability—and organizational authority—to question assumptions, test outputs, identify inconsistencies, request evidence, and stop or escalate activity when something doesn’t make sense.

    That is a much higher standard.

    The Rubber-Stamp Problem

    There is a dangerous failure mode hiding inside many AI implementations:

    Automation increases faster than independent verification.

    The AI gets faster.

    The workflow gets faster.

    The employee gets more outputs to review.

    Eventually, the reviewer isn’t really evaluating each decision.

    They’re approving a stream of machine-generated decisions based on trust, familiarity, or workload.

    At that point, “human oversight” can become little more than a procedural checkbox.

    And checkboxes don’t create effective controls.

    They create evidence that someone checked a box.

    Those are very different things.

    Greater Autonomy Should Require Greater Challenge

    This is where the governance model needs to evolve.

    AI autonomy should not be treated as a reason to reduce oversight.

    It should be treated as a reason to increase the quality and independence of challenge.

    Think about the progression:

    AI recommends

    Human validates the recommendation.

    ↓

    AI executes

    Human validates the conditions under which execution occurs.

    ↓

    AI acts autonomously

    The organization needs stronger preventive controls, monitoring, exception handling, evidence capture, and independent testing.

    The more decision-making authority the system receives, the more important it becomes to understand:

    What assumptions is the system making?

    What evidence supports its output?

    What happens when the system is wrong?

    Who can override it?

    What prevents unauthorized behavior?

    Can the organization reconstruct what happened afterward?

    That’s not an argument against autonomy.

    It’s an argument for building a control environment that is capable of supporting it.

    Challenge Must Become Part of the System

    There is also a larger opportunity here.

    Effective challenge doesn’t necessarily have to come exclusively from another human.

    Organizations can increasingly build systems that challenge AI systems through independent validation, rule-based controls, anomaly detection, evidence requirements, policy checks, logging, and secondary models.

    That creates a more interesting architecture:

    AI produces.

    Controls challenge.

    Humans govern exceptions and accountability.

    The objective isn’t to put a person in front of every AI decision.

    That would defeat much of the value of automation.

    The objective is to build an environment where consequential AI behavior is subject to meaningful challenge.

    That distinction matters.

    The Infrastructure Investment May Not Be Enough

    Organizations are already thinking seriously about the infrastructure required to support AI.

    Compute.

    Data.

    Networking.

    Storage.

    Energy.

    Specialized hardware.

    But there is another infrastructure layer that receives considerably less attention:

    Governance infrastructure.

    An organization may have enough compute to run thousands of agents.

    That doesn’t mean it has enough governance capacity to understand what those agents are doing.

    The constraint may eventually shift from:

    “Can we run the AI?”

    to:

    “Can we responsibly supervise everything we’ve allowed the AI to do?”

    That is a very different problem.

    The New AI Governance Question

    For years, organizations have asked whether they have a human in the loop.

    That question is becoming insufficient.

    A better question is:

    Can the organization independently challenge the AI when the AI is wrong, uncertain, unexpected, or operating outside its intended boundaries?

    That question changes the conversation.

    It moves governance away from demonstrating that a policy exists and toward demonstrating that the control environment can actually function.

    Because the real risk isn’t simply that AI makes mistakes.

    Organizations already know AI can make mistakes.

    The bigger risk is creating an operating environment where those mistakes become difficult to detect, difficult to challenge, and increasingly expensive to reverse.

    AI capability is scaling.

    The next competitive advantage may belong to organizations that can scale something alongside it:

    the ability to challenge the machine.

    And as AI becomes more autonomous, effective challenge may become less of a governance preference and more of an operational necessity.

  • Beyond The Model: The AI Dependency Stack

    I Started Studying AI and Somehow Ended Up Studying the Power Grid

    I didn’t start studying artificial intelligence because I wanted to understand the electrical grid.

    Yet somehow, that’s where the trail led.

    What began as an effort to better understand AI models gradually expanded into compute infrastructure, data centers, electricity generation, transmission constraints, capital investment and eventually governance.

    Looking back, the progression makes perfect sense.

    Models → Compute → Data Centers → Electricity → Transmission → Capital → Governance

    Each subject exposed a dependency underneath the previous one.

    And that has changed the way I think about AI.

    Follow the Dependency

    When most conversations about artificial intelligence begin, they start with the model.

    How capable is it?

    How quickly is performance improving?

    What tasks can it automate?

    What happens when agents become more autonomous?

    Those are important questions. But eventually I started asking another one:

    What does this depend on?

    AI models depend on compute.

    Compute depends on chips, servers, cooling systems and data centers.

    Data centers depend on enormous amounts of reliable electricity.

    Electricity depends on generation capacity and the infrastructure required to move that power where it is needed.

    That means transmission capacity, substations, interconnection queues, permitting and infrastructure investment suddenly become part of the AI conversation.

    Keep following the chain and another layer appears.

    Capital.

    Someone has to finance all of this infrastructure.

    Companies have to decide how much capacity to build, where to build it, what assumptions justify the investment and whether expected AI demand will generate sufficient returns.

    Now the conversation isn’t simply technological.

    It is operational, financial and strategic.

    And eventually it becomes a governance problem.

    Capability Is Only One Layer of the System

    One of the most important lessons I’ve taken from this progression is that technological capability doesn’t guarantee system-level capability.

    A model might become significantly more powerful while the infrastructure supporting its deployment remains constrained.

    More demand for compute can create pressure on data-center capacity.

    Additional data centers increase electricity demand.

    New generation capacity doesn’t necessarily solve the problem if transmission infrastructure cannot deliver that electricity where demand is growing.

    And even if every physical constraint could be removed, organizations still have to determine whether the resulting investments make economic sense and whether the systems being deployed can be governed responsibly.

    This creates an interesting possibility:

    The biggest constraint on the next stage of AI might not always be AI.

    It could appear somewhere else in the dependency chain.

    Systems Don’t Respect Industry Categories

    We tend to organize knowledge into categories.

    Artificial intelligence.

    Energy.

    Finance.

    Infrastructure.

    Risk.

    Governance.

    But real systems don’t care about those categories.

    Their dependencies cross them constantly.

    An AI strategy can become an infrastructure strategy.

    An infrastructure strategy can become an energy strategy.

    An energy constraint can become a capital-allocation problem.

    A capital decision can create new operational risks.

    And those risks eventually require governance.

    What looked like several unrelated subjects was actually one interconnected system viewed from different layers.

    That’s what systems thinking started changing for me.

    Instead of asking only:

    What can this technology do?

    I became increasingly interested in another question:

    What has to remain true for this entire system to work?

    That question exposes a very different landscape.

    The Bottleneck Moves

    Another realization followed.

    Bottlenecks don’t disappear simply because technology improves.

    They move.

    If models become more efficient, another constraint may become more important.

    If compute capacity expands, electricity availability may become the limiting factor.

    If generation expands, transmission may become the constraint.

    If infrastructure expands rapidly, capital discipline or governance may become the limiting factor.

    This means leaders evaluating AI cannot look exclusively at technological capability.

    They also have to understand the dependency chain supporting that capability.

    Otherwise, it becomes easy to mistake progress in one layer for readiness across the entire system.

    The Infrastructure Underneath the Intelligence

    I still study AI.

    But now I find myself paying much more attention to what surrounds it.

    Energy infrastructure.

    Data-center development.

    Transmission.

    Capital allocation.

    Operational risk.

    Governance.

    Not because I abandoned the original subject, but because following it seriously kept expanding the boundaries of the system.

    That’s probably the biggest lesson this journey has taught me:

    You don’t really understand a system until you understand what it depends on.

    AI may appear to be a software revolution when viewed from the surface.

    Follow the dependency chain downward, however, and you eventually reach physical infrastructure, energy, economics and governance.

    And sometimes the most important part of a technology isn’t the technology itself.

    It’s everything that has to work underneath it.

  • The AI Agent Isn’t the Risk. Unmanaged Autonomy Is.

    A Shift Most Organizations Haven’t Fully Recognized

    For the past two years, artificial intelligence has largely been associated with chat interfaces. Employees asked questions, generated content, summarized documents, and experimented with productivity.

    That phase is ending.

    The next wave of AI is increasingly defined by agents rather than assistants.

    Unlike traditional chatbots, AI agents perform tasks. They don’t simply recommend actions—they increasingly execute them.

    This represents a meaningful shift in enterprise risk.

    From Information to Action

    A chatbot produces information.

    An AI agent may:

    • Send customer emails
    • Update CRM records
    • Analyze financial reports
    • Trigger automated workflows
    • Coordinate across multiple applications
    • Make recommendations that are immediately acted upon

    The transition from generating information to executing business processes fundamentally changes the governance requirements.

    Execution introduces accountability.

    The HR Analogy

    Imagine hiring a new employee.

    Before their first day, HR and management establish:

    • Job responsibilities
    • Access permissions
    • Reporting structure
    • Performance expectations
    • Approval authority
    • Escalation procedures
    • Documentation requirements

    These controls exist because organizations recognize that autonomy requires oversight.

    Ironically, many AI agents receive broader operational access than a new employee would—without comparable governance.

    Governance Must Scale With Autonomy

    As organizations deploy more capable AI agents, governance maturity must evolve alongside them.

    Key questions include:

    Who approves the agent?

    Who authorized deployment?

    Who owns ongoing oversight?

    What decisions can it make independently?

    Every autonomous action should have clearly defined boundaries.

    Not every task deserves full automation.

    What systems can it access?

    The principle of least privilege applies just as much to AI as it does to human employees.

    Who reviews its work?

    Human oversight remains critical for high-impact decisions.

    The objective is not removing humans.

    The objective is placing humans at the correct control points.

    How are mistakes investigated?

    Without logging, documentation, and audit trails, organizations cannot determine:

    • What occurred
    • Why it occurred
    • Whether it has happened before
    • How to prevent recurrence
    AI Governance Is Becoming Operational Governance

    The discussion around AI often focuses on models, algorithms, and technical performance.

    Those remain important.

    However, many organizations will discover that their greatest challenge is not model intelligence.

    It is operational accountability.

    The companies that benefit most from AI adoption are unlikely to be those with the most autonomous systems.

    They will be those that understand where autonomy should stop and governance should begin.

    Final Thought

    Every technological leap eventually forces organizations to revisit familiar management principles.

    AI agents are no exception.

    The question isn’t whether AI can perform work.

    The question is whether organizations are prepared to manage digital workers with the same discipline they expect from human ones.

    Because as AI gains autonomy, governance can no longer remain optional.

    Demetrius New

    Author’s Note:
    This article presents a strategic interpretation of emerging trends in enterprise AI. While individual organizations will vary, the observations reflect broader shifts in compute demand, infrastructure investment, energy planning, and AI governance that are increasingly influencing enterprise adoption.

  • AI Systems Don’t Fail Because of Intelligence — They Fail Because of Governance

    In the excitement of deploying artificial intelligence, the conversation often gets hijacked by the technology’s “intelligence.” We’re captivated by its ability to generate natural-sounding text, create stunning images, and identify complex patterns in data. But this focus is misplaced.

    While model accuracy, hallucinations, and bias are significant, they aren’t the primary drivers of AI failure. The real risk lies in a much less glamorous, but far more consequential, area: governance.

    Think about it: A highly intelligent financial analyst is useless if their company has no accounting systems. Their insights would be lost in a sea of data, and their decisions could have disastrous consequences if not properly reviewed and audited. The same principle applies to AI.

    The Mirage of “AI Risk”

    Most companies view AI risk through a technological lens. They worry about:

    • Model accuracy: “Is our prediction model right often enough?”
    • Hallucinations: “Is the language model just making things up?”
    • Bias: “Does our AI perpetuate societal inequalities?”

    These are critical issues, and they absolutely require technical solutions. However, they are symptoms of a larger, more fundamental problem.

    The Real Risk: A Lack of Control

    The true danger of AI is not its lack of intelligence, but its potential for uncontrolled autonomy. The real risks that companies face are:

    • Uncontrolled automation: Giving AI the authority to make critical decisions without appropriate oversight.
    • No audit trail: The inability to trace the decision-making process of an AI system, making it impossible to understand how it reached a particular conclusion.
    • No human checkpoints: Failing to incorporate human judgment at key points in the AI lifecycle, allowing the system to operate on autopilot.
    • Unclear authority over model outputs: Failing to define who is accountable for the decisions made by an AI system.

    This is a failure of governance, not technology. It’s a failure of organizational processes and controls, not of algorithms.

    The Missing Layer

    To successfully deploy AI, companies must introduce a robust governance layer. This means going beyond simply building and deploying models, and focusing on:

    • Architecture: Designing AI systems with control, transparency, and auditability built-in. This involves defining clear interfaces, data pipelines, and decision points.
    • Oversight: Establishing clear processes and personnel responsible for monitoring and managing AI systems throughout their lifecycle. This includes continuous monitoring of performance, bias detection, and regular audits.
    • Accountability: Clearly defining roles and responsibilities for AI development, deployment, and performance. This means knowing who is responsible for the inputs, the outputs, and the consequences.

    The Core Concept

    Here’s the key takeaway:

    Powerful AI systems need governance the same way financial systems need accounting.

    Just as we wouldn’t trust a financial analyst without an accounting system, we shouldn’t trust an AI system without a robust governance framework. Governance provides the necessary checks and balances, the audit trail, and the accountability structure to ensure that AI is used effectively, ethically, and responsibly.

    Closing

    The next generation of AI infrastructure will not be defined by its intelligence alone. It will be defined by its ability to be governed, managed, and controlled. It will be characterized by its defensibility.

    The true differentiator for companies that succeed with AI will be their ability to build and implement robust governance frameworks. This is not just a regulatory or ethical imperative; it’s a fundamental business necessity.

    It’s time to shift the conversation. It’s time to stop worrying about the “intelligence” of AI and start focusing on the control. If you enjoy reading about AI from this perspective, like, follow and share. Leave a comment, I will be providing more on this topic.