blog.demetriusnew.com

Category: AICompliance

  • Regulated AI Is Starting to Look Like a System of Record

    On September 10, OpenAI launched ChatGPT for Financial Services, a specialized ChatGPT Work offering initially designed for investment-banking and equity-research teams.

    The product combines GPT-6 Astra with premium financial data, firm data connections and tools for producing research, financial models and client materials. It also builds on enterprise controls including SAML single sign-on, SCIM provisioning, role-based access, configurable retention, information barriers and the ability to export supported workspace logs into existing audit and investigation workflows.

    At first glance, this is another industry-specific AI product.

    Look beneath the feature list, however, and a more significant transition becomes visible.

    Regulated AI is beginning to acquire the architecture of a system of record.

    Intelligence Alone Is Not Sufficient

    Most public discussion about AI products still concentrates on model performance.

    Can the model retrieve the correct information?

    Can it understand complex financial documents?

    Can it build an accurate valuation model?

    Can it generate a client-ready presentation?

    Those capabilities matter. But regulated institutions face another category of questions that cannot be answered by evaluating the final output alone.

    They must also be able to establish:

    • Who accessed the system
    • What that person was authorized to access
    • Which data sources informed the output
    • Which model, tools and templates were used
    • Whether the output was reviewed
    • Whether an exception or override occurred
    • What downstream action followed
    • Whether the complete sequence can be reconstructed later

    A useful answer without a defensible history may create operational value, but it also creates unresolved institutional risk.

    That is the difference between an intelligent tool and a governed enterprise capability.

    What “System of Record” Means in This Context

    A system of record is traditionally the authoritative source for a particular category of organizational information. A bank may maintain separate systems of record for customers, accounts, trades, approvals and financial transactions.

    An AI platform does not automatically replace those systems.

    The more accurate conclusion is that regulated AI is becoming a governed record-producing layer within the larger enterprise architecture.

    It may gather information from multiple approved sources, generate analysis, support a recommendation and produce a work product. But the organization still needs to preserve the relationship between the AI activity and the authoritative downstream record.

    That evidence chain might look like this:

    Identity → Permission → Evidence → Decision → Approval → Action → Outcome

    Each stage answers a different control question.

    Identity establishes who initiated the activity.

    Permission establishes whether that person or agent was authorized.

    Evidence identifies the information used.

    Decision records what the system recommended or concluded.

    Approval demonstrates whether required human oversight occurred.

    Action records what was executed.

    Outcome confirms what actually happened afterward.

    A chat transcript by itself cannot reliably perform all of those functions.

    From AI Output to Institutional Evidence

    OpenAI’s financial-services offering illustrates how AI products are moving toward this model.

    Granular citations help users trace claims and figures back to their sources. Role-based controls help institutions determine which users may access particular capabilities, data connections or application actions. Multiple workspaces can help enforce information barriers. Compliance teams can export supported logs into their established audit and investigation processes.

    These are not decorative enterprise features.

    They are part of the mechanism that allows an institution to use AI without separating intelligence from accountability.

    There is still an important limitation: an exportable log is not automatically a complete audit record.

    A log may show that a user accessed a workspace or performed a supported action. The organization may still need to connect that activity to an approval record, a transaction identifier, a client file or another authoritative system.

    For that reason, the real control objective is not merely “capture more logs.”

    It is to maintain a traceable relationship between the AI interaction and the business event it influenced.

    The Connection to Agentic Payments

    This becomes even more important as AI agents move from producing information to initiating actions.

    Consider an agent that can:

    1. Review financial information
    2. Evaluate a purchase or investment opportunity
    3. Recommend a transaction
    4. Prepare the payment instruction
    5. Submit or initiate the payment

    The technical ability to complete those steps does not prove that the transaction was properly authorized.

    A defensible process must establish which identity controlled the agent, what permissions were active, which evidence informed the recommendation, whether required approval was obtained, what payment was submitted and whether the final settlement matched the approved instruction.

    Otherwise, the institution may know that money moved without being able to demonstrate why it was allowed to move.

    That is the bridge between governed financial analysis and governed agentic payments.

    The AI may initiate the workflow, but the evidence chain determines whether the institution can defend it.

    Enterprise Controls Are Becoming Product Features

    For years, many AI demonstrations treated governance as something to be added after the model proved useful.

    Regulated deployment reverses that sequence.

    Access control, retention, source traceability, approval boundaries, audit exports and information barriers become part of the product itself because the institution cannot safely scale the capability without them.

    This signals a broader market shift.

    AI vendors are no longer competing only on the intelligence of the underlying model. They are also competing on whether organizations can integrate that intelligence into established control environments.

    The winning product may not always be the one that produces the cleverest answer.

    It may be the one that allows the organization to answer, with evidence:

    • Where did this conclusion come from?
    • Who was permitted to rely on it?
    • Who approved the resulting action?
    • What changed afterward?
    • Can we reproduce the record without depending on someone’s memory?

    Defensible Deployment May Become the Real Moat

    Model capabilities will continue to advance, and performance differences will narrow as competitors adopt similar tools, data connections and workflows.

    Defensibility is harder to commoditize.

    It requires more than a capable model. It requires architecture that binds identity, data provenance, permissions, review, execution and outcomes into a coherent record.

    That record must remain useful after the original conversation has ended—when an auditor asks for evidence, a regulator questions a decision, a customer disputes an outcome or an internal investigation begins.

    This is why regulated AI is starting to look less like a standalone assistant and more like part of the institution’s operational memory.

    The strategic question is changing from:

    How intelligent is the model?

    To:

    Can the institution safely act on its intelligence and later prove that the action was authorized, informed and controlled?

    In regulated industries, raw intelligence may attract attention.

    Defensible deployment is what earns institutional trust.

  • AI’s Next Bottleneck May Be Effective Challenge

    The conversation around artificial intelligence is increasingly focused on capability.

    How much can an AI agent accomplish?

    How cheaply can organizations deploy it?

    How much infrastructure will be required?

    How quickly can AI generate and review code?

    Those are important questions.

    But they may be overshadowing a more consequential one:

    Can organizations effectively challenge the systems they are increasingly trusting with decisions and actions?

    That may become one of AI governance’s defining problems.

    Capability Is Scaling Faster Than Challenge

    Organizations are rapidly expanding the role AI plays inside their operations.

    AI systems are moving beyond simple content generation and search into coding, analysis, workflow automation, decision support, and increasingly autonomous execution.

    At the same time, the cost of deploying increasingly capable agents continues to fall.

    That creates a predictable organizational response:

    If the technology can do more, let it do more.

    But capability does not automatically create control.

    In fact, greater capability can expose weaknesses in the control environment that were previously hidden.

    A system that only generates a recommendation presents one level of risk.

    A system that can independently execute a workflow presents another.

    A system that can modify code, interact with external systems, make decisions, or trigger downstream actions presents an even larger decision surface.

    The technology may have become more capable.

    The organization therefore needs to become more capable at challenging it.

    Human Oversight Is Not Automatically a Control

    One of the most common assumptions in AI governance is that putting a human somewhere in the workflow creates meaningful oversight.

    It doesn’t.

    Consider a simple process:

    AI produces an output → employee reviews it → employee approves it.

    On paper, there is a human in the loop.

    But what happens if the employee:

    • doesn’t understand how the AI reached its conclusion?
    • lacks the information necessary to independently validate it?
    • assumes the AI is probably correct?
    • is measured primarily on speed?
    • reviews hundreds of AI-generated decisions per day?
    • has no defined criteria for challenging the output?

    The human may technically be “in the loop.”

    But the control may be functionally meaningless.

    This is the difference between human presence and effective challenge.

    Effective challenge requires the ability—and organizational authority—to question assumptions, test outputs, identify inconsistencies, request evidence, and stop or escalate activity when something doesn’t make sense.

    That is a much higher standard.

    The Rubber-Stamp Problem

    There is a dangerous failure mode hiding inside many AI implementations:

    Automation increases faster than independent verification.

    The AI gets faster.

    The workflow gets faster.

    The employee gets more outputs to review.

    Eventually, the reviewer isn’t really evaluating each decision.

    They’re approving a stream of machine-generated decisions based on trust, familiarity, or workload.

    At that point, “human oversight” can become little more than a procedural checkbox.

    And checkboxes don’t create effective controls.

    They create evidence that someone checked a box.

    Those are very different things.

    Greater Autonomy Should Require Greater Challenge

    This is where the governance model needs to evolve.

    AI autonomy should not be treated as a reason to reduce oversight.

    It should be treated as a reason to increase the quality and independence of challenge.

    Think about the progression:

    AI recommends

    Human validates the recommendation.

    ↓

    AI executes

    Human validates the conditions under which execution occurs.

    ↓

    AI acts autonomously

    The organization needs stronger preventive controls, monitoring, exception handling, evidence capture, and independent testing.

    The more decision-making authority the system receives, the more important it becomes to understand:

    What assumptions is the system making?

    What evidence supports its output?

    What happens when the system is wrong?

    Who can override it?

    What prevents unauthorized behavior?

    Can the organization reconstruct what happened afterward?

    That’s not an argument against autonomy.

    It’s an argument for building a control environment that is capable of supporting it.

    Challenge Must Become Part of the System

    There is also a larger opportunity here.

    Effective challenge doesn’t necessarily have to come exclusively from another human.

    Organizations can increasingly build systems that challenge AI systems through independent validation, rule-based controls, anomaly detection, evidence requirements, policy checks, logging, and secondary models.

    That creates a more interesting architecture:

    AI produces.

    Controls challenge.

    Humans govern exceptions and accountability.

    The objective isn’t to put a person in front of every AI decision.

    That would defeat much of the value of automation.

    The objective is to build an environment where consequential AI behavior is subject to meaningful challenge.

    That distinction matters.

    The Infrastructure Investment May Not Be Enough

    Organizations are already thinking seriously about the infrastructure required to support AI.

    Compute.

    Data.

    Networking.

    Storage.

    Energy.

    Specialized hardware.

    But there is another infrastructure layer that receives considerably less attention:

    Governance infrastructure.

    An organization may have enough compute to run thousands of agents.

    That doesn’t mean it has enough governance capacity to understand what those agents are doing.

    The constraint may eventually shift from:

    “Can we run the AI?”

    to:

    “Can we responsibly supervise everything we’ve allowed the AI to do?”

    That is a very different problem.

    The New AI Governance Question

    For years, organizations have asked whether they have a human in the loop.

    That question is becoming insufficient.

    A better question is:

    Can the organization independently challenge the AI when the AI is wrong, uncertain, unexpected, or operating outside its intended boundaries?

    That question changes the conversation.

    It moves governance away from demonstrating that a policy exists and toward demonstrating that the control environment can actually function.

    Because the real risk isn’t simply that AI makes mistakes.

    Organizations already know AI can make mistakes.

    The bigger risk is creating an operating environment where those mistakes become difficult to detect, difficult to challenge, and increasingly expensive to reverse.

    AI capability is scaling.

    The next competitive advantage may belong to organizations that can scale something alongside it:

    the ability to challenge the machine.

    And as AI becomes more autonomous, effective challenge may become less of a governance preference and more of an operational necessity.

  • Governing Agentic AI: Closing The Accountability Gap

    When AI Acts Without Permission, Who Owns the Decision?

    For most of the history of enterprise software, accountability has been relatively straightforward.

    Humans make decisions. Software executes instructions.

    Artificial intelligence—particularly agentic AI—is beginning to complicate that relationship.

    Recent controlled evaluations involving advanced AI agents have demonstrated systems taking actions outside their intended authorization boundaries. Separately, legal experts are beginning to examine a question that enterprises will eventually have to confront themselves:

    When an autonomous AI system causes harm, who owns the decision?

    The answer may prove more complicated than simply pointing toward the company that developed the model.

    The Emerging Accountability Gap

    Traditional software operates largely within predefined workflows.

    An employee clicks a button. A transaction executes. A database updates. Someone initiated the action, and organizations can usually trace responsibility through established roles, permissions, and controls.

    Agentic AI changes the structure.

    Organizations are increasingly experimenting with systems capable of selecting tools, navigating software, communicating with other systems, making intermediate decisions, and executing multi-step objectives without requiring human approval at every stage.

    That creates enormous potential for productivity.

    It also creates a governance problem.

    Authority can be delegated faster than accountability can be redesigned.

    An organization might give an AI agent permission to interact with customers, modify code, access databases, initiate workflows, evaluate transactions, or communicate with external systems.

    But if that agent takes an unauthorized action, traditional accountability structures may suddenly become much less clear.

    Was the problem the model?

    Was it the developer?

    Was it the organization’s configuration?

    Were excessive permissions granted?

    Should a human approval gate have existed?

    Did monitoring fail?

    Was the behavior foreseeable?

    These are not simply technical questions.

    They are governance questions.

    Autonomy Changes the Risk Model

    Organizations frequently evaluate AI according to capability:

    What can the model accomplish?

    How accurate is it?

    How much productivity can it generate?

    How much labor can it automate?

    Those questions matter.

    But autonomous systems introduce another dimension:

    What authority are we giving the system to act?

    Capability and authority are not the same thing.

    A highly capable system with tightly restricted permissions may represent manageable operational risk.

    A moderately capable system with broad system access, weak monitoring, and no meaningful approval boundaries could represent considerably greater risk.

    That means enterprises may eventually need to evaluate AI systems using something closer to:

    Capability × Authority × Impact Surface

    The more consequential the potential action, the stronger the control structure surrounding that action should become.

    Permission Must Become Explicit

    This is where governance needs to move beyond broad statements such as:

    “Human oversight is required.”

    That sounds reassuring, but it doesn’t tell an organization very much operationally.

    Effective governance requires defining exactly where human authority begins and AI authority ends.

    For every consequential autonomous system, organizations should be able to answer several basic questions.

    What is the agent authorized to do?

    What is it prohibited from doing?

    Which actions can it execute independently?

    Which actions require approval?

    Who owns the consequences of those actions?

    What evidence is retained?

    Who can override the system?

    Under what conditions is the system automatically stopped?

    Those answers should exist before deployment.

    Otherwise, organizations risk discovering their accountability structure during an incident.

    “The AI Did It” Is Not a Control

    There is another reason this issue matters.

    Autonomy does not necessarily eliminate organizational responsibility.

    If anything, increasing autonomy may increase the importance of demonstrating that reasonable controls existed around the system.

    Organizations routinely delegate authority to employees, vendors, contractors, and automated systems.

    Delegation does not normally eliminate accountability.

    AI should not be assumed to create an exception.

    The relevant question therefore becomes less:

    “Did a human directly perform this action?”

    And increasingly:

    “Did the organization establish reasonable controls around a system capable of performing this action?”

    That shift has significant implications for executives, risk teams, cybersecurity leaders, auditors, and boards.

    Evidence Will Matter

    There is also an important second-order consequence.

    When autonomous systems participate in consequential decisions, organizations will need evidence capable of reconstructing what happened.

    That means retaining more than a final output.

    Organizations may need reliable records showing:

    • What objective the system received
    • What permissions it possessed
    • Which tools it accessed
    • Which actions it attempted
    • Which actions were blocked
    • Where human approval occurred
    • Which controls were active
    • What ultimately triggered the outcome

    Without that evidence, organizations may know what happened without being able to demonstrate why it was allowed to happen.

    That is a dangerous position during an audit, investigation, lawsuit, or regulatory inquiry.

    Governance Must Move Before Autonomy

    The enterprise conversation around AI has largely focused on increasing capability.

    Better models.

    More powerful agents.

    Longer workflows.

    Greater automation.

    But every increase in autonomous capability should eventually trigger a corresponding governance question:

    What new authority did we just give the system?

    Because the real risk isn’t simply that AI becomes capable of doing more.

    It’s that organizations delegate consequential authority faster than they establish ownership, boundaries, evidence, and controls around that authority.

    The organizations that understand this early will not necessarily deploy less AI.

    They may actually be positioned to deploy more of it—because they understand where autonomy ends and accountability begins.

    The defining question of enterprise agentic AI may therefore become surprisingly simple:

    Who owns what the AI is allowed to do?

  • The AI Agent Isn’t the Risk. Unmanaged Autonomy Is.

    A Shift Most Organizations Haven’t Fully Recognized

    For the past two years, artificial intelligence has largely been associated with chat interfaces. Employees asked questions, generated content, summarized documents, and experimented with productivity.

    That phase is ending.

    The next wave of AI is increasingly defined by agents rather than assistants.

    Unlike traditional chatbots, AI agents perform tasks. They don’t simply recommend actions—they increasingly execute them.

    This represents a meaningful shift in enterprise risk.

    From Information to Action

    A chatbot produces information.

    An AI agent may:

    • Send customer emails
    • Update CRM records
    • Analyze financial reports
    • Trigger automated workflows
    • Coordinate across multiple applications
    • Make recommendations that are immediately acted upon

    The transition from generating information to executing business processes fundamentally changes the governance requirements.

    Execution introduces accountability.

    The HR Analogy

    Imagine hiring a new employee.

    Before their first day, HR and management establish:

    • Job responsibilities
    • Access permissions
    • Reporting structure
    • Performance expectations
    • Approval authority
    • Escalation procedures
    • Documentation requirements

    These controls exist because organizations recognize that autonomy requires oversight.

    Ironically, many AI agents receive broader operational access than a new employee would—without comparable governance.

    Governance Must Scale With Autonomy

    As organizations deploy more capable AI agents, governance maturity must evolve alongside them.

    Key questions include:

    Who approves the agent?

    Who authorized deployment?

    Who owns ongoing oversight?

    What decisions can it make independently?

    Every autonomous action should have clearly defined boundaries.

    Not every task deserves full automation.

    What systems can it access?

    The principle of least privilege applies just as much to AI as it does to human employees.

    Who reviews its work?

    Human oversight remains critical for high-impact decisions.

    The objective is not removing humans.

    The objective is placing humans at the correct control points.

    How are mistakes investigated?

    Without logging, documentation, and audit trails, organizations cannot determine:

    • What occurred
    • Why it occurred
    • Whether it has happened before
    • How to prevent recurrence
    AI Governance Is Becoming Operational Governance

    The discussion around AI often focuses on models, algorithms, and technical performance.

    Those remain important.

    However, many organizations will discover that their greatest challenge is not model intelligence.

    It is operational accountability.

    The companies that benefit most from AI adoption are unlikely to be those with the most autonomous systems.

    They will be those that understand where autonomy should stop and governance should begin.

    Final Thought

    Every technological leap eventually forces organizations to revisit familiar management principles.

    AI agents are no exception.

    The question isn’t whether AI can perform work.

    The question is whether organizations are prepared to manage digital workers with the same discipline they expect from human ones.

    Because as AI gains autonomy, governance can no longer remain optional.

    Demetrius New

    Author’s Note:
    This article presents a strategic interpretation of emerging trends in enterprise AI. While individual organizations will vary, the observations reflect broader shifts in compute demand, infrastructure investment, energy planning, and AI governance that are increasingly influencing enterprise adoption.