Dhivya Nagasubramanian is VP of AI Transformation and Innovation at a significant U.S. monetary establishment, the place she leads the design, deployment, and governance of manufacturing agentic AI programs. She is the creator of Agentic AI for Engineers (Apress/Springer Nature), a sensible information to constructing autonomous AI programs that may be trusted in manufacturing. Since its launch, the e book has recorded greater than 6,000 institutional accesses on SpringerLink, holdings in over 260 libraries worldwide, and adoption throughout universities. She is a USPTO-granted patent holder in utilized machine studying. Her analysis pursuits embody constructing functions which can be immune to adversarial jailbreak assaults and contributing to broader trade efforts to develop higher fashions for multicultural security and safety. She is a sought-after skilled speaker and panelist throughout a number of trade and educational conferences.
Your profession has progressed from enterprise consulting and knowledge science into main AI transformation inside monetary companies. How did these experiences form Agentic AI for Engineers, and what recurring hole between AI analysis and enterprise deployment satisfied you that this e book wanted to be written?
I began in 2008 constructing portfolio accounting and efficiency measurement programs for banking platforms. A kind of initiatives was a GIPS-compliant engine for calculating time-weighted returns, which monetary establishments in additional than 80 nations ultimately used. That work taught me a lesson that has formed my complete profession. In regulated finance, probably the most harmful failure is a improper quantity that appears proper. A subtly incorrect calculation will get trusted, reported, and acted on for years, as a result of nothing seems damaged.
I noticed the identical sample once more later. I discovered a structural hole within the Markov mannequin of a broadly used advertising attribution mannequin. The framework was utilized by lots of of hundreds of customers, and the error had survived for years for one purpose: the outputs appeared cheap. Once I moved into AI transformation in monetary companies, I watched agentic programs repeat this sample with a lot greater stakes. An agent doesn’t simply produce a improper outcome. It takes motion primarily based on it.
The hole that satisfied me to jot down the e book is that analysis and enterprise deployment care about various things. Analysis measures functionality on benchmarks. Enterprises depend upon how a system behaves below ambiguity, altering knowledge, and adversarial stress. Most writing about brokers stops on the demo stage. I wrote Agentic AI for Engineers for the engineer who has to place their title on a system that can run with restricted supervision inside a regulated establishment.
The e book describes the shift from automation to autonomy as a brand new engineering mindset. What really separates an autonomous AI agent from a complicated automated workflow, and the place are organizations almost definitely to confuse the 2?
I give groups a easy check. Are you able to checklist each path the system can take earlier than you run it? If sure, you’ve gotten automation. It could be very subtle automation with a language mannequin inside it, however it’s nonetheless automation. If the system breaks down targets, picks instruments, and decides the order of actions at runtime primarily based on context you didn’t script, you’ve gotten autonomy. The distinction has nothing to do with which mannequin you employ. It comes right down to the place the choices are made.
Organizations confuse the 2 in each instructions, however just one course is dear. Calling a scripted LLM workflow agentic is generally a advertising drawback. The pricey mistake is the reverse. Groups deploy genuinely autonomous decision-making below governance that was designed for deterministic automation. Their testing assumes repeatability. Their monitoring assumes recognized failure patterns. Their audit path assumes there’s a traceable rule behind each motion. None of these assumptions maintain for an agent.
That’s the mindset shift I argue for within the e book. With automation, the principle query is whether or not the system executed appropriately. With autonomy, the query turns into whether or not the system determined appropriately. Answering that requires totally different instrumentation, totally different analysis strategies, and a unique degree of warning.
Many agentic AI programs carry out impressively in demonstrations however battle when uncovered to actual customers, altering knowledge, and unpredictable instruments. What parts ought to be thought of necessary in a production-ready agent structure?
The trustworthy reply is that manufacturing wants every little thing a demo permits you to skip. An illustration works as a result of somebody managed the setting. Manufacturing takes that management away.
Within the e book, I describe a seven-layer reference structure, however these are the parts I take into account necessary. Device contracts with typed inputs and specific permissions, so the agent’s attain is restricted by design quite than by hope. State administration that survives interruptions and failures. Structured failure dealing with with outlined escalation paths. An analysis harness that runs constantly, not as soon as earlier than launch. And an audit path that information selections and the reasoning behind them, not simply the actions taken.
The element groups skip most frequently is runtime verification. This can be a layer that checks each motion in opposition to the agent’s acknowledged goal and the insurance policies that apply to it. Normal infrastructure tells you whether or not a name succeeded. Verification asks whether or not the decision ought to have been made in any respect. It by no means proves its worth in a demo, as a result of nothing in a demo must be caught. In manufacturing, it’s the very first thing you’ll want you had.
Your e book covers security, alignment, suggestions loops, testing, debugging, analysis, and deployment. How ought to engineering groups mix these components right into a steady assurance course of quite than treating security as a closing pre-launch evaluate?
A closing evaluate rests on the idea that the system you reviewed is the system that can run. For agentic AI, that assumption fails in 3 ways. The underlying fashions get up to date. The instruments and APIs the agent will depend on change. And the information and setting the agent operates in maintain shifting. A one-time security evaluate describes a system that now not exists just a few months later.
Steady assurance means closing the loop. Behavioral evaluations run in CI (steady integration) the identical manner unit checks do, they usually gate each change to prompts, instruments, and fashions. Runtime monitoring feeds manufacturing traces again into the analysis units. I describe 4 monitoring patterns within the e book, as a result of no single sample covers the total failure floor. Each incident produces a brand new test, the identical manner each bug ought to produce a regression check. And adversarial testing runs on an everyday schedule as an alternative of as soon as earlier than launch.
The organizational aspect issues as a lot because the technical aspect. Security can’t be a separate group that exhibits up on the finish with a guidelines. The engineers who construct the agent ought to personal its analysis suites, its invariants, and its escalation design, as a result of they know higher than anybody the place the system’s judgment is weakest.
You have got emphasised semantic monitoring of agent instrument calls. What does semantic monitoring reveal that typical logs and observability instruments can’t, and the way can it detect when an agent is technically functioning however pursuing the improper goal?
Standard observability solutions one query: did the decision succeed? It covers standing codes, schemas, latency, and error charges. It can’t reply the query that issues most for brokers, which is whether or not the decision ought to have occurred. A instrument name may be technically excellent and nonetheless be the improper motion. Proper format, legitimate credentials, profitable response, improper factor to do.
Semantic monitoring evaluates each motion in opposition to the agent’s acknowledged goal and the insurance policies that apply to it. Within the e book, I describe this as defining semantic invariants over agent instrument calls. These are properties that should maintain it doesn’t matter what path the agent takes. An agent researching one buyer’s case ought to by no means entry information outdoors that case. An agent ought to flag a discrepancy above a sure threshold, by no means resolve it by itself. When an invariant is violated, you catch the issue in the mean time of motion as an alternative of in a quarterly audit.
That is additionally the way you detect purpose drift, which I take into account the defining failure of agentic programs. Purpose drift seems to be like an agent that completes each step appropriately in keeping with the logs whereas quietly working towards the improper goal. It by no means exhibits up in a log, as a result of nothing failed. Semantic monitoring treats intent as one thing you measure straight, and intent is the place brokers go improper.
When an agent encounters ambiguous directions, conflicting insurance policies, unavailable instruments, or inadequate confidence, what ought to structured failure dealing with appear to be? How can builders make sure the system pauses, escalates, or recovers as an alternative of improvising?
The issue to design in opposition to is that almost all brokers have just one technique to finish a activity, which is to provide a solution. When directions are ambiguous or insurance policies battle, the agent improvises. Not as a result of the mannequin is reckless, however as a result of improvising is the one habits the structure permits. Structured failure dealing with means giving the system higher choices and making them simpler to succeed in than a made-up reply.
In follow, which means refusal and escalation are designed and examined with the identical care as success. I like to recommend defining typed failure states: ambiguous instruction, conflicting coverage, unavailable instrument, inadequate confidence. Each maps to a particular habits. Ask a clarifying query. Hand off to a human with full context connected. Drop to learn solely operation. Retry inside tighter limits. Or cease. Confidence thresholds ought to match the danger of the precise motion. The bar for drafting an inside abstract and the bar for touching a buyer account ought to by no means be the identical quantity.
Two practices make this actual. First, design the escalation path earlier than the joyful path. If a human will obtain the handoff, determine what context they get and what authority they’ve earlier than you write the primary immediate. Second, check failure the best way you check options. Inject ambiguous directions. Disable instruments in staging. Feed the agent conflicting insurance policies. A system that has by no means practiced failing will act with out steering the primary time it occurs in manufacturing.
Monetary establishments should stability experimentation with strict necessities round privateness, auditability, mannequin threat, and regulatory compliance. The place ought to human approval stay necessary, and the place can brokers safely be given higher operational autonomy?
My rule is that human approval ought to keep necessary for any motion that’s irreversible, that strikes cash or impacts a affected person’s consequence, that entails buyer communications carrying regulatory weight, that feeds regulatory reporting, or that modifications the agent’s personal instruments, insurance policies, or permissions. Brokers can safely tackle extra autonomy in learn and purpose work: analysis, retrieval, doc evaluation, triage, drafting, and reconciliations that flag discrepancies for human evaluate as an alternative of resolving them.
The widespread mistake is requiring human approval at each step. That doesn’t make the system safer. It recreates the guide course of with further steps and trains reviewers to rubber stamp. Approvals belong at actual resolution factors. There ought to be few of them, and every ought to be staffed by somebody with precise authority to say no.
The opposite precept is that autonomy ought to be earned quite than granted. Begin slim. Develop because the system proves itself below monitoring. Maintain the proof. In mannequin threat phrases, your management setting ought to allow you to present a regulator not solely what the agent did, however why you had been justified in giving it that degree of independence. That report is the true license to function.
By way of your work on synthetic intelligence safeguards and requirements, you’re serving to outline guidelines for programs whose capabilities proceed to alter. Which safeguards can realistically be standardized right now, and which areas stay too depending on context, trade, or use case?
The safeguards we will standardize right now are structural. They require that sure controls exist, are documented, and may be inspected. Documented functionality and permission boundaries for autonomous programs. Outlined human oversight and escalation mechanisms. Incident detection and reporting obligations. Analysis strategies and proof necessities. Clear accountability throughout the system lifecycle, in order that when one thing goes improper, the reply to who’s accountable is rarely simply the AI. These may be standardized as a result of they concern whether or not the controls exist and the way good they’re, not the precise values they take.
What we can’t but standardize are the settings themselves. Particular confidence thresholds, hurt taxonomies, and applicable autonomy ranges rely closely on context. The appropriate failure tolerance for a advertising content material agent and for a system that touches scientific or monetary selections are usually not two factors on the identical scale. They belong to totally different regimes, formed by the area, the jurisdiction, and who bears the hurt when one thing fails.
Monetary controls are a helpful mannequin right here. Audit necessities are standardized globally, however materiality is at all times judged in context. Requirements that respect that break up are likely to get adopted. Requirements that attempt to dictate context-dependent values are likely to get ignored, and a security commonplace no one follows protects nobody.
Your latest analysis has explored multicultural and adversarial robustness in AI security analysis. What sorts of mannequin failures are prone to be missed by Western-centric or globally averaged benchmarks, and the way ought to enterprises consider programs supposed for multicultural deployment?
Wanting solely at averaged benchmarks can miss the failures that matter most. A mannequin can submit a robust total security rating whereas failing badly for a particular language, dialect, or cultural context, as a result of these failures disappear into the typical. Western-centric analysis provides one other blind spot. It misses harms which can be culturally particular, similar to idioms, gestures, and non secular or regional contexts the place the identical output is innocent in a single tradition and damaging in one other. It additionally under-tests code-switching, transliteration, and non-Western names and entities. There may be an adversarial angle as effectively. Attackers don’t goal your common efficiency. They discover your weakest slice, and for many fashions that slice is a lower-resource language or an under-evaluated cultural context.
That is what drew me into contributing to multicultural AI security benchmark analysis. The core discovering is simple. Judging security throughout cultures requires analysis knowledge and human judgment drawn from these cultures. You can’t assess hurt from outdoors the context by which it happens.
For enterprises, I recommend three guidelines. Consider each phase you serve, and by no means settle for a mean as proof. Construct analysis units out of your precise buyer populations. Run adversarial testing within the languages your prospects use. In the event you serve prospects in forty nations and consider solely in English, you’ve gotten measured your system for another person’s deployment.
You additionally maintain a granted United States Patent and Trademark Workplace patent in utilized AI. What drawback had been you making an attempt to resolve, what did the method educate you about turning AI analysis into sensible innovation, and which unresolved agentic AI problem are you most considering addressing subsequent?
The patent that was just lately authorised addresses a scale-and-consistency drawback in monetary companies high quality evaluate — typical name evaluate is guide, so human reviewers can solely pattern a fraction of calls, and their assessments fluctuate from individual to individual. My strategy routes every high quality and compliance query to the strategy finest suited to reply it, with each resolution logged and traceable for audit.
You possibly can see the identical themes I’ve described all through this interview. Route every activity to the least highly effective instrument that may do it effectively. Maintain people within the loop the place judgment issues. And make each automated resolution traceable, as a result of in a regulated establishment a solution with out proof just isn’t a solution.
What the patent course of taught me is that the self-discipline is within the specifics. You can’t patent a imprecise thought. It’s a must to present precisely what your technique does in a different way, and precisely the place it applies. That compelled precision made the underlying work higher.
The unsolved problem I care most about now could be runtime verification for multi-agent programs. Defining semantic invariants for a single agent is manageable. I do know, as a result of we do it in manufacturing. When brokers delegate work to different brokers, habits turns into emergent, and the failures transfer into the handoffs. An instruction will get barely reinterpreted at every step. A coverage applies to at least one agent however to not the agent it recruits. Verifying the interplay between brokers, not simply the actions of every one, is the place I count on the subsequent technology of silent failures to look. That’s the drawback I need to work on subsequent.
Thanks for the good interview, readers may need to order her e book Agentic AI for Engineers.
