For early-stage open-source initiatives, the “Getting began” information is usually the primary actual interplay a developer has with the venture. If a command fails, an output doesn’t match, or a step is unclear, most customers gained’t file a bug report, they may simply transfer on.
Drasi, a CNCF sandbox venture that detects modifications in your information and triggers speedy reactions, is supported by our small staff of 4 engineers in Microsoft Azure’s Workplace of the Chief Know-how Officer. We have now complete tutorials, however we’re transport code quicker than we will manually check them.
The staff didn’t notice how large this hole was till late 2025, when GitHub up to date its Dev Container infrastructure, bumping the minimal Docker model. The replace broke the Docker daemon connection, and each single tutorial stopped working. As a result of we relied on handbook testing, we didn’t instantly know the extent of the injury. Any developer attempting Drasi throughout that window would have hit a wall.
This incident pressured a realization: with superior AI coding assistants, documentation testing could be transformed to a monitoring downside.
The issue: Why does documentation break?
Documentation normally breaks for 2 causes:
1. The curse of information
Skilled builders write documentation with implicit context. After we write “anticipate the question to bootstrap,” we all know to run `drasi listing question` and look ahead to the `Working` standing, and even higher to run the `drasi wait` command. A brand new person has no such context. Neither does an AI agent. They learn the directions actually and don’t know what to do. They get caught on the “how,” whereas we solely doc the “what.”
2. Silent drift
Documentation doesn’t fail loudly like code does. Whenever you rename a configuration file in your codebase, the construct fails instantly. However when your documentation nonetheless references the outdated filename, nothing occurs. The drift accumulates silently till a person studies confusion.
That is compounded for tutorials like ours, which spin up sandbox environments with Docker, k3d, and pattern databases. When any upstream dependency modifications—a deprecated flag, a bumped model, or a brand new default—our tutorials can break silently.
The answer: Brokers as artificial customers
To unravel this, we handled tutorial testing as a simulation downside. We constructed an AI agent that acts as a “artificial new person.”
This agent has three vital traits:
- It’s naïve: It has no prior information of Drasi—it is aware of solely what’s explicitly written within the tutorial.
- It’s literal: It executes each command precisely as written. If a step is lacking, it fails.
- It’s unforgiving: It verifies each anticipated output. If the doc says, “It is best to see ‘Success’”, and the command line interface (CLI) simply returns silently, the agent flags it and fails quick.
The stack: GitHub Copilot CLI and Dev Containers
We constructed an answer utilizing GitHub Actions, Dev Containers, Playwright, and the GitHub Copilot CLI.
Our tutorials require heavy infrastructure:
- A full Kubernetes cluster (k3d)
- Docker-in-Docker
- Actual databases (akin to PostgreSQL and MySQL)
We would have liked an atmosphere that precisely matches what our human customers expertise. If customers run in a particular Dev Container on GitHub Codespaces, our check should run in that identical Dev Container.
The structure
Contained in the container, we invoke the Copilot CLI with a specialised system immediate (view the complete immediate right here):
This immediate utilizing the immediate mode (-p) of the CLI agent offers us an agent that may execute terminal instructions, write recordsdata, and run browser scripts—identical to a human developer sitting at their terminal. For the agent to simulate an actual person, it wants these capabilities.
To allow the brokers to open webpages and work together with them as any human following the tutorial steps would, we additionally set up Playwright on the Dev Container. The agent additionally takes screenshots which it then compares in opposition to these supplied within the documentation.
Safety mannequin
Our safety mannequin is constructed round one precept: the container is the boundary.
Slightly than attempting to limit particular person instructions (a shedding recreation when the agent must run arbitrary node scripts for Playwright), we deal with all the Dev Container as an remoted sandbox and management what crosses its boundaries: no outbound community entry past localhost, a Private Entry Token (PAT) with solely “Copilot Requests” permission, ephemeral containers destroyed after every run, and a maintainer-approval gate for triggering workflows.
Coping with non-determinism
One of many greatest challenges with AI-based testing is non-determinism. Giant language fashions (LLMs) are probabilistic—generally the agent retries a command; different occasions it offers up.
We dealt with this with a three-stage retry with mannequin escalation (begin with Gemini-Professional, on failure attempt with Claude Opus), semantic comparability for screenshots as a substitute of pixel-matching, and verification of core-data fields relatively than risky values.
We even have a listing of tight constraints in our prompts that forestall the agent from happening a debugging journey, directives to manage the construction of the ultimate report, and likewise skip directives that inform the agent to bypass optionally available tutorial sections like establishing exterior companies.
Artifacts for debugging
When a run fails, we have to know why. For the reason that agent is working in a transient container, we will’t simply Safe Shell (SSH) in and go searching.
So, our agent preserves proof of each run, screenshots of internet UIs, terminal output of vital instructions, and a remaining markdown report detailing its reasoning like proven right here:
These artifacts are uploaded to the GitHub Motion run abstract, permitting us to “time journey” again to the precise second of failure and see what the agent noticed.
Parsing the agent’s report
With LLMs, getting a definitive “Go/Fail” sign {that a} machine can perceive could be difficult. An agent may write a protracted, nuanced conclusion like:
To make this actionable in a CI/CD pipeline, we needed to do some immediate engineering. We explicitly instructed the agent:
In our GitHub Motion, we then merely grep for this particular string to set the exit code of the workflow.
Easy methods like this bridge the hole between AI’s fuzzy, probabilistic outputs and CI’s binary go/fail expectations.
Automation
We now have an automated model of the workflow which runs weekly. This model evaluates all our tutorials each week in parallel—every tutorial will get its personal sandbox container and a recent perspective from the agent appearing as an artificial person. If any of the tutorial analysis fails, the workflow is configured to file a difficulty on our GitHub repo.
This workflow can optionally even be run on pull-requests, however to forestall assaults we now have added a maintainer-approval requirement and a `pull_request_target` set off, which signifies that even on pull-requests by exterior contributors, the workflow that executes would be the one in our essential department.
Working the Copilot CLI requires a PAT token which is saved within the atmosphere secrets and techniques for our repo. To verify this doesn’t leak, every run requires maintainer approval—besides the automated weekly run which solely runs on the `essential` department of our repo.
What we discovered: Bugs that matter
Since implementing this method, we now have run over 200 “artificial person” classes. The agent recognized 18 distinct points together with some critical atmosphere points and different documentation points like these. Fixing them improved the docs for everybody, not simply the bot.
- Implicit dependencies: In a single tutorial, we instructed customers to create a tunnel to a service. The agent ran the command, after which—following the subsequent instruction—killed the method to run the subsequent command.
The repair: We realized we hadn’t instructed the person to maintain that terminal open. We added a warning: “This command blocks. Open a brand new terminal for subsequent steps.” - Lacking verification steps: We wrote: “Confirm the question is working.” The agent received caught: “How, precisely?”
The repair: We changed the imprecise instruction with an specific command: `drasi wait -f question.yaml`. - Format drift: Our CLI output had advanced. New columns have been added; older fields have been deprecated. The documentation screenshots nonetheless confirmed the 2024 model of the interface. A human tester may gloss over this (“it appears largely proper”). The agent flagged each mismatch, forcing us to maintain our examples updated.
AI as a power multiplier
We frequently hear about AI changing people, however on this case, the AI is offering us with a workforce we by no means had.
To duplicate what our system does—working six tutorials throughout recent environments each week—we would wish a devoted QA useful resource or a big funds for handbook testing. For a four-person staff, that’s not possible. By deploying these Artificial Customers, we now have successfully employed a tireless QA engineer who works nights, weekends, and holidays.
Our tutorials are actually validated weekly by artificial customers. attempt the Getting Began information your self and see the outcomes firsthand. And if you happen to’re dealing with the identical documentation drift in your individual venture, take into account GitHub Copilot CLI not simply as a coding assistant, however as an agent—give it a immediate, a container, and a objective—and let it do the work a human doesn’t have time for.
