{"id":25365,"date":"2026-04-15T03:57:25","date_gmt":"2026-04-14T18:57:25","guid":{"rendered":"https:\/\/aireviewirush.com\/?p=25365"},"modified":"2026-04-15T03:57:27","modified_gmt":"2026-04-14T18:57:27","slug":"how-drasi-used-github-copilot-to-search-out-documentation-bugs","status":"publish","type":"post","link":"https:\/\/aireviewirush.com\/?p=25365","title":{"rendered":"How Drasi used GitHub Copilot to search out documentation bugs"},"content":{"rendered":"<p> <br \/>\n<\/p>\n<div>\n<p>For early-stage open-source initiatives, the \u201cGetting began\u201d information is usually the primary actual interplay a developer has with the venture. If a command fails, an output doesn\u2019t match, or a step is unclear, most customers gained\u2019t file a bug report, they may simply transfer on.<\/p>\n<p><a href=\"https:\/\/drasi.io\/\" target=\"_blank\" rel=\"noopener\">Drasi<\/a>, a CNCF sandbox venture that detects modifications in your information and triggers speedy reactions, is supported by our small staff of 4 engineers in Microsoft Azure\u2019s Workplace of the Chief Know-how Officer. We have now complete tutorials, however we&#8217;re transport code quicker than we will manually check them.<\/p>\n<p>The staff <a id=\"post-98436-OLE_LINK33\"\/>didn\u2019t notice how large this hole was till late 2025, when GitHub up to date its Dev Container infrastructure, bumping the minimal Docker model. The replace broke the Docker daemon connection, and each single tutorial stopped working. As a result of we relied on handbook testing, we didn\u2019t instantly know the extent of the injury. Any developer attempting Drasi throughout that window would have hit a wall.<a id=\"post-98436-OLE_LINK10\"\/><a id=\"post-98436-OLE_LINK11\"\/><a id=\"post-98436-OLE_LINK34\"\/><\/p>\n<p>This incident pressured a realization: <strong>with superior AI coding assistants, documentation testing could be transformed to a monitoring downside<\/strong>.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_53 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title \" >Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\" role=\"button\"><label for=\"item-6a6d9ff3314ca\" ><span class=\"\"><span style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input aria-label=\"Toggle\" aria-label=\"item-6a6d9ff3314ca\"  type=\"checkbox\" id=\"item-6a6d9ff3314ca\"><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#The_issue_Why_does_documentation_break\" title=\"The issue: Why does documentation break?\">The issue: Why does documentation break?<\/a><ul class='ez-toc-list-level-3'><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#1_The_curse_of_information\" title=\"1. The curse of information\">1. The curse of information<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#2_Silent_drift\" title=\"2. Silent drift\">2. Silent drift<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#The_answer_Brokers_as_artificial_customers\" title=\"The answer: Brokers as artificial customers\">The answer: Brokers as artificial customers<\/a><ul class='ez-toc-list-level-3'><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#The_stack_GitHub_Copilot_CLI_and_Dev_Containers\" title=\"The stack: GitHub Copilot CLI and Dev Containers\">The stack: GitHub Copilot CLI and Dev Containers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#The_structure\" title=\"The structure\">The structure<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#Safety_mannequin\" title=\"Safety mannequin\">Safety mannequin<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#Coping_with_non-determinism\" title=\"Coping with non-determinism\">Coping with non-determinism<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#Artifacts_for_debugging\" title=\"Artifacts for debugging\">Artifacts for debugging<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#Parsing_the_agent%E2%80%99s_report\" title=\"Parsing the agent\u2019s report\">Parsing the agent\u2019s report<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#Automation\" title=\"Automation\">Automation<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#What_we_discovered_Bugs_that_matter\" title=\"What we discovered: Bugs that matter\">What we discovered: Bugs that matter<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/aireviewirush.com\/?p=25365\/#AI_as_a_power_multiplier\" title=\"AI as a power multiplier\">AI as a power multiplier<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\" id=\"the-problem-why-does-documentation-break\"><span class=\"ez-toc-section\" id=\"The_issue_Why_does_documentation_break\"><\/span><a id=\"post-98436-OLE_LINK12\"\/><a id=\"post-98436-OLE_LINK14\"\/>The issue: Why does documentation break?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>\n  Documentation normally breaks for 2 causes:\n<\/p>\n<h3 class=\"wp-block-heading\" id=\"1-the-curse-of-knowledge\"><span class=\"ez-toc-section\" id=\"1_The_curse_of_information\"><\/span>1. The curse of information<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Skilled builders write documentation with implicit context. After we write \u201canticipate the question to bootstrap,\u201d we all know to run `drasi listing question` and look ahead to the `Working` standing, and even higher to run the `drasi wait` command. A brand new person has no such context. Neither does an AI agent. They learn the directions actually and don\u2019t know what to do. They get caught on the \u201chow,\u201d whereas we solely doc the \u201cwhat.\u201d <\/p>\n<h3 class=\"wp-block-heading\" id=\"2-silent-drift\"><span class=\"ez-toc-section\" id=\"2_Silent_drift\"><\/span><a id=\"post-98436-OLE_LINK13\"\/>2. Silent drift<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>\n  Documentation doesn\u2019t fail loudly like code does. Whenever you rename a configuration file in your codebase, the construct fails instantly. However when your documentation nonetheless references the outdated filename, nothing occurs. The drift accumulates silently till a person studies confusion.\n<\/p>\n<p>\n  That is compounded for tutorials like ours, which spin up sandbox environments with Docker, k3d, and pattern databases. When any upstream dependency modifications\u2014a deprecated flag, a bumped model, or a brand new default\u2014our tutorials can break silently.\n<\/p>\n<h2 class=\"wp-block-heading\" id=\"the-solution-agents-as-synthetic-users\"><span class=\"ez-toc-section\" id=\"The_answer_Brokers_as_artificial_customers\"><\/span><a id=\"post-98436-OLE_LINK15\"\/>The answer: Brokers as artificial customers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To unravel this, we handled tutorial testing as a simulation downside. We constructed an AI agent that acts as a \u201cartificial new person.\u201d<\/p>\n<p>\n  This agent has three vital traits:\n<\/p>\n<ol class=\"wp-block-list\">\n<li><strong>It&#8217;s na\u00efve<\/strong>: It has no prior information of Drasi\u2014it is aware of solely what&#8217;s explicitly written within the tutorial.<\/li>\n<li><strong>It&#8217;s literal<\/strong>: It executes each command precisely as written. If a step is lacking, it fails.<\/li>\n<li><strong>It&#8217;s unforgiving<\/strong>: It verifies each anticipated output. If the doc says, \u201cIt is best to see \u2018Success\u2019\u201d, and the command line interface (CLI) simply returns silently, the agent flags it and fails quick.<\/li>\n<\/ol>\n<h3 class=\"wp-block-heading\" id=\"the-stack-github-copilot-cli-and-dev-containers\"><span class=\"ez-toc-section\" id=\"The_stack_GitHub_Copilot_CLI_and_Dev_Containers\"><\/span><a id=\"post-98436-OLE_LINK16\"\/>The stack: GitHub Copilot CLI and Dev Containers<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>We constructed an answer utilizing GitHub Actions, Dev Containers, Playwright, and the GitHub Copilot CLI. <\/p>\n<p>\n  Our tutorials require heavy infrastructure:\n<\/p>\n<ul class=\"wp-block-list\">\n<li>A full Kubernetes cluster (k3d)  <\/li>\n<li>Docker-in-Docker  <\/li>\n<li>Actual databases (akin to PostgreSQL and MySQL)  <\/li>\n<\/ul>\n<p>We would have liked an atmosphere that precisely matches what our human customers expertise. If customers run in a particular Dev Container on GitHub Codespaces, our check should run in that <strong>identical<\/strong> Dev Container.<\/p>\n<h3 class=\"wp-block-heading\" id=\"the-architecture\"><span class=\"ez-toc-section\" id=\"The_structure\"><\/span><a id=\"post-98436-OLE_LINK17\"\/>The structure<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>\n  Contained in the container, we invoke the Copilot CLI with a specialised system immediate (<a href=\"https:\/\/github.com\/drasi-project\/learning\/blame\/main\/.github\/prompts\/tutorial-evaluation.md\" target=\"_blank\" rel=\"noopener\">view the complete immediate right here<\/a>):\n<\/p>\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69de8e1583978&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image wp-lightbox-container\"><img decoding=\"async\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/a-screen-shot-of-a-computer-terminal-bash-co.webp\" alt=\"A screen shot of a computer terminal:&#10;&#10;bash &#10; &#10;&#10;copilot -p &quot;$(cat prompt.md)&quot;  &#10;  --allow-all-tools  &#10;  --allow-all-paths  &#10;  --deny-tool 'fetch'  &#10;  --deny-tool 'websearch'  &#10;  --deny-tool 'githubRepo'  &#10;  --deny-tool 'shell(curl *)'  &#10; &#10;&#10;    # ... additional deny-tool flags &#10; &#10;  --allow-url localhost  &#10;  --allow-url 127.0.0.1 \" class=\"wp-image-98438 webp-format\" srcset=\"\" data-orig-src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/a-screen-shot-of-a-computer-terminal-bash-co.webp\"\/><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\"><br \/>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"\/>\n\t\t\t<\/svg><br \/>\n\t\t<\/button><\/figure>\n<p>This immediate utilizing the immediate mode (-p) of the CLI agent offers us an agent that may execute terminal instructions, write recordsdata, and run browser scripts\u2014identical to a human developer sitting at their terminal. For the agent to simulate an actual person, it wants these capabilities.<\/p>\n<p>To allow the brokers to open webpages and work together with them as any human following the tutorial steps would, we additionally set up Playwright on the Dev Container. The agent additionally takes screenshots which it then compares in opposition to these supplied within the documentation. <\/p>\n<h3 class=\"wp-block-heading\" id=\"security-model\"><span class=\"ez-toc-section\" id=\"Safety_mannequin\"><\/span>Safety mannequin<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Our safety mannequin is constructed round one precept: <strong>the container is the boundary<\/strong>.<\/p>\n<p>Slightly than attempting to limit particular person instructions (a shedding recreation when the agent must run arbitrary node scripts for Playwright), we deal with all the Dev Container as an remoted sandbox and management what crosses its boundaries: no outbound community entry past localhost, a Private Entry Token (PAT) with solely \u201cCopilot Requests\u201d permission, ephemeral containers destroyed after every run, and a maintainer-approval gate for triggering workflows. <\/p>\n<h3 class=\"wp-block-heading\" id=\"dealing-with-non-determinism\"><span class=\"ez-toc-section\" id=\"Coping_with_non-determinism\"><\/span>Coping with non-determinism<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>One of many greatest challenges with AI-based testing is non-determinism. Giant language fashions (LLMs) are probabilistic\u2014generally the agent retries a command; different occasions it offers up.<\/p>\n<p>\n  We dealt with this with a three-stage <a href=\"https:\/\/github.com\/drasi-project\/learning\/blob\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/workflows\/tutorial-evaluation-scheduled.yml#L95\" target=\"_blank\" rel=\"noopener\">retry with mannequin escalation<\/a> (begin with Gemini-Professional, on failure attempt with Claude Opus), <a href=\"https:\/\/github.com\/drasi-project\/learning\/blame\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/prompts\/tutorial-evaluation.md#L26C7-L26C7\" target=\"_blank\" rel=\"noopener\">semantic comparability for screenshots<\/a> as a substitute of pixel-matching, and verification of core-data fields <a href=\"https:\/\/github.com\/drasi-project\/learning\/blame\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/prompts\/tutorial-evaluation.md#L25\" target=\"_blank\" rel=\"noopener\">relatively than risky values<\/a>.\n<\/p>\n<p>\n  We even have a <a href=\"https:\/\/github.com\/drasi-project\/learning\/blame\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/prompts\/tutorial-evaluation.md#L49-L49C12\" target=\"_blank\" rel=\"noopener\">listing of tight constraints<\/a> in our prompts that forestall the agent from happening a debugging journey, <a href=\"https:\/\/github.com\/drasi-project\/learning\/blame\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/prompts\/tutorial-evaluation.md#L31\" target=\"_blank\" rel=\"noopener\">directives to manage<\/a> the construction of the ultimate report, and likewise <a href=\"https:\/\/github.com\/drasi-project\/learning\/blame\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/prompts\/tutorial-evaluation.md#L43-L43C17\" target=\"_blank\" rel=\"noopener\">skip directives<\/a> that inform the agent to bypass optionally available tutorial sections like establishing exterior companies.\n<\/p>\n<h3 class=\"wp-block-heading\" id=\"artifacts-for-debugging\"><span class=\"ez-toc-section\" id=\"Artifacts_for_debugging\"><\/span><a id=\"post-98436-OLE_LINK19\"\/>Artifacts for debugging<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>When a run fails, we have to know why. For the reason that agent is working in a transient container, we will\u2019t simply Safe Shell (SSH) in and go searching.<\/p>\n<p>So, our agent preserves proof of each run, screenshots of internet UIs, terminal output of vital instructions, and a remaining markdown report detailing its reasoning like proven right here:<\/p>\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69de8e15848ca&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image wp-lightbox-container\"><img decoding=\"async\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/drasi-getting-started-tutorial-evaluation-e.webp\" alt=\"# Drasi Getting Started Tutorial Evaluation &#10;&#10;## Environment &#10;- Timestamp: 2026-02-20T13:32:07.998Z &#10;- Directory: \/workspaces\/learning\/tutorial\/getting-started &#10;&#10;## Step 1: Setup Drasi Environment &#10;- Skipped as per instructions (already in DevContainer). &#10;- Verified environment setup by checking `resources` folder existence. &#10;&#10;## Step 2: Create PostgreSQL Source &#10;- Command: `drasi apply -f .\/resources\/hello-world-source.yaml`&#10;&#10;.................................... &#10;............  more steps  .......... &#10;.................................... &#10;&#10;### Scenario 1: hello-world-from &#10;- Initial check: \u201cBrian Kernighan\u201d present. (Screenshot: `09_hello-world-from.png`) &#10;- Action: Insert \u2018Allen\u2019, \u2018Hello World\u2019. &#10;- Verification: \u201cAllen\u201d appeared in UI. (Screenshot: `10_hello-world-from-updated.png`) &#10;- Result: **PASSED** &#10;&#10;............................................................ &#10;..... more validation by playwright taking screenshots ..... &#10;............................................................ &#10;&#10;## Conclusion &#10;The tutorial instructions were clear and the commands executed successfully. The expected behavior matches the actual behavior observed via the Debug Reaction UI. &#10;&#10;## STATUS: SUCCESS \" class=\"wp-image-98439 webp-format\" srcset=\"\" data-orig-src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/drasi-getting-started-tutorial-evaluation-e.webp\"\/><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\"><br \/>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"\/>\n\t\t\t<\/svg><br \/>\n\t\t<\/button><\/figure>\n<p>\n  These <a href=\"https:\/\/github.com\/drasi-project\/learning\/blob\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/workflows\/tutorial-evaluation-scheduled.yml#L156\" target=\"_blank\" rel=\"noopener\">artifacts are uploaded<\/a> to the GitHub Motion run abstract, permitting us to \u201ctime journey\u201d again to the precise second of failure and see what the agent noticed.\n<\/p>\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69de8e158590d&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image wp-lightbox-container\"><img decoding=\"async\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/screenshot-of-agents-report-output-in-a-folder-wit.webp\" alt=\"Screenshot of Agents report output in a folder with other files. \" class=\"wp-image-98440 webp-format\" srcset=\"\" data-orig-src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/screenshot-of-agents-report-output-in-a-folder-wit.webp\"\/><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\"><br \/>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"\/>\n\t\t\t<\/svg><br \/>\n\t\t<\/button><\/figure>\n<h3 class=\"wp-block-heading\" id=\"parsing-the-agent-s-report\"><span class=\"ez-toc-section\" id=\"Parsing_the_agent%E2%80%99s_report\"><\/span><a id=\"post-98436-OLE_LINK18\"\/>Parsing the agent\u2019s report<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>\n  With LLMs, getting a definitive \u201cGo\/Fail\u201d sign {that a} machine can perceive could be difficult. An agent may write a protracted, nuanced conclusion like:\n<\/p>\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69de8e158671a&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img decoding=\"async\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1024x119.webp\" alt=\"\" class=\"wp-image-98447 webp-format\" srcset=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1024x119.webp 1024w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-388x45.webp 388w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-768x90.webp 768w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-450x53.webp 450w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-650x76.webp 650w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image.webp 1080w\" data-orig-src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1024x119.webp\"><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\"><br \/>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"\/>\n\t\t\t<\/svg><br \/>\n\t\t<\/button><\/figure>\n<p>To make this actionable in a CI\/CD pipeline, we needed to do some immediate engineering. We explicitly instructed the agent:<\/p>\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69de8e1587453&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img decoding=\"async\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-1024x128.webp\" alt=\"\" class=\"wp-image-98454 webp-format\" srcset=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-1024x128.webp 1024w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-388x49.webp 388w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-768x96.webp 768w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-1536x193.webp 1536w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-450x56.webp 450w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-650x82.webp 650w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1.webp 1650w\" data-orig-src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/Picture1-1024x128.webp\"><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\"><br \/>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"\/>\n\t\t\t<\/svg><br \/>\n\t\t<\/button><\/figure>\n<p>\n  In our GitHub Motion, we then merely <a href=\"https:\/\/github.com\/drasi-project\/learning\/blob\/b826ab8a7e1116a274d3a670a270516d9eaa81d2\/.github\/workflows\/tutorial-evaluation-scheduled.yml#L143C11-L154C13\" target=\"_blank\" rel=\"noopener\">grep for this particular string<\/a> to set the exit code of the workflow.\n<\/p>\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;69de8e1588316&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img decoding=\"async\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1-1024x336.webp\" alt=\"\" class=\"wp-image-98448 webp-format\" srcset=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1-1024x336.webp 1024w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1-388x127.webp 388w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1-768x252.webp 768w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1-450x148.webp 450w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1-650x213.webp 650w, https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1.webp 1080w\" data-orig-src=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2026\/04\/image-1-1024x336.webp\"><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\"><br \/>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"\/>\n\t\t\t<\/svg><br \/>\n\t\t<\/button><\/figure>\n<p>Easy methods like this bridge the hole between AI\u2019s fuzzy, probabilistic outputs and CI\u2019s binary go\/fail expectations.<\/p>\n<h3 class=\"wp-block-heading\" id=\"automation\"><span class=\"ez-toc-section\" id=\"Automation\"><\/span>Automation<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>We now have an <a href=\"https:\/\/github.com\/drasi-project\/learning\/blob\/main\/.github\/workflows\/tutorial-evaluation-scheduled.yml\" target=\"_blank\" rel=\"noopener\">automated model of the workflow<\/a> which <a href=\"https:\/\/github.com\/drasi-project\/learning\/actions\/runs\/22225944239\/\" target=\"_blank\" rel=\"noopener\">runs weekly<\/a>. This model evaluates all our tutorials each week in parallel\u2014every tutorial will get its personal sandbox container and a recent perspective from the agent appearing as an artificial person. If any of the tutorial analysis fails, the workflow is configured to file a difficulty on our GitHub repo.<\/p>\n<p>This workflow can optionally even be run on pull-requests, however to forestall assaults we now have added a maintainer-approval requirement and a `pull_request_target` set off, which signifies that even on pull-requests by exterior contributors, the workflow that executes would be the one in our essential department. <\/p>\n<p>\n  Working the Copilot CLI requires a PAT token which is saved within the atmosphere secrets and techniques for our repo. To verify this doesn&#8217;t leak, every run requires maintainer approval\u2014besides the automated weekly run which solely runs on the `essential` department of our repo.\n<\/p>\n<h2 class=\"wp-block-heading\" id=\"what-we-found-bugs-that-matter\"><span class=\"ez-toc-section\" id=\"What_we_discovered_Bugs_that_matter\"><\/span><a id=\"post-98436-OLE_LINK21\"\/><a id=\"post-98436-OLE_LINK20\"\/>What we discovered: Bugs that matter<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Since implementing this method, we now have run over 200 \u201cartificial person\u201d classes. The agent recognized 18 distinct points together with some critical atmosphere points and different documentation points like these. <strong>Fixing them improved the docs for everybody, not simply the bot<\/strong>.<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Implicit dependencies<\/strong>: In a single tutorial, we instructed customers to create a tunnel to a service. The agent ran the command, after which\u2014following the subsequent instruction\u2014killed the method to run the subsequent command.<br \/><strong>The repair<\/strong>: We realized we hadn\u2019t instructed the person to maintain that terminal open. We added a warning: <em>\u201cThis command blocks. Open a brand new terminal for subsequent steps.\u201d<\/em><\/li>\n<li><strong>Lacking verification steps<\/strong>: We wrote: \u201cConfirm the question is working.\u201d The agent received caught: \u201cHow, precisely?\u201d<br \/><strong>The repair<\/strong>: We changed the imprecise instruction with an specific command: `drasi wait -f question.yaml`. <\/li>\n<li><strong>Format drift<\/strong>: Our CLI output had advanced. New columns have been added; older fields have been deprecated. The documentation screenshots nonetheless confirmed the 2024 model of the interface. A human tester may gloss over this (\u201cit appears largely proper\u201d). The agent flagged each mismatch, forcing us to maintain our examples updated.<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\" id=\"ai-as-a-force-multiplier\"><span class=\"ez-toc-section\" id=\"AI_as_a_power_multiplier\"><\/span><a id=\"post-98436-OLE_LINK23\"\/><a id=\"post-98436-OLE_LINK28\"\/><a id=\"post-98436-OLE_LINK29\"\/>AI as a power multiplier<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>\n  We frequently hear about AI changing people, however on this case, <strong>the AI is offering us with a workforce we by no means had<\/strong>.\n<\/p>\n<p>\n  To duplicate what our system does\u2014working six tutorials throughout recent environments each week\u2014we would wish a devoted QA useful resource or a big funds for handbook testing. For a four-person staff, that&#8217;s not possible. By deploying these <strong>Artificial Customers<\/strong>, we now have successfully employed a tireless QA engineer who works nights, weekends, and holidays.\n<\/p>\n<p>Our tutorials are actually validated weekly by artificial customers. <a href=\"https:\/\/drasi.io\/drasi-kubernetes\/getting-started\/\" target=\"_blank\" rel=\"noopener\">attempt the Getting Began information your self<\/a> and see the outcomes firsthand. And if you happen to\u2019re dealing with the identical documentation drift in your individual venture, take into account <a href=\"https:\/\/github.com\/features\/copilot\/cli\" target=\"_blank\" rel=\"noopener\">GitHub Copilot CLI<\/a> not simply as a coding assistant, however as an agent\u2014give it a immediate, a container, and a objective\u2014and let it do the work a human doesn\u2019t have time for.<\/p>\n<\/div>\n<p><script>\n\t\tfunction facebookTracking() {\n\t\t\t\/\/ If GPC or AMC Signal is enabled, do not fire Facebook Pixel\n\t\t\tif ( navigator.globalPrivacyControl || document.cookie.includes('3PAdsOptOut=1') ) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\t!function(f,b,e,v,n,t,s){if(f.fbq)return;n=f.fbq=function(){n.callMethod?\n\t\t\t\tn.callMethod.apply(n,arguments):n.queue.push(arguments)};if(!f._fbq)f._fbq=n;\n\t\t\t\tn.push=n;n.loaded=!0;n.version='2.0';n.queue=[];t=b.createElement(e);t.async=!0;\n\t\t\t\tt.src=v;t.type=\"ms-delay-type\";t.setAttribute('data-ms-type','text\/javascript');\n\t\t\t\ts=b.getElementsByTagName(e)[0];s.parentNode.insertBefore(t,s)}(window,\n\t\t\t\tdocument,'script','https:\/\/connect.facebook.net\/en_US\/fbevents.js');\n\t\t\tfbq('init', '10154859804840582');\n\t\t\t\t\t\tfbq('track', 'PageView');\n\t\t\t\t\t}\n\t<\/script><br \/>\n<br \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>For early-stage open-source initiatives, the \u201cGetting began\u201d information is usually the primary actual interplay a developer has with the venture. If a command fails, an output doesn\u2019t match, or a step is unclear, most customers gained\u2019t file a bug report, they may simply transfer on. Drasi, a CNCF sandbox venture that detects modifications in your [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":25367,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[],"class_list":["post-25365","post","type-post","status-publish","format-standard","has-post-thumbnail","category-iot"],"_links":{"self":[{"href":"https:\/\/aireviewirush.com\/index.php?rest_route=\/wp\/v2\/posts\/25365","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aireviewirush.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aireviewirush.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aireviewirush.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aireviewirush.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=25365"}],"version-history":[{"count":1,"href":"https:\/\/aireviewirush.com\/index.php?rest_route=\/wp\/v2\/posts\/25365\/revisions"}],"predecessor-version":[{"id":25366,"href":"https:\/\/aireviewirush.com\/index.php?rest_route=\/wp\/v2\/posts\/25365\/revisions\/25366"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aireviewirush.com\/index.php?rest_route=\/wp\/v2\/media\/25367"}],"wp:attachment":[{"href":"https:\/\/aireviewirush.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=25365"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aireviewirush.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=25365"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aireviewirush.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=25365"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}