Breaking: New AI-Driven Approach to Documentation Testing Emerges from Open-Source Crisis

The CNCF sandbox project Drasi has pioneered a method to automatically detect documentation bugs using AI agents, after a silent dependency update caused all its tutorials to fail simultaneously. The team at Microsoft Azure's Office of the CTO deployed GitHub Copilot to act as a "synthetic new user" that naively and literally follows instructions, flagging inconsistencies instantly.

“This incident forced a realization: with advanced AI coding assistants, documentation testing can be converted to a monitoring problem,” said Mark Johnson, lead engineer at Drasi. The approach could reshape how open-source projects maintain their getting-started guides.

Background

Drasi is an early-stage open-source project that detects changes in data and triggers immediate reactions. It is maintained by a small team of four engineers within Microsoft Azure’s Office of the Chief Technology Officer.

The project ships code faster than it can manually test tutorials. In late 2025, a GitHub update to its Dev Container infrastructure bumped the minimum Docker version, breaking the Docker daemon connection—and every single tutorial stopped working. The team didn’t know the extent of the damage immediately because they relied on manual testing.

“Any developer trying Drasi during that window would have hit a wall,” Johnson noted. The incident exposed a critical gap: documentation breaks silently and often goes undetected until users abandon the project.

What This Means

This new AI-driven method transforms documentation testing from a manual chore into a continuous monitoring task. Other open-source projects can adopt similar synthetic user agents to catch breaks before developers notice.

The technique addresses two common causes of documentation failure: the curse of knowledge (where writers omit implicit steps) and silent drift (where changes in dependencies or code are not reflected in docs). By using an agent that is naïve, literal, and unforgiving, teams can verify every command and expected output automatically.

The Problem: Why Documentation Breaks

Documentation usually breaks for two reasons. First, experienced developers write with implicit context—they know to run drasi list query to check status, but a new user does not. The instructions say the "what" but skip the "how."

Second, documentation doesn’t fail loudly like code. When a configuration file is renamed, the build breaks immediately. But outdated documentation references accumulate silently until a user complains. For tutorials that spin up sandbox environments with Docker, k3d, and sample databases, any upstream dependency change—a deprecated flag, a bumped version, or a new default—can break tutorials silently.

The Solution: AI Agents as Synthetic Users

To solve this, the Drasi team treated tutorial testing as a simulation problem. They built an AI agent that acts as a "synthetic new user." The agent has three critical characteristics: it is naïve (no prior knowledge of Drasi), literal (executes every command exactly as written), and unforgiving (verifies every expected output and fails fast if something is off).

“Our agent doesn’t guess or fill gaps—it follows the guide blindly,” said Johnson. “If the doc says ‘you should see Success’ and the CLI returns nothing, the agent flags it immediately.”

The Stack: GitHub Copilot CLI and Dev Containers

The team built the solution using GitHub Copilot in combination with Dev Containers. The AI copilot helps generate and validate tutorial steps, while Dev Containers provide isolated, reproducible environments for running the agent’s tests.

This stack allows the team to run the synthetic user on-demand or as part of a continuous integration pipeline. Any break in the tutorial is detected instantly, preventing silent drift from reaching end users.

For other project maintainers, the Drasi team recommends starting small: pick one critical tutorial, create an agent that mimics a new user, and run it after every deployment. “It’s a shift in mindset—from hoping your docs are right to monitoring that they stay right,” Johnson added.