How We Catch Documentation Drift with Pi and GitHub Actions
According to Linear, teams using AI agents now ship 5x as many pull requests as two years ago, while Cursor’s data shows developers produce 2.5x as much code as 18 months ago.
More code and faster shipping make documentation drift from the product. Documentation describes one behavior, while code produces another. This hinders adoption, breaks onboarding workflows, and can drive prospective users to competitors or frustrate existing ones.
Tools such as Mintlify offer features to catch documentation drift and automatically update your docs. However, Mintlify’s implementation may not fit your existing workflow, and it locks you into its platform. This guide shows you how to build the same capability without locking into any specific product.
This guide walks through the GitHub Actions workflow we use to catch documentation drift on pull requests with Pi, an open source agent harness. You can replicate the workflow on your own repository.
We’re already running this setup on VectorLint, an open source content review harness we maintain at TinyRocket.
Let’s get started.
What You’ll Build
By the end of this guide, you’ll have three pieces wired together:
- An agent skill that tells the agent what counts as documentation drift
- A message builder script that constructs the complete agent instruction
- A GitHub Actions workflow that orchestrates the check on demand and posts the result as a PR comment
Note: The workflow flags documentation drift; it doesn’t create a PR or issue. That’s intentional. There could be false positives where the agent flags drift that isn’t user-facing. Automatically creating a pull request for each false positive only adds review work. You can build a downstream workflow that takes this workflow’s output and creates an issue or PR with an agent.
Before You Start
You need a few things in place:
- A GitHub repository with documentation alongside the code (for example,
/docsor a README markdown file) - Owner permission on the repository
- An API key from an AI provider that Pi supports (Anthropic, OpenAI, Google, and others). This guide uses Amazon Bedrock. If you use a different provider, the only change is the authentication parameters.
Pi is an open source coding agent that you run from the command line. It’s lightweight, provider-agnostic, and highly customizable, which is why we chose it for this guide. If you want to test the skill before committing it to CI, install Pi locally.
Note: You can replace Pi with any other agent and get similar results. You’ll need to adapt the setup to how your agent works.
Step 1: Create the Doc-Drift Skill
Copy the doc-drift skill from the VectorLint repository into your project at .agents/skills/doc-drift/. The directory contains:
SKILL.md: the main instruction the agent followsreferences/user-facing-criteria.md: defines what counts as a user-facing changereferences/comment.md: defines the output format for drift reports
The skill tells the agent to extract user-facing behavioral changes from a PR diff, search the documentation for claims those changes invalidate, and write a report for each finding.
A user-facing change could be a renamed command-line flag, a new environment variable, a changed config key, or a rewritten error message. The skill ignores internal refactors, test changes, and anything with no observable effect on the user.
Note: We built this skill for VectorLint, so it has hardcoded scope paths and repository references. To adapt it, change the scope paths and repository name in
SKILL.mdand update the criteria inreferences/user-facing-criteria.mdto match your product. You can also hand the skill to an agent and ask it to adapt the references to your codebase.
Step 2: Build the GitHub Actions Workflow
Create a file at .github/workflows/doc-drift.yml. The sections below walk through each block of this file, then assemble the full workflow at the end.
The Trigger and the Authorization Guard
To guard against cost and security overhead, a documentation check should only run when someone triggers it on a pull request, not on every push.
name: Doc Drift Check
on:
issue_comment:
types: [created]
jobs:
authorize:
name: Check authorization
runs-on: ubuntu-latest
if: >
github.event.issue.pull_request != null &&
startsWith(github.event.comment.body, '/check-docs')
permissions:
issues: write
pull-requests: write
contents: read
outputs:
authorized: ${{ steps.auth.outputs.authorized }}
The on: issue_comment trigger fires whenever a comment is created. The if: condition narrows triggering to comments on pull requests that start with /check-docs, ignoring issue comments. The permissions block grants the minimum access the job needs (reading repository contents and writing to issues and pull requests), while the authorized output lets downstream jobs decide whether to continue.
To guard against unauthorized reviews, only repository owners should be able to trigger the drift check. This limits the cost of running AI models and reduces the surface area for prompt injection.
steps:
- name: Check authorization
id: auth
uses: actions/github-script@v7
with:
script: |
const association = context.payload.comment.author_association;
if (association !== 'OWNER') {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: @${context.payload.comment.user.login} Only the repo owner can trigger doc drift checks.
});
core.setOutput('authorized', 'false');
} else {
core.setOutput('authorized', 'true');
}
The step reads author_association from the comment payload and compares it to OWNER. If the commenter isn’t the repository owner, the step posts a rejection comment explaining why and sets the authorized output to false. Otherwise, it sets the output to true, allowing downstream jobs to proceed.
This mitigates the risk of prompt injection since pull request titles, bodies, and comments are user-controlled strings.
Note: Running an agent in a workflow carries some security risk. A malicious actor could create a pull request containing instructions to get your agent to expose secrets. To avoid that, only grant the minimum permissions your agents need to run. For more on this, read GitHub’s guide on mitigating cloud agent risk.
Check Out the PR and Fetch the Diff
The agent needs the diff to know what changed in the pull request and to check for behavioral changes.
check-docs:
name: Check documentation drift
runs-on: ubuntu-latest
needs: authorize
if: needs.authorize.outputs.authorized == 'true'
permissions:
issues: write
pull-requests: write
contents: read
steps:
- name: React to comment
uses: actions/github-script@v7
with:
script: |
await github.rest.reactions.createForIssueComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
content: 'eyes'
});
- name: Get PR head SHA
id: pr
uses: actions/github-script@v7
with:
script: |
const pr = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number
});
core.setOutput('head_sha', pr.data.head.sha);
- name: Checkout PR branch
uses: actions/checkout@v4
with:
ref: ${{ steps.pr.outputs.head_sha }}
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Fetch PR diff
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr diff ${{ github.event.issue.number }} > /tmp/pr.diff
echo "Diff size: $(wc -l < /tmp/pr.diff) lines"
The eye’s reaction gives the reviewer immediate feedback that the command landed, before the run finishes. The Get PR head SHA step resolves the pull request’s current head commit, so the checkout targets the right ref. fetch-depth: 0 gives the agent the full repository history, which it uses to reason about how code and documentation relate. setup-node installs Node version 20, which Pi’s runtime needs to run. Writing the diff to a file keeps the agent’s prompt clean and lets you inspect or format the diff separately.
Create the Message Builder Script
You need to tell Pi where to find the diff, which skill to invoke, and where to write its reports. Keeping that logic in a standalone script rather than inline YAML makes the script testable on its own and easier to modify later.
Create the file at .github/scripts/build-doc-drift-message.ts:
import { writeFile } from 'node:fs/promises';
const [, , DIFF_PATH, OUTPUT_PATH] = process.argv;
if (!DIFF_PATH || !OUTPUT_PATH) {
throw new Error('Usage: tsx .github/scripts/build-doc-drift-message.ts <diff-path> <output-path>');
}
const WORKSPACE = process.env.GITHUB_WORKSPACE;
if (!WORKSPACE) {
throw new Error('GITHUB_WORKSPACE is required');
}
const MESSAGE = You are running a doc drift check on a pull request in the VectorLint repository.
The pull request checkout to inspect is located at:
${WORKSPACE}
Read the PR diff from this file:
${DIFF_PATH}
Use the doc-drift skill. When you have finished, write one report file
per behavioral change you identified, named sequentially:
${WORKSPACE}/.doc-drift-1.md
${WORKSPACE}/.doc-drift-2.md
... and so on.
If there are no issues to report, write a single file ${WORKSPACE}/.doc-drift-1.md
containing the no-issues-found report.
Do not post anything to GitHub directly. The workflow will handle posting.
;
await writeFile(OUTPUT_PATH, MESSAGE);
The script takes two arguments, the path to the saved PR diff and the path to write the output message. It resolves both against GITHUB_WORKSPACE, so the paths are absolute. The output is a short routing message that tells Pi where to find the diff, where to write its reports, and that the workflow handles the GitHub posting. The script never inlines the diff itself; Pi reads it from the file at runtime.
The workflow invokes this script as:
npx tsx .github/scripts/build-doc-drift-message.ts /tmp/pr.diff /tmp/pi-message.txt
The doc-drift skill it references lives at .agents/skills/doc-drift/ – the same skill you created in Step 1.
Install Pi and Run the Drift Check
With the diff on disk, the message builder output ready, and Node.js available, the workflow can install Pi and hand it the routing message. This is where the agent actually reasons about whether documentation needs updating.
A CI runner has no interactive terminal, so Pi runs in a headless, non-interactive mode, with file actions pre-approved.
- name: Install Pi
run: npm install -g @earendil-works/pi-coding-agent
- name: Build agent message
env:
GITHUB_WORKSPACE: ${{ github.workspace }}
run: |
npx tsx .github/scripts/build-doc-drift-message.ts /tmp/pr.diff /tmp/pi-message.txt
- name: Run doc drift check
env:
AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.PI_BEDROCK_API_KEY }}
AWS_REGION: ${{ secrets.PI_BEDROCK_REGION }}
PI_MODEL: ${{ secrets.PI_MODEL }}
run: |
pi --approve --no-session -p \
--provider amazon-bedrock \
--model "$PI_MODEL" \
--skill .agents/skills/doc-drift \
"/skill:doc-drift $(cat /tmp/pi-message.txt)"
npm install -g makes the pi command-line tool available for the rest of the job. You don’t need a session log, so --no-session makes sure the session isn’t saved, and -p prints the response and exits. Both --no-session and -p give you non-interactive execution.
--approve trusts the project’s files for the run so Pi doesn’t prompt for approval. In this setup, the agent only reads files and writes its report, so there is nothing destructive to guard against. --skill .agents/skills/doc-drift points Pi at the doc-drift skill you created in Step 1. When the agent finishes, it writes one report file per behavioral change to numbered paths like .doc-drift-1.md, which the next step picks up. The three environment variables – AWS_BEARER_TOKEN_BEDROCK, AWS_REGION, and PI_MODEL– come from repository secrets you configure in Step 3.
Post the Findings as PR Comments
The agent is a language model, inherently nondeterministic. Letting it post directly to GitHub would make posting unpredictable. It could emit partial output, retry mid-run, or fail halfway through a comment. A file-based handoff sidesteps that.
- name: Post report comments
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
REPORT_FILES=$(ls $GITHUB_WORKSPACE/.doc-drift-*.md 2>/dev/null | sort -V)
if [ -z "$REPORT_FILES" ]; then
gh pr comment ${{ github.event.issue.number }} \
--body "Doc drift check completed but produced no output. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
else
for file in $REPORT_FILES; do
gh pr comment ${{ github.event.issue.number }} --body-file "$file"
done
fi
- name: Post failure comment
if: failure()
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr comment ${{ github.event.issue.number }} \
--body "Doc drift check failed. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
The step lists all .doc-drift-*.md files on the runner and loops through them, posting each one as a PR comment with gh pr comment --body-file. The agent never posts to GitHub itself. It only writes files, and the shell loop posts the comments to the PR, producing the same behavior on every run.
The workflow owning all GitHub writes keeps the agent’s permissions minimal and makes every comment traceable.
Two fallback paths handle edge cases. If the run produces no report files, a comment tells the reviewer and links to the Actions log. If the run fails outright, the failure() step catches the failure and posts a failure notice with the same log link.
The Complete Workflow
Here’s a copy-paste version of the workflow for your convenience.
name: Doc Drift Check
on:
issue_comment:
types: [created]
jobs:
authorize:
name: Check authorization
runs-on: ubuntu-latest
if: >
github.event.issue.pull_request != null &&
startsWith(github.event.comment.body, '/check-docs')
permissions:
issues: write
pull-requests: write
contents: read
outputs:
authorized: ${{ steps.auth.outputs.authorized }}
steps:
- name: Check authorization
id: auth
uses: actions/github-script@v7
with:
script: |
const association = context.payload.comment.author_association;
if (association !== 'OWNER') {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: @${context.payload.comment.user.login} Only the repo owner can trigger doc drift checks.
});
core.setOutput('authorized', 'false');
} else {
core.setOutput('authorized', 'true');
}
check-docs:
name: Check documentation drift
runs-on: ubuntu-latest
needs: authorize
if: needs.authorize.outputs.authorized == 'true'
permissions:
issues: write
pull-requests: write
contents: read
steps:
- name: React to comment
uses: actions/github-script@v7
with:
script: |
await github.rest.reactions.createForIssueComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
content: 'eyes'
});
- name: Get PR head SHA
id: pr
uses: actions/github-script@v7
with:
script: |
const pr = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number
});
core.setOutput('head_sha', pr.data.head.sha);
- name: Checkout PR branch
uses: actions/checkout@v4
with:
ref: ${{ steps.pr.outputs.head_sha }}
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install Pi
run: npm install -g @earendil-works/pi-coding-agent
- name: Fetch PR diff
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr diff ${{ github.event.issue.number }} > /tmp/pr.diff
echo "Diff size: $(wc -l < /tmp/pr.diff) lines"
- name: Build agent message
env:
GITHUB_WORKSPACE: ${{ github.workspace }}
run: |
npx tsx .github/scripts/build-doc-drift-message.ts /tmp/pr.diff /tmp/pi-message.txt
- name: Run doc drift check
env:
AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.PI_BEDROCK_API_KEY }}
AWS_REGION: ${{ secrets.PI_BEDROCK_REGION }}
PI_MODEL: ${{ secrets.PI_MODEL }}
run: |
pi --approve --no-session -p \
--provider amazon-bedrock \
--model "$PI_MODEL" \
--skill .agents/skills/doc-drift \
"/skill:doc-drift $(cat /tmp/pi-message.txt)"
- name: Post report comments
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
REPORT_FILES=$(ls $GITHUB_WORKSPACE/.doc-drift-*.md 2>/dev/null | sort -V)
if [ -z "$REPORT_FILES" ]; then
gh pr comment ${{ github.event.issue.number }} \
--body "Doc drift check completed but produced no output. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
else
for file in $REPORT_FILES; do
gh pr comment ${{ github.event.issue.number }} --body-file "$file"
done
fi
- name: Post failure comment
if: failure()
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr comment ${{ github.event.issue.number }} \
--body "Doc drift check failed. Check the [Actions log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
Step 3: Configure the Provider Secrets
Add PI_BEDROCK_API_KEY, PI_BEDROCK_REGION, and PI_MODEL as repository secrets in your GitHub repository settings under Settings → Secrets and variables → Actions.
For other providers, change the --provider flag in the workflow and the environment variable names to match. For example, Anthropic needs --provider anthropic and only two secrets (ANTHROPIC_API_KEY and PI_MODEL). Check the Pi provider documentation for the environment variable your provider expects.
Step 4: Test the Workflow
On any pull request, leave a comment starting with /check-docs as the repo owner. The workflow confirms you’re authorized, runs the agent against the PR diff, and posts one comment per behavioral change the agent found. Here’s a drift finding:
## ⚠️ Doc drift - renamed the --output flag to --format
### docs/guides/cli-reference.md - Output formats
**What the doc claims:** "Run vectorlint doc.md --output line for terminal output."
**What's now true:** The flag is --format, not --output.
Fix prompt:
~~~
docs/guides/cli-reference.md, Output formats: "--output" is no longer accurate - the flag is now --format. Update it. Keep all existing structure, tone, and style.
~~~
Every surfaced drift finding has a fix prompt. The check tells you what broke, and the prompt gives you a starting point to fix it.
Not every PR drifts documentation. When the agent finds a new user-facing change that isn’t documented yet, the comment names the change and its location. When nothing needs attention, you get a confirmation comment with the files and changes it checked.
Handling False Positives and Negatives
AI models are probabilistic, so there’s bound to be some false positives and negatives.
The agent might flag product changes as documentation drift when they aren’t, or miss drifts when a PR genuinely invalidates documentation.
When you notice either problem, update the instruction in the docs-drift skill. Add a concrete instruction that names the pattern, states what the agent should do differently, and includes an example if the rule isn’t obvious. Then test it locally on the same code change to verify it covers the edge case.
These misses are less frequent with more capable models. If you observe consistent false positives or negatives, consider upgrading to a more capable model.
Resolving the Doc Drift
At this point, your workflow catches documentation drift at the PR level. When a PR changes documented behavior, the agent flags it in the PR itself, no platform required.
The simplest approach to resolving the drift is to update the documentation in the same PR. You can hand the fix message to an agent to update the relevant docs, or build an agent workflow to create a PR based on detected drift.
You’ll need guardrails, constraints, and verification signals to keep the agent’s output accurate, on-style, and useful. These would let you quickly confirm the agent’s work is correct and provide feedback so the agent can self-correct.
Building those guardrails is a project in itself and requires iteration. You need style-aware instructions, verification checks, and feedback loops that let you quickly assess the agent’s output and improve performance over time.
For teams without a dedicated docs engineer, hiring an external documentation engineering service is often more cost-effective and efficient.
At TinyRocket, we build automated workflows, agent skills, verification loops, and tools that help teams keep documentation in sync with their product. Our agentic workflows let your team focus on building while reducing documentation work to a matter of quick judgment. Book a call today to get a free documentation workflow audit.