Driftline

See how your prompts change before an AI model is retired.

Driftline is a GitHub Action. It finds the OpenAI, Anthropic and Google model IDs in your code, checks them against the providers' published retirement dates, and runs your real prompts on both the current and the replacement model so you can see what changes before you switch.

It runs inside your own CI with your own API keys. There is no Driftline server: prompts go straight from your runner to your AI provider.

What you get

• Annotations on the exact file and line of retired or retiring models (the 50 most urgent; the rest are in the report).
• A report in the job summary, and optionally a GitHub issue that updates itself on each run and closes itself once nothing is left to fix.
• Drift tests (Pro): each prompt in .driftline/prompts.yml runs on both models. You get a similarity score per prompt, and a hard failure if a JSON answer stops parsing.

Quick start

1. Copy examples/driftline.yml to .github/workflows/driftline.yml.

2. Optional, for drift tests (Pro): copy examples/prompts.yml to .driftline/prompts.yml and add your provider API keys as repository secrets.

3. Activate your Pro key once per GitHub account or organization, from inside one of its repositories:

   npx @aidriftline/driftline activate <your-license-key>
   

   It prints a license code. Save that code as the secret DRIFTLINE_LICENSE_KEY. A key works on up to 3 GitHub accounts or organizations; npx @aidriftline/driftline deactivate <code> frees one.

Inputs

Input	Default	What it does
path	.	Directory to scan
warn-days	90	Models retiring within this many days count as "retires soon" (0 turns this off)
fail-on	retired	Fail the job on retired, at-risk, or never
prompts-file	.driftline/prompts.yml	Prompts for drift tests (YAML or JSON). Looked up from the repository root, then from path
license-key		Driftline Pro license code from activate; without it only the scan runs
create-issue	false	Open or update one issue labelled driftline. Model answers are never put in the issue
show-answers	true	Show model answers in the job summary. In a public repository job summaries are public, so set false if your prompts contain private data
github-token	github.token	Needs issues: write when create-issue is on

Outputs: findings (model references found) and at-risk (references to retired or soon-to-retire models).

Model IDs written the Amazon Bedrock way (anthropic.claude-…-v1:0, including ARNs), the Google Vertex AI way (claude-…@20240620) or with a router prefix (openrouter/openai/gpt-4, vertex_ai/…) are recognized too. Azure OpenAI deployments follow Microsoft's own retirement dates and are not checked. Mentions in Markdown and text files, such as a changelog, are reported as notices but never fail the build.

Drift tests make two API calls per prompt: one to the current model, one to the replacement. A prompt without model is tested on the model in your code that retires first. Calls time out after two minutes and are retried once if the provider is busy. When a provider names a replacement that is itself being retired, Driftline points to the model at the end of that chain.

API keys are read from OPENAI_API_KEY, ANTHROPIC_API_KEY and GEMINI_API_KEY (or GOOGLE_API_KEY).

How results are scored

Result	Meaning
✅ Same	Similarity 0.90 or higher
🟡 Changed	Similarity 0.60–0.90, or the current model is already retired and the new answer needs review
🔴 Broken	Similarity below 0.60, or expect: json and the new answer is not valid JSON

Text is compared with word and word-pair overlap. JSON is compared field by field.

Run it locally

No install, no key:

npx @aidriftline/driftline path/to/your/repo          # report
npx @aidriftline/driftline path/to/your/repo --json   # same, as JSON

Exit code 1 means a retired model is used in code.

For AI coding agents

If you are an agent (Claude Code, Cursor, Copilot, Codex…) changing code that calls OpenAI, Anthropic or Google: run npx -y @aidriftline/driftline . --json before you pick or keep a model ID, and do not introduce an ID whose status is retired or at-risk. Without a shell, look the ID up in registry/models.json or at https://aidriftline.com/models/<model-id>/. More, including a snippet for AGENTS.md / CLAUDE.md: aidriftline.com/agents.

MCP server

Driftline also runs as an MCP server, so your agent can ask it directly. Tools: check_models (status, date and replacement for a list of IDs), scan_project (retired and retiring IDs in a folder, with file and line) and upcoming_retirements (what retires in the next N days). It reads the daily-updated list from aidriftline.com and falls back to the copy in the package when offline.

Claude Code:

claude mcp add driftline -- npx -y @aidriftline/driftline mcp

Cursor, Windsurf, Claude Desktop and other clients (mcp.json / claude_desktop_config.json):

{
  "mcpServers": {
    "driftline": { "command": "npx", "args": ["-y", "@aidriftline/driftline", "mcp"] }
  }
}

Needs Node 20+. Free, no key.

Retirement data

registry/models.json is maintained by hand from the official pages (OpenAI, Anthropic, Google) and records the date it was last checked. Google publishes only the earliest possible shutdown date, so Driftline reports those models as "may stop working" rather than "retired". Anthropic's "not sooner than" dates are informational and never trigger a warning. Audio, image, video and embedding models are scanned and reported too, but only text models are drift-tested. Some Google models are not shut down but routed: after the date, requests to the old ID are answered by the replacement model (routes: true in the registry). Driftline flags those too, because the call keeps working while the answers change.

Every model also has its own page with the date, replacement and source: aidriftline.com/models.

Development

npm test

No dependencies. Node 20 or later.

License

Elastic License 2.0. You can use Driftline for free in your own CI. You may not offer it as a hosted service, or remove or work around the Pro license-key check.