A few days ago I published aidevops — the terminal client for an AI DevOps / QA copilot — to PyPI: pipx install aidevops and you are done. The source is on GitHub under MIT.
This post walks through what it solves, how the data flows, how the risk score is computed, what the AI actually does (and why it is kept in a cage), how each command is used, and what I deliberately left out.
explain: the full chain behind one commitA team produces a lot of development data every day: requirements live in Jira, commits in GitHub, test results in CI, bugs in a defect tracker, release records in a pipeline. All of it exists — scattered across systems that do not know each other.
So every developer does the same manual work: which modules did this change touch? What bugs did those modules have historically? Are they covered by tests? If we ship this release, where is the risk concentrated? The answers are all in the systems, but assembling them means five browser tabs, three exported reports and a memory test.
aidevops exists to link the chain requirement → code → test → bug → release, and then answer four concrete questions:
The terminal client puts those answers where the developer already is. Nobody should have to switch to a browser to read one risk report.
One sentence: aidevops is a terminal client (CLI + full-screen TUI), not a server. It contains no business logic; every capability is an HTTP call to a backend (Django + PostgreSQL + Redis, with pgvector for vector search).
The boundary is deliberate, for three reasons:
/api/v1), so logic cannot drift.If you install the client without a backend, login fails on the first step. That is architecture, not a bug. Two ways to run the backend locally: cp .env.example .env then docker compose up -d --build (Swagger at /api/docs/), or make install && make migrate && make run plus make worker / make beat for Celery.
Client stack: Typer (CLI), Textual (full-screen TUI), Rich (tables and output), httpx (HTTP). Pure Python, no compiled extensions, Python ≥ 3.11.
Without data everything downstream is an empty state. Ingestion is the least glamorous part of the system, and it has three channels:
GitHub REST sync. An incremental cursor and a backfill cursor are maintained separately — new commits go through the incremental path, the first sync of a repository walks the full history. Rate limits are handled with backoff, because GitHub's limits are a fact of life rather than an exception.
Webhook ingestion. Push, CI results and issue changes arrive in real time, behind two gates: HMAC-SHA256 signature verification (the request really came from GitHub) and delivery_id idempotency (GitHub redelivers events; the same delivery must not produce two records).
Celery beat reconciliation. Webhooks get lost — network blips, restarts — so a scheduled job periodically reconciles and guarantees eventual consistency.
Module attribution and test correlation happen asynchronously after ingestion. The trade-off is explicit: ingestion prefers slow over lost or duplicated, because every downstream correlation assumes the data is trustworthy.
The first step of correlation is answering "which modules did this commit touch". It sounds trivial and is actually a parsing problem: src/auth/ clearly belongs to auth, but who owns utils/date.py? Does a change under tests/ count as touching a business module?
The answer is a three-level attribution strategy, applied in priority order:
src/ or lib/);The important half is the last sentence: every attribution carries a citable reason. "This file belongs to auth because it matched the override src/auth/**" — attribution is not a guess, it can be opened and inspected. Risk scoring and regression candidates are all built on this traceable layer.
explain: the full chain behind one commitaidevops explain <sha> is the signature command. Give it a commit SHA and it returns the commit's place in the development chain:
The last point deserves emphasis: data_gaps distinguishes "nothing was found" from "we have no data". The fastest way for a tool to lose trust is to dress ignorance as a conclusion — in a repository that never ingested test results, "no regression candidates" must mean "unknown", not "safe". That is the product's honesty floor, and explain makes it a first-class output.
aidevops risk <sha> (and aidevops module-risk <id> for a module view) returns a normalised risk score. It is deliberately explainable: the ten signals are listed with their individual contributions, and the contributions sum exactly to the score.
| Signal | What it measures |
|---|---|
change_volume | Size of the change (lines, files) |
historical_bug_density | Historical bug density of the modules involved |
test_failure_rate | Recent failure rate of the related tests |
module_centrality | Centrality of the touched modules in the dependency graph |
change_frequency | How often this area changes lately (hotspot detection) |
coverage_gap | Coverage gap on the changed paths |
open_high_severity_bugs | Open high-severity bugs on the modules |
test_coverage_gap | Test coverage gap for the module |
release_proximity | How close the next release is (risk is more expensive near the cut) |
data_freshness_penalty | An honest penalty when the data is too stale to trust |
Three engineering decisions worth spelling out:
release_proximity; a team with weak test infrastructure can emphasise coverage_gap.The AI layer answers "given all this data, can something be automated". Its structure is considerably more involved than "call an LLM".
Pluggable providers. Model credentials are managed per organisation (aidevops providers / provider-add / provider-verify) over an OpenAI-compatible protocol, with presets for DeepSeek, Qwen and Ollama, and credentials encrypted with Fernet. Model keys belong on the server, exactly like Git PATs.
Five built-in agents, each with an explicit input/output contract:
| Agent | What it does |
|---|---|
| Code Impact | Analyses the blast radius of a commit |
| Test Generation | Generates test cases along a scenario matrix (11 categories, not a shotgun blast) |
| Bug Investigation | Builds an evidence chain around one bug |
| Root Cause Analysis | Works backwards from symptoms to plausible root-cause areas |
| Requirement Analysis | Extracts verifiable acceptance points from requirement text |
Start one with aidevops analyse -a test-generation --wait; browse history with analyses / job. aidevops trace <id> is the debugging tool: every model call and tool call behind one analysis, with timings. The agent is not a black box.
aidevops findings / finding <id> / triage <id> <status> form the workspace for AI findings. Every finding carries evidence references: whatever the model says must resolve back to real data (a commit, a bug, a test); unresolvable references never make it into the output. triage makes the disposition (open / confirmed / false positive) an explicit state transition, and that trail becomes team knowledge.
aidevops tui is the full-screen dashboard (Textual): commit list, finding stream, suggestion queue — for people who would rather not memorise commands.
Evidence validation is the hard constraint of this layer: every reference the model emits must resolve to real data, or it is dropped. That mechanically eliminates incidents like "the model hallucinated a bug that never existed".
The tool surface is closed. The model can call 13 read-only tools, all forced through project scoping; no write operation appears in its tool list. The AI can see data but cannot touch state — which is the precondition for the next section.
Everything the AI produces is a proposal: a batch of generated tests, an investigation conclusion, a remediation suggestion. None of it takes effect by itself.
aidevops proposals # pending proposals
aidevops approve <id> # approve → executed by a fixed executor
aidevops reject <id> # reject → discarded
Three hard rules:
approve executes. Execution lives in a hard-coded executor, not in the model. The model can propose; it cannot act.AuditLog. Who approved what and when, and what the executor did, is queryable with aidevops audit.For QA the trade-off is clear: the trust balance for letting AI mutate state is not there yet. Tests are a quality line of defence, and you do not put a black box that acts on its own on the line.
Multi-tenant safety follows the same logic: tenant id comes only from the URL (never inferred from the request body), non-member access returns 404 (without leaking existence), insufficient role returns 403, and every query goes through the single scoped_queryset entry point — leaving no room for "someone forgot the filter".
pipx install aidevops # recommended: isolated venv, command on PATH
# or
pip install aidevops
Login uses the GitHub device flow: type a code in the browser, never paste a password into a terminal. On remote servers and containers use login --no-browser.
aidevops login # GitHub device-flow login
aidevops org-add "My Team" # first run: create org (creator is admin)
aidevops projects # list projects (* marks the current one)
aidevops use qa-copilot-platform # select a project for later commands
aidevops status # current backend / account / project
Connecting a repository is the real entry point; skip it and commits / risk / explain are all empty states:
aidevops connections # list Git credentials, note the connection id
aidevops connection-add -l my-github -t <PAT> # private repos need repo read access
aidevops connection-verify <connection-id> # verify the credential first
aidevops repo-add -c <connection-id> -f owner/repo # register the repository
aidevops sync <repository-id> --wait # pull history, ready when it returns
A connection is an organisation-level credential (managed by admins) and can register multiple repositories; repo-update adjusts the sync window and attribution depth.
Upgrading is aidevops upgrade. The client compares itself against the latest PyPI release on every run and warns in red when behind (result cached for 24 hours; AIDEVOPS_NO_UPDATE_CHECK=1 disables it). One guardrail: upgrade refuses to run inside a pip install -e development checkout, because that would silently replace your working tree with the released version — it tells you to git pull instead.
Config and tokens live in the user config directory (Windows %LOCALAPPDATA%\aidevops\, Linux ~/.config/aidevops/), mode 0600.
--json is a global option and goes before the subcommand:
aidevops --json findings | jq '.[] | select(.severity == "critical")'
Two details exist for pipelines: in --json mode the upgrade notice goes to stderr so stdout stays clean, and AIDEVOPS_JSON=1 is equivalent to the global flag. In CI, inject the token with AIDEVOPS_TOKEN (no interactive login) and point AIDEVOPS_BASE_URL at a test backend for integration checks.
A realistic CI flow: run aidevops --json risk $GITHUB_SHA in the pipeline; when the score crosses a threshold, post the regression candidate list as a PR comment. The risk engine and the AI suggestions become first-class citizens of the pipeline.
The client lives in the repository's cli/ directory and the engineering setup is complete:
python -m pytest # 50 client tests, including Textual pilot TUI tests
python -m ruff check .
python -m mypy ai_devops_cli
The TUI is tested with Textual's pilot — keystroke flows and panel switches in a full-screen app are covered without manual clicking. The backend has 569 tests (4 network cases skipped by default), none of which need the internet, so CI stays fast.
Every multi-tenant write path funnels through scoped_queryset, and combined with the URL-derived tenant semantics, tests can assert case by case that "tenant A can never see tenant B's data".
Releases to PyPI go through a tag-triggered trusted publishing workflow: push a tag → GitHub Actions builds → OIDC short-lived credentials upload to PyPI. No local packaging, no long-lived tokens.
It is Alpha (Development Status 3), with a few explicit limits:
Roadmap priority: finish the terminal detail views → real-LLM smoke test → production validation of the Celery worker shape.
If you are building a QA platform too, or you are interested in development-data correlation and caged AI agents, issues are welcome at github.com/zhoujungis/ai_devops. One line to install: pipx install aidevops.