aitlc Field Guide

Setup

Three steps. aitlc runs outside your project's virtualenv and shells into it, so it never shares dependencies with the suite under test.

Install

uv tool install aitlc      # or: pipx install aitlc
aitlc --help

The same tool is published under two distribution names — aitlc and dax-aitlc. They install identical code and both provide the aitlc command; pick whichever your organisation's index prefers, and install only one.

Generate `aitlc.toml`

cd /path/to/your/project
aitlc init --dry-run   # see what it detected, write nothing
aitlc init             # write the file

init reads the repo and fills the file in — feature and step directories from where the files actually are, the issue-key prefix from feature filenames, and scenario_setup from the call inside your own before_scenario. [env] is populated from your .env: variable names only, no value is ever read or stored. Anything undetected is written as a commented placeholder rather than guessed.

aitlc searches upward from the current directory, so one file at the repo root covers every subdirectory. The result looks like this:

[project]
name = "myproject"
issue_key_prefix = "PROJ-"
feature_dir = "features"
step_dir = "features/steps"
locators_dir = "config/web_locators"

# Your own per-scenario setup, called with behave's (context, scenario)
# signature. Without it, a step slice gets no before_scenario at all.
scenario_setup = "features.environment_helpers:populate_scenario_data"

# The class your steps actually talk to. Most suites wrap Playwright's Page
# in one; name it and a step slice gets the same context.browser a real run
# would. Leave unset if your steps use the Page directly.
browser_actions = "helpers.browser:BrowserActions"

[env]
# Maps aitlc's generic names -> your actual env var names.
# aitlc stores no secrets; it only learns which variables to read.
jira_token = "JIRA_TEST_TOKEN"
jira_xray_client_id = "JIRA_XRAY_CLIENT_ID"
jira_xray_client_secret = "JIRA_XRAY_CLIENT_SECRET"

What each `[project]` setting controls

Setting	Detected?	What it changes
`name`	auto	Labels output only.
`issue_key_prefix`	auto	Lets you type `1234` instead of `PROJ-1234`.
`feature_dir`	auto	Bare-ID resolution searches here, recursively.
`step_dir`	auto	Feeds `steps unused` and the step console's registry.
`locators_dir`	auto	`record --suggest-steps` uses it to tell a newly recorded selector from one you already have.
`scenario_setup`	auto	Makes `steps run` slices work on data-dependent steps.
`browser_actions`	manual	The class assigned to `context.browser` in a step slice. Without it the raw `Page` is used, and steps written against a wrapper fail on their first call.
`browser_factory`	manual	A class exposing `launch_local_mobile_browser_via_cdp(playwright, cdp_url, device_name)`. Only needed to combine `--mobile` with `--cdp-url`.

The last two are undetected on purpose: there is no reliable signal for which of a project's many classes is the browser wrapper, and guessing wrong produces a confusing failure several steps later. init writes them as commented placeholders instead.

Verify

aitlc doctor                    # environment checks
aitlc run <TEST-ID> --dry-run   # steps resolve, no browser

--dry-run is the cheapest real signal that config, paths and step imports are all correct.

How it stays codebase-independent

A debugging tool that requires editing the suite it debugs cannot be adopted incrementally, cannot be uninstalled cleanly, and silently becomes a no-op for anyone who installs the tool but not the edit.

aitlc attaches instead, in three layers:

behave's own runner option. aitlc ships a Runner subclass passed via behave's documented --runner-class / --runner flag. It calls super().run_hook(...) for every hook, so all your project hooks still run, unchanged and in order.
Version detection. aitlc asks behave --help what it supports rather than parsing a version string — forks and vendored builds don't follow upstream numbering.
A universal fallback. Where no flag exists, aitlc writes a sitecustomize.py onto PYTHONPATH, which Python imports at startup and which patches ModelRunner.run_hook in place. It no-ops unless AITLC_INSTRUMENT=1, because that file lands on the path of every child process.

behave	Flag	Class-path format
1.2.7.dev	`--runner-class`	`pkg.module.Class`
1.3+	`--runner` / `-r`	`pkg.module:Class`
1.2.6	neither	uses the fallback

Migrating from a hook block? If your own hooks already carry a pause-on-failure block it is now redundant, but harmless — aitlc's runner gates on its own variable, so the two can never both fire. Delete it when convenient.

Running tests

Bare test IDs resolve recursively, so you never need a full path — and never need to tag other features to narrow a run.

aitlc run <TEST-ID>

Run one feature file and report structured results.

aitlc run PROJ-24026
aitlc run PROJ-24026 --dry-run          # steps resolve, no browser
aitlc run PROJ-25466:47                 # one Examples row
aitlc run PROJ-24026 --debug            # halt on failure, browser stays open
aitlc run PROJ-24026 --retry 2 --retry-only-if-known-flake

FILE:LINE selects a scenario, not a resume point. behave runs the scenario containing that line, from its first step. It pays off for a Scenario Outline — one Examples row instead of all of them (measured: 37 passed / 37 untested instead of 74).

Agent use

Exit code 0/1; stdout is one JSON object with steps_by_status and a failures[] array carrying step and error. Add --toon for a compact table. Every run is appended to history automatically.

aitlc parallel run [IDS...]

Run many features concurrently — without editing tags.

aitlc parallel run                    # everything discovered
aitlc parallel run PROJ-1 PROJ-2 -j 4
aitlc parallel run --list             # preview selection, run nothing
aitlc parallel run --debug --isolated -j 4

Skip tags already in your files are honored, and skipped files are reported with the reason — "skipped by tag" must never look identical to "never discovered".

Agent use

--list returns the exact selection as JSON before anything runs — use it to confirm scope. The summary carries per-feature results plus source: focus, args or discovery.

aitlc parallel focus

Pin what the bare command runs, so you stop typing filenames.

aitlc parallel focus PROJ-24026 PROJ-25931   # set once
aitlc parallel run                            # then just run
aitlc parallel focus --clear

This replaces the "tag every other file, then revert" habit. The selection lives under reports/, so nothing in the repo changes and nothing can be committed by accident.

The debugging cycle

Five commands, in this order. Each replaces a habit that is slower or, on a suite that creates users and moves balances, destructive.

aitlc s3 triage-run → debug start → retry → next → certify

Take a CI failure, drive one kept browser to that point, iterate on the broken step, move forward, then prove it in a fresh instance.

aitlc s3 triage-run --suite <plan>        # 1. what CI actually failed on
aitlc debug start PROJ-7449 --at 12       # 2. isolated browser, driven to the step
aitlc debug retry PROJ-7449               # 3. edit -> re-run THAT step -> repeat
aitlc debug next PROJ-7449                # 4. forward, from the state you have
aitlc debug certify PROJ-7449 --times 2   # 5. fresh instance, real feature, twice

Never re-run locally just to see where it failed. triage-run reads the run's per-execution Behave JSON — a few hundred KB against a multi-megabyte HTML report — and prints the failing step, the real error and the locator.
debug start always launches an isolated browser. A CDP attach reuses an existing browser context, so a long-lived profile keeps its sessions and eventually fails a run at your project's own login — a failure that reads as a test bug.
retry re-runs one step, not the scenario. After a fix, the state you already have is worth more than a clean start.
certify is separate and defaults to two passes. One pass does not disprove a race.

When the DOM and an assertion disagree, instrument rather than theorise. One log.info("%r", text) at the read site, for one run, beats another round of inspection — it shows what the code actually read, which is not always what the page shows.

Debugging live

Keep one browser open across many iterations instead of paying setup and login on every change.

aitlc cdp launch | status | list | stop

Own a long-lived debug Chrome that outlives the shell that started it.

aitlc cdp launch          # detached, mobile-sized by default
aitlc cdp status          # is it actually answering?
aitlc cdp list            # every tracked instance, alive or dead
aitlc cdp launch --new    # isolated: own port + own profile
aitlc cdp stop --all

Backgrounding Chrome from a shell ties it to that shell; when the shell exits, the next attach fails with a bare ECONNREFUSED.
Liveness is the port, not the PID — list shows tracked-but-dead instances as "running": false rather than hiding them.
stop verifies the PID is still our Chrome before signalling, so a stale state file can't kill an unrelated process group.

Use --new when several browsers must run at once. Separate profiles matter, not just ports: a shared profile directory corrupts under concurrent Chromes and leaks cookies between tests.

aitlc steps run <ID> --range A-B

Resume a scenario partway through, in an already-open browser. This replaces commenting out the steps that already passed.

aitlc cdp launch
# setup + login once (~2 min)
aitlc steps run PROJ-24026 --range 6-13 \
      --cdp-url http://127.0.0.1:9333 --mobile "Galaxy S8"

# iterate on later steps in the SAME session — seconds per cycle
aitlc steps run PROJ-24026 --range 14-19 \
      --cdp-url http://127.0.0.1:9333 --mobile "Galaxy S8" \
      --scenario-setup none

Use --scenario-setup none after the first slice. Setup mints fresh per-scenario data; re-running it on a resume invalidates the session the earlier slice established.

A step slice gets no before_scenario — which is where most suites generate per-scenario data. aitlc invokes your real hook via scenario_setup and stops immediately if it fails, rather than running on into a failure that surfaces several steps later looking like an app bug.

--trace out.zip writes a Playwright trace — screenshots and DOM snapshots per action — which aitlc trace show opens. Tracing is otherwise only produced on the remote grid, so a local session leaves no timeline. --capture-network records the API responses seen during the slice: a slice gets no before_scenario, so any collector your suite installs there is absent.

Agent use

Emits JSON-lines: one object per step with status, duration_s and error, plus a scenario_setup record showing what setup actually produced. --range 31- means "line 31 to the end". A run that reaches no steps is an error, not an empty success — the child's stderr is surfaced and the exit code is non-zero.

Reading a page cheaply

Answer "is X on screen" as structured text instead of pixels.

aitlc cdp inspect --a11y

aitlc cdp inspect --cdp-url http://127.0.0.1:9333 --a11y
aitlc cdp inspect --cdp-url ... --a11y --a11y-query "Apply filters"
aitlc cdp inspect --cdp-url ... --a11y --a11y-selector "#panel"
aitlc cdp inspect --cdp-url ... --check "#saveBtn,//button[text()='Close']"

Form	Size	Assertable?
Screenshot PNG	~55 KB	needs vision
Full a11y tree	1,961 chars	yes — text
Targeted query	20 chars	yes — text

The tree also carries what a screenshot cannot express: nesting, control state ([expanded]), and field values (textbox "Search filters": City). Built on page.aria_snapshot(); page.accessibility was deprecated for three years then removed, so aitlc falls back to the CDP Accessibility domain only on older Playwright.

Agent use

The cheapest way to check presence, text or state. Start with --a11y-query; the response reports chars and full_chars so you can see what the query saved and tighten it. Reach for a screenshot only when the question is genuinely visual — layout, overlap, styling.

Suite health

Two things behave gives you no answer for on its own.

aitlc steps unused

Report step definitions that no feature file uses. behave has no equivalent of Cucumber's unused-step report.

aitlc steps unused
aitlc steps unused --no-include-composite   # Cucumber's false positives

Matching goes through behave's own registry, so the answer agrees with what the runner would dispatch. Steps invoked via context.execute_steps(...) are extracted from the AST and counted as used.

Check the corpus before believing the number. The result is only as complete as the feature files it can see. Where canonical Gherkin lives in a test manager, most steps look dead — measured here, 51 local feature files reported 83% of definitions as unused. The command warns when the ratio is implausible. Treat that as "incomplete corpus", not "delete these".

aitlc history show

Flake rate from observed outcomes, not hand-written signatures.

aitlc history show --flaky-only
aitlc history show --last 200
aitlc history clear

aitlc run appends every outcome automatically, so a new flake is visible the second time it happens. Flaky means it has both passed and failed — a test that has only ever failed is broken, and retrying it just spends time to reach the same answer.

Agent use

Consult before retrying. is_flaky plus flake_rate are stronger evidence than a signature match, because they come from what actually happened in this repo.

Xray sync

Xray stores only the step body — no Feature: line, no tags, no Scenario: header, no comments.

aitlc xray get-gherkin PROJ-24026
aitlc xray compare-gherkin PROJ-24026            # local vs live
aitlc xray update-gherkin PROJ-24026 --file body.txt
aitlc xray fetch-features PROJ-29026 --status FAILED
aitlc xray find-step-usage "click on audience tab"

compare-gherkin normalizes your local file before diffing so it compares like for like. update-gherkin re-fetches after writing to confirm the write persisted — the mutation echoing your input back is not proof. fetch-features resolves a whole Test Execution or Plan into runnable feature files: the "everything that failed last night" entry point.

Evidence

Escalate in order — a single frame explains most failures, and the interactive viewer costs far more to open and read.

aitlc trace extract-frame trace.zip   # last frame as an image (cheap)
aitlc trace show trace.zip            # full interactive viewer (expensive)
aitlc s3 report-summary               # counts + failures, no download
aitlc report <TEST-ID>                # run + replayable terminal recording

Journal & cache

Answer a follow-up question by reading, not by running the thing again.

aitlc journal list | show | diff | cache

Every run is recorded — argv, exit code, duration, payload — and fetched reports are cached by their source key.

aitlc journal list --last 5 --command run
aitlc journal show 20260817T104512-run
aitlc journal diff <earlier> <later>      # did the fix work, or was that luck?
aitlc journal cache                        # size on disk; --clear empties it

Payloads pass through the same redaction used for command output before touching disk — a journal that writes tokens into an unremarkable directory is worse than no journal.
Entries are size-capped and pruned by count; cache keys are hashed, so two test plans holding the same execution filename cannot collide.

Agent use

diff is the cheap answer to "is this flaky or fixed": two runs of the same command, compared, with no browser involved.

Locator hygiene

Selector shapes that pass, read the wrong element, and fail somewhere else entirely.

aitlc locators lint | rules

Advisory, never fatal. Each finding carries the rewrite, not just the diagnosis.

aitlc locators lint                    # high severity only
aitlc locators lint --severity low     # everything, deliberately noisy
aitlc locators rules                   # what is checked, and why

positional-index — aria-rowindex='2' follows whatever sorts first, not the record you meant.
grid-cell-without-role — a grid header carries data-field too, so the selector can return the column title where a value was expected.
unanchored-xpath — //* or //div with nothing pinning it; with .first this silently takes an element from another container.

A relative predicate — //div[@role='row'][.//div[@role='cell' …]], the xpath analogue of locator.filter(has=…) — counts as anchored. That is the shape the lint steers toward, so it is never flagged.

Escape hatches

aitlc's own commands cover the paths worth making easy. These cover everything else, so adopting aitlc never means losing a flag it doesn't wrap.

aitlc behave --aitlc-debug features/x.feature   # any behave args, forwarded
aitlc behave --print-command <args>             # show command, run nothing
aitlc pw show-trace trace.zip
aitlc pw codegen https://example.com
aitlc pw install chromium

What they add over typing behave directly: your .env loaded first, the right interpreter and working directory, and aitlc's optional instrumentation. aitlc's own flags are prefixed --aitlc- so they can never collide with a behave flag, now or in a future release.

Agent use

--print-command prints the exact invocation as JSON. Use it to hand a reproduction to someone without aitlc, and to state what you're about to run before running it.

Everything else

Command	Purpose
`aitlc start`	Bootstrap briefing for a fresh agent or engineer
`aitlc classify-failure`	Match a failure against `patterns.yaml`
`aitlc propose-fix`	Assemble the evidence needed to propose a fix
`aitlc record --suggest-steps`	Record a session, diff selectors against existing locators
`aitlc notify-teams`	Post a run summary to a Teams webhook
`aitlc jira create-task`	Create a Jira Task
`aitlc tunnel status \| restart`	LambdaTest tunnel health
`aitlc users validate \| generate`	Pooled test-user maintenance — requires `--yes`

aitlc users acts on a shared pool and refuses to run without --yes; the underlying scripts have no prompt and no dry run.

Troubleshooting

`--debug` did nothing

Check the instrumentation line printed to stderr. If it says sitecustomize, aitlc couldn't use behave's runner option — the detail field distinguishes "this build has no such option" from "behave couldn't be probed at all" (usually a missing virtualenv).

Instrumentation silently ignored

Instrumentation flags must precede the positional feature path. behave's positional argument ends option parsing, so anything after it is treated as another path.

`No module named 'aitlc.runtime.runner:AitlcRunner'`

A path-format mismatch, not a missing install — behave 1.2.7 needs the dotted form. aitlc picks the right shape per detected flag, so seeing this means the flag was overridden by hand.

A step slice fails on data-dependent steps

scenario_setup is unset, so no per-scenario data exists. The scenario_setup record in the output reads skipped when this happens.

Setup

Install

Generate aitlc.toml

What each [project] setting controls

Verify

How it stays codebase-independent

Running tests

The debugging cycle

Debugging live

Reading a page cheaply

Suite health

Xray sync

Evidence

Journal & cache

Locator hygiene

Escape hatches

Everything else

Troubleshooting

--debug did nothing

Instrumentation silently ignored

No module named 'aitlc.runtime.runner:AitlcRunner'

A step slice fails on data-dependent steps

Generate `aitlc.toml`

What each `[project]` setting controls

`--debug` did nothing

`No module named 'aitlc.runtime.runner:AitlcRunner'`