Troubleshooting and Error Codes¶

Command Reference

Practical guide for debugging qasight failures: exit codes, CI payloads, log locations, and step-by-step fixes. Implementation detail: Deep Dive - Execution Logic.

Exit code reference¶

Defined in qasight/engine/exit_codes.py as EngineExitCode. The process exit code is propagated via typer.Exit in command handlers and qasight/cli/app.py main().

Code	Enum	Meaning	Typical causes
`0`	`SUCCESS`	Run completed; all stage return codes zero	Pass; or cycle skipped by trigger (resting)
`1`	`DOMAIN_FAILURE`	Tests or framework reported failure	Non-zero pytest/behave/behavex exit
`2`	`INVALID_INPUT`	Config or CLI validation failed	Missing YAML, bad schema, unknown cycle, tag mismatch; a mistyped command or option, or bare `qasight` (prints the error or the help, no traceback)
`3`	`INFRA_FAILURE`	Infrastructure / runtime dependency failure	Return code `127` (missing binary); `124` (stage timeout); report DB archive failure; Docker/Allure CLI failures
`4`	`INTERNAL_ERROR`	Unexpected engine error	Orchestrator `internal_failure` on uncaught engine exception; empty return-code list in classifier
`130`	`INTERRUPTED`	Stopped by Ctrl-C (SIGINT)	The running stages were stopped, later stages skipped, and the run recorded as failed with `interrupted_by: SIGINT` in its metadata
`143`	`TERMINATED`	Stopped by SIGTERM	A CI job's cancel button or `kill`; handled like `130`

A run killed outright (kill -9, out of memory, the machine or the API server going down) can't record itself. Its marker stays in the cycle's lock file (<artifacts>/<cycle>/.lock), and the next run of that cycle, or the API server when it starts, records it as a failed run with error: interrupted and exit 3 ("stopped without finishing"). It has no test results.

Classification logic¶

classify_exit_code(returncodes, infra_error=None, internal_failure=False):

infra_error set → 3
internal_failure set → 4 (orchestrator caught an unexpected engine exception; not a framework subprocess exit)
Empty returncodes → 4
Any stage return code in (124, 127) → 3
Any other non-zero return code → 1
Otherwise → 0

Stage timeouts emit returncode=124 and timed_out=true. Each host stage runs in its own process group, so a timeout stops everything the stage started (a server it launched, pytest-xdist workers, browsers), not just the direct child. A stage stopped by a signal gets returncode 130 or 143 and error: stopped by SIGINT (or SIGTERM); 130 / 143 come from run_plan, never from classify_exit_code. A second Ctrl-C kills the stages outright, a third raises KeyboardInterrupt. Engine internal failures set internal_failure=true on the stage result (NDJSON stage_finished includes both fields).

Commands that use exit codes¶

Command	Uses `EngineExitCode`
`qasight run`	Yes — engine result
`qasight ingest`	Yes — engine result locally; with `--server`, the server's result, or 3 when the upload fails (unreachable, wrong token)
`qasight report`	Yes — missing artifacts, Allure failure, I/O errors
`qasight doctor`	2 on hard FAIL; 0 on pass
`qasight config validate`	2 on failure (literal in some handlers)
`qasight clean`	2 / 3
`qasight config db`	2 / 3

Error output channels¶

Interactive (default)¶

Config errors: [fail]error: <message>
Run failure: Run exited with code <N>.
Success: Run finished successfully.
Trigger skip: Rich panel titled Resting
Stage failure: post-mortem panel with output_tail and log path

CI (`--ci`)¶

All machine-readable lines go to stdout as NDJSON (one JSON object per line). No Rich markup.

Config error:

{"event":"error","code":"invalid_input","message":"cycle 'foo' not found. Available: bar, baz."}

Trigger evaluation:

{"event":"cycle_trigger","cycle":"sample-pytests","status":"resting","reason":"...","matched":[],"mode":"git"}

Dry run skip (roadmap — no --dry-run CLI flag exists yet, see Command Reference):

{"event":"dry_run","cycle":"sample-pytests","status":"skipped","reason":"trigger"}

Stdout is reserved for NDJSON in CI mode — do not pipe other tools onto the same stream without filtering.

Artifact mirror¶

artifacts/<cycle>/events.ndjson duplicates engine events (plan_started, stage_started, stage_finished, plan_finished, plan_aborted) even in interactive mode. Use this file when CI logs are truncated.

NDJSON event schema (`qasight run --ci`)¶

Emitted by CIRenderer and _NdjsonRecorder.

`plan_started`¶

{"event":"plan_started","plan":"sample-pytests","stage_count":1}

`stage_started`¶

{"event":"stage_started","stage":"pytest-sample","framework":"pytest","index":1,"count":1}

`stage_finished`¶

{
  "event":"stage_finished",
  "stage":"pytest-sample",
  "framework":"pytest",
  "returncode":1,
  "duration_s":12.4,
  "log_path":"/path/artifacts/sample-pytests/pytest-sample/run.log",
  "timed_out":false
}

When the executor sets an error string (timeout, missing binary):

error may appear on the in-memory result; the NDJSON mirror in orchestrator includes error when present on StageResult. The mirror also carries internal_failure (true only for orchestrator-caught engine exceptions). CIRenderer stdout omits both keys.
crash_kind ("stack_overflow", "crash" or null) names a failed stage that died of a stack overflow or a crash; the orchestrator mirror adds crash_detail, the line that showed it. See Stack overflow or crash.

`plan_finished`¶

{
  "event":"plan_finished",
  "plan":"sample-pytests",
  "aggregate_returncode":1,
  "exit_code":1,
  "duration_s":12.5,
  "stages":[...],
  "error":null
}

`plan_aborted` (engine `fail_fast`)¶

Written to the artifact events.ndjson when run_plan(fail_fast=True) aborts after a failing stage. Reachable today via the API layer (POST /api/v1/cycles/{cycle}/executions); the --fail-fast CLI flag is a roadmap item.

{"event":"plan_aborted","plan":"sample-pytests","reason":"fail_fast","completed_stages":1}

A run stopped by Ctrl-C or SIGTERM writes the same event with "reason":"signal" and "signal":"SIGINT" (or "SIGTERM"), then plan_finished with exit_code 130 or 143.

A run stopped from the UI's Stop button (POST /api/v1/cycle-executions/{id}/cancel) writes "reason":"stopped" with "signal":"SIGTERM" and exits 143; its stages' error says stopped by a stop request, the recorded run is failed with interrupted_by: a stop request, and the execution's status is cancelled. Only that run's stages are stopped, not other runs on the same server. A run stopped while still queued writes no events: its event stream sends {"event":"error","code":"cancelled"}.

Log and artifact map¶

artifacts/<cycle>/
  events.ndjson              # Full event timeline
  plan_result.json           # Summary JSON (best-effort)
  <stage>/
    run.log                  # Complete subprocess stdout+stderr
    allure-results/
      pytest/ | behave/ | behavex/
        *-result.json

Post-run reports (when configured): often under reports/allure/, reports/extent/, etc. — see qasight.yaml reporters:.

Report DB archives: require qasight[db] and database.url or DATABASE_URL; UUID from qasight report list.

Failure-point playbook¶

A module is missing (exit 2)¶

No module named 'fastapi'. This needs the api extra (fastapi): pip install 'qasight[api]': a command needs an optional extra that isn't installed. qasight's own metadata names the extra, so the line is the fix. For a core dependency (yaml, typer, ...) it says to pip install --force-reinstall qasight: the environment is broken, often by an interrupted upgrade. A module qasight doesn't depend on still shows the full traceback.

Config not found (exit 2)¶

Symptoms

error: no qasight.yaml / qasight.yml / pyproject.toml [tool.qasight] found under /your/cwd

Debug

Confirm working directory: pwd
Or pass explicit path: qasight run --cycle <name> --config /path/to/qasight.yaml
CI: parse NDJSON {"event":"error","code":"invalid_input",...}

Fix

Add qasight.yaml at repo root or set --config
Use qasight config init / qasight init to scaffold

Invalid YAML or schema (exit 2)¶

Symptoms

invalid YAML in ...
unknown equipment 'foo'
Pydantic-style validation messages from loader

Debug

qasight config validate
qasight config validate --config path/to/qasight.yaml

Fix

Correct syntax and keys per Architecture Overview § Configuration as the single source of truth
Supported frameworks: pytest, behave, behavex

Unknown cycle or no enabled stages (exit 2)¶

Symptoms

cycle 'x' not found. Available: ...
the config defines several cycles; pick one with --cycle. Available: ... (qasight run with no --cycle)
Error: No such command 'x'. or Error: No such option: --x after a Usage: line (a mistyped command or flag; also exit 2)
plan 'x' has no stages enabled in this environment

Debug

qasight cycles list
qasight cycles show <name>

Fix

Use a defined cycle name from cycles: in YAML
Check stage if_expr — stage may be disabled for current env vars

Tag filter mismatch (exit 2)¶

Symptoms

cycle 'foo' does not include tag 'smoke'
No cycles match --tag 'smoke'

Fix

Add tag to cycle in YAML, or run without --tag
For qasight run --cycle all --tag smoke, ensure at least one cycle lists smoke under tags:

Trigger skip — resting (exit 0)¶

Symptoms

Panel: Cycle <name> skipped: No stimulus detected...
NDJSON: "status":"resting" on cycle_trigger
No stages run; no new failures

Debug

Inspect trigger: block on the cycle (paths, since_ref)
Snapshot state under artifacts/ for the cycle

Fix

Expected behavior when no matching file changes
Force run: qasight run --cycle <name> --force

Framework executable missing (exit 3 or 1)¶

Symptoms

Stage error: executable not found: ...
Stage returncode: 127 → should yield exit 3 at plan level
qasight doctor shows CLI pytest FAIL — not found on PATH

Debug

qasight doctor
cat artifacts/<cycle>/<stage>/run.log

Fix

Install framework in active venv: pip install pytest; for behave / behavex stages, pip install 'qasight[bdd]' (since 1.2 a plain pip install qasight leaves them out, and the stage error says so)
Ensure CI image activates the same venv before qasight run

Stage timeout (exit 3)¶

Symptoms

timed_out: true and returncode: 124 in stage_finished NDJSON / events.ndjson
error: stage exceeded timeout_s=<N>
Truncated or abrupt end in run.log
Process exit code 3 when timeout is the only failure

Debug

# Inspect event file
grep timed_out artifacts/<cycle>/events.ndjson
tail -100 artifacts/<cycle>/<stage>/run.log

Fix

Increase timeout_s in defaults: or per-stage in YAML
Fix hanging test; use qasight run --stream locally to watch live output

Stack overflow or crash (exit 1)¶

Symptoms

A red Stack overflow or Crashed badge on the stage or the failing test on the run page, and 💥 stack overflow / 💥 crashed in the PR comment
crash_kind / crash_detail on the stage (stage_finished NDJSON, plan_result.json, the run record), a crash: line in the CLI's stage panel, and [qasight] crash detection: … at the end of run.log

QASight looks for each runtime's own stack overflow line (Python RecursionError, Java/Kotlin StackOverflowError, JavaScript Maximum call stack size exceeded, Go goroutine stack exceeds, Rust has overflowed its stack, .NET Stack overflow., Ruby SystemStackError, AddressSanitizer stack-overflow), for a native crash (Segmentation fault, AddressSanitizer SEGV), and for a process killed by SIGSEGV, SIGBUS, SIGILL, SIGABRT or SIGFPE (or Windows 0xC00000FD / 0xC0000005). Only failed stages and failed tests are flagged. The table is in qasight/engine/crash_signatures.py; why it works this way: Crash Detection - 2026-10-31.

Fix

Stack overflow: look for recursion with no base case, or two functions calling each other, in the repeating frames of the trace
Crash: run the test under a debugger or AddressSanitizer; a crash is a bug in native code (or a C extension), not a failed assertion

Test / assertion failures (exit 1)¶

Symptoms

Non-zero stage returncode (often 1)
Framework traceback in run.log
Allure *-result.json with failed status

Debug

qasight run --cycle <name> --stream    # live output
qasight report --cycle <name>          # Allure dashboard

Fix

Address failing tests in target_repo
Use --fail-fast to stop after first failing stage

Allure Report 3 CLI missing (exit 3 on report)¶

Symptoms

qasight report prints failure from AllureReporter
qasight doctor shows Allure Report 3 CLI WARN

Debug

node --version
qasight doctor
ls node_modules/.bin/allure   # after npm install in repo root

Fix

Install Node.js 18+, then in the QASight repo root: npm install (installs the allure npm package pinned in package.json)
Or set TESTO_ALLURE_BIN to a custom Allure 3 executable
Or use qasight report --format json which does not need the Node CLI

Command mapping (Allure 2 → 3)

Allure 2 (Java)	Allure 3 (Node)
`allure generate <dirs> -o <out> --clean`	`allure generate <dirs> -o <out> --config ./allurerc.mjs`
`allure generate … --single-file`	`allure awesome <dirs> -o <out> --single-file`
`allure serve <dirs>`	`allure open <dirs> --port <n>` (generate + serve)
`allure open <reportDir>`	`allure open <reportDir> --port <n>`

Hosted reports: {ALLURE_SERVER_URL}/reports/<run_id>/index.html (any static file server).

No artifacts for report (exit 2)¶

Symptoms

no results found under artifacts — run qasight run --cycle … first

Debug

ls -la artifacts/<cycle>/
find artifacts -name '*-result.json' | head

Fix

Run the cycle first: qasight run --cycle <name>
Match --artifacts path if non-default
Reporters skipped when no Allure JSON exists (see run_configured_reporters)

Run Detail page shows "No artifacts recorded" / "No reports for this run" despite a passing cycle: fixed 2026-07-06 — DbBackend.persist() never set snapshot_dir on the RunRecord, so GET /api/v1/runs/{run_id}/reports always returned artifact_links: []. See Report Links and Artifacts Missing Fix - 2026-07-06. Native Allure/Behave HTML static_links still require a reporters: block in qasight.yaml — that part is a config gap, not a bug.

Report database / archive failures (exit 3)¶

Symptoms

qasight report list fails or doctor DB row FAIL
No UUID printed after run when archive expected

Debug

qasight doctor
echo "$DATABASE_URL"
qasight config db show    # if configured

Fix

Install extra: pip install 'qasight[db]'
Set database.url in YAML or DATABASE_URL
In CI, archive always runs synchronously (--ci ignores --async-report-db)
A failed required archive after a green run exits 3
Use --no-report-db to skip archive when DB not needed

Reporter partial failure (exit 0)¶

Symptoms

Tests passed but Slack/ReportPortal/Extent missing
[fail] lines from reporter factory but run still exits 0

Debug

Check reporter-specific env vars (REPORTPORTAL_TOKEN, webhooks in YAML)
Inspect reports/ output dirs

Fix

Validate reporters: block in qasight.yaml
Run qasight report separately to isolate Allure vs integrations

Argparse / Typer help (non-engine)¶

Symptoms

Typer prints help when required options missing (e.g. qasight run without --cycle)
May not use EngineExitCode — often exit 2 from Click/Typer

Fix

Pass required flags per Command Reference

Removed: `uqo run` headless contract¶

v1.0 shipped a second CLI contract, uqo run --config <runs.yaml> with --ghost, --json and --stream-json, backed by a Docker-based runner. It was removed in v1.1 together with the uqo command, and CI integrations use qasight run --cycle <name> --ci (NDJSON events above, plan_finished last). Exit codes 0–4 mean the same thing in both.

Quick diagnostic commands¶

qasight doctor
qasight config validate
qasight cycles list
qasight run --cycle <name> --dry-run
qasight run --cycle <name> --ci 2>/dev/null | tee run.ndjson
qasight run --cycle <name> --stream

Command Reference — flags and subcommands
Deep Dive - Execution Logic — session lifecycle and race conditions
Technical Debt Tracker — known contract gaps and refactors
QA Strategies — triggers and CI patterns

Troubleshooting and Error Codes¶

Exit code reference¶

Classification logic¶

Commands that use exit codes¶

Error output channels¶

Interactive (default)¶

CI (--ci)¶

Artifact mirror¶

NDJSON event schema (qasight run --ci)¶

plan_started¶

stage_started¶

stage_finished¶

plan_finished¶

plan_aborted (engine fail_fast)¶

Log and artifact map¶

Failure-point playbook¶

A module is missing (exit 2)¶

Config not found (exit 2)¶

Invalid YAML or schema (exit 2)¶

Unknown cycle or no enabled stages (exit 2)¶

Tag filter mismatch (exit 2)¶

Trigger skip — resting (exit 0)¶

Framework executable missing (exit 3 or 1)¶

Stage timeout (exit 3)¶

Stack overflow or crash (exit 1)¶

Test / assertion failures (exit 1)¶

Allure Report 3 CLI missing (exit 3 on report)¶

No artifacts for report (exit 2)¶

Report database / archive failures (exit 3)¶

Reporter partial failure (exit 0)¶

Argparse / Typer help (non-engine)¶

Removed: uqo run headless contract¶

Quick diagnostic commands¶

Related notes¶

CI (`--ci`)¶

NDJSON event schema (`qasight run --ci`)¶

`plan_started`¶

`stage_started`¶

`stage_finished`¶

`plan_finished`¶

`plan_aborted` (engine `fail_fast`)¶

Removed: `uqo run` headless contract¶