Deep Dive — Execution Logic¶

Architecture Overview

This note maps how QASight (qasight run) initializes a test session, executes it stage-by-stage on the host, preserves state, and tears down. It is the implementation companion to Architecture Overview and QA Strategies.

Every run uses host subprocesses — no Docker. The Docker-based execution stack (the project's original "UQO" design) that existed until v1.1 is summarized at the end.

Executive summary¶

Property	Behavior
Entry	`qasight run` → Typer → `execute_plan_command`
Config	`qasight.yaml` (or discovery chain)
Execution unit	Cycle (plan) → ordered stages
Stage runtime	One `subprocess.Popen` per stage
Stage ordering	Sequential in `run_plan()` by default; up to `max_parallel_stages` at once when a cycle sets it
Parallelism	Framework-internal only, via `workers` (BehaveX; pytest with pytest-xdist) — see Framework level
Durability	`artifacts/<cycle>/` — logs, NDJSON events, Allure JSON
Exit codes	`EngineExitCode` 0–4 via `classify_exit_code()`

End-to-end lifecycle¶

sequenceDiagram
  participant CLI as testo_run
  participant Runner as cli_runner
  participant Svc as CycleRunService
  participant Config as config_loader
  participant Trig as triggers
  participant Orch as orchestrator
  participant Exec as executor
  participant FW as framework_adapter

  CLI->>Runner: execute_plan_command
  Runner->>Config: discover_and_load / resolve_plan
  Runner->>Svc: run(cfg, plan, renderer, options)
  Svc->>Config: resolve_stages_for_plan
  Svc->>Trig: evaluate_cycle_trigger optional
  Svc->>Orch: run_plan
  loop each stage
    Orch->>Exec: run_stage
    Exec->>FW: build_argv
    Exec->>Exec: Popen + LogBuffer + timeout
    Exec-->>Orch: StageResult
  end
  Orch->>Orch: plan_result.json + events.ndjson
  Svc->>Svc: reporters + optional DB archive + trigger snapshot

Phase map¶

Phase	Module	What happens
1. CLI parse	`qasight/cli/commands/run.py`	Validates flags; defers heavy imports
2. Config load	`qasight/config/loader.py`	`discover_and_load()` → `QASightConfig`
3. Plan resolve	`qasight/config/resolver.py`	`resolve_plan()` / `resolve_stages_for_plan()`
4. Trigger gate	`qasight/services/cycle_run.py` → `config/triggers.py`	Optional skip (exit 0) unless `--force`
5. Renderer pick	`qasight/cli/runner.py`	Buffered / Stream / CI (NDJSON)
6. Engine run	`qasight/engine/orchestrator.py`	`run_plan()` — sequential stages, or parallel with `max_parallel_stages`
7. Subprocess	`qasight/engine/executor.py`	`run_stage()` per stage
8. Post-run	`qasight/services/cycle_run.py`	Reporters, native report snapshot, DB archive, trigger snapshot

Initialization¶

CLI → runner bridge¶

qasight run calls execute_plan_command() in qasight/cli/runner.py:

--cycle all — runs every cycle name (sorted); each trigger evaluated separately; fail_fast can stop after the first failing cycle.
Single cycle — resolve_plan(cfg, plan_name=...).
--tag — filters cycles when using all, or validates tag membership for one cycle.
--dry-run — prints resolved argv/cwd table (or NDJSON dry_run_stage events); no subprocesses.

Each resolved cycle is handed to CycleRunService.run() (qasight/services/cycle_run.py), which owns the trigger gate, the engine call and every post-run step. The API's qasight_api/cycle_execution_manager.py calls the same service (for named cycles and for ad-hoc one-stage runs built by single_stage_plan()), so a run started from the dashboard goes through exactly the same steps; the two callers differ only in the renderer and the listener that presents trigger/archive messages (Rich panels or NDJSON on stdout for the CLI, lines in events.ndjson for the API).

Config errors return exit code 2 (EngineExitCode.INVALID_INPUT). In --ci mode, errors emit:

{"event":"error","code":"invalid_input","message":"..."}

Config discovery order¶

From qasight/config/loader.py (first match wins):

--config PATH
./qasight.yaml
./qasight.yml
[tool.qasight] in ./pyproject.toml

Legacy flat runs: schemas are wrapped into a single plan named default.

Stage resolution¶

resolve_stages_for_plan(plan) applies:

Default timeouts and paths from defaults: in YAML
Per-stage if_expr gating (environment-based enable/disable)
${env:VAR} interpolation in config strings

An empty resolved stage list is a hard error (exit 2).

Renderer selection¶

Mode	Class	`wants_streaming`	Output
Default	`BufferedRenderer`	`false`	Rich progress + post-mortem panels
`--stream`	`StreamRenderer`	`true`	Same panels + live stdout chunks
`--ci`	`CIRenderer`	`false`	NDJSON lines on stdout only

Workers override: _apply_workers_override() clones the plan with workers= set on every stage (only frameworks that use workers act on it).

Session execution (`run_plan`)¶

qasight/engine/orchestrator.py owns the main loop.

Setup (per cycle)¶

Resolve artifacts_root (from config defaults.artifacts_root, default artifacts/).
Create artifacts/<cycle>/ and open events.ndjson (append-only NDJSON).
Copy parent_env (typically os.environ) for all stages.
Emit PlanStarted to the renderer.

Per-stage loop¶

For each stage in order:

Emit StageStarted (renderer + NDJSON stage_started).
Call run_stage() inside a broad try/except — unexpected exceptions become StageResult with returncode=4 and error="internal error: ...".
Append StageResult; emit StageFinished.
Write NDJSON stage_finished (includes returncode, timed_out, error, log_path, quarantined, waived_returncode).
If --fail-fast and returncode != 0, write plan_aborted and break.

Aggregation¶

aggregate_returncode = max(stage returncodes, default=0)
exit_code = classify_exit_code(returncodes, infra_error=None)
Emit PlanFinished; write plan_finished NDJSON.
Best-effort plan_result.json via _try_persist() (JSON snapshot; full DB persistence is not wired here yet).

Health % computation¶

Both persistence backends (qasight/persistence/db_backend.py, qasight/persistence/json_backend.py) call qasight/persistence/health.py::compute_stage_health() before writing the run record. It re-parses each stage's Allure result JSON (via qasight.reporting.allure_results.parse_collected_results, the same machinery the Extent/ReportPortal/TestBeats reporters use) to get real per-test passed/failed/broken/skipped/total counts:

Per-stage health_pct — passed / total * 100 for that stage's own Allure results, stored on each entry in the persisted stages metadata.
Overall run health_pct — a single weighted pass rate: sum of passed across every stage divided by the sum of total across every stage (not an average of the per-stage percentages). This is what the Run Detail page's Summary card and the Dashboard/Runs list health figures show.
Crashed stages — a stage that exited non-zero without producing any results (e.g. it crashed at startup) has no pass rate, so the overall figure is scaled by the share of stages that did not crash: one stage at 100% plus one crashed stage gives 50%, not the 100% the other stage alone would show.
Fallback — if no stage produced any parseable Allure results (empty total everywhere), the overall figure falls back to the older binary estimate (passed_stages / len(stages) * 100, i.e. did each stage subprocess exit 0) so a run with no Allure output still gets an approximate health instead of a bare 0%. Runs persisted before this change only have that binary estimate — stage_health is empty for them, and the frontend shows "Per-stage breakdown not available for this run."

Test time¶

The same parse gives each test's duration (Allure stop - start), so both backends also store test time, summed by run_test_totals() in qasight/persistence/health.py:

test_time_ms on each stage: the sum of that stage's test durations.
metrics_duration_ms on the run: the sum over every test. It is not wall time (duration_s), which also counts collection, fixtures and process start.
avg_case_ms on the run: metrics_duration_ms / total_tests, null when the run has no tests.

Compare shows these as "Test time (sum)" and "Avg per test". Runs persisted before 2026-10-10 have neither key and show n/a there. tests/contract/qasight/test_run_metadata_contract.py fails when qasight/history/views.py reads a metadata key that persistence never writes.

Subprocess execution (`run_stage`)¶

qasight/engine/executor.py is a pure subprocess wrapper.

Layout before spawn¶

artifacts/<cycle>/<stage>/
  run.log
  allure-results/<framework>/   # wiped and recreated each stage

When the run ends, run_plan removes any other stage folder under artifacts/<cycle>/ (one with a run.log or allure-results/) that this run didn't write: a stage removed, renamed, or skipped by if: or fail_fast. So the reporters, archive and history snapshot that read the cycle folder afterwards only see this run's stages. A second run of the same cycle waits for the first (Engine Fixes - 2026-11-01).

Steps:

get_adapter(stage.framework) → PytestAdapter | BehaveAdapter | BehaveXAdapter | PlaywrightAdapter | CommandAdapter (argv = stage.args verbatim) | JunitAdapter (no process, see the post-stage hook below), or a qasight.frameworks plugin adapter
adapter.build_argv(target_repo, results_dir, stage_args, workers)
merged_env(parent_env, stage.extra_env) plus injected vars:
TESTO_SHARED_ALLURE_RESULTS_DIR → Allure output dir
TESTO_ARTIFACTS_ROOT → artifacts root
TESTO_LAST_TEST_TYPE → framework name

extra_env values are read like a shell would want them: YAML true / false become true / false, null becomes an empty string, and a list or mapping is a config error. A stage's if: (evaluated by qasight/config/if_expr.py before the run) is ${env:VAR} alone or one == / != comparison; the operator is found before ${env:...} is replaced, so a value containing == can't change it, and and / or / not / parentheses are a config error.

Process model¶

subprocess.Popen(argv, cwd=target_repo, stdout=PIPE, stderr=STDOUT)
    │
    ├── daemon thread: drain_stream_into_buffer → LogBuffer.feed
    └── main thread: proc.wait(timeout=stage.timeout_s)

LogBuffer — tees all bytes to run.log; keeps a 64 KiB in-memory ring for output_tail (Rich panels).
Timeout — subprocess.TimeoutExpired → SIGTERM → 5s grace → SIGKILL via _terminate(); sets timed_out=True and normalizes returncode=124.
Missing executable — FileNotFoundError → returncode=127, error describes missing binary.

Post-stage hook¶

For behavex, _finish_behavex_reports runs best-effort. The adapter starts BehaveX without --formatter (BehaveX skips report.html whenever one is set), so BehaveX writes its own HTML report and report.json to behave_reports/. The hook then converts report.json into Allure results in allure-results/behavex/ with BehaveX's own Allure formatter (qasight/reporting/behavex_allure.py), and calls ensure_behavex_report_html, which renders report.html only if a stage passed its own --formatter. A failure in either step is logged, appended to run.log and set as the stage's error (unless a timeout or launch error is already there). The exit code is unchanged. See BehaveX Reports - 2026-10-21.

A junit stage skips the subprocess entirely: _record_junit_stage imports its junit_xml reports without the age filter (they predate the stage by definition), sets the returncode to 1 when a case failed or broke or no file matched (0 otherwise), uses the reports' summed test time as the stage duration, and writes what it read to run.log as [qasight] junit: .... The plan's duration_s is the larger of the wall clock and the summed stage durations, so an ingested run shows its test time rather than the milliseconds the import took.

For any stage with junit_xml globs (typically equipment: command), _import_junit converts the matching result files (only files inside target_repo, only files written since the stage started) into Allure *-result.json files in allure-results/<framework>/. JUnit XML gives one result per <testcase>: <failure> → failed, <error> → broken, <skipped> → skipped, otherwise passed. Other formats are recognised by content and read by their own importer (Runner Coverage). It is best-effort, like the BehaveX hook. The process exit code still decides the stage result, and the import outcome (count, or malformed files skipped) is appended to run.log as [qasight] junit_xml: ....

Last, when the plan has quarantine: rules and the process exited 1, qasight/engine/quarantine.py reads the stage's Allure results. If every failed or broken test matches a rule, the stage's returncode becomes 0 and the original goes to waived_returncode; the matched names go to StageResult.quarantined either way and to run.log as [qasight] quarantine: .... The result files are not changed, so the failures still count in health, reports and Compare. See Flaky Test Quarantine - 2026-10-22.

Runtime state preservation¶

What survives across stages¶

State	Mechanism
Environment	`parent_env` dict passed unchanged between stages (plus per-stage `extra_env` overlay)
Cycle artifacts	Same `artifacts/<cycle>/` directory; `events.ndjson` appended
Trigger snapshot	After successful snapshot-mode run, `persist_trigger_snapshot()` writes JSON under artifacts
Allure per stage	Isolated — `results_dir` is `rmtree`'d before each stage

What does not carry over¶

Subprocess stdout (only persisted in per-stage run.log)
In-memory LogBuffer ring (recreated per stage)
Framework process state (each stage is a fresh process)

Event stream durability¶

Two consumers write the same logical events:

Renderer — human UI or CI stdout (CIRenderer → emit_ndjson)
_NdjsonRecorder — append-only artifacts/<cycle>/events.ndjson

plan_result.json is written after the loop completes (best-effort; OSError ignored).

Concurrency and parallelism¶

Orchestrator level — opt-in parallel stages¶

Stages run one at a time, in config order, unless the cycle sets max_parallel_stages: N. Then run_plan() runs up to N stages at once on a thread pool (testo-stage-* threads, each waiting on its stage's subprocess):

Stages start in config order as slots free up; stage_finished events are written as stages end, and PlanResult.stages stays in config order.
Only the calling thread writes events.ndjson and calls the renderer, so both stay consistent without locks.
Output is buffered: each stage's panel prints when it finishes (no live interleaving).
fail_fast starts no new stage after a failure; running ones finish, then plan_aborted is written.
Each stage thread runs in a copy of the caller's contextvars context, so it sees the run's stop scope (below).

Stopping a run¶

qasight/engine/interrupt.py keeps a StopScope per run: the stage processes it runs and what asked it to stop. The CLI uses one process-wide scope, which the SIGINT/SIGTERM handlers pull. Each API run enters its own scope (interrupt.scoped, in CycleExecutionManager), so POST /api/v1/cycle-executions/{id}/cancel terminates that run's stages only (a second call kills them), and the run's next stages are skipped. A queued run is dropped before it starts. Interpreter exit still kills every scope's stages. - Stage names must be unique (each writes artifacts/<cycle>/<stage>/). Stages sharing a target_repo share its caches, ports and databases, so only mark a cycle parallel when its stages are independent. N stages with workers: 4 can start 4N test processes.

See Scale Runs and Stages - 2026-10-24.

Framework level — optional¶

workers: (stage, or defaults:, default 4; CLI --workers overrides every stage) means something different per framework:

Framework	What `workers: N` does
`behavex`	`--parallel-processes N --parallel-scheme feature`, unless `args:` already set them.
`pytest`	`-n N` when N > 1 and pytest-xdist is importable by the `pytest` on PATH (the adapter reads its shebang and probes that interpreter). Without xdist it runs serially: with one warning when the stage or `defaults:` sets `workers`, quietly on the built-in default of 4. A `-n`/`--numprocesses` or `-p no:xdist` in `args:` wins.
`playwright`	`--workers=N`, unless `args:` set `--workers`/`-j` (this replaces `workers` in `playwright.config`).
`jest`, `vitest`	`--maxWorkers=N`, unless `args:` set it (Jest: also `-w`, `--runInBand`/`-i`; Vitest: also `--pool`, `--no-file-parallelism`).
`behave`	Nothing — native behave is single-process. Setting `workers:` on the stage loads fine but logs a warning.
`command`	Nothing — the command owns its own flags (same warning).
`cypress`	Nothing — Cypress runs specs one after another without its paid cloud (same warning).

The API's GET /api/v1/cycles/{cycle} returns workers: null for behave, cypress and command stages, so the UI only shows Workers where it can apply (pytest's is labelled "with pytest-xdist").

Threading in the engine¶

Thread	Role	Daemon
`testo-log-reader-<stage>`	Read subprocess pipe → `LogBuffer`	Yes
`testo-report-archive`	`try_persist_cycle_report()` when `--async-report-db` (non-CI)	No (joined, 30s timeout)
`testo-stage-<n>`	Runs one stage when `max_parallel_stages > 1`	No (the pool is joined before `plan_finished`)
`testo-run-worker-<n>`	API only: `TESTO_MAX_CONCURRENT_RUNS` workers that take queued executions	Yes

The log reader is joined with timeout=2.0 after the process exits. If the reader is still draining, some tail bytes may be lost before the next stage starts.

Multi-cycle (`--cycle all`)¶

Cycles run one after another in sorted name order. No thread pool across cycles.

Teardown¶

Per-stage subprocess¶

Process exits or is killed after timeout.
Log reader joined (2s cap).
LogBuffer closed (flushes run.log).
StageResult returned to orchestrator.

Per-cycle¶

plan_finished event and JSON summary.
Reporters — run_configured_reporters() if YAML reporters: or --reporter set (Allure generate, Extent, ReportPortal, TestBeats).
Report DB archive — try_persist_cycle_report() unless --no-report-db or --no-persist:
Sync (default, and always under --ci) — blocks until archive completes; failure bumps exit to 3.
--async-report-db (interactive only) — background thread joined up to 30s before process exit; failure bumps exit to 3.
Trigger snapshot — if snapshot-mode trigger fired and run succeeded, persist catalog for next diff.

Process exit¶

qasight run raises typer.Exit(code=exit_int). qasight/cli/app.py main() returns that code to the shell.

Architectural bottlenecks and race conditions¶

These are current code behaviors worth knowing for CI design and future refactors. See also Technical Debt Tracker and Troubleshooting and Error Codes.

Issue	Location	Impact
Broad `except Exception` around `run_stage`	`orchestrator.py`	Surprises become synthetic stage failure instead of aborting the plan
Sequential multi-cycle	`execute_plan_command` with `plan_name == "all"`	Wall-clock time grows linearly with cycle count
Per-stage Allure wipe	`executor` `shutil.rmtree(results_dir)`	Correct isolation; cannot accumulate Allure results across retries within one stage without code changes
Log reader join timeout	`executor` `reader.join(timeout=2.0)`	Rare truncated tail in `output_tail` / last lines of `run.log` under heavy I/O
Reporter failures don't fail the run	`ReporterFactory` partial failures	A failed reporter is logged and returned as `ok=False`, but the exit code stays green

Exit classification reference¶

qasight/engine/exit_codes.py:

# internal_failure → INTERNAL_ERROR (4)
# 124, 127 → INFRA_FAILURE (3)
# any other non-zero → DOMAIN_FAILURE (1)
# empty returncodes list → INTERNAL_ERROR (4)

Stage timeouts emit 124 after _terminate(); orchestrator sets internal_failure=True for uncaught engine exceptions (plan exit 4).

Removed: the UQO headless / Docker path¶

Until v1.1 a second stack ran beside the engine: uqo run --config and the API's /api/v1/executions called HeadlessEngineService, which ran each framework in a one-off Docker container via qasight/runners.py and wrote history through run_history.record_completed_run. It duplicated trigger-less execution, persistence and exit-code logic, so behaviour drifted between surfaces. It was removed; what only it provided now lives on the engine path:

Former legacy-only feature	Now
Run one framework without a cycle	`POST /api/v1/adhoc-executions` → `single_stage_plan()` → `CycleRunService`
CI provenance on run records	`DbBackend` + `services/ci_provenance.py`
Failed cases / traceback / log tail for AI summaries	`persistence/failure_context.py` via `DbBackend`
InfluxDB / Prometheus push	`integrations.push_run_metrics_if_configured()` after every cycle

The compose stack (Postgres only) is described in repo ARCHITECTURE.md and is not required for qasight run.

Official documentation¶

Topic	Reference
Allure result format	https://docs.qameta.io/allure/
Docker Compose	https://docs.docker.com/compose/
Compose file spec	https://docs.docker.com/compose/compose-file/

Architecture Overview — module map and artifact layout
QA Strategies — triggers, CI output, typical flows
Command Reference — flags and exit codes
Troubleshooting and Error Codes — failure playbook
Technical Debt Tracker — prioritized refactor backlog