Skip to content

Test Labels, Environments and Releases - 2026-11-02

Context

On 2026-11-02 Taltal asked whether QASight misses a key feature next to Allure TestOps and similar tools (TestRail, ReportPortal, Qualflare, BrowserStack, Datadog, Buildkite, Launchable, Xray, Zephyr). The comparison built on Feature Gaps and Quick Wins - 2026-10-31 and Paid Features to Offer Free - 2026-10-31 and found one real gap: QASight didn't know what a test is about. Every tool above keeps a test's feature, story, severity, owner, tags and issue links and builds its trees, filters and dashboards on them. QASight received those labels (Allure results, Playwright tags, JUnit properties) and threw them away.

The comparison ranked five gaps, and Taltal chose to build all of them in waves:

Id Gap Status
N1 Test labels and issue links Built (this note)
N2 Environments per run, pass and flake rate split by environment Built (this note)
N3 Run a chosen set of tests Built (this note)
N4 Releases: runs tagged with an app version, a release page Built (this note)
N5 Which issues have red tests Built (this note)

Manual test case management stays out of scope, as the comparison page says.

Decision

  • One parser. qasight/reporting/allure_cases.py reads labels and links off each Allure result, next to name and status. Labels that only say where a test lives (package, suite, test class, host, thread, framework, language) are dropped; the rest (feature, story, epic, severity, owner, tag, layer and custom ones) are kept as {name, value}, lower-case names, at most 20 per test. Only http(s) links are kept (at most 10), so a javascript: URL in a result never reaches the UI.
  • Importers fill them. A JUnit <testcase>'s own <properties> (pytest's record_property, Surefire) become labels: tag/tags split on commas, an http(s) value becomes a link, and issue/jira/story/requirement/tms without a URL stay an issue label. Playwright annotations whose description is a URL become links (type: "issue" is an issue link). Playwright tags, allure-pytest markers and allure-behave tags were already Allure labels.
  • Stored per test. Revision 0006 adds nullable JSON columns labels and links to test_result. Rows from before keep NULL, read as no labels. Runs recorded before this change gain labels only if their snapshot is parsed again.
  • Read side. TestSummary (test list and test page) and FailingTest (run page) carry labels and links from the latest run. GET /api/v1/analytics/tests takes label= (feature:Checkout, or a bare value such as smoke, case-insensitive) and q= also matches label values. GET /api/v1/analytics/labels?cycle= (services/test_labels.py) counts each label's tests, failing and flaky ones, features first.
  • UI. Label chips and issue links on the run page's failing tests and the test page; a chip links to the cycle page with ?label=, where the Tests card lists only those tests. A "By label" card on the cycle page shows the counts and is hidden when no test has a label. The static demo export writes labels/<cycle>.json; the UI filters by label on the client, so the static site works too.
  • Plain names. The UI and API say labels and links, never a competitor's feature name.

N2: environments per run

  • One string per run. services/run_environment.py stores what a run tested on as run metadata environment, "browser=firefox, os=linux": keys cleaned to lower case and sorted, at most 8 pairs, so two runs on the same environment compare equal as plain strings. No migration: it lives in the run's metadata JSON like the CI provenance.
  • Where it comes from, first to last: --env KEY=VALUE on qasight run and qasight ingest (merged into TESTO_ENV for the process, so the stages and an upload see it too); the TESTO_ENV variable; Allure's own environment.properties in the run's results. POST /api/v1/ingest takes an environment field, which qasight ingest --server fills from the same places. The GitHub Action has an environment input that sets TESTO_ENV. Nothing is filled in automatically (no OS or Python version), because splitting runs nobody asked to split would only weaken flaky detection.
  • Flaky detection keeps environments apart. flaky_service lanes were per branch; they are now per branch and environment, a same-commit conflict needs the same environment, and a failing streak counts runs of the latest run's environment. So a test that always fails on firefox and passes on chromium is a failure on firefox, not flaky. Runs without an environment share one lane, as before.
  • Run failures. A run's failing streak and "new" baseline use runs of the same environment (or with none recorded); a pull request's target-branch baseline prefers the run on the same environment.
  • UI and API. environment on GET /api/v1/runs, the run detail, and each point of the test history; the test history adds environments, pass and fail counts per environment. The run page and the cycle's runs list show it; the test page has a "By environment" card that says "Fails only on browser=firefox" when that's the case.

N3: run a chosen set of tests

  • One selection, many runners. services/test_selection.py holds a selection (test names, and the stage each ran in when it comes from a recorded run) and narrow_plan() turns it into the same cycle with each stage told to run only those tests: pytest node ids (or -k on the names when the stage's own arguments name a path, since node ids next to tests/ would run all of tests/), Playwright --grep with --pass-with-no-tests, Behave and BehaveX --name. Other stages run as they are, with two variables a script or adapter can use: ALLURE_TESTPLAN_PATH (Allure's public test plan file, which Allure's own adapters for JUnit 5, TestNG, Cucumber and pytest read to run only the listed tests) and TESTO_SELECTED_TESTS.
  • Entry points. qasight run --test NAME (repeatable) and --rerun-failed RUN_ID on the CLI; tests and rerun_failed on POST /api/v1/cycles/{cycle}/executions (kept in the run queue's JSON). The UI's run page has "Run these tests again" and the cycle page's label filter has "Run these" for a cycle defined in qasight.yaml; both open the cycle's run panel with the selection shown and a "Run every test instead" way out.
  • Same engine. The narrowed plan goes through CycleRunService and run_plan() like any run; nothing else changes. The run records selected_tests (how many) and selected_from in its metadata.
  • No match is not a failure. pytest exits 5 when it collects nothing; with a selection the executor counts that as passed for the stage, because it only means none of the chosen tests lives there. A selection from a recorded run skips stages with no failing test.
  • Not a scheduler. Nothing here splits tests across machines or calls a CI provider's "re-run failed jobs" (F11 in the paid-features note); the run happens where qasight run or the team server runs.

N4: releases

  • One value per run. services/run_release.py stores the release a run tested as run metadata release ("2.3"). It comes from --release on qasight run and qasight ingest, else TESTO_RELEASE; POST /api/v1/ingest takes a release field and the GitHub Action a release input. Names are letters, digits, ., -, + and _ (up to 60), because they go into URLs and the static demo's file names. Nothing is guessed from git tags: a run without one belongs to no release. Like the environment, it needs no migration.
  • A release is its latest runs. services/releases.py groups every cycle's stored runs by release. A release's pass rate and failing count add up the latest run of each cycle that tested it; its flaky count is the tests flaky detection flags across that cycle's runs of the release. Releases are ordered by when they were first tested, not by parsing version numbers, so 2.10 after 2.9 and date-style names both work.
  • Compared with the release before. Each release's page lists its failing tests, marking the ones that passed (or did not run) in the previous release's latest run of the same cycle as new, and the tests the previous release failed and this one passes as fixed.
  • UI and API. GET /api/v1/releases and GET /api/v1/releases/{release}, release on run items and the run detail. A Releases page in the navigation lists them; a release page shows pass rate, failing and flaky counts, the change from the release before, each cycle's latest run, and the failing and fixed tests. The run page links its release. The static demo export writes releases.json and releases/<name>.json.

N5: which issues have red tests

  • Built on N1. services/test_issues.py groups a cycle's tests by the issue they point at: issue links (type issue or tms: Allure links, Playwright issue annotations, a JUnit issue/jira property holding a URL) and issue labels without a URL (record_property("jira", "PAY-8")). A link and a label with the same name (case-insensitive) are one issue, keeping the link's URL. Plain link links (docs, dashboards) are not issues.
  • State from the latest run. An issue is failing when one of its tests failed in its latest run, else flaky when one of them is flaky, else passing. Failing issues come first. The tracker is never called: nothing here needs a Jira or GitHub token, and an issue's own open/closed state stays in the tracker.
  • UI and API. GET /api/v1/analytics/issues?cycle= and an "Issues" card on the cycle page ("2 issues have red tests"), each issue linking to the tracker and its failing tests to their test page. Hidden when no test points at an issue. The static demo export writes issues/<cycle>.json.

Consequences

  • A suite that already uses Allure, Playwright tags or JUnit properties gets the breakdown with no config change. A plain pytest JUnit file has no labels until tests call record_property("feature", "Checkout") or use allure-pytest.
  • test_result rows are a little larger (two small JSON values per test).
  • N5 builds directly on the issue labels and links stored here; a suite that names no issues sees no Issues card.
  • A run of chosen tests is a run like any other in the history. Its absent tests are simply absent: flaky detection and per-test history skip runs a test isn't in, and Compare lists them as not run.
  • A release page is only as complete as the runs tagged with it: a cycle that never ran with --release 2.3 is not part of 2.3.
  • Environments split flaky lanes only when runs record one. A team that sets TESTO_ENV on some jobs and not others gets an extra "not set" lane until every job sets it.