Skip to content

Easy Setup - 2026-11-01

Context

A team with tests and CI already had to read Getting Started, write qasight.yaml by hand, find the right result-file flag for each runner and write a workflow before seeing anything. qasight demo and the /try page show what QASight does on sample or pasted results; nothing set up the team's own project. Four changes were approved together, each its own pull request:

  1. qasight init detects the stack, writes the config and the CI workflow, and records a first run.
  2. The GitHub Action works with no config file: it finds result files itself and posts the pull request comment.
  3. Copy-paste setup for each stack and each CI system (GitHub Actions, GitLab CI, Jenkins, CircleCI).
  4. A dry run that shows the pull request comment the last results would get.

Decision

  • One place knows the stacks. qasight/services/project_setup.py reads project files only (nothing is run): detect_project() returns a Stack per runner with the qasight.yaml stage that runs it so its results are read (native equipment for pytest, Jest, Vitest and Playwright; a command stage with junit_xml for Go, .NET, Maven, Gradle, RSpec and PHPUnit) and how to install its dependencies. find_result_files() finds results a test run left behind by content, skipping files git tracks (fixtures) and dependency folders. qasight/services/ci_templates.py turns the same facts into CI files. qasight init, the Action's zero-config mode and the docs snippets all use these, so they can't disagree.
  • qasight init writes, then records. It writes qasight.yaml, .github/workflows/qasight.yml and missing .gitignore lines, keeps any file that exists unless --force, and records a first run (existing results through the ingest path, else one qasight run). A failing test still counts as recorded (exit 0). The old wizard stays as qasight init --interactive. Reference: Command Reference ยง qasight init.
  • The Action needs no config. integrations/github-action/run_testo_action.py picks qasight ingest --ci over qasight run --ci when the workspace has no config or the results input is set, and qasight ingest with no file arguments uses find_result_files(), so the Action, a GitLab job and a laptop find results the same way. comment-on-pr: auto is on in that mode only, so configured users who never asked for a comment don't start getting one. History is the part a fresh runner lacks: a prepare-history step points DATABASE_URL at a SQLite file under ~/.cache/qasight and the composite action restores and saves it with actions/cache (new key per run and step, restored by workflow-and-job prefix), unless the user already set a database. Only the database is cached so the cache doesn't grow with report copies.
  • Snippets are generated. ci_templates.py also renders GitLab CI, Jenkins and CircleCI files: the project's own test command with a result-file flag (ci_test_command()), then qasight ingest after it whatever the outcome (after_script, post { always }, when: always), with the history cached per branch where the CI has a cache. Outside GitHub, qasight is installed with uv so a Node, Go or JVM image needs no Python. Setup Snippets is written by scripts/export_setup_snippets.py from those functions for a typical project of each stack, and tests/contract/ci/test_setup_snippets.py fails when it is stale, so the page shows exactly what qasight init --ci <system> writes. qasight init never overwrites an existing pipeline file; it writes its own next to it.
  • The preview is a real run in a throwaway history. qasight try (qasight/cli/commands/try_cmd.py) doesn't fake the comment: it runs qasight ingest and qasight report markdown as child processes with DATABASE_URL and TESTO_HOME pointing at a temporary folder, as qasight demo does, after copying a local SQLite history in so the "since the last run on main" part is real. A server or Postgres history isn't copied; the preview is then a first run's comment. --from-github uses gh run download rather than its own API client, so sign-in, GitHub Enterprise hosts and tokens are gh's job. It reuses the /try page's idea (see what you'd get before setting anything up) for your own repository instead of a pasted file.
  • A monorepo is scanned, not guessed (qasight init --scan). Plain qasight init reads the project files at the top of the repository only, so api/pyproject.toml, web/package.json and services/billing/go.mod gave one wrong stage or none. qasight/services/repo_scan.py walks the folders (skipping dependency and hidden folders, 6 levels deep), runs the same detect_project() in each folder that has a project file, and writes a stage per runner per folder with target_repo set and the folder in its name (api-pytest). When an outer folder's run already picks a folder up (a Maven parent and its modules, Gradle, a root Jest or Vitest, a bare pytest or pytest tests above it) the outer stage is kept and the inner one is listed as left out. dotnet test starts at a .sln, else in each test project's folder, since it refuses a folder with several projects. CI install and test commands run in their folder in a subshell ((cd api && ...)), because GitLab runs a job's script lines in one shell. --scan prints the stages and asks before it writes; no answer (no terminal) writes nothing, --yes skips the question and --dry-run shows the files.
  • Proof instead of an AI guess (--check). An optional AI pass that refines the proposal was considered and dropped: it reads the same files the scanner reads and can't tell whether a command works. qasight/services/stage_check.py asks each runner to list its tests instead (pytest --collect-only, jest --listTests, vitest list, playwright test --list, go test -list, dotnet test --list-tests, rspec --dry-run, phpunit --list-tests) and counts them. A stage that lists none is left out; one whose runner isn't installed or fails to list is kept and flagged. Listing loads test code (conftest, compilation), so it's opt-in and never runs a test. Node runners run from the project's own node_modules/.bin, so nothing is downloaded. Maven and Gradle have no listing mode that doesn't run tests and are not checked.
  • Generated YAML is text, not a dump. The templates write comments and key order the way a person would, and the tests load the output back (discover_and_load, yaml.safe_load) so it stays valid.

Consequences

  • The detection rules are heuristics. A project that installs test dependencies some other way gets a workflow whose install step needs a tweak; --dry-run shows it before anything is written.
  • Cypress is detected but not added, since it needs the app running first.
  • --scan only looks at a local folder. Scanning a GitHub repository without cloning and running its code (for example from the results its CI already publishes) is a possible later step.
  • Adding a stack means one Stack entry in project_setup.py and, for CI, its language setup in ci_templates.py.