Skip to content

End-to-end tests ​

Pitlane is tested the way it is used: by running real GitHub Actions jobs on a real Pitlane instance. The suite in e2e/ is a set of workflow_dispatch workflows, one per scenario, that each assert their own success, and a script that pushes them to a test repository, runs them and reports the results.

Run it ​

You need a Pitlane instance that serves a test repository, and on your machine bash 4+, git, jq and the GitHub CLI authenticated with push access to that repository (gh auth login).

sh
e2e/run.sh                              # every scenario on SheepHost/pitlane-org-test
e2e/run.sh my-org/pitlane-e2e           # another test repository
e2e/run.sh my-org/pitlane-e2e basic cache   # a selection
e2e/run.sh --twice                      # everything twice, to see warm caches
e2e/run.sh --label pitlane-staging      # another runs-on label
e2e/run.sh --help

The script:

  1. clones the test repository, copies e2e/workflows/*.yml to .github/workflows/ and e2e/fixtures/ to e2e/fixtures/, removes e2e-*.yml workflows that no longer exist here, and pushes to the default branch only if something changed (--no-sync skips this step);
  2. dispatches the scenarios at the same time, except matrix, which runs alone afterwards because it measures how many jobs run in parallel;
  3. waits for every run, then prints one line per scenario run: conclusion, wall time, the longest time a job waited for a runner (which includes waiting for a free slot), the timings the jobs reported, and a link to the run;
  4. exits with status 1 if any run did not succeed.

Use a throwaway private repository: the suite pushes to its default branch and uploads short-lived artifacts and caches. No token is read or printed: git and API calls go through gh's own credential handling.

Scenarios ​

ScenarioWhat it checks
basicA plain job: env precedence (workflow, job, step), GITHUB_ENV, GITHUB_PATH, step and job outputs (multiline too), and real JavaScript actions with hyphenated inputs: actions/checkout fetch-depth: 2, actions/setup-node node-version: 22.11.0, actions/upload-artifact include-hidden-files: true, checked by a dependent job. A runner that loses INPUT_* variables with a dash in their name falls back to each input's default, and these assertions fail.
servicesservices: Postgres and Redis with health checks: both healthy, a SQL and a Redis round trip on localhost, and a step container on the slot's host network.
containerA container: job on node:22-bookworm-slim: steps run inside it, actions/checkout and actions/github-script run in the container, a service is reached by its name on the job network.
dockerdocker buildx build twice: the second build must reuse every layer (under 8 s, CACHED on the slow layer); the first one is reported as warm or cold depending on what the slot built before. docker compose up --wait, requests to the stack, down -v.
cacheactions/cache/save in one job, then in a dependent job a restore by exact key, a restore by restore-keys prefix (cache-hit: false, matched key checked), and a clean miss. actions/setup-node with cache: npm and npm ci on a fixture lockfile.
artifactsTwo jobs upload artifacts (50 small files in nested directories, and 100 MiB); a third downloads one by name and all by pattern, and checks every SHA-256.
matrix4 matrix legs that each hold a runner for 30 s: every leg must run on a distinct runner, and at least 4 (--min-concurrency) must overlap.
logs200,000 log lines (about 20 MB), log groups, add-mask, a failing step with continue-on-error, failure() and always() conditions; the job must succeed.
multiarchdocker/setup-qemu-action and docker/setup-buildx-action, then a linux/arm64 build on an amd64 runner, and the image run under emulation.

Job timeouts are not covered: they depend on the instance's settings.

Expected results ​

Every scenario concludes success. Measured on the instance that serves SheepHost/pitlane-org-test (Pitlane v0.1.1, GitHub's cache backend):

ScenarioFirst run (all dispatched together)Rerun, idle instance
basic141 s36 s
services124 s25 s
container136 s39 s
docker125 s30 s
cache139 s67 s
artifacts154 s
logs141 s
multiarch160 s
matrix (alone)58 s, peak concurrency 4/4

When the scenarios are dispatched together, most of their wall time is jobs waiting for a free slot (80–115 s in the "max queue" column); the jobs themselves take seconds to a minute.

Timings to look at in the report:

  • docker build 2 is 0–1 s. docker build 1 is cold (10–30 s) the first time a slot builds the fixture image, warm after that.
  • cache save / restore of 64 MiB: about 15 s and 4 s.
  • npm ci takes 1–8 s: Pitlane keeps the npm cache on disk, so it is fast even when setup-node reports cache-hit=false on a first run.
  • arm64 build under QEMU: about 16 s for the fixture image.
  • matrix: the 4 legs start within a second of each other.

multiarch registers QEMU binfmt_misc handlers, which are global to the host kernel. On a host where that is not allowed (some unprivileged containers), it fails at setup-qemu-action: exclude it with a scenario selection.

Add a scenario ​

  1. Add e2e/workflows/e2e-<name>.yml with on: workflow_dispatch, a label input and runs-on: ${{ inputs.label || 'pitlane' }} on every job.
  2. Make the workflow fail on its own when the behaviour breaks: assert with test, not by eye.
  3. Report a timing with echo "::notice title=e2e-timing::<what>: <n>s"; the script collects these annotations.
  4. Add <name> to ALL_SCENARIOS in e2e/run.sh and a row to the table above. Pin every action by full commit SHA, as in the other scenarios.

Released under the MIT License.