End-to-end tests
Pitlane is tested the way it is used: by running real GitHub Actions jobs on a real Pitlane instance. The suite in e2e/ is a set of workflow_dispatch workflows, one per scenario, that each assert their own success, and a script that pushes them to a test repository, runs them and reports the results.
Run it
You need a Pitlane instance that serves a test repository, and on your machine bash 4+, git, jq and the GitHub CLI authenticated with push access to that repository (gh auth login).
e2e/run.sh # every scenario on SheepHost/pitlane-org-test
e2e/run.sh my-org/pitlane-e2e # another test repository
e2e/run.sh my-org/pitlane-e2e basic cache # a selection
e2e/run.sh --twice # everything twice, to see warm caches
e2e/run.sh --label pitlane-staging # another runs-on label
e2e/run.sh --helpThe script:
- clones the test repository, copies
e2e/workflows/*.ymlto.github/workflows/ande2e/fixtures/toe2e/fixtures/, removese2e-*.ymlworkflows that no longer exist here, and pushes to the default branch only if something changed (--no-syncskips this step); - dispatches the scenarios at the same time, except
matrix, which runs alone afterwards because it measures how many jobs run in parallel; - waits for every run, then prints one line per scenario run: conclusion, wall time, the longest time a job waited for a runner (which includes waiting for a free slot), the timings the jobs reported, and a link to the run;
- exits with status 1 if any run did not succeed.
Use a throwaway private repository: the suite pushes to its default branch and uploads short-lived artifacts and caches. No token is read or printed: git and API calls go through gh's own credential handling.
Scenarios
| Scenario | What it checks |
|---|---|
basic | A plain job: env precedence (workflow, job, step), GITHUB_ENV, GITHUB_PATH, step and job outputs (multiline too), and real JavaScript actions with hyphenated inputs: actions/checkout fetch-depth: 2, actions/setup-node node-version: 22.11.0, actions/upload-artifact include-hidden-files: true, checked by a dependent job. A runner that loses INPUT_* variables with a dash in their name falls back to each input's default, and these assertions fail. |
services | services: Postgres and Redis with health checks: both healthy, a SQL and a Redis round trip on localhost, and a step container on the slot's host network. |
container | A container: job on node:22-bookworm-slim: steps run inside it, actions/checkout and actions/github-script run in the container, a service is reached by its name on the job network. |
docker | docker buildx build twice: the second build must reuse every layer (under 8 s, CACHED on the slow layer); the first one is reported as warm or cold depending on what the slot built before. docker compose up --wait, requests to the stack, down -v. |
cache | actions/cache/save in one job, then in a dependent job a restore by exact key, a restore by restore-keys prefix (cache-hit: false, matched key checked), and a clean miss. actions/setup-node with cache: npm and npm ci on a fixture lockfile. |
artifacts | Two jobs upload artifacts (50 small files in nested directories, and 100 MiB); a third downloads one by name and all by pattern, and checks every SHA-256. |
matrix | 4 matrix legs that each hold a runner for 30 s: every leg must run on a distinct runner, and at least 4 (--min-concurrency) must overlap. |
logs | 200,000 log lines (about 20 MB), log groups, add-mask, a failing step with continue-on-error, failure() and always() conditions; the job must succeed. |
multiarch | docker/setup-qemu-action and docker/setup-buildx-action, then a linux/arm64 build on an amd64 runner, and the image run under emulation. |
Job timeouts are not covered: they depend on the instance's settings.
Expected results
Every scenario concludes success. Measured on the instance that serves SheepHost/pitlane-org-test (Pitlane v0.1.1, GitHub's cache backend):
| Scenario | First run (all dispatched together) | Rerun, idle instance |
|---|---|---|
basic | 141 s | 36 s |
services | 124 s | 25 s |
container | 136 s | 39 s |
docker | 125 s | 30 s |
cache | 139 s | 67 s |
artifacts | 154 s | |
logs | 141 s | |
multiarch | 160 s | |
matrix (alone) | 58 s, peak concurrency 4/4 |
When the scenarios are dispatched together, most of their wall time is jobs waiting for a free slot (80–115 s in the "max queue" column); the jobs themselves take seconds to a minute.
Timings to look at in the report:
- docker build 2 is 0–1 s. docker build 1 is cold (10–30 s) the first time a slot builds the fixture image, warm after that.
- cache save / restore of 64 MiB: about 15 s and 4 s.
- npm ci takes 1–8 s: Pitlane keeps the npm cache on disk, so it is fast even when setup-node reports
cache-hit=falseon a first run. - arm64 build under QEMU: about 16 s for the fixture image.
- matrix: the 4 legs start within a second of each other.
multiarch registers QEMU binfmt_misc handlers, which are global to the host kernel. On a host where that is not allowed (some unprivileged containers), it fails at setup-qemu-action: exclude it with a scenario selection.
Add a scenario
- Add
e2e/workflows/e2e-<name>.ymlwithon: workflow_dispatch, alabelinput andruns-on: ${{ inputs.label || 'pitlane' }}on every job. - Make the workflow fail on its own when the behaviour breaks: assert with
test, not by eye. - Report a timing with
echo "::notice title=e2e-timing::<what>: <n>s"; the script collects these annotations. - Add
<name>toALL_SCENARIOSine2e/run.shand a row to the table above. Pin every action by full commit SHA, as in the other scenarios.