Test WordPress and WooCommerce functionality with GitHub actions

When I found the Reflect.run test automation SaaS tool years ago, they had a free tier that allowed you to visually construct website functionality tests and run them on a regular basis.

I used them to test our newspaper website’s basic functionality that went beyond a “does the page load” check: are articles paywalled appropriately, can you load and submit our contact form, if you have a subscription and log in can you see articles and our print archives, etc. It was pretty slick:

Then the service removed their free tier and my budget didn’t support their asking price, so we didn’t have those tests for a while. A recent bug caused by plugin update reminded me of their usefulness and importance, so I revisited the issue.

I’ve known about Playwright for a while now but have never worked with it directly. Claude helped me think through implementation details and we arrived at a plan for a smoke suite run on GitHub actions. It works pretty well!

Running 5 tests using 1 worker
  ✓  1 [chromium] › tests/wp-contact-form.spec.ts:8:9 › WWN WP: contact form › WWN WP: contact page loads and empty submit shows validation errors (24.5s)
  ✘  2 [chromium] › tests/wp-gift-subscription-cart.spec.ts:12:9 › WWN WP: gift subscription › WWN WP: gift subscription can be added to cart and reaches checkout (24.8s)
  ✓  3 [chromium] › tests/wp-gift-subscription-cart.spec.ts:12:9 › WWN WP: gift subscription › WWN WP: gift subscription can be added to cart and reaches checkout (retry #1) (25.1s)
  ✘  4 [chromium] › tests/wp-metered-content-login.spec.ts:7:9 › WWN WP: metered content › WWN WP: metered paywall gate appears, then login unlocks article (1.2m)
  ✓  5 [chromium] › tests/wp-metered-content-login.spec.ts:7:9 › WWN WP: metered content › WWN WP: metered paywall gate appears, then login unlocks article (retry #1) (58.9s)
  ✓  6 [chromium] › tests/wp-online-access-account.spec.ts:11:9 › WWN WP: online access and account › WWN WP: print edition requires login, subscriber can open an issue and view account (58.0s)
  ✓  7 [chromium] › tests/wp-subscribe-checkout.spec.ts:8:9 › WWN WP: subscribe to checkout › WWN WP: subscription can be added to cart and reaches checkout (17.0s)

First, I set up Playwright (Typescript) test definitions for each thing I wanted to check. For example, here’s the one that tests our contact form:

import { test, expect } from './support/fixtures';
import { NAV, humanPause, assertNotChallenged } from './support/wp';

test.describe('WWN WP: contact form', () => {
    test('WWN WP: contact page loads and empty submit shows validation errors', async ({ page }) => {
        await test.step('Home page -> Contact Us', async () => {
            const response = await page.goto('/', NAV);
            await assertNotChallenged(page);
            expect(response?.ok(), 'WWN WordPress: home page did not load').toBeTruthy();
            await humanPause(page);

            // Contact Us appears in the header, the secondary nav and the footer, plus a hidden mobile copy.
            // Scope to the desktop secondary nav and take the visible one.
            await page
                .locator('#secondary-nav-contain')
                .getByRole('link', { name: /^contact us$/i })
                .click();
            await expect(page).toHaveURL(/\/contact\/?$/);
            await assertNotChallenged(page);
        });

        await test.step('Form is displayed', async () => {
            await expect(page.getByText('Your email address (required)')).toBeVisible();
        });

        await test.step('Empty submit shows validation errors', async () => {
            await humanPause(page);
            await page.locator('#csbutton').click();

            // Inline per-field tips are what users see. Scope to them to avoid the hidden SR summary list.
            const tips = page.locator('.wpcf7-not-valid-tip');
            await expect(tips.first()).toBeVisible({ timeout: 20_000 });
            await expect(tips.filter({ hasText: 'The field is required.' }).first()).toBeVisible();

            // Three required fields on this form: name, email, message.
            await expect(tips).toHaveCount(3);

            // The form-level message is aria-hidden, so assert text rather than visibility.
            await expect(page.locator('.wpcf7-response-output')).toContainText(
                'One or more fields have an error. Please check and try again.',
            );
        });
    });
});

You’ll notice some logic around human-like pauses and detecting whether Cloudflare challenges appear. The Cloudflare problem (love you, Cloudflare, kisses) evolved into the need to try to bypass it in a more comprehensive way for these tests. So, at the Cloudflare domain level, there’s a WAF rule that bypasses all bot detection when a given header and secret value is present in the request.

import { test as base, expect } from '@playwright/test';

const SITE_HOST = new URL(process.env.WWN_WP_BASE_URL ?? 'https://oursite.com').hostname;

export const test = base.extend({
    context: async ({ context }, use) => {
        const secret = process.env.OUR_SECRET;
        if (secret) {
            await context.route(
                (url) => url.hostname === SITE_HOST,
                (route) =>
                    route.continue({
                        headers: { ...route.request().headers(), 'X-Our-Special-Header': secret },
                    }),
            );
        }
        await use(context);
    },
});

export { expect };

Once the tests were written, the GitHub action workflow file pulls it all together:

name: WWN WordPress Monitor

permissions:
  contents: read

on:
  schedule:
    - cron: '30 9 * * *'
  workflow_dispatch:

jobs:
  wordpress-tests:
    name: WWN WordPress site tests
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      OUR_USER: ${{ secrets.OUR_USER }}
      OUR_PASS: ${{ secrets.OUR_PASS }}
      OUR_SECRET: ${{ secrets.OUR_SECRET }}
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm

      - run: npm ci

      - run: npx playwright install --with-deps chromium

      - name: Run Playwright tests
        run: npx playwright test

      - name: Annotate failure for Nagios
        if: failure()
        run: |
          titles=$(jq -r '[.. | objects | select(has("specs")) | .specs[] | select(.ok == false) | .title] | unique | join(", ")' test-results/results.json 2>/dev/null || true)
          msg="WWN WordPress: failed - ${titles:-see run logs}"
          echo "::error title=WWN WordPress::${msg:0:200}"
      - name: Upload reports
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: wwn-wordpress-monitor-reports
          path: |
            playwright-report
            test-results
          retention-days: 14

As it says, the workflow annotates what happens in each run to be useful for monitoring by Nagios, which we use to know when things are broken in our small business infrastructure.

So, then a “simple” Nagios custom check script can ask the GitHub API how things went in the latest run:

#!/usr/bin/env bash
# Nagios plugin: WWN WordPress Monitor (GitHub Actions workflow status, pull model).
# Deps: bash, curl, jq. Exit: 0 OK, 1 WARNING, 2 CRITICAL, 3 UNKNOWN.

PREFIX="WWN WordPress:"
API="https://api.github.com"
REPO=""; WORKFLOW="wordpress-monitor.yml"; TOKEN=""
MAX_AGE=93600; BRANCH="main"; EXPIRE_DAYS=14

usage() {
  cat <<USAGE
Usage: $0 -r owner/repo -t token [-w workflow_file] [-a max_age_seconds] [-b branch] [-e days] [-h]
  -r  owner/repo (required)
  -t  GitHub fine-grained PAT (required, probably from a Nagios $USER var)
  -w  workflow file (default: $WORKFLOW)
  -a  max age in seconds of last completed run (default: $MAX_AGE)
  -b  branch (default: $BRANCH)
  -e  warn if token expires within this many days (default: $EXPIRE_DAYS)
  -h  this help
USAGE
}

unknown() { echo "$PREFIX UNKNOWN - $1"; exit 3; }

while getopts "r:w:t:a:b:e:h" opt; do
  case $opt in
    r) REPO=$OPTARG ;; w) WORKFLOW=$OPTARG ;; t) TOKEN=$OPTARG ;;
    a) MAX_AGE=$OPTARG ;; b) BRANCH=$OPTARG ;; e) EXPIRE_DAYS=$OPTARG ;;
    h) usage; exit 3 ;;
    *) usage >&2; exit 3 ;;
  esac
done

[ -n "$REPO" ] || unknown "missing required option -r owner/repo"
[ -n "$TOKEN" ] || unknown "missing required option -t token"
[[ "$MAX_AGE" =~ ^[0-9]+$ && "$EXPIRE_DAYS" =~ ^[0-9]+$ ]] || unknown "-a and -e must be integers"
command -v curl >/dev/null 2>&1 || unknown "curl not installed"
command -v jq >/dev/null 2>&1 || unknown "jq not installed"

HDRS=$(mktemp) ; BODY=$(mktemp)
trap 'rm -f "$HDRS" "$BODY"' EXIT

# gh_get <path-and-query>; sets HTTP_CODE; body in $BODY, headers in $HDRS.
# Auth header goes through a curl config on stdin so it is not in the process list.
gh_get() {
  HTTP_CODE=$(printf 'header = "Authorization: Bearer %s"\n' "$TOKEN" | curl -sS -K - \
    --max-time 20 -o "$BODY" -D "$HDRS" -w '%{http_code}' \
    -H "Accept: application/vnd.github+json" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    -H "User-Agent: check_wwn_wordpress_monitor" \
    "$API$1" 2>/dev/null) || { HTTP_CODE=000; return 1; }
}

human() { # seconds -> 5h12m / 2h / 14m / 30s
  local s=$1 d h m
  d=$((s/86400)); h=$(((s%86400)/3600)); m=$(((s%3600)/60))
  if [ "$d" -gt 0 ]; then echo "${d}d${h}h"
  elif [ "$h" -gt 0 ]; then if [ "$m" -gt 0 ]; then echo "${h}h${m}m"; else echo "${h}h"; fi
  elif [ "$m" -gt 0 ]; then echo "${m}m"
  else echo "${s}s"; fi
}

gh_get "/repos/$REPO/actions/workflows/$WORKFLOW/runs?branch=$BRANCH&status=completed&per_page=1&exclude_pull_requests=true" \
  || unknown "network error or timeout contacting api.github.com"

case $HTTP_CODE in
  200) ;;
  401) unknown "HTTP 401: token expired or revoked, or lacks access to the repo" ;;
  403|429)
    if grep -qi '^x-ratelimit-remaining: 0' "$HDRS" || jq -e '.message|test("rate limit";"i")' "$BODY" >/dev/null 2>&1; then
      unknown "GitHub API rate limit hit"
    fi
    unknown "HTTP $HTTP_CODE: token lacks permission (needs Actions: read) or org approval pending" ;;
  404) unknown "HTTP 404: repo or workflow not found, or token lacks access (check -r, -w, token scope)" ;;
  *) unknown "unexpected HTTP status $HTTP_CODE from GitHub" ;;
esac

jq -e '.workflow_runs|type=="array"' "$BODY" >/dev/null 2>&1 || unknown "malformed JSON from GitHub"
[ "$(jq '.workflow_runs|length' "$BODY")" -gt 0 ] || unknown "no completed runs found for $WORKFLOW on branch $BRANCH"

RUN=$(jq -c '.workflow_runs[0]' "$BODY")
IFS=$'\t' read -r RUN_ID RUN_NUM CONCLUSION URL AGE < <(jq -r --argjson now "$(date +%s)" \
  '[.id,.run_number,(.conclusion//"unknown"),.html_url,($now - (.updated_at|fromdateiso8601))]|@tsv' <<<"$RUN" 2>/dev/null) \
  || unknown "malformed run data from GitHub"
[[ "$AGE" =~ ^-?[0-9]+$ && -n "$RUN_ID" ]] || unknown "malformed run data from GitHub"
[ "$AGE" -lt 0 ] && AGE=0

# Token expiry (header present only for fine-grained PATs with expiry)
EXP_NOTE=""; EXP_WARN=0
EXP_HDR=$(grep -i '^github-authentication-token-expiration:' "$HDRS" | head -1 | cut -d: -f2- | tr -d '\r' | sed 's/^ *//')
if [ -n "$EXP_HDR" ]; then
  # Format: "2026-11-01 12:00:00 UTC"
  EXP_ISO=$(sed -E 's/^([0-9-]+) ([0-9:]+).*/\1T\2Z/' <<<"$EXP_HDR")
  EXP_EPOCH=$(jq -rn --arg t "$EXP_ISO" '$t|fromdateiso8601' 2>/dev/null)
  if [[ "$EXP_EPOCH" =~ ^[0-9]+$ ]]; then
    LEFT=$((EXP_EPOCH - $(date +%s)))
    if [ "$LEFT" -lt $((EXPIRE_DAYS*86400)) ]; then
      EXP_WARN=1; EXP_NOTE="token expires ${EXP_HDR%% *}"
    fi
  fi
fi

AGE_H=$(human "$AGE")
PERF="| age=${AGE}s;${MAX_AGE};;"
finish() { # state code, message
  local msg=$2
  [ -n "$EXP_NOTE" ] && msg="$msg (WARNING: $EXP_NOTE)"
  echo "$PREFIX $1 - $msg $PERF"; exit "$3"
}

if [ "$AGE" -gt "$MAX_AGE" ]; then
  finish CRITICAL "stale: last completed run was $AGE_H ago" 2
fi

case $CONCLUSION in
  success)
    if [ "$EXP_WARN" -eq 1 ]; then
      finish WARNING "last run passed $AGE_H ago (run #$RUN_NUM) but $EXP_NOTE" 1
    fi
    finish OK "last run passed $AGE_H ago (run #$RUN_NUM)" 0 ;;
  failure|timed_out|startup_failure)
    DETAIL=""
    if gh_get "/repos/$REPO/actions/runs/$RUN_ID/jobs?per_page=100" && [ "$HTTP_CODE" = 200 ]; then
      JOBINFO=$(jq -r '[.jobs[]?|select(.conclusion=="failure" or .conclusion=="timed_out")][0]
        | if .==null then empty else [.id,.name,([.steps[]?|select(.conclusion=="failure" or .conclusion=="timed_out")][0].name//"")]|@tsv end' "$BODY" 2>/dev/null)
      if [ -n "$JOBINFO" ]; then
        IFS=$'\t' read -r JOB_ID JOB_NAME STEP_NAME <<<"$JOBINFO"
        if [ -n "$STEP_NAME" ]; then DETAIL=" at step '$STEP_NAME'"; else DETAIL=" in job '$JOB_NAME'"; fi
        if gh_get "/repos/$REPO/check-runs/$JOB_ID/annotations" && [ "$HTTP_CODE" = 200 ]; then
          ANN=$(jq -r '[.[]?|select((.message//"")|startswith("WWN WordPress:"))][0].message//empty' "$BODY" 2>/dev/null | head -c 200)
          [ -n "$ANN" ] && DETAIL="$DETAIL: $ANN"
        fi
      fi
    fi
    finish CRITICAL "last run ${CONCLUSION/failure/failed} $AGE_H ago$DETAIL $URL" 2 ;;
  cancelled|skipped|neutral|action_required|stale)
    finish WARNING "last run $CONCLUSION $AGE_H ago (run #$RUN_NUM) $URL" 1 ;;
  *)
    unknown "unrecognized run conclusion '$CONCLUSION'" ;;
esac

Then Nagios can alert me if things are falling apart, or if things have recovered after problems, e.g. via Pushover:

And if there is a problem that needs investigating, the “artifact” available in the GitHub action run details is a zipfile of the Playwright run report, a self-contained HTML and image bundle with all the details and screenshots. Amazing.

The Cloudflare bypass is not perfect, it seems some of GitHub’s action infrastructure IPs trigger “are you human” checks with Cloudflare no matter what. And the run timing seems to be a bit inconsistent (based on whether GitHub actions are having issues, I think) so the monitoring we’re doing has had some false alarms for delayed runs. But otherwise, I’m pretty happy with it.

Published by

Chris Hardie

Journalist, publisher, software developer, entrepreneur

Leave a Reply

Your email address will not be published. Required fields are marked *