CI recipes

The API exists so that a pipeline can assert on email. Here is what that looks like in full, with nothing left as an exercise.

A GitHub Actions job

One ephemeral inbox per run, its SMTP pair handed to the test suite as environment variables, a verdict at the end, and a delete that runs even when the tests fail. The only secret in the repository is the API token; the SMTP credentials are minted per run and die with the inbox.

name: email

on: [pull_request]

env:
  MAILCLUTCH_API: https://app.mailclutch.com/api/v1

jobs:
  signup-email:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Create an inbox for this run
        id: inbox
        run: |
          RESPONSE=$(curl -sS --fail "$MAILCLUTCH_API/inboxes" \
            -H "Authorization: Bearer ${{ secrets.MAILCLUTCH_TOKEN }}" \
            -H "Content-Type: application/json" \
            -H "Idempotency-Key: gha-${{ github.run_id }}-${{ github.run_attempt }}" \
            -d '{"name":"pr-${{ github.event.number }}","ttl":"1h"}')
          echo "id=$(echo "$RESPONSE" | jq -r .id)" >> "$GITHUB_OUTPUT"
          echo "SMTP_HOST=$(echo "$RESPONSE" | jq -r .smtp.host)" >> "$GITHUB_ENV"
          echo "SMTP_PORT=$(echo "$RESPONSE" | jq -r .smtp.port)" >> "$GITHUB_ENV"
          echo "SMTP_USERNAME=$(echo "$RESPONSE" | jq -r .smtp.username)" >> "$GITHUB_ENV"
          echo "::add-mask::$(echo "$RESPONSE" | jq -r .smtp.password)"
          echo "SMTP_PASSWORD=$(echo "$RESPONSE" | jq -r .smtp.password)" >> "$GITHUB_ENV"
          echo "RUN_STARTED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_ENV"

      - name: Run the flow that sends the email
        run: npm ci && npm run test:signup

      - name: The email arrived and is right
        run: |
          curl -sS --fail "$MAILCLUTCH_API/inboxes/${{ steps.inbox.outputs.id }}/expect" \
            -H "Authorization: Bearer ${{ secrets.MAILCLUTCH_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d @- <<JSON | tee verdict.json
          {
            "timeout": 30,
            "since": "$RUN_STARTED_AT",
            "expect": {
              "to": "signup-test@example.com",
              "subject_matches": "^Verify your email",
              "has_link_matching": "^https://app\\\\.acme\\\\.dev/verify/[A-Za-z0-9]+$",
              "no_unrendered_variables": true,
              "deliverability_at_least": 80
            }
          }
          JSON

      - name: Post the rendered email on the pull request
        if: always() && steps.inbox.outputs.id != ''
        run: |
          EMAIL_ID=$(jq -r '.email.id // empty' verdict.json 2>/dev/null || true)
          [ -z "$EMAIL_ID" ] && exit 0
          URL=$(curl -sS --fail "$MAILCLUTCH_API/emails/$EMAIL_ID/shares" \
            -H "Authorization: Bearer ${{ secrets.MAILCLUTCH_TOKEN }}" \
            -H "Content-Type: application/json" -d '{"expires_in":"7d"}' | jq -r .url)
          gh pr comment ${{ github.event.number }} --body "Sign-up email on this branch: $URL"
        env:
          GH_TOKEN: ${{ github.token }}

      - name: Delete the inbox
        if: always() && steps.inbox.outputs.id != ''
        run: |
          curl -sS "$MAILCLUTCH_API/inboxes/${{ steps.inbox.outputs.id }}" -X DELETE \
            -H "Authorization: Bearer ${{ secrets.MAILCLUTCH_TOKEN }}"

Three things worth noticing. The Idempotency-Key includes the run attempt, so a retried job gets its own inbox while a retried request gets the same one. The expectation carries since, because the email was sent before the wait began. And the last step is not strictly needed — an hour later the inbox is gone on its own — but a job that cleans up after itself keeps the live count far from the limit of 50.

The token needs inboxes:create, inboxes:delete, emails:read, analysis:run and shares:create. Give it an expiry, and rotate it from the settings screen when it nears.

The same flow inside a test

When the test suite itself talks to the API — a Playwright or Vitest run that owns the whole flow — the shape is the same, without the shell.

const API = 'https://app.mailclutch.com/api/v1';
const headers = {
  Authorization: `Bearer ${process.env.MAILCLUTCH_TOKEN}`,
  'Content-Type': 'application/json',
};

const call = async (path, init = {}) => {
  const res = await fetch(`${API}${path}`, { ...init, headers });
  const body = await res.json();
  if (!res.ok && res.status !== 422) throw new Error(`${res.status} ${body.message}`);
  return body;
};

test('the password reset email is right', async () => {
  const inbox = await call('/inboxes', {
    method: 'POST',
    body: JSON.stringify({ name: 'reset-flow', ttl: '10m' }),
  });

  try {
    const startedAt = new Date().toISOString();
    await requestPasswordReset('user@example.com', inbox.smtp);

    const verdict = await call(`/inboxes/${inbox.id}/expect`, {
      method: 'POST',
      body: JSON.stringify({
        timeout: 30,
        since: startedAt,
        expect: {
          to: 'user@example.com',
          subject_contains: 'Reset your password',
          has_link_matching: '^https://app\\.acme\\.dev/reset/',
          no_unrendered_variables: true,
        },
      }),
    });

    // The failures array is the assertion message: expected and actual, per rule.
    expect(verdict.failures).toEqual([]);

    // Follow the link the same call extracted.
    await page.goto(verdict.extracted.matched_link);
    await expect(page.getByRole('heading', { name: 'Choose a new password' })).toBeVisible();
  } finally {
    await call(`/inboxes/${inbox.id}`, { method: 'DELETE' });
  }
});

A 422 from expect is a verdict, not a transport error, which is why the helper lets it through: the test reads failures and fails with them. Everything else about the calls — pagination, keys, limits — is in Conventions.