CI AWS credentials runbook

System: AWS IAM — Woodpecker CI long-lived access keys Owner: operator (sre-agent for rotation automation) Card: #4108 ADR: docs/architecture/adr/0136-gha-deliberate-holds-cutover.md (Decision 2) Last incident: — Last reviewed: 2026-07-09

Context

Three least-privilege IAM users are provisioned in infra/ci/ci-artifacts.tf. These replace GHA OIDC short-lived tokens for the Woodpecker pipelines that need AWS access. See ADR-0136 Decision 2 for the formal security posture downgrade rationale and accepted mitigations.

This is a documented security posture downgrade from OIDC short-lived tokens. Mitigations: separate plan/apply users, 90-day rotation, least-privilege policies. Do NOT expand these users' permissions without filing a new ADR section.

IAM users

Secret names at each vault path: ACCESS_KEY_ID and SECRET_ACCESS_KEY.

How to tell it's broken

• Symptom 1: WP pipeline step fails with InvalidClientTokenId or NoCredentialProviders — vault read succeeded but key is invalid or deleted.
• Symptom 2: WP pipeline step fails with AccessDenied: ... is not authorized to perform: ... — key is valid but the IAM policy is too narrow for the action attempted.
• Symptom 3: WP wpc_load_vault_secrets.py returns HTTP 404 for ACCESS_KEY_ID or SECRET_ACCESS_KEY — vault path is missing (folder was deleted or never created).
• Symptom 4: WP pipeline step fails with ExpiredTokenException — key is still Active in AWS but has been in use past the 90-day rotation window.

How to diagnose (in order)

1. Check key age: aws iam list-access-keys --user-name raxx-tf-email-stack-plan-ci aws iam list-access-keys --user-name raxx-tf-email-stack-apply-ci aws iam list-access-keys --user-name raxx-freescout-ssm-reader-ci Check CreateDate. If any key is older than 90 days, rotate immediately.

2. Check key status (Active/Inactive): Same command as above — look at Status field.

3. Verify vault paths exist: # Via Infisical web UI at vault.raxx.app, project raxx-secrets, prod environment: # /MooseQuest/aws/terraform/email-delivery-stack-plan/ACCESS_KEY_ID # /MooseQuest/aws/terraform/email-delivery-stack-apply/ACCESS_KEY_ID # /MooseQuest/aws/freescout/ssm-reader/ACCESS_KEY_ID A 404 from wpc_load_vault_secrets.py means the folder is missing — re-run the mint script to recreate it.

4. Test caller identity using loaded keys: Load ACCESS_KEY_ID + SECRET_ACCESS_KEY from vault, then: AWS_ACCESS_KEY_ID=<loaded> AWS_SECRET_ACCESS_KEY=<loaded> \ aws sts get-caller-identity Expected: returns a JSON with UserId, Account, and the user ARN.

5. Check CloudTrail for the user's recent calls: aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=Username,AttributeValue=raxx-tf-email-stack-plan-ci \ --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ) \ --region us-east-2 (macOS: date -u -v-1H; Linux: date -u -d '1 hour ago')

Known failure modes

Failure mode A: key expired / Inactive

Symptom: InvalidClientTokenId or ExpiredTokenException in WP step log. Cause: Key is over 90 days old and the rotation job did not run, or the key was manually deactivated. Fix: Run the rotation procedure below. Verification: aws iam list-access-keys --user-name <user> shows a new Active key with a CreateDate within the last 24 hours. WP test step aws sts get-caller-identity returns the expected user ARN.

Failure mode B: vault path missing (404)

Symptom: WP pipeline log shows HTTP 404 from vault, pipeline step exits non-zero. Cause: Vault folder was deleted manually, or the mint script was never run after terraform apply. Fix:

bash scripts/ops/secrets/mint_ci_aws_keys.sh

The script creates folders before writing secrets (feedback_vault_folder_must_exist). Verification: wpc_load_vault_secrets.py with the correct path returns a 200 and the ACCESS_KEY_ID field is non-empty.

Failure mode C: AccessDenied on a needed action

Symptom: AWS returns AccessDenied: ... is not authorized to perform: <action> Cause: A new Terraform resource type was added to the email-delivery-stack root but the IAM policy in ci-artifacts.tf was not updated to allow the new actions. Fix: Add the required IAM actions to the inline policy for the affected user in infra/ci/ci-artifacts.tf, open a PR, run plan+apply. Verification: Re-run the failing WP step after apply confirms the new policy is active (aws iam get-user-policy --user-name <user> --policy-name <name>).

Failure mode D: two existing active keys (rotation blocker)

Symptom: aws iam create-access-key returns LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2 Cause: A prior rotation attempt created a new key but did not delete the old one. AWS allows a maximum of 2 active keys per user. Fix: List keys, identify the old one (earlier CreateDate), deactivate and delete:

aws iam list-access-keys --user-name raxx-tf-email-stack-plan-ci
aws iam update-access-key --user-name raxx-tf-email-stack-plan-ci \
  --access-key-id <OLD_KEY_ID> --status Inactive
aws iam delete-access-key --user-name raxx-tf-email-stack-plan-ci \
  --access-key-id <OLD_KEY_ID>

Then re-run mint_ci_aws_keys.sh --rotate. Verification: aws iam list-access-keys shows exactly one Active key.

Rotation procedure (90-day cadence)

Rotate all three users at once. Target: < 90 days between rotations. Schedule: first Tuesday of every third month.

# Ensure vault credentials are in the environment
export INFISICAL_CLIENT_ID="..."
export INFISICAL_CLIENT_SECRET="..."
export INFISICAL_PROJECT_ID="29b77751-f761-4afa-b3fa-2c842988f95c"
export CF_ACCESS_CLIENT_ID="..."
export CF_ACCESS_CLIENT_SECRET="..."

# Dry-run first to verify connectivity
bash scripts/ops/secrets/mint_ci_aws_keys.sh --dry-run

# Execute rotation (creates new key, writes to vault, deletes old key)
bash scripts/ops/secrets/mint_ci_aws_keys.sh --rotate

The --rotate flag: 1. Lists each user's existing Active keys. 2. Mints a new key for the user. 3. Writes the new key pair to vault (POST, or PATCH to update existing secret). 4. Deactivates the old key. 5. Deletes the old key.

After rotation, verify:

aws iam list-access-keys --user-name raxx-tf-email-stack-plan-ci
aws iam list-access-keys --user-name raxx-tf-email-stack-apply-ci
aws iam list-access-keys --user-name raxx-freescout-ssm-reader-ci

Each user should show exactly one Active key with a CreateDate within the last hour.

Then trigger a WP test build to confirm the new keys load and aws sts get-caller-identity returns the expected ARN.

Verification after first mint or rotation

1. Vault path health check — confirm no 404: Load secrets via scripts/ci/wpc_load_vault_secrets.py targeting each vault path.

2. Caller identity smoke test — confirm key is accepted by AWS: AWS_ACCESS_KEY_ID=<loaded> AWS_SECRET_ACCESS_KEY=<loaded> \ aws sts get-caller-identity --region us-east-2 Expected ARN format: arn:aws:iam::521228113048:user/ci/raxx-*-ci

3. WP test step — trigger the affected WP pipeline manually and confirm the vault-load step and AWS call step both return exit 0.

S3 artifact bucket

Bucket: raxx-ci-artifacts (us-east-2) Provisioned by: infra/ci/ci-artifacts.tf

Lifecycle rules:

To verify lifecycle configuration is intact:

aws s3api get-bucket-lifecycle-configuration --bucket raxx-ci-artifacts

To test write access from the FreeScout user:

AWS_ACCESS_KEY_ID=<freescout-key> AWS_SECRET_ACCESS_KEY=<freescout-secret> \
  aws s3 cp /dev/null s3://raxx-ci-artifacts/freescout-smoke-logs/test.txt \
  --region us-east-2

Emergency stop

To immediately disable all three CI users (revoke AWS access without deleting keys):

for user in raxx-tf-email-stack-plan-ci raxx-tf-email-stack-apply-ci raxx-freescout-ssm-reader-ci; do
  keys=$(aws iam list-access-keys --user-name "${user}" \
    --query 'AccessKeyMetadata[].AccessKeyId' --output text 2>/dev/null || true)
  for key_id in ${keys}; do
    aws iam update-access-key --user-name "${user}" \
      --access-key-id "${key_id}" --status Inactive
    echo "Deactivated ${key_id} for ${user}"
  done
done

To re-enable after the incident is resolved, run the rotation procedure above.

Escalation

Wake the operator when: - A key has been active for more than 120 days (30-day grace past the 90-day target). - CloudTrail shows API calls from unexpected IP ranges or outside business hours. - A key appears in a Git commit, Sentry log, or any public channel — treat as a credential leak incident (SEV-1). Deactivate immediately; do not wait for rotation. - aws iam create-access-key returns LimitExceeded and the cleanup procedure above does not resolve it.

Contact: ops@raxx.app