Raxx · internal docs

internal · gated

Rotation SOP — Google Workspace Service Account Key

Mode: programmatic Last validated: 2026-04-24 UTC Validation method: read-only-docs (sandbox-rotation feasible — create a test SA, rotate keys, delete) Average duration: 5m Required role: ops

Applies to: any Google Cloud / Google Workspace service account JSON key file used by Raxx automation (e.g., Drive sync, Calendar integration, Gmail API). Stored in Infisical as the full JSON blob (e.g., GOOGLE_WORKSPACE_SA_KEY_JSON).

Confirmed: gcloud CLI supports full programmatic rotation. A service account can have up to 10 keys simultaneously, allowing easy overlap. The recommended pattern is create-new → propagate → disable-old → delete-old.

When to run

Prerequisites

Steps

1. Pre-rotation checks

SA_EMAIL="raxx-drive-bot@<project>.iam.gserviceaccount.com"
PROJECT_ID="<project>"

# List existing keys
gcloud iam service-accounts keys list \
  --iam-account="$SA_EMAIL" \
  --project="$PROJECT_ID"
# Expect: at least one user-managed key. System-managed keys are separate and not rotated by us.

# Confirm the current key file authenticates
gcloud auth activate-service-account --key-file=/path/to/current-sa-key.json
gcloud projects describe "$PROJECT_ID" | head -5
# Expect: project metadata

2. Generate the new credential

NEW_KEY_FILE="/tmp/sa-key-rotation-$(date -u +%Y%m%dT%H%M%SZ).json"
gcloud iam service-accounts keys create "$NEW_KEY_FILE" \
  --iam-account="$SA_EMAIL" \
  --project="$PROJECT_ID"
# Output includes the new key's `name` (full resource path); the JSON contents at $NEW_KEY_FILE include the private key.
NEW_KEY_ID=$(gcloud iam service-accounts keys list \
  --iam-account="$SA_EMAIL" --project="$PROJECT_ID" --managed-by=user \
  --sort-by=~validAfterTime --format="value(KEY_ID)" --limit=1)
echo "New key ID: $NEW_KEY_ID"

3. Validate the new credential

gcloud auth activate-service-account --key-file="$NEW_KEY_FILE"
gcloud projects describe "$PROJECT_ID" | head -5
# Expect: same project metadata as step 1.

For a Google Workspace API consumer, validate a representative call:

# Example: Drive API list
gcloud auth print-access-token --key-file="$NEW_KEY_FILE" \
  | xargs -I {} curl -sS -H "Authorization: Bearer {}" \
    "https://www.googleapis.com/drive/v3/files?pageSize=1" | jq '.files | length'
# Expect: 0 or 1, no auth error.

4. Store in Infisical

infisical secrets set GOOGLE_WORKSPACE_SA_KEY_JSON="$(cat $NEW_KEY_FILE)" \
  --projectId="$INFISICAL_PROJECT_ID" --env=prod

The full JSON blob is the credential. Some consumers prefer base64-encoded — encode if needed:

base64 < "$NEW_KEY_FILE" | tr -d '\n'

5. Propagate to downstream consumers

Consumer How
Heroku app (env-var JSON) heroku config:set GOOGLE_WORKSPACE_SA_KEY_JSON="$(cat $NEW_KEY_FILE)" -a <app>
Heroku app (file path) Use heroku config:set GOOGLE_APPLICATION_CREDENTIALS=/app/sa-key.json and write the file from the env at boot
GitHub Actions gh secret set GOOGLE_WORKSPACE_SA_KEY_JSON -b "$(cat $NEW_KEY_FILE)"
Local dev DM operator via Slack D0AJ7K184TV

After propagation, shred the local file:

shred -u "$NEW_KEY_FILE" 2>/dev/null || rm -P "$NEW_KEY_FILE" 2>/dev/null || rm "$NEW_KEY_FILE"

6. Verify downstream

After consumer restart:

heroku run --app <app> python -m scripts.gws_sync_dry_run
# Expect: no auth errors.

For long-running consumers, wait for at least one full operation cycle.

7. Disable the old key (NOT delete yet)

gcloud iam service-accounts keys disable "$OLD_KEY_ID" \
  --iam-account="$SA_EMAIL" \
  --project="$PROJECT_ID"

Wait at least 24h to confirm no consumer is still using the old key:

# Inspect SA key usage in Cloud Audit Logs for the old key ID
gcloud logging read "protoPayload.authenticationInfo.serviceAccountKeyName=\"$OLD_KEY_ID\"" \
  --project="$PROJECT_ID" --limit=10 --freshness=24h
# Expect: no entries (or only entries from before the rotation).

7b. Delete the old key

After observation period:

gcloud iam service-accounts keys delete "$OLD_KEY_ID" \
  --iam-account="$SA_EMAIL" \
  --project="$PROJECT_ID"

8. Audit log entry

action: secret.rotate.completed
actor: <admin_id>
context: {
  "secret_name": "GOOGLE_WORKSPACE_SA_KEY_JSON",
  "service_account": "<email>",
  "old_key_id": "<...>",
  "new_key_id": "<...>",
  "method": "programmatic"
}

Rollback

The old key is Disabled (not deleted) until step 7b. To re-enable:

gcloud iam service-accounts keys enable "$OLD_KEY_ID" \
  --iam-account="$SA_EMAIL" \
  --project="$PROJECT_ID"

Then revert Heroku config to the old key JSON from Infisical history.

After step 7b (delete), the old key is unrecoverable. Note: deleted keys remain billable / counted toward the 10-key limit for up to 30 days per Google's eventual-consistency cleanup. Plan rotation cadence accordingly.

Vendor doc references

Known gotchas