Skip to main content

Stripe Go-Live Runbook

This runbook documents the procedure for migrating the production backend from Stripe test mode to live mode. It is based on the actual go-live performed in April 2026 and captures every gotcha discovered along the way.

This is a one-time operation per Stripe account, but the steps are also valid for:

  • Recreating the live environment after rotating to a new Stripe account
  • Setting up live mode in a new region or new business entity
  • Disaster recovery if the live webhook secret is lost
Production impact

Once executed, every checkout against the prod backend charges real money. Run a real smoke transaction (small one-time pack) before announcing the change to customers.

Overview​

Prerequisites​

Before starting:

  1. Stripe account fully activated. Live mode toggle in the dashboard must be available — charges_enabled=true and payouts_enabled=true on the account. New Stripe accounts may require KYC/business verification before live mode is unlocked.
  2. AWS SSO active — aws sso login --profile Administrator-842697652860
  3. Direct DB access to the production RDS via SSM port forwarding. See RDS for the tunnel command.
  4. One super admin user in the production DB (needed for the admin GraphQL mutations). Verify with: 
    SELECT id, email FROM "User" WHERE "isSuperAdmin" = true;
  5. Live Stripe secret key revealed once in the Stripe dashboard → Developers → API keys → Reveal sk_live_*. The Stripe API does not allow retrieving an existing secret key value programmatically — you must copy it from the dashboard at least once.

Step 1: Verify the Live Key​

export STRIPE_SK='sk_live_...'
curl -s https://api.stripe.com/v1/account -u "${STRIPE_SK}:" | jq '{id, business_name: .business_profile.name, charges_enabled, payouts_enabled, default_currency, country}'

Expected output: account details with charges_enabled: true and payouts_enabled: true. If either is false, complete Stripe activation in the dashboard before proceeding.

Step 2: Audit and Replace the Webhook Endpoint​

List existing live webhooks:

curl -s https://api.stripe.com/v1/webhook_endpoints -u "${STRIPE_SK}:" | jq '.data[] | {id, url, status, api_version, events: (.enabled_events | length)}'

If any endpoint exists with the wrong URL (e.g. pointing at the marketing site instead of api.aiqlick.com/stripe/webhook), delete it — Stripe does not allow expanding secret on retrieve, so updating in place loses you the signing secret.

# Delete the broken endpoint
curl -s -X DELETE "https://api.stripe.com/v1/webhook_endpoints/we_..." -u "${STRIPE_SK}:"

Create the correct live endpoint with all 14 events the backend handles:

curl -s -X POST https://api.stripe.com/v1/webhook_endpoints -u "${STRIPE_SK}:" \
  -d "url=https://api.aiqlick.com/stripe/webhook" \
  -d "api_version=2023-10-16" \
  -d "description=aiqlick-backend production webhook" \
  -d "enabled_events[]=checkout.session.completed" \
  -d "enabled_events[]=invoice.paid" \
  -d "enabled_events[]=invoice.payment_failed" \
  -d "enabled_events[]=invoice.finalized" \
  -d "enabled_events[]=invoice.finalization_failed" \
  -d "enabled_events[]=invoice.payment_action_required" \
  -d "enabled_events[]=invoice.upcoming" \
  -d "enabled_events[]=customer.subscription.created" \
  -d "enabled_events[]=customer.subscription.updated" \
  -d "enabled_events[]=customer.subscription.deleted" \
  -d "enabled_events[]=customer.subscription.trial_will_end" \
  -d "enabled_events[]=customer.subscription.paused" \
  -d "enabled_events[]=customer.subscription.resumed" \
  -d "enabled_events[]=charge.refunded"

Capture the secret field from the response — this is the live whsec_* signing secret. It is only returned once at creation time and cannot be retrieved later. If you lose it, you must delete and recreate the endpoint.

The api_version must match STRIPE_API_VERSION in the backend secret (2023-10-16 as of this writing). If you change one, change the other.

Step 3: Update Secrets Manager​

Fetch the current production secret, patch only the Stripe fields, and put the updated version back:

# Fetch
aws secretsmanager get-secret-value \
  --profile Administrator-842697652860 --region eu-north-1 \
  --secret-id aiqlick-backend/production \
  --query SecretString --output text > /tmp/prod_secret.json

# Patch with Python (preserves all 90+ other keys unchanged)
python3 << 'PY'
import json
with open('/tmp/prod_secret.json') as f: d = json.load(f)
d['STRIPE_SECRET_KEY'] = 'sk_live_...'
d['STRIPE_WEBHOOK_SECRET'] = 'whsec_...'
d['STRIPE_PUBLISHABLE_KEY'] = 'pk_live_...'  # add for parity with dev
with open('/tmp/prod_secret_new.json', 'w') as f: json.dump(d, f)
PY

# Push new version
aws secretsmanager put-secret-value \
  --profile Administrator-842697652860 --region eu-north-1 \
  --secret-id aiqlick-backend/production \
  --secret-string file:///tmp/prod_secret_new.json

# Securely delete temp files
shred -u /tmp/prod_secret.json /tmp/prod_secret_new.json
Always patch — never rewrite from scratch

The aiqlick-backend/production secret contains 90+ unrelated keys (DB URL, Redis, JWT, SMTP, Bedrock, etc.). Always fetch → patch → put. Never construct the secret string from scratch or you will wipe non-Stripe configuration.

Step 4: Force a backend redeploy to pick up the new secret​

ECS reads the secret bundle (_BUNDLE_JSON) at task startup. A service redeploy forces a fresh task with the new value — see Secrets Management. The deployment circuit breaker rolls back automatically on health-check failure.

aws ecs update-service \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --service backend-prod --force-new-deployment

# Wait for service stability
aws ecs wait services-stable \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --services backend-prod

# Verify the new task is reading the new STRIPE_SECRET_KEY (first 12 chars only)
TASK=$(aws ecs list-tasks --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --service-name backend-prod --query 'taskArns[0]' --output text)
aws ecs execute-command \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --task "$TASK" --container aiqlick-backend \
  --interactive --command \
  "sh -c 'echo \"\$_BUNDLE_JSON\" | python3 -c \"import json,sys; print(json.loads(sys.stdin.read()).get(\\\"STRIPE_SECRET_KEY\\\", \\\"\\\")[:12])\"'"

Verify externally:

curl -s -o /dev/null -w "%{http_code}\n" https://api.aiqlick.com/health
curl -s -X POST https://api.aiqlick.com/graphql -H "Content-Type: application/json" -d '{"query":"{ __typename }"}'

# Webhook endpoint should reject unsigned requests with 400
curl -s -X POST https://api.aiqlick.com/stripe/webhook -H "Content-Type: application/json" -d '{}'
# Expected: "Webhook Error: No stripe-signature header value was provided."

Step 5: Re-sync Plans to Live Mode​

The DB Plan rows still hold test-mode stripeProductId and stripePriceId values that don't exist in live mode. They must be cleared and recreated.

The syncAllPlansStripePricing admin mutation only operates on plans where stripePriceId IS NULL, so you must null the stale values first. Then either call the mutation as a super admin, or do it directly via the Stripe API + DB UPDATE if no super admin credentials are available.

Path A: Via the admin mutation (preferred)​

Sign in as a super admin and call:

mutation {
  syncAllPlansStripePricing {
    id
    name
    stripePriceId
    stripeProductId
  }
}

Beforehand, null out the stale IDs in the DB so the mutation has work to do:

UPDATE "Plan"
SET "stripeProductId" = NULL, "stripePriceId" = NULL, "updatedAt" = NOW()
WHERE active = true AND "billingMode" != 'POSTPAID';

Path B: No super admin credentials available​

Mint a short-lived super admin JWT manually (the JWT_SECRET is in Secrets Manager):

JWT_SECRET=$(aws secretsmanager get-secret-value \
  --profile Administrator-842697652860 --region eu-north-1 \
  --secret-id aiqlick-backend/production \
  --query SecretString --output text | jq -r .JWT_SECRET)

TOKEN=$(python3 -c "
import jwt, time
payload = {
    'sub': '<super-admin-user-id>',
    'email': '<super-admin-email>',
    'iat': int(time.time()),
    'exp': int(time.time()) + 1800
}
print(jwt.encode(payload, '${JWT_SECRET}', algorithm='HS256'))
")

# Verify the token
curl -s -X POST https://api.aiqlick.com/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -d '{"query":"{ whoAmI { id email isSuperAdmin } }"}'
JWT minting depends on a weak default secret

This works only because JWT_SECRET is currently a static string in Secrets Manager. After this go-live procedure, the proper fix is to rotate JWT_SECRET to a long random value (e.g. openssl rand -hex 64) — once that happens, this manual minting path will no longer work and you must instead use real super admin credentials. See the security hygiene backlog.

Step 6: Re-sync Credit Packs to Live Mode​

Same approach as Plans — null the stale Stripe IDs, then call the admin mutation. There is no syncAllCreditPacks mutation, so you must call createCreditPack for each pack you want, or use the Stripe API directly.

Using createCreditPack for a fresh catalog (assuming the prior pack rows have already been deleted or were never present):

mutation {
  createCreditPack(input: {
    name: "Starter",
    credits: 50,
    price: 50,
    currency: "usd",
    sortOrder: 1
  }) {
    id name credits price currency sortOrder
  }
}

The resolver auto-creates the Stripe Product + one-time Price and writes the IDs back to the DB row. See the Credit Pack Catalog for the production-default tiers.

The output type CreditPackOutput does not expose stripePriceId/stripeProductId/status — querying those fields will fail validation and the mutation will not run. Stick to the fields actually defined in credit-pack.output.ts.

Step 7: Clean Up Orphans​

After the switch, you typically have residual data on both sides:

Stripe side — orphan products and prices from previous platform iterations:

# List all active products/prices and identify any without metadata.planId or metadata.creditPackId
curl -s "https://api.stripe.com/v1/products?limit=100&active=true" -u "${STRIPE_SK}:" | jq '.data[] | {id, name, "creditPackId": .metadata.creditPackId, "planId": .metadata.planId}'

Archive any product without metadata pointing to a current DB row:

# Archive a product (cannot be hard-deleted once it has prices)
curl -s -X POST "https://api.stripe.com/v1/products/prod_..." -u "${STRIPE_SK}:" -d "active=false"

Default-price gotcha: archiving a price that is currently the default_price of its product fails with "This price cannot be archived because it is the default price of its product.". Clear the default first:

curl -s -X POST "https://api.stripe.com/v1/products/prod_..." -u "${STRIPE_SK}:" -d "default_price="
# Now the price can be archived
curl -s -X POST "https://api.stripe.com/v1/prices/price_..." -u "${STRIPE_SK}:" -d "active=false"

Stripe side — orphan subscriptions on legacy prices:

Cancel any active/trialing subscriptions that reference legacy products:

curl -s -X DELETE "https://api.stripe.com/v1/subscriptions/sub_..." -u "${STRIPE_SK}:"

DB side — orphan test-mode subscriptions:

The DB may contain Subscription rows with stripeSubscriptionId values that are test-mode sub_* IDs. They will silently stop receiving webhook updates. Deactivate them:

UPDATE "Subscription"
SET status = 'CANCELED',
    "isActive" = false,
    "endDate" = NOW(),
    "updatedAt" = NOW()
WHERE "stripeSubscriptionId" LIKE 'sub_%'
  AND "isActive" = true;

The affected users can re-checkout in live mode whenever they want.

Step 8: Smoke Test​

  1. Webhook reachability:

    curl -s -o /dev/null -w "%{http_code}\n" -X POST https://api.aiqlick.com/stripe/webhook \
      -H "Content-Type: application/json" -d '{}'

    Expect 400 with body Webhook Error: No stripe-signature header value was provided. — confirms the route is wired and signature verification is enforced.

  2. Plan and pack visibility: sign in as any user and load the pricing page. The Plan and CreditPack lists should show the new live products.

  3. Real transaction: purchase the smallest credit pack (Starter, $50) with a real card. Verify in order:

    • checkout.session.completed event arrives in the live webhook (visible in Stripe dashboard → Developers → Events)
    • CreditPackPurchase row transitions PENDING → COMPLETED
    • CreditTransaction row of type PACK_PURCHASE is created
    • CreditBalance row reflects the new balance
    • Invoice row is created with status='PAID'

  4. Refund test (optional): issue a refund from the Stripe dashboard. Verify charge.refunded webhook fires and a CreditTransaction of type REFUND is created.

Final Alignment Verification​

Run this to confirm DB and Stripe agree on every Plan and CreditPack:

-- DB side
SELECT 'Plan' AS kind, name, "billingMode", "stripeProductId", "stripePriceId" FROM "Plan" WHERE active = true
UNION ALL
SELECT 'CreditPack', name, status::text, "stripeProductId", "stripePriceId" FROM "CreditPack" WHERE status = 'ACTIVE'
ORDER BY kind, name;
# Stripe side — every product should have metadata.planId or metadata.creditPackId
curl -s "https://api.stripe.com/v1/products?limit=100&active=true" -u "${STRIPE_SK}:" | \
  jq '.data[] | {id, name, "match": (.metadata.planId // .metadata.creditPackId // "ORPHAN")}'

Every active Stripe product should resolve to a current DB row, and every active DB Plan/CreditPack row (where needsStripePricing is true) should resolve to a live Stripe product.

Common Pitfalls​

SymptomCauseFix
Container still uses test keys after secret updateUsed docker restart instead of stop/rm/runRecreate the container, see Secrets Management
Webhook signature verification failswhsec_* from a different endpoint or test modeRecreate the live webhook to get the correct signing secret
syncAllPlansStripePricing returns same plans unchangedThe mutation only fills plans with stripePriceId IS NULLNull the stale IDs first via SQL
createCreditPack mutation returns GraphQL validation errorOutput type only exposes a subset of fieldsStick to fields defined in credit-pack.output.ts (no status, stripeProductId, stripePriceId)
Cannot archive a Stripe priceIt's the default_price of its productClear default_price on the product first
Live charges succeed but DB shows nothingWebhook URL points at the wrong host (e.g. marketing site)Recreate the webhook with url=https://api.aiqlick.com/stripe/webhook
STRIPE_API_VERSION mismatch warningsBackend env var and webhook endpoint api_version differSet both to the same value (currently 2023-10-16)
Currency mismatch on payoutsPlans priced in USD but Stripe account default_currency=eurStripe handles FX at payout. To avoid conversion, recreate prices in EUR.

Rollback​

If something goes wrong post-flip and you need to revert to test mode quickly:

  1. Restore the previous Secrets Manager version: 
    aws secretsmanager list-secret-version-ids \
      --profile Administrator-842697652860 --region eu-north-1 \
      --secret-id aiqlick-backend/production
    aws secretsmanager update-secret-version-stage \
      --profile Administrator-842697652860 --region eu-north-1 \
      --secret-id aiqlick-backend/production \
      --version-stage AWSCURRENT \
      --move-to-version-id <previous-version-id>
  2. Recreate the backend container (Step 4) so it picks up the rolled-back env-file.
  3. The DB Plan rows will now hold live-mode Stripe IDs that don't exist in test mode — checkout will fail with No such price. Either re-run Step 5 against test mode, or accept that subscription checkout is broken until you flip back to live.

Rollback is destructive to the live mode setup — you will lose the live webhook signing secret and have to recreate it on the next forward attempt. Avoid rollback unless absolutely necessary. Forward-fix is almost always the right call.