Railway’s incident: rotate keys, then prove recovery.

What the original disclosure said

On 21 October, Railway reported possible unauthorized access to encrypted environment variables and decrypted contents for some of them. It advised users to rotate stored secrets and review service logs as a precaution. That disclosure did not establish that every customer was affected, or that a particular customer’s database contents had been taken. Treat the vendor’s notification as a reason to investigate your exposure, not as a substitute for that investigation.

What the later report added

Railway’s preliminary report distinguished access to internal systems from individual account logins. It described a subset of workspaces with potentially compromised variables, broader precautionary upstream revocations, and existing customer workloads continuing to run. Its 26 October update called the incident resolved for existing users. These are Railway’s findings; they do not independently certify a customer’s downstream services. The practical question now is whether your team completed and verified its own response.

Map what each secret can reach

Our suggested first deliverable is a small credential register. Start with production, then include staging projects, shared variables and integrations maintained by another team. A variable named TOKEN says little about its impact. Ask which provider issued it, what it can read or change, and which application uses it. A credential that signs sessions deserves a different recovery plan from a read-only analytics key. Do not paste values into the register.

• Record the credential identifier, issuer, purpose and responsible person.
• List all consuming services, scheduled jobs, CI systems and external tools.
• Record privileges and whether the service is reachable from outside the private network.
• Track replacement, revocation, deployment and verification times separately.

Rotate at the source of authority

Plan the change with the issuing provider’s current procedure. Replacing a value in a hosting dashboard does not itself revoke the former credential at the service it accesses. If there is evidence of active misuse, the incident lead should prioritize containment over uninterrupted availability. Where the provider supports a controlled overlap, coordinate the replacement across consumers and finish by invalidating the old credential. Keep rollback focused on the application release; do not restore a potentially exposed key to make a broken integration work. Our recommendation is to name one owner for the entire change, including external tools.

Use Railway’s database-specific procedure

For Railway’s PostgreSQL template, the vendor recommends Regenerate in Database View, followed by redeployment of consumers and connection verification. Manually changing a password variable alone can leave the database and its configuration inconsistent. Include reporting tools and jobs outside Railway. This procedure is specific to that template: use the appropriate instructions for another engine or an externally managed database.

Review AI API usage independently

Railway’s OpenAI response guide covers revoking a potentially exposed key, deploying its replacement and reviewing usage by project and key. Compare request volume, models and spend with your application’s own traffic. Keep unexplained events for provider support. An unexpected bill is a useful lead, but by itself does not establish who used a key or whether the incident caused that usage.

Check the deployment path too

Railway’s recovery guide says commits pushed while deployments were paused were not queued. It also describes reconnecting GitHub and replacing revoked CLI or CI credentials. Check the actual running commit and deployment time rather than assuming the latest push went live. Follow the provider’s recovery instructions for the deployment method you use.

Define evidence for closing the incident

For your own recovery review, collect a small, redacted evidence pack. It should let a second engineer understand what changed without receiving a secret. Missing logs are a limitation to record, not a clean finding. Escalate suspicious data access to the responsible incident, privacy and legal owners, who can determine the applicable reporting obligations.

• Confirm the intended release is serving the production domain.
• Exercise sign-in and an authorized test record through the real application.
• Run a representative background job and each integration whose key changed.
• Confirm former credentials are no longer accepted using the issuer’s safe verification method.
• Record remaining log gaps, unresolved events, owners and follow-up dates.

Make the next rotation less disruptive

OWASP’s secrets-management guidance supports least privilege, lifecycle management and automated rotation where appropriate. Our practical next step is a rehearsal in a non-production environment: choose one integration, replace its credential, update consumers and measure the recovery. Use separate credentials for separate environments. A secrets manager helps organize the process, but cannot compensate for a key that grants far more access than the application needs.