Railway asked for new secrets while its customers could not deploy them

On 21 October Railway, a cloud hosting platform that says more than 2 million developers build on it, disclosed that an unauthorized third party may have accessed the encrypted environment variables its customers store with it and, for some of them, obtained the decrypted contents. Those variables can hold the API keys and the database and registry credentials that customers’ applications run on.

The disclosure, posted on Railway’s engineering blog under the name Jacob Cooper, gave the company’s reading of the motive: “We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.” Railway said it had told those upstream providers to revoke all keys, and that its initial analysis suggested the impact “may be limited to a subset of users”. It still told every customer to roll every environment variable and private registry credential stored on the platform, and to review their logs.

The same notice set out the catch. “All deployments are paused while we investigate,” it said. “Until builds are restored, you will not be able to roll out new versions.” Eleven minutes after its status page linked the notice, Railway spelled out the consequence for the credentials it was replacing itself: “Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored.”

Railway’s incident shows what happens when the supplier that stores a company’s secrets is also the only route to putting new ones into production. The pause Railway imposed while it investigated also held up each customer’s own response, and Railway’s restoration order meant a customer’s plan decided how long that lasted.

A day and a half without builds

Railway’s status page shows new builds paused in all regions by 09:28 UTC on 21 October, more than 12 hours before the security notice was linked there at 21:59 UTC. That evening, at 19:15 UTC, Railway set out how recovery would run: “When we re-enable builds, we will do it in stages to keep the platform stable as capacity comes back: Enterprise workspaces first, followed by Pro, then Hobby, then Free and Trial.”

It kept to that order. Builds returned for Enterprise workspaces at 03:32 UTC on 22 October, for Pro at 06:29, for Hobby at 21:08 and for Free and Trial at 22:49. Measured from the 21:59 UTC status update that linked the notice, an Enterprise customer waited about five and a half hours to deploy replacement credentials, and a Hobby customer about 23. From the first status entry that reported new builds paused in every region to the last tier coming back was more than 37 hours.

Railway acknowledged the bind while it lasted. “We understand you are unable to update your applications with new credentials at this time and sincerely apologize for the inconvenience,” its status page said at 21:59 UTC on 21 October.

What the exposed keys reach

Railway called its instruction to all users a precaution while it completed its review. By 01:22 UTC on 22 October it had published 13 guides. Among them are guides on checking for unexpected usage at OpenAI, OpenRouter, Grok and Stripe, checking for anomalous access to GitHub, AWS and Google Cloud, and rotating PostgreSQL, MySQL, MongoDB and Redis credentials.

The cloud guides describe how long a leaked key stays useful. “Access keys are long-lived, so an exposed key works until you deactivate it,” the AWS guide says, and it points readers to billing as well as to AWS’s CloudTrail logs. The Google Cloud guide notes that “by default, service account keys don’t expire”. A key that has left the platform keeps working until its owner turns it off at the provider that issued it, wherever the owner’s deployments stand.

Railway moved faster on the credentials it controlled. It began rotating every storage bucket credential at 22:10 UTC on 21 October, and at 04:17 UTC the next day it invalidated all Railway API tokens, which broke any integration with its public API until customers created new ones. At 19:06 UTC on 22 October it paused new signups, and they were still paused on the morning of 23 October.

One supplier, two jobs

Railway describes itself as “the all-in-one intelligent cloud provider for deploying web apps, servers, databases, and more”. For a startup or a small software company without its own operations team, the database passwords, cloud keys, payment keys and deploy button all sit with the same supplier.

Railway could withdraw what it had issued, bucket credentials and API tokens, on its own timetable. The keys customers had brought from OpenAI, AWS or Stripe could be revoked only at those providers, and their replacements could reach a running service only through Railway’s paused pipeline.

On 23 October Railway said it had emailed customers it judged to have “a very high likelihood of impact” and was still carrying out forensic analysis to find more. It said it planned to publish a preliminary incident report around midday Eastern Time that day, and its status page did not yet link one at 15:45 UTC.

Railway’s status log shows that on 22 October Enterprise customers could deploy replacement credentials about 18 hours before Hobby customers, after a notice that asked every customer, on every plan, to replace all of them.