Post Mortem: Patching the PHP and Node.js Vulnerabilities

On the evening of 20 October, we got an early warning that PHP and Node.js were about to ship security fixes for critical vulnerabilities. Less than 24 hours later, the fixes were public, and every Laravel Cloud application needed PHP patched. We found no evidence that any customer was exploited as a result of these vulnerabilities.

This post covers what we did, what went well, and where we fell short. Laravel Cloud, Forge, Envoyer, and Vapor all needed different work, and the products that took the longest are where most of our lessons are.

#What happened

PHP and Node.js each announced security releases on 21 October. The PHP releases fixed two critical remote code execution vulnerabilities, and the Node.js releases fixed ten vulnerabilities, including a Brotli decoder flaw that could allow remote code execution.

The PHP project gave us a pre-announcement at 19:58 UTC on 20 October, and the public releases followed at 13:38 UTC the next day. Details were scarce on purpose, which is how responsible disclosure is supposed to work. But knowing a vulnerability exists is enough for people to start hunting, and word was already spreading online. We had under 24 hours to patch the fleet.

#Why our usual playbook did not fit

Most of the time, we handle a CVE by updating our base images and notify customers they need to deploy to pull in the latest image and pick up the fix. We generally avoid forcing redeployments, for two reasons. A fleet-wide rebuild is slow, and and could take many hours during a critical period. Applications also run their own deploy commands, which can include database migrations, notifications, and file changes, and we can't know which of those are safe to trigger on someone's behalf.

This vulnerability was severe enough that we wanted every workload we manage patched, so we needed a second playbook.

#Adding a layer of protection first

While we built the real fix, we added a layer of protection that did not depend on any patch. At 08:55 UTC on 21 October, hours before the PHP release was public, we turned on Cloudflare's managed firewall ruleset for our Cloud domains, including newer rules that flag remote code execution and anomalous code execution in incoming requests. By 10:24 UTC we had added a smaller set of those newer rules for Forge and our dedicated zones. A firewall can't cover every route to a vulnerability like this, so the rules reduced exposure and did not replace the patch. We are keeping them because they inspect traffic before it reaches a customer's application, which gives every application, patched or not, an extra layer against this kind of request.

#Getting PHP in time

We normally get PHP through the official Docker Hub images. With so many people about to pull PHP and Node.js at once, we weren't sure how quickly those would be rebuilt.

So we planned two routes and decided we would commit to whichever worked first. Docker Hub open sources the way it builds its images, so we could copy the compilation process and build the patched PHP ourselves the moment the release landed. In the end, we shipped our own image at 14:24 UTC, and we used it. Docker Hub didn't have patched images until hours after the release, and some of the third-party download hosts our image builds rely on were down for part of the day. Building our own image allowed us to move faster.

#Live patching running workloads

Cloud runs on Kubernetes, which works by reconciliation. Think of it as a loop that compares the desired state of each workload with its current state, then makes changes until they match. We write Kubernetes operators to add our own logic to that loop, and one of them, the app operator, manages customer applications in each cluster.

That gave us a place to put the patch. We updated the app operator to drop the patched PHP binaries into each workload when it starts, instead of rebuilding and redeploying customer images. Kubernetes workloads are immutable, so the operator restarts each one to apply the change, and the platform absorbs the restart so customer applications see no downtime.

A built-in pacer stops the operator from restarting everything at once, which protects downstream systems like our source of truth and our billing pipeline. We sped it up by 5x for this rollout, which our estimates put at about 35 minutes of pacing instead of about three hours. We stopped short of removing the pacer entirely, because a rush of restarts could overload those systems.

We first proved the mechanism in our dev and staging environments, then canaried it at 11:17 UTC in two of our lowest-density regions before the PHP release was even public. The first wave began at 14:31 UTC, starting in Canada and Tokyo and verified on a small group first. From about 15:30 UTC the patch rolled out in waves across the remaining regions, so we could watch for failures before it reached our larger regions and so the restarts did not all land at once. That staging caught problems before they reached most customers, and the mechanism is now a standard tool for us. By 18:05 UTC, every region was at 100%. We paused new deployments on Cloud for about three and a half hours while this happened, from 13:54 to 17:33 UTC.

Planning took longer than the code. We already had the building blocks.

#Timeline

All times are UTC. Times marked ~ are reconstructed from call recordings and are accurate to about 10 minutes.

#Forge, Envoyer, and Vapor needed more

Cloud was the easier case, because we control the runtime. The other products ask customers to update their own servers or applications, which meant different work and a longer day.

We require Laravel Forge customers to upgrade PHP on each of their servers, and the release triggered roughly 10 times the normal volume of upgrade jobs. Many of those jobs failed and retried against flaky upstream package repositories and outdated third-party sources. Deployments shared a queue with them, so they were starved until they timed out. The Forge control panel was slow for most of the day. Separately, the real-time updates in the UI stalled, and we moved them to a Reverb cluster on Laravel Cloud to recover.

Envoyer servers couldn't be patched, so we took it offline at 13:52 UTC and upgraded the servers. It was back by 22:58 UTC, and deployment health checks followed on 22 October.

Vapor needed new runtimes and images, and customers had to redeploy to pick them up.

#What went well

Laravel Cloud customers did not need to do anything. The runtime patch reached every running application without a redeploy in under 5 hours, and the canary gave us confidence before the real release. Building PHP ourselves removed our dependence on someone else's publishing schedule, and testing on a small group first caught problems before they reached most customers.

The team also shipped both the PHP and Node.js patches across a response that ran about 27 hours and handed off between time zones.

#What did not go well

The largest miss was on Forge. At about 17:45 UTC, we told customers the queue was cleared, and deployments should work. About 1.5 hours later, the queue was climbing again, and roughly 4,800 deployments were canceled as they had been queued for too long between 19:30 and 21:27 UTC. Those customers had to redeploy themselves, and our recovery message had come too early.

We first told Vapor customers that we would redeploy their applications, then changed the guidance at 19:25 UTC and asked them to redeploy themselves.

Behind these mistakes were a few causes. We didn’t have clear owners for each communication channel when the incident first began, which led to unclear messaging to users. We fixed this by assigning one owner to each channel. The same small group worked the first response for about 27 hours and owners changed as the work moved across time zones without always holding the full history.

The Forge, Envoyer, and Vapor teams are also smaller and have less monitoring than Cloud, and some of their systems ran end-of-life software that we had to rebuild instead of patch. We had no reliable record of which software versions every application was running.

#What we are changing

Much of how we ran communications held up but needs to be improved upon, and we intend to keep doing it:

• Incident commander sign-off: Every customer message went through the incident commander before it was sent. We are keeping that gate and intend to add more thorough claim-by-claim checks against the real status of each product before comms go out.

• Stable channel owners: After the early confusion, we gave each communication channel one owner and kept changes to a minimum.

Two things we are adding:

• A soak period for recoveries: We will wait a set time before announcing that something has recovered, so we know the recovery is real.

• Recorded war rooms: Every SEV1 and SEV2 war room will run on a call that keeps a transcript, and the playbook will require it.

Each product is also making its own changes:

• Cloud: We are keeping the in-place patch mechanism as a standard tool, so we can respond at this speed again, and keeping the Cloudflare firewall rules. We are tracking the exact runtime each application runs more strictly and removing end-of-life versions from the platform.

• Forge: Deployments now run on their own queue, so a surge of other jobs can't starve them. We repaired our package sources and repository keys, stopped retrying jobs for records that no longer exist, and added capacity and controls to absorb a surge of upgrade jobs.

• Envoyer: We are keeping it on a supported operating system and PHP version, so a future patch applies without downtime, and restoring and modernizing its deployment health checks.

• Vapor: We marked the legacy al2 runtime as unsupported in our documentation and are contacting customers still on it.

We are also reviewing our redeployment policy so there is a written answer for what happens when we can't patch a workload or automatically redeploy it.

#What we need to be prepared for

This will keep happening. Disclosures like these are now frequent, and AI is shortening the time between a vulnerability becoming known and someone trying to exploit it. We built a playbook this time, and we are working on making it routine, which we'll cover in a future post.

If you run Laravel Forge, Vapor, or Envoyer and haven't updated PHP yet, the Forge knowledge base walks you through checking your version. If you were affected by anything described above, thank you for your patience while we worked through it, and contact our support team if you'd like to talk it through.

#A note from the team

We want to end with recognition of the people who did this work. A small group of us carried the response from the moment the pre-announcement arrived on the evening of 20 October until Forge was back to normal two days later. When someone in one time zone had to stop, someone in the next one read back through hours of channel history and kept the work going. A few days later, many of us did it all again for the Node.js follow-up.

In about 27 hours, we built a new way to patch applications while they were still running. We tested it in two quiet regions, then rolled it out in waves while we watched every dashboard we had. We had never done that on Cloud before. It worked because we trusted each other to make calls, and because we stopped overnight when we were too tired to make good ones.

We also read every kind note from customers and partners about how we handled the response. Thank you. Those hugops notes meant more than you’d guess in the middle of a long day. To everyone who waited on a Forge deployment or a straight answer about Vapor while we worked: we’re sorry for the wait, and thank you for staying with us.