Remediation at Scale: Lessons from PayPal for the Equifax Security Team

In January 2016, Laksh Raghavan of PayPal wrote about how his company managed their response to a critical Java open source component vulnerability in one of their applications.  Similar to the vulnerable Struts2 component being implicated in last week's Equifax breach of 143 million consumer records, the vulnerability at PayPal would allow for remote execution of code -- the most critical form of software vulnerability.

Laksh described how their team discovered the vulnerability through an alert originating from their bug bounty program.  He then addressed how the team remediated the issue at scale, including short- and long-term efforts.  Critical to their success and speed of remediation were code repositories, automated application analysis, and prioritization of fixes.

The first thing they did was to take an inventory of their applications to assess which ones used the vulnerable component.  Laksh commented:

• If your organization has a good app inventory, it makes your life easy.
• Validate the manual inventory with automated scans and make sure you have a solid list of servers to be patched.
• If you have a centralized code repository, it definitely makes things easy.
• Make sure you don’t miss one-off apps that are not on your core Java frameworks.
• Additionally, reach out to the security and product contacts within all of your subsidiaries (if they are not fully integrated) and ask them to repeat the above steps.

He then recommended the following remediation steps specific to their use of the vulnerable commons collection component, but application to many vulnerable components:

Short-term

• First up, drive the patching of your high-risk Internet facing systems – specifically COTS products that have published exploits.
• For custom apps that use the vulnerable component, you can either remove the vulnerable classes if not used or upgrade to the latest version.
• If the owners of those systems are different, a lot can be accomplished in parallel.

Long-term

• Again, this vulnerability is not limited to commons-collections or Java per se. This is an entire class of vulnerability like XSS. Your approach should be holistic accordingly.
• Your best bet is to completely turn off object serialization everywhere.
• If that is not feasible, this post from Terse Systems has good guidance on dealing with custom code.

While we can't turn back time on the Equifax breach, we can learn from others who performed successful discovery and remediation of their vulnerable components at scale.

For those interested in assessing what, if any, vulnerable open source components have been used in your applications, Sonatype offers a free application health check service.  Analysis of an application can be accomplished in under 5 minutes.  Try it.

Readers interested in the full PayPal remediation process can read Laksh's original blog here.