Wicked Good Development Episode 13: Hacks and Ax, July edition

Wicked Good Development is dedicated to the future of open source. This space is to learn about the latest in the developer community and talk shop with open source software innovators and experts in the industry.

Ax Sharma, a security researcher at Sonatype and tech journalist at large, joins Kadi and Omar for his monthly malware update. Ax breaks down the latest on protestware and ransomware.

Listen to the episode

Wicked Good Development is available wherever you find your podcasts. Visit our page on Spotify's anchor.fm

Show notes

Hosts

• Kadi Grigg - Host - (Twitter: @kadigrigg)
• Omar Torres - Co-host

Panelists

• Ax Sharma - Security Researcher at Sonatype

References

• Protestware on the rise: Why developers are sabotaging their own code
• The Story behind colors.js and fakers.js
• FBI recovers $500,000 healthcare orgs paid to Maui ransomware

Transcript

Kadi 0:10
Hi, my name is Kadi Grigg, and welcome to another episode of Wicked Good Development, where we talk shop with OSS innovators, experts in industry and dig into really what's happening in the developer community.

Omar 0:21
Hola, my name is Omar, and I'll be your co-host for Wicked Good Development and today, we have an awesome Ax Sharma update.

Ax 0:35
Yes, hi, Kadi and Omar, everyone. Thanks for having me here. I'm Ax Sharma, a security researcher and a tech journalist. I just love to analyze malware and novel vulnerabilities and write about them. So here I am with you today.

Kadi 0:50
Well, thanks for coming back on, Ax. Let's dive in. The last time I spoke with you, I believe we were both reporting on the Log4Shell or Spring4Shell event that happened back in April. Can you bring us up to date on what you've seen happening in the cybersecurity space since then?

Ax 1:02
Yes, absolutely. So log4Shell was a long time ago. Since then, I will say this year started with a bang, when one of the most popular libraries, “colors” and “fakers”, these are npm libraries used by 1000s of applications, and these started freezing applications that were using them. And this theme evolved into initially, people thought this was a hijack, a malicious hijack, but it turned out to be protestware. A developer sabotages his or her own work just to make a point or protest an issue. And that team has not slowed down even in this into this month. I'm seeing in July, we are seeing protestware.

Kadi 1:36
So what is the point of protestware if you're going to be purposely harming your own team? Like I'm, I'm trying to understand what the point you'd be making is by doing that.

Ax 1:45
Yeah, absolutely. That's a good question. It's not an easy decision for a developer to, as you know, many open source developers, they are really, you know, volunteers and they're doing this activity in their spare time, they're not getting paid for it. So for a developer to dedicate, I don't know, hours, weeks, or sometimes years of their time to developing these applications that then become popular and to sabotage their own creation, you have to understand from their point of view, they must have been in some ways pushed to that direction that they thought that was the only choice left. And let me just give you a more concrete example. So with Log4J, as we know, many, of the maintainers of Log4J framework were too busy. They were burdened with the CVEs. You know, over the holidays, they were almost expected to patch the vulnerabilities and read them and issue guidelines immediately, and it's not their full-time job. So this kind of cascaded over to the maintainer of colors and Faker who said that, you know, he had previously said that he doesn't want to support fortune 500 companies with this framework that here he is building these libraries. So many companies use it, but nobody has supported his library or contributed to it financially or otherwise. So he just withdrew all the code. And actually, with colors with faker, he read through the code with colors, he introduced a malicious file, I should say, a problematic piece of code that froze your applications.

Kadi 3:10
Now, is this prevalent in a lot of like enterprise-wide applications?

Ax 3:14
Yeah, great question. So with these, some of these dependencies, right, if you're an Application Builder, and big corporations, you know, they're using these basic dependencies, maybe not colors, Faker, but say there was a node IPC dependency, you're getting very popular. So yes, any small changes made to these projects, especially if they're malicious, not all of these changes are necessarily malicious. But if they're malicious, they will affect your production. If unless you have some security measures in place, we can talk about that in a bit.
But one point I want to emphasize is when I say the trend hasn't slowed down, this started with colors and faker that we thought was a one-off incident this year. Then we saw node IPC library, actually deleting code. So this wasn't just, you know, freezing applications, temporarily. This was actually deleting all your files if you're a Russian or Belarusian user. And the maintainer said, you know, it was to protest against Russia's invasion of Ukraine. So a very polarized reaction, obviously, many people praise the move, but some people were like, skeptical of it that this has eroded trust in the OSS community. And then the next trend I saw was more peaceful protests. So we have packages like style components, event source polyfill, and es 5x. And there are one or two more I'll reveal in the next few days that basically, these packages don't harm your computer. But if you install them, and if you happen to be based in Russia or Belarus, let’s say those regions, they'll just pop up messages like encouraging users to check out more reliable sources of news. They have links to the BBC site, that kind of thing. And then, very recently, this month, an incident did happen, which was compared by some to the left-pad incidents of 2016.

Kadi 4:55
Now can you remind us what left-pad was in 2016?

Ax 5:00
Yes, absolutely. So in 2016, what happened was, there was a KIK npm package called KIK, literally K I K, and its maintainer had been, you know, maintaining it for quite some time. And then there's this company called KIK. They said, you know, their lawyers reached out first of all nicely asking, you know, this name conflicts with our trademark. So would you be willing to, you know, give up your package name? And he just flat-out refused the request. He's like,
‘No, I've been maintaining this package far before you guys. And now you're just asking me to give away the ownership, which can actually break applications using the kick dependency.” But anyway, fast forward, npm decided to side with KIK. Just because of, you know, they said that they did have the trademark to whatever the legal paperwork was. And they took away his package. So in protest, this developer decided to withdraw all his code and dependencies from npm. And that dependency included left-pad. So this broke the internet because maybe KIK package created by this developer wasn't as popular, but the left-pad dependency was used by many, many applications. So let's say that so many applications broke, then I believe somebody had to intervene and restore this dependency. That was 2016.
Now this year, what happened particularly this month, was Markus Unterwaditzer. I hope I'm saying that right. Like he's the developer of atomicwritess Python library. And he deleted his code from the PyPi registry. And the reason is quite bizarre, because…

Ax 6:25
So PyPi, because there have been so many hijacks and stuff going on, right, lately. So PyPi mandated two-factor authentication for maintainers who were maintaining critical projects. And critical projects, according to PyPI are any software projects that make up for the top 1% of downloads on their website. So this atomicwrites Project fell into those top 1% projects. And so the maintainer got this email,” Congrats, you have been selected as maintainer of a critical project. Please secure your account, but two-factor authentication.” So in response, the developer decided to just delete his code from PyPI and republish it, just to get rid of this top 1% or critical project status. So some people criticized them, and some understood why he did this.

Omar 7:09
I have two questions, but I don't know where we should go first. Because on one side, I'm curious about the colors and fakers, like how widespread was that? How many, like do you know how much it affected? And then, my other question is more on the philosophical side regarding the open source community and if you have any insight regarding how does the open source community feel about this? Do they feel like these protests where folks have tapped into something true? And maybe they're feeling it too, like, oh, man, it's not cool for people to be using our work and be making money off of it? Right? Because that seems kind of like, that seems like a real struggle. And do you see there being a bigger tide of folks doing this kind of stuff?

Ax 7:57
Yeah, so great question. First of all, to answer the impact or the extent of impact of these protests. Well, let's take an example. The colors library receives over 20 million weekly downloads on npm alone. It's got 19,000 projects relying on it. This is according to GitHub. And faker was in the similar range, like 2.8 million weekly downloads. It's got 2500 dependents, that's 2500 projects, depending on faker. So maybe faker is not such a critical application. But then you look at node IPC. And maybe that one is right. And when you talk about impact, I think is with so much widespread usage, and we can be almost sure at least some of them or many of them wouldn't have the best security practices.
So we can expect that they would have their dependency version pinned to a safe version. Very likely, they're just pulling the latest version. So that's the first part of the question, the impact can be very widespread for a big library. If a small scale library developer does protest, and it's not harmful, it's not deleting anything, maybe it's not that big of a deal. But these cases that we are seeing into this month, same with the atomicwrites Python library, right? These are some big frameworks, I mean, top 1% critical projects, right, like top 1% of projects. According to the most downloaded projects, these are the critical projects, if somebody, a developer of a critical project does such a thing on Python or npm. I think that that will have some echoes.
Secondly, to answer the community reaction thing. It really depends on what kind of protest it is. So if it's just like, temporarily freezing your application or printing a message, I think that is forgivable on the community side. But if you're actually deleting files, and the user had no idea about this, right, then people may not like it. Because you know, you're eroding trust in the OSS community. And obviously, the developer's reputation is ruined. Let's face it, because in future not distant future. Nobody will trust you or any code you have ever written. They'll be like, “Oh, did he or she put backdoors in it.” or “Will they do that in the future?”

Ax 10:00
Yeah, so the later on protests were themes that we are seeing, which are just about like, Russia's invasion of Ukraine. These themes are, they offer a more peaceful approach. So they'll print like, yeah, like peace message, or “stop the war,” or “look at more accurate news sources” kind of thing. I think that sits well with the community.

Kadi 10:27
So I want to switch gears a little bit, actually. You brought up both npm and PyPi. And it's no secret that npm has always been riddled with, you know, vulnerabilities and these types of attacks. But what I have started to notice, and I would like your thoughts on this, is I'm starting to see more traction within the PyPI community. And I'm trying to figure out what that could be because of. So, what are your thoughts? You know, is this true that you've been seeing more traction in PyPI regarding vulnerabilities and hacks? Or, you know, Is that…. am I just losing my mind, which isn't a far journey?

Ax 11:03
Oh, I don't believe that. I think your observation is spot on. I don't think it's so much about npm and PyPI. I think there are many things going on here. So first things first, I think there are a lot of copycat actors, right? So threat actors when they've seen success, or just they have, you know, seen too much of security incidents impacting npm, they'll be like, “Ah, why aren't we looking over there. And there, we could also, in fact, RubyGems, or PyPiI or this or that or NuGet.” So they'll try everything, you know, whatever is the lowest hanging fruit. Now, the thing with many Open Source Repositories is they want to keep the barriers to entry to a minimum. So that's why say, npm, or PyPI, make it quite easy for anybody to publish a package. Because the whole idea is, as a developer, you shouldn't be struggling too much just to distribute your projects. But the downside of that is that also lose the bad guys and who now want to abuse the system.

Ax 11:54
And secondly, I think Sonatype, in particular, has been catching a lot more malware across both like PyPI and npm. Because our detection capabilities have improved recently, and we have expanded to other ecosystems, including PyPI. So that could also be the reason we are catching much more bad things in both ecosystems now.

Omar 12:05
Could you give us the latest? We know ransomware has been occurring a lot. It's not new, but I think it's rising in occasions. Could you give us the latest on ransomware? What's going on there?

Ax 12:18
Yeah, quite interesting updates just from last week or so. I mean, obviously, we know like Conte and Samir had, you know, encrypted the Costa Rican government, which led to national emergency basically, that's what their President declared, because multiple departments like Department of Health, Treasury, everything, was impacted by it. And it just, it's a it's a problem that's just not going away. Now, there's some good news there, like FBI did recover $500,000 that healthcare organizations have paid to one particular ransomware gang called Maui. So the Department of Justice is on this.

Ax 12:57
So the Department of Justice has announced the seizure of $500,000 in Bitcoin paid by American healthcare providers to Maui ransomware. I'm reading this article from Bleeping Computer. But I think that is a positive moment there. But when you look at the scale of ransomware, and how much organizations are losing to it, because one thing you have to understand, not every victim makes it to the news. Really don't.
I mean, if you're following the communities internally, you're looking at all the threat intel. Most people for ethical reasons, journalists or security researchers, they will not report on ransomware, myself included. I mean, victims are updated daily, but we're not chasing every story. Because obviously, you don't want to give publicity in the wrong way. That's understandable. But if it's a big, big company, and it's in the public interest and say they're hiding something, or like we feel like people need to know about it, yes. But the reason I'm emphasizing this is the ransomware you hear in the news; it's just a very small chunk of the pie to all the attacks that keep going on every day and negotiations that appear on these internal dark websites. And then they disappear within minutes. That tells you somebody got hacked, but they actually paid the ransom. So now their name is off the list.

Kadi 14:00
So Ax, you've talked about some of these types of different attacks. What is the solution for some of them? What do you do?

Ax 14:07
Right so, for example, when you cannot trust your dependencies, or say, to prevent malicious hijacks, and protestware, I think it's safe to pin your dependency versions. That's the simplest solution, even if you have no automation in place. In other words, if you are using, say, I don't know, CTX dependency version one, and even if later versions get hijacked, you will not be fetching those, you'll just have one. Now the caveat to that is you will also have to maintain it yourself. You will have to make sure it's working with the latest version of your other dependencies. So without automation, this can get a bit challenging.
The other thing you need to be sure of when you're appending dependencies not all ecosystems are immutable. So in npm, would not publish, I could not republish version one of a project, even if I hijacked it as far as I know, but in PyPi I saw, the CTX library got hijack version 0.1.2, which is a legitimate version, it was republished, and it had the altered malicious code. So, that's why I'm saying not all ecosystems are immutable. So even when you're building your dependency, some automation is necessary. But say for npm, you should be okay. If you just want to pin your dependency or make a copy of it, like fork, the dependency, maintain it yourself, much safer out. But again, you'll have to, you know, vet any future updates yourself. That's the caveat, but it keeps you safe.
Now say for phishing, or like when you receive macros and stuff if you if you follow best practices, or you know, for your team members and employees if you have some security training of phishing simulators, phishing simulator exercises, I know some companies, like Ninjio, curriculum does that. It helps your employees get better equipped with such attack vectors and defend against them.

Kadi 15:55
Any final thoughts?

Ax 15:56
Yeah, a lot going on in InfoSec land. Just watch out, and no need to panic, but just follow best practices. Have your antivirus software installed and monitor your environment all the time. And yeah, just don't click any links or attachments you do not trust.

Kadi 16:10
Well, Ax, till next time. Thank you so much for coming on today. We hope, we look forward to seeing you next month.

Ax 16:15
Thank you, guys.

Kadi 16:17
Thanks for listening to another episode of wicked good development brought to you by Sonatype. This show was co-produced by Katie Grigg and Omar Torres and made possible in partnership with our collaborators. Let us know what you think, and leave us a review on Apple Podcasts or Spotify. If you have any questions or comments, please feel free to leave us a message. If you think this was valuable content, share this episode with your friends. Till next time.