065 - Discussing the SolarWinds attack with Aaron Rosenmund

Daniel speaks with Pluralsight author and cybersecurity professional Aaron Rosenmund about the SolarWinds supply chain attack. Aaron explains how it happened, what the response has been and what lessons can be learned.

• Aaron's Pluralsight courses
• Aaron's LinkedIn profile

If you enjoy this episode, please consider leaving a review on Apple Podcasts or wherever you listen.

Please send any questions or comments to podcast@pluralsight.com.

Daniel Blazer:

Hello, and welcome to All Hands on Tech, I'm Daniel Blazer. On today's episode, I speak with Pluralsight author, Aaron Rosenmund about the SolarWinds supply chain attack from a few weeks ago. We discuss how it happened, what the response has been, and what lessons are emerging as the dust settles. This episode is obviously a great one for security professionals, but it should also prove fascinating for anyone who's simply curious about security. Well, thank you so much for chatting with me today, Aaron, I'm really excited to learn a little bit more about what's been on with SolarWinds.

Aaron Rosenmund:

Yeah, absolutely. It's definitely timely and important and kind of been a crazy time of the year for people in my industry, honestly.

Daniel Blazer:

Yeah, definitely. I think we have a wide range of listeners to this show, so we have, definitely, security professionals that are listening, they probably already know quite a bit about this. But we also have some people maybe like me, they're interested in security, but they're not always up to date with everything. So, maybe to get things started, could you just kind of provide us with an overview of the SolarWinds supply chain attack? What's happened up to this point?

Aaron Rosenmund:

Yeah, absolutely. So, in December it essentially came out that FireEye had been compromised. Now, FireEye is one of the USs premier cyber firms, they do instant response and malware reverse engineering and all kinds of other stuff and products. But what's interesting is they also do red team operations. Red team operations are where you emulate attackers and you do that against organizations that have defenses that they want to test, and then you cyclically get better. You attack, you improve defenses. So, you'd pay someone like FireEye to do that. I bring that up because they had announced that they were breached, saying that they had lost their tools. So, they had developed custom red team tools, custom attack tools and those were stolen and everybody's like, oh man, FireEye, how irresponsible that you allowed your tools to get stolen.

And there's all this conversation, LinkedIn was blowing up, at least in my LinkedIn because of the people I follow, about, it's irresponsible and those are cyber weapons of war and all this crazy stuff. And so, then a week or two passes and I don't remember the exact timeline, but I remember I was out shopping with my family in the mall, over Christmas break, well, I was reading it, and then it comes out that, oh, just kidding, FireEye got breached, but so did everyone. And because FireEye recognized that they were breached, they were just the first one to figure it out. And when they went back and found out what we call the root cause, or they did root cause analysis to figure out how their tools were stolen, they found out that it was from the SolarWinds Orion tool.

And so, SolarWinds, a lot of people are familiar with that. It's a IT management and infrastructure management tool among other things, I think they have a ton of different plugins and capabilities. But the SolarWinds Orion tool specifically was the product that the root cause led them back to. And what they found is that updates from SolarWinds that were digitally signed. So, when you digitally sign software, it essentially uses PKI infrastructure to say that this update, this piece of software that I downloaded, hasn't been compromised, it is from the people who they say it's from. And it just validates that software is not changed in between the time that the manufacturer made it and that you installed it on your computer. And so, a lot of unsigned software won't run on devices, so it's a nice safeguard. But in this case, they tracked it back to signed code that was running as part of the SolarWinds Orion update.

That was actually the root cause in installing malware that's now being called solar burst malware. And so, attackers had gained access to FireEye through that solar burst malware, and then spread laterally through their network. And there's a bunch of other ways that they spread laterally, not with the same malware, and then gotten access to those tools. The problem is that the update source was compromised. And so, that's why we called it a supply chain compromise. The supply of the update to that SolarWinds Orion tool, the place where you download that update from was actually, you were downloading software that worked as an update, but it also included the solar burst malware. And so, that's what we're talking about with the supply chain compromise. And that had been happening, the dates are still getting completely figured out, but basically from March, 2020 through May, 2020. All of the updates that were released for that tool were all compromised with this malware.

Daniel Blazer:

One question that came to mind as you're talking about this, the supply chain attack, that's how you categorize it, is this one of the largest supply chain attacks of the last couple years or something like that?

Aaron Rosenmund:

Yeah. So, I guess to steal a quote from... I don't know where I heard it, but I'm sure it's somebody else's quote, I just want to reference that I'm stealing it, is that, "I think 10% of what happens on the internet that's bad, no one's ever found." So, when we say things like largest supply chain attack that's ever happened, I'm like, yeah, probably the largest supply chain attack we've ever found. Fair enough, this attacker's activity, it's been attributed to Russia, but attribution's weird, so I try to avoid attribution in that. There's a campaign called Olympic Destroyer, just a little side note here, and the malware had so many layers of signatures that were all false flags that all the experts in the world of the period of a year basically went saying, oh, it's Korea, no, it's Russia, no, it's China.

And just kept hopping back and forth on who it was attributed to. And I don't know... It finalized on one of the countries. But the main point is that for six months of period of time, the whole world thought it was Korea. And for another six months, they're like, no, maybe it's actually the NSA. And then another six months, it's like, no, it's the GRU in Russia. And so, attribution, I leave that to the governments to figure out with their job. But the main point was that, largest supply chain attack to ever happen. There was... CCleaner is a really good example. So, it's freeware that you can download and it'll clean your PC. And it was a good tool.

I mean, for many, many years people used it, I'm sure many people are still using it. But again, in the same way that software update was compromised, and then anyone who installed software update during the time that it was compromised now had malware on their box. We've seen this before. I do want to make a point that this is a software supply chain compromise, so it's a difference from a hardware supply chain compromise. So, there's not a chip that's on a box that you installed into your network that's now calling back to some malware organization. Instead, it's a software supply chain compromise, and that somewhere in the development of the software, code that wasn't supposed to be there, not from the company, was injected and then signed and distributed. That's the difference.

Daniel Blazer:

Yeah, I think usually when I've heard supply chain compromise, like you said, I think of a hardware supply chain, so it's interesting to make that distinction. With this being as widespread as it was, I read a thing that it was 425 out of the fortune 500 companies, so a huge percentage of them were affected by this. What is being done right now to just try to get some sort of resolution, try to stop the bleeding, so to speak?

Aaron Rosenmund:

Yeah. So, when an attack like this happens very quickly, and especially FireEye being the first ones to find it, and Microsoft jumped on this as well. So, Microsoft Defender has signatures for it that'll detect if malware is on your system. And FireEye released a excellent article, Reversing the Malware and How it Works. We're lucky for, not to get too technical, for those of you who understand .NET (dot net). When you reverse .NET, it actually reverses back into readable code. So, it's a bit different from trying to reverse a C++ compiled program, or a golang compiled program where there's a lot of figuring out what the assembly's doing. With .NET when you reverse it, it compiles back to the code, so that makes that easier. But you still have to understand what's happening. A lot of this being wide spread is attributed to a few different things.

And the first thing is that, a lot of people use SolarWinds, so good for SolarWinds. The rest of it is that that was very intentional targeting from whoever created this malware. And not everyone who was infected was breached, so to speak. So, what has happened so far is that the little bit of investigation that's been done, we found that, whether it's 425 of the top fortune 500 companies, or if it's 30,000 companies or whatever it is... I was actually trying to look up a good number before we got to this, but it's all over the place and not everybody's going to raise their hand and say, yeah, I was breached. Because what they want to do is find out yeah, I was breached, that means there was a connection back to the attacker.

We call that a C2 server, a command-and-control server. So, you install the update, how this works is, 12 to 14 days later, mainly so you don't attribute it to the update, the malware will then execute. It's not even an executable, it's a dynamic link library. So, it's loaded in conjunction with the Orion software in-memory, and it's going to call back out to domain controller or a domain server looking for a specific domain. And it's always the same domain, so that's interesting. I'm going to bring up a few key points. And I think as you continue to ask me questions, I'll try to bring that back around to why those things were important, especially from a security skills perspective. What did we miss? But when it calls back out, it then is given a very specific, additional domain to continue the command-and-control to.

So, it's going to go back out to a different server than that original one. And that was assigned based on the name of the organization, and the domain name of the organization and the name of the box that it was running on. And so, what the attacker can then do is go through and choose. So, that's now siting in a database somewhere, and the attacker can go through and choose, hey, oh, this is the Treasury, or this is FireEye, or this is whoever, and now they have these backdoor access into all these companies. So, just because they were infected, doesn't mean that things happened after that point, is the point I wanted to make.

Daniel Blazer:

Yeah. That makes sense. Is this all automated, as far as them choosing who they want to breach? Is it they've set this all up beforehand or is it literally whoever is responsible for this manually going through and browsing their options?

Aaron Rosenmund:

Yeah. So, this is Aaron Rosenmund speaking, my guess is it's manual. When you have an attacker this sophisticated, they have very specific targets that they're looking for. And if you automate something, it's more likely it'll be caught. And they took a lot of care not to be caught, for the most part. I mean, malware's still doing malware things, like calling back over C2 channels and using HEDPS instead of HTTPs. I mean, all of those things were there, so you can see the communication, it's not an encrypted channel on the network. But yeah, the main point is that they're definitely going through and manually picking who they want to take after action on. So, the automation portion of this was that if you install the SolarWinds Orion software, you were compromised. Malware was installed.

Now, did the author of the malware then go back in and using the information that they had that was automatically pulled from your network, now go and laterally move, we call it. So, they'd move from the server or the infrastructure piece that had the SolarWinds software on it and they'd scrape credentials, use valid credentials, go log in. And that's part of their TTPs, or tactics, techniques, and procedures. Actually, that initial compromise, though you may have cleared it now, so we get to like, what are we doing now, though that may have been cleared because we have a lot of information about, hey, if you have this specific version of the software update, you need to delete it and all that stuff, and probably wipe that box. That's great. But the advice from the DHS was actually revised to say, you need to go investigate it.

Pull the image, identify where that compromise box was talking to afterwards, and what artifacts you can pull, because if there was after action, it's not going to be through the same malware. They didn't then go like, okay, I'm also going to install on this box and this box and this box, and it's all calling back using the same malware with the same signature. Instead, they simply would find valid credentials, there's ways of doing that once you're already inside a network, and then they'd log in over RDP in normal ways that you log into your environment with valid credentials, from IPs that were in the country where you're operating. So, you can't look at a globe like, oh, there's a red network from another country, that's the one, that's bad. That level of security analysis is now not going to catch it.

Daniel Blazer:

And that was all just intentionally to avoid being caught, that strategy?

Aaron Rosenmund:

Yeah. This is really my favorite part of the conversation. So, this malware did a number of things. And when we talk about what's really advanced, it's a matter of perspective. So, you probably hear my perspective a bit, I'm not saying it's not advanced, usually malware authors are kind of lazy, in that they're looking to quickly develop malware, get it out and accomplish their goals. Now, instead, now we're looking at a very advanced threat actor, and this campaign, as far as we know now, actually started with initial access attempts into SolarWinds back in September of 2019. So, this wasn't something that just happened. This has been going on for over a year now. Yep. So, that being said, there is a lot of care taken in the way that they built the malware and the way it was deployed to not be caught.

So, making the effort to compromise SolarWinds to get into their CI/CD process, however that was done. It's still not confirmed whether it was through IDE or if it was an insider that allowed them to gain access. Getting their code into sign code, that was then put on a valid distribution point for updates, all of that is a lot of work and really advanced. It takes time to develop and make that happen. Now, the piece after that is, now that that software's downloaded and executed. When the malware executed, the care that was taken is that it would use domain names for access, like the box that you would see access in your environment would have a very similar domain name to the devices that were in your environment. Okay. So, what does that do?

For our really base level, and this is what this gets at for what we're doing about it now with people, we have a people gap, a skills gap here, is at the really base level when you're doing security analysis... So, you're looking through a list of devices connected to your network, and it doesn't say evil solar burst malware box. But it doesn't say that. Instead, it says, if our boxes are jeff1@domain.com, it's going to be jeff2@domain.com. That'll be the name of the box that's connected. So, if you do a quick scan through that list, you don't identify that as being something that's potentially malicious. So, that care was taken. Same thing for the IPs, that were connected for remote access, is they took care that the IPs that were connecting were assigned to the country where the organization was.

So, if it's a US based organization and you're accessing assets in the US, the attacker IP was also in the US, which now is like cloud infrastructure, it isn't very hard to do either. But again, that removes the... Baseline skill level as a security analyst, is there a weird connection way over here? If I'm in the US and there's a strange connection to China, it's the only one, then that's probably bad. Well, okay. Now, that level of analysis won't catch it. And so, things like that. If they made files, when they're laterally moving, they did have to use SMB, which is a way to just move files around inside a network that's domain joined. When they made files, they delete them afterwards, so it'd be a temporary file use. And so, there's things that make it where I can't now just find a bunch of artifacts, but what I can do is look for more advanced stuff.

So, I can look for why is the SolarWinds Orion executable talking to the... Let me see what the domain was, it's AVS VMC cloud.com. So, why is the executable talking to that? Should it be? And if I have proper network inspection and I'm tying that to executable network connections on devices, then I can do that. But that level of detection is the next bump up, and there's not really a good way. I call it behavioral detections. It's not really a good way for malware to get away from it if they want to execute command-and-control, if they want to be able to accomplish their goals.

Daniel Blazer:

Everything that you've touched on, you've mentioned some fairly advanced ways to cover their tracks, the people responsible for this attack. If I am just at a regular average sized company, what could I have done to avoid this?

Aaron Rosenmund:

Yeah. And that's the sentiment I'm getting from a lot of people, is well, they were super advanced, so there's nothing we could have done. And too bad now. And there's two problems to that. The first one is, our current response. And you can read this anywhere, you look it up, what's going on with our current responses saying, hey, if you have this installed, wipe that computer and then reinstall it with the new version now that the NSA is actually approved, doesn't have compromised software in it. So, if you want to keep using SolarWinds, you can, it just has to be this new version. And go ahead and delete the box that was likely compromised, and then rebuild it. Like we said before, that's great. But it doesn't allow you to identify.

They started with solar burst malware, they're actually, when they're moving around, they're deploying different kinds of malware, so you likely now have an infected network. You need to go threat hunt through that network. And I just brought up the word threat hunt. And a threat hunt team isn't in a small, medium sized business, you don't have a team to go do threat hunting. And turns out, and you can read this in many different places, across the US, across the world, threat hunting teams are more advanced security teams going and hunting adversaries or hunting attackers. There's not enough of those in general. There's not enough people with the skills to be able to identify, was there lateral movement? Or what happened? And this compromise was so large, it really pointed out our gap. And I think we've been making gains. And do we have enough people in security in the first place?

We have warm bodies. But now we need to close the gap on the difference between an entry level skill and teams of people that are advanced enough to go find this stuff. And that's our next challenge and something I see being a challenge as this continue. And it's been really highlighted by this. So, back to your actual question, which was, what can you do about it? I think the first thing is, there's some really easy stuff to see before you delete that box. What else is it talking to? You can run some simple commands. I mean, obviously we both work for Pluralsight, there's training to figure out how to run netstat and understand what executables are talking to what other boxes. Just that level of stuff will allow you to keep that information somewhere. And if you do decide that you have time to go do a threat hunt, you at least now have somewhere to start. So, that's the first thing.

If you then need to wipe it and reinstall, and that's the level you're at, that's fine. And that's kind of what the DHS revised to say is, if you have a team that can do threat hunting and can do forensics, then do that first, then turn it off. If you don't, just wipe it. And we're kind of hoping for the best now, because you're likely compromised if you run that target list for after actions, then you likely do have other malware now. But the other malware, if you look at other malware that's been found, and there's little information on this because you already have to be advanced enough to find the solar burst attack, and then you have to have a team come in and actually hunt through your network to see what else happened. And then it has to actually be attributed to the SolarWinds attack in the first place.

Just because there's malware, it doesn't mean it was the SolarWinds attack. So, being able to tie those things together is a whole nother level. But when you do that, they're using something called TEARDROP, which is just a in-memory non-file based malware that's dropping what's called a beacon. And beacon is actually from a tool called Cobalt Strike. Cobalt Strike was made here in the US by Raphael Mudge. I think they just got bought by Core Impact. It's a red team operations tool, it's used all the time and automates a lot of the red team operations capabilities. And there's known signatures for Cobalt Strike beacons. And this isn't the first place Cobalt Strike... I use it every year. I run a military exercise for the red team every year, we use Cobalt Strike. You can change the way the beacon looks, but there's very specific detections for how you find that beacon in-memory.

How that software looks in-memory is very specific, the Cobalt Strike. And you can go look for that in your environment. So that's really some basic stuff. Threat hunting starts to sound intimidating, but you can just go and say, hey, let me look across my network connections and see if there's any weird ones, first of all. If I can look at what domains are being resolved, can I find a domain that is referenced in the documentation associated with this compromise? There's some really basic stuff and that's not too difficult to get to, even for small businesses.

Daniel Blazer:

If I were working at a company and we were doing all of those things that would've prevented what's happened with SolarWinds, or at least plant something like it?

Aaron Rosenmund:

I don't want to say prevented. What you really are depending on here is time to detection. That's what we're worried about. In this case, no one detected it for roughly a year. And that points out that, yes, if you just, as an attacker, you follow what we call good trade craft, and this is good OPSEC, I'm doing the things I should do. Everything I told you this malware does, that's not like, boom, no one thought of that before, that's so crazy. It's all just like, oh yeah, if I'm going to attack somebody, I'm not going to come from some weird IP that they shouldn't normally see. The box names I use should probably be the same as their box names, that way it doesn't look weird in the list somewhere. And that kind of stuff isn't crazy, it's just the attention to detail. They did it, in every case where there was a place to say, this is good trade craft and this is how I should operate, they did it.

And down to, even on the box with the files that they drop or all names, something that makes sense for SolarWinds, they changed. Yes, I mentioned they used HTTP. So, on the network, if you're looking at things, honestly, that's one big here as I look through everything. If you just look at your network connections, almost all of them should be HEDPS. You should be using SSL. I don't think I've browsed to a website that wasn't using SSL for two years. So, if you see a connection that's not using SSL, that's weird. That's probably something everybody can go do right now. So, if it's just HTTP, that means when you look at that stream, it's clear text. Now, was everything inside it not encrypted? Kind of. They use base 64, and I don't want to get too technical, but I mean, that's encoding it's not really encryption, so you can reverse that.

They used XOR with a single bit. I mean, it was al really basic stuff. So, you couldn't see the exact commands they're running on your C2, but still it's all clear. It's like random JSON going back and forth on HTTP in and out of your network, that's bad. You probably should check that out. And so, because of the HTTP access, they even went down and renamed the protocol that it shows up in a protocol analyzer to be OIP, so Orion infrastructure protocol. Just that level of attention to detail. It didn't actually change it, it's still HTTP, you can still read everything that the C2 or the commanding controls sending back and forth. But what it did do, is make it where if you're just, again, if you're just looking through a list of what protocols are in my network, it's like, oh, it's just Orion infrastructure protocol, it's fine, coming from the Orion executable. And so, everything was very surface, and if you went one layer more deep, then it's all the same bad malware stuff that you would normally see. But it was very, very good attention to detail.

Daniel Blazer:

We discovered this about a month ago, where does this story go from here? Is this something that will take years to get to the bottom of, to find out who's responsible? Or is it something that maybe in the next couple of months, we'll have a clearer picture?

Aaron Rosenmund:

I've worked in a lot of different domains, whether enterprise or government and the reality of will we really understand who's responsible for this? I don't know. And I think even if we do, I don't know that we the public will understand. Or maybe we will, I'm not really sure. And so, when you get to speculation, it's like, I'm not sure that this will ever publicly be released, that's these people responsible. That's happened in the past. We've done a... But that's really a country to country thing. And is the government going to assign blame? We've even assigned blame down to specific people who worked in military units associated with other countries, that's happened. But it takes years. And even then, any real expert will tell you it's really difficult to do that attribution piece, especially in an attacker that's this advance.

I mean, if they covered their tracks just in the forethought in how this malware operates that well, they likely cover tracks that leads back to specific people really well also. But that gets out of the domain of cybersecurity. And I think that's the other reason I like to avoid it, is because the way you find this out, we're now talking about human intelligence. It's governments, intelligence networks will know people who are basically double agents who work in other governments teams and that's how they find out. So, if that real attribution comes out, we likely won't know about it. You don't want to burn that asset, so to speak. Which sounds like really cool and spy, but it's really just, instead of tracing back the packets, we're just asking someone a question, is the real basics of it.

Daniel Blazer:

That's fair enough. It does start to sound a little James Bond-ish, but that definitely makes sense. We've talked about the enterprise response, what about just the security community as a whole? What lessons are being learned from this whole experience? What conversations are being sparked?

Aaron Rosenmund:

I'll try not to color my response to mention my own personal opinion, but I think one really needed spark, or one really needed point of contention that this is bringing to the forefront is a lot of companies have focus. If they do start focusing on security, it's focused on meeting a audit or a compliance requirement or it's focused on, am I meeting my regulatory requirement? What kind of data am I storing? Do I meet HIPAA? All that stuff. And if when you look at what that does is, it gives us a piece of paper and we can go down the piece of paper and the company has to have a team that looks at this stuff and say, am I checking this box as a control? And it's down to, if I'm storing HIPAA data, is it encrypted in the right way? Do I have passwords on my machines?

But when you look at it, every single one of these companies that was compromised met all of those controls. A lot of the government organizations have yearly requirements for pen tests and for audits and all kinds of stuff. They have tons of pieces of paper and policy saying that, here's how we're protecting ourselves in cybersecurity and in our cyberspace and our IT infrastructure and all of our data. And then this definitely still happened to them. It still happened to them and lasted over a year and no one caught it. And if FireEye didn't catch it, they definitely weren't going to catch it. So, that's the thing that I see changing the tide here, is that more advanced companies with a serious focus on security are looking at detection. Now, I said, there's not really a lot of prevention we can do for this kind of thing.

And this, it happens all the time. We see in the news, there's constantly breaches coming out, and this person got breached and that person got breached. I think intrusions and breaches are going to happen when you have attackers as smart and persistent and well funded as whoever's responsible for solar burst. They're going to find a way around it, whatever policy or protection you have. And whether that's signed code, well, they found a way into the signed code. They found a way into an executable that you install every day. You installed the malware for them, in this instance. And so, that's not going to go away. There's always going to be that cat and mouse or the attackers versus defenders technology advancement there. It's like the cold war of cybersecurity. But what can change is, do we have sufficiently advanced people in place to detect it?

And if we're doing those things, why is this executable talking to this domain? There's no reason it should be talking with that domain. That level of thought at all of these organizations and what we're calling security operations is really how I refer to all of that. It's the cycle of... It's the line between, I have a policy that says here's how cybersecurity should work, and then security operations is saying, I'm monitoring the network traffic and actively looking for detections. I'm running red team operations that simulate people that I'm concerned about, or similarly adversaries that I'm concerned about, and then making changes to my detections and making changes to my environment based on what we find. I have a team that's actively doing threat hunts. That team requires higher level skills and actively doing things on the network for an active defense. But I think that's where this conversation's going to go in the future, because that's where we would've went from, no one caught it over a year, to, we caught it in a few days after it happened.

Daniel Blazer:

What I'm hearing from you is, maybe the biggest takeaway from this whole story, is moving from more of a reactive to a proactive stance, a little more from defensive to offensive. Is that fair to say that?

Aaron Rosenmund:

I don't know if I want to call it defensive to offensive. I think it's more of a passive to active. So, we passively set policy and we passively check boxes that meet regulatory requirements, but we're not actively checking that we're defending against attacks. We're not actively checking the network to make sure that there's nothing weird going on. We're not actively investigating executables or lateral movement or to see if there's beacons from really well known... I brought up Cobalt Strike. The reason I brought it up is that's an extremely well known software. There's detections for those beacons, even if at different points in the stages of this attack. So, solar burst happened, yeah, that was super advanced. But then they actively used that C2 channel to then migrate to another box. But before they did that, they'd have to run some sort of scraper to pull credentials.

Probably should have caught that with something on the box, that they run the credential scraping. And then when they laterally move, probably should have caught that they're making weird temp files, they're executing commands to pull down a known software, that's bad. Probably should have caught that. They're now beaconing out with a new kind of malware that's not as advanced, probably should have caught. It's all those points that we probably should have caught those things, and if we had active defenses in, we would've.

Daniel Blazer:

Along those same lines, do pen testers need to change their tactics in response to this whole event?

Aaron Rosenmund:

A pen test wasn't going to stop this from happening. So, a pen test, at least in our definition, is different from a red team operation. Okay. So, when you're doing red team operations or adversary emulation, you're saying here's a known adversary... Say I'm in banking. This is a known adversary, they attack the banking industry. I do need to be concerned about them. I understand what their tech techniques and procedures are. I'm now going to have a red team emulate that adversary over time and see how my defenses and mitigations hold up. And then at the end of that, we'll wash out and we'll say, here's where we succeeded, let's keep this up, I really like whatever network appliance I paid for is working great. Or it's not, and here's where we failed. And I need to... This firewall solution is not really working for us. Or we don't have a detection in place for lateral movement that we should have, that kind of thing. We'll find those things out when we do that red team emulation.

Pen test, generally speaking, and I say generally, because some pen tests do go that far, they're just usually not as targeted, so that's where we draw that line. Pen testing is going to say, hey, can you get into my company's stuff? And they'll say, yeah, let me try. And then you try to get in. That's a bit of a oversimplification, but really the main difference is in the detail. Now, if they tried to get in, would they have tried to get in through a software supply chain attack through SolarWinds with signed code? Absolutely not. There's no way, that would've been way outside the scope of a pen test for them to use another company's software update to attack you. So, you shouldn't expect that a pen test would've identified that vector for you.

Now, if you scoped either a pen test or if we're getting into red team operations territory, if at this time you say, I am a company that needs to worry about a actor that's this sophisticated, I need to worry about software supply chain attacks, then you can pay a team to come simulate this attack. And they can do it in a number of different ways. You can get that set up so that it is in scope, and now you can see if you can detect it. So there's teams right now, I guarantee it, that are going around saying, yeah, we're going to emulate this entire attack chain, and you can see if you're good to go or not.

Daniel Blazer:

I think it goes back to what you were saying before, about shifting your perspective to be more active rather than checking the boxes, right?

Aaron Rosenmund:

Yeah, absolutely. Well, it's also, how hard is it to find something that no one realizes is a thing? And I guess to rephrase that, no one's ever seen what happened with SolarWinds before, so I can't come emulate that activity because there's no threat actor who did it for me to emulate it.

Daniel Blazer:

Fair enough. I know this is overall this is a complicated story, there's a lot of different pieces to it. But as we wrap up here, if you were going to distill this down and you were going to leave some final words of advice, encouragement, et cetera, with the people listening to this podcast, what would you want to say?

Aaron Rosenmund:

The biggest thing for me is to take this as a lesson learned. So, part of the instant response process is, you get in, you figure out what happened, you find out the scope of everything that happened. And I think as a community, responding to this as a community, we're going through just a massive instant response process and will be over this year. What we really need to make sure is, as we identify these root causes, and maybe the root cause is we didn't have sufficient detection, or maybe the root cause is we shouldn't just trust every digitally signed software that comes in. Or whatever we, as a community, decide, here's the best practices for what was the root cause of this incident. We need to take that and apply those to our environments.

And that's where the lessons learned phase of instant response, is when we take those lessons, we need to actually apply them to make change in the way that we do security, in the way that... All the way from the organization we just talked about. It's like, yes, we should have policy that this... This will generate policy, a hundred percent. I'm not saying it won't, but we need to understand that the generation of policy in itself doesn't actually protect us. So, I'd like to see that policy backed up by that active defense that we talked about. That is the lesson learned that comes out of this event.

Daniel Blazer:

It just seems too easy for me to also tie in there what you said about making sure you have people with the right level of skills. Like you said, there's more security professionals out there now than there once was, but are they at that right skill level to really be as helpful as they need to be in situations like this?

Aaron Rosenmund:

Honestly, you're right, it's too easy of a pedestal. But while it's here, this isn't a hiring problem. I do want to make that clear. This isn't, we need to hire the right people, it's, we all agree the right people don't yet exist. So, what does that mean? It means it's a training problem. And we can hire people who have an interest and a passion for cybersecurity, but they need a source that can train them in those advanced skills that we talked about. And the thing about cybersecurity is, that doesn't mean you're good to go. You don't watch one course, even from Pluralsight and you're like, all right, I'm good now. It's a ever moving target. Advanced is current. Those are the same meaning in cybersecurity. So, I have to understand what my current threats are and I have to understand what the current ways to mitigate and detect those are. And that means we have to go from, okay, I'm entry level in the cybersecurity, I now need to get trained up to current.

Daniel Blazer:

I guess what we were saying before about a company being active, it's also up to each individual security professional to think in terms of being active for their own knowledge and their own skillset.

Aaron Rosenmund:

Yeah, absolutely. And it'll help too, as this helps define that shift of what cybersecurity is for a organization, which I hope it does. It's a big enough event to be a catalyst for change, to say, hey, it's more of that active defense stuff that we all need to worry about. And if as an industry that's more well known, then that makes people as individuals understand that that's the stuff that they need to start getting into.

Daniel Blazer:

Yeah. That makes sense. Well, thanks so much, Aaron, for chatting. For me personally, this has been informative. But I think hopefully for everyone listening, it has been as well. It's also, I feel, brought a little bit of clarity to a story that otherwise is just a little bit complex for people to digest.

Aaron Rosenmund:

Yeah, absolutely. And I think in... I'm sure there'll always be someone who'll be able to point out, oh, that's not quite correct. But the main point is we wanted to get the big [inaudible 00:40:26]. And it's an ever changing story. So, I appreciate you having me.

Daniel Blazer:

Thank you for listening to All Hands on Tech. To see show notes and more info, visit pluralsight.com/podcast.