jerry: [00:00:00] All right, here we go today. Sunday, July 17th. 2022. And this is episode 268. Of the defensive security podcast. My name is Jerry Bell and joining me tonight as always is Mr. Andrew Kellett.
Andy: Hello, Jerry. How are you, sir?
jerry: great. How are you doing?
Andy: I’m doing good. I see nobody else can see it, but I see this amazing background that you’ve done with your studio and all sorts of cool pictures. Did you take those.
jerry: I It did not take those. They are straight off Amazon actually. It’s.
jerry: I’ll have to post the picture at some [00:01:00] point, but the pictures are actually sound absorbing panels.
Andy: Wow. I there’s jokes. I’m not going to make them, but anyway, I’m doing great. Good to see ya..
jerry: Awesome. Just a reminder that the thoughts and opinions we express on the show are ours and do not represent those of our employers. But as you are apt to point out, they could be for the right price.
Andy: That’s true. That’s true. And that, and by the way, what that really means is you’re not going to change our opinions. You’re just going to to hire them.
jerry: Correct. right. Sponsor our existing opinions.
Andy: Someday that’ll work.
jerry: All right. So we have some interesting stories today. The first one comes from SC magazine dot com. The title is why solar winds just might be one of the most secure software companies. In the tech universe.
Andy: It’s a pretty interesting one. I went into this a little.
Andy: Cynical. But there’s a lot of [00:02:00] really interesting stuff in here.
jerry: Yeah there, there is, I think
jerry: What I found interesting. A couple of things. One is very obvious. That this is a. Planted attempt to get back into the good graces of the it world. But at the same time, It is very clear that they have made some pretty significant improvements in their security posture. And I think for that, it deserves a.
jerry: A discussion.
Andy: Yeah, not only improvements, but they’re also.
Andy: Having these strong appearance of transparency and sharing lessons learned. Which we appreciate.
jerry: Correct. The one thing that I so we’ll get into it a little bit, but they still don’t really tell you. How. The thing happened.
jerry: Obviously it was aliens. They did tell you what happened. And so in the. Article here they describe this the [00:03:00] CISO of solar winds describes that the attack didn’t actually. Change their code base. So the attack wasn’t against their code repository. It was actually against one of their build systems.
jerry: And so they were the adversary here. Was injecting code. At build time, basically. So it wasn’t something that they could detect through code reviews. It was actually being added as part of the build process. And by inference the head. Pretty good control. At least they assert they had good control over their
jerry: source code, but they did not have good control. Over the build process and in the article they go through. The security uplifts they’ve made to their build process, which are quite interesting. Like they I would describe it as they have three parallel. Build channels that are run by three different teams.
jerry: And at the end of, at the [00:04:00] end of each of those, there’s a comparison. And if they don’t. They don’t match, if the. They call it a deterministic build. So there are like their security team does one, a dev ops team does another and the QA team does a third. And all building.
jerry: The same set of code. They should end up with the same final. Final product. All of the systems are are central to themselves. They don’t commingle. They don’t have access to each others. So there should be a very low opportunity for for an adversary to have access to all three.
jerry: Environments and do the same thing they did without being able to detect at the end, when they do the comparison between the three builds, whether it’s a novel approach. I hadn’t thought about it. It seems.
jerry: My first blush was, it seemed excessive, but as the more I think about it, It’s probably not a huge amount of [00:05:00] resources to do so maybe it makes sense.
Andy: And also, they mentioned that three different people are in charge of it. And so to corrupt it. Or somehow injected. Into all three would take. Somehow corrupting three different individuals, somehow some way.
jerry: Yeah, they would have to clue the three teams would have to collude.
Andy: Which. Is difficult.
jerry: Yep. Absolutely.
jerry: So they actually I haven’t looked into it, but they actually say that they’ve open sourced their their approach to this the multi kind of multi what I’ll just call multi-channel build. I thought that was. Interesting.
jerry: So There’s a, it’s a good read that they talk about how they changed from their prior model of having one centralized SOC under the. The company CISO to three different SOCs that monitor different. Different aspects of the environment. They went from having a kind of a part-time.
jerry: Red team to a [00:06:00] dedicated red team who’s focused on the build environment. I will say the one. Reservation I have is this kind of feels maybe a little bit like they’re fighting the. The last war. And so all the stuff that they’re describing is very focused on. Addressing the thing that failed last time.
jerry: And, are they making equal improvements in other areas?
Andy: Could be, I would say that.
Andy: They’re stuck in a bit of a pickle here where they need to address. The common question is how do you stop this from happening again? That is. That is what most people are going to ask them. It’s what the government’s asking them. That’s what customers asking them. And so there. There’s somewhat forced, whether that’s the most.
Andy: Efficient use of resources, not to deal with that problem right there. They have no choice. But I also feel like a lot of the changes they met, build change to their build process. I would catch. A great many other supply chain type. [00:07:00] Attack outcomes.
Andy: It seems to me.
jerry: Fair. Fair enough.
Andy: It’s also interesting because a lot of these things are easy to somewhat. Explain. I bet there’s a lot of devil’s in the details if they had to figure out, they mentioned that they did. They halted all new development of any new features for seven months and turned all attention to security.
jerry: Yeah, so it sounded like they moved from I think an on-prem. Dev and build environment to one that was up in AWS so that they could dynamically. Create and destroy them as needed.
Andy: Yeah, it’s. It’s an interesting, the fundamental concept that this article is saying is, Hey, once you’ve been breached, And you secure yourself.
Andy: Do you have a lower likelihood of being breached in the future. Are you like Dell? You have the board’s attention. Now you have the budget. Now you have the people now have the mandate to secure the company.
Andy: And is that true?
jerry: think it is situational. that there are some, [00:08:00] I’m drawing a blank. I think that’s one of the hotel change. don’t want to say the wrong name, but I I believe that there are. There are also instances. We’re readily available. Where the contrast true. Like they just keep getting hacked over and over.
Andy: And I sometimes wonder if that has to do with the complexity of their environment and the legacy stuff in their environment. If you look at a company like, I don’t know anything about solar winds, but I’m guessing. You know that there is somewhat of a. Fairly modern it footprint that. Maybe somewhat easy to retrofit as opposed to, hotel chain.
Andy: Probably some huge data centers that are incredibly archaic in their potential architecture and design and.
jerry: That’s a good point. It’s a very good point. It’s a different, it’s very different business model, right?
Andy: And they talked about how they’re spending, they’ve got three different tiers of socks now outsourcing two of them. They’re spending a crap ton of money on security.
Andy: Whether with CrowdStrike watching all their end point [00:09:00] stuff. They mentioned it here. I’m sure that CrowdStrike appreciated that. Their own. Tier three SOC. They’ve got a lot of stuff and they also talking to that now their retention rates for customers are back up in the nineties, which is pretty, pretty good. So I don’t know. Yeah. Clearly this is a PR thing.
Andy: But at the same time, I really do appreciate. A company that’s gone through this sharing as much as they’re sharing because the rest of us can learn from it.
jerry: Yeah, absolutely.
Andy: And the other thing it’s interesting because I look at this, cause I work for software company now. And it’s a small company. It’s nothing the size of these guys. And we don’t have the resources these guys have, but. I think about how many points in our dev chain. Probably could be easily corrupted in a supply chain attack.
Andy: That they’re stopping with their model. That, I wonder what. What could I do? Like how much of this could you do on a budget? There’s a huge amount of people environment here. There’s a huge amount of. Of red tape and [00:10:00] bureaucracy and checks and balances that must add tremendously to the cost.
Andy: Probably slow things down a little bit, probably gonna, would get pushed back. If you just tried to show up at your dev shop and say, Hey, we’re doing this now without having gone through this sort of event. So what I’m dancing around here is the concept of culture. Have, post-breach, you now have a culture that is probably more willing to accept what could be perceived as draconian security mandate over how they do things.
Andy: As opposed to pre breach.
jerry: Yeah. It probably doesn’t scale down very well.
jerry: With the. The overhead that they’ve poured on. Any, they also. In the article point out that you. It remains to be seen. How well solar winds continues carrying on, but it does, like you said, it does seem like. They’ve they’ve definitely taken this and learned from it and not only learn from it, but also have like we see in this article,
jerry: I’m trying to [00:11:00] help the rest of The rest of the industry learned, which is, by the way, like what we’re trying to do here on the show. Kudos to them.
jerry: For that.
Andy: Yeah. I also wonder how many other dev development shops.
Andy: We’ll learn from this and adopt some of these practices. So they’re not the next supply chain attack. Cause that’s really where the benefit comes.
jerry: Yeah. Yeah, absolutely.
jerry: All right. Onto the next story, which comes from computer weekly.com and the title here is log for shell on its way to becoming endemic. So the the us government after. Joe Biden’s president Joe Biden’s cyber executive order in, I think it was. 2021. Maybe. Formed this cyber security.
jerry: What is it called? The
Andy: safety review board.
jerry: safety review board. I could remember the S.
jerry: Which I think was modeled after the [00:12:00] NTSB or what have you. But they released this report last week, which describes. What happened in, or at least their analysis of what happened. In the log4j. Incident that happened last year. And. So I have mixed. Mixed emotions.
jerry: About this one. You know that one of the, one of the key findings is that. Open source development. Doesn’t have the same level of maturity and resources that, that. Commercial software does. And, on the one hand, one of the promises of open source was, many eyes makes.
jerry: Bugs. Very shallow. Which I think we’ve seen is not really holding water very well. But I think the other problem is it’s asserting that. Open source developers are uniquely making security mistakes in their development. [00:13:00] In the last I checked every single month for the past 20 plus years. Microsoft releases.
jerry: Set of patches For security bugs in their software and they are not open source. And so I, I think it w what’s a little frustrating to me was they didn’t. It feels like they didn’t address the elephant in the room. Which was not necessarily that the. Th that the open source developers here did
jerry: a bad job. They didn’t understand how to. Code securely. It’s self-evident that they made a, they made some mistakes. But the bigger problem is the fact that it was rolled up into so freaking many. Other open source. In non open source packages in and multi-tiered right. It’s.
jerry: Combined into a package that’s combined into another package. That’s combined into another package. That’s. [00:14:00] Combined into this commercial software. And the big challenge we had as an industry. Was figuring out where they, where all that stuff was. And then even after that Trying to beat on your vendors.
jerry: To come to terms with the fact that they actually have log4j in there environment, and then having to make these like painful decisions do we stop using. For instance, VMware, because we know that they have yet that they have log4j and they haven’t released the patch. At the time they have, since, by the way,
jerry: But. Th that is I think that’s the more concerning problem. Not just obviously for log4j but when you look across the industry, we have lots of things like log4j that are. Pretty managed by either a single person or a very small team on a best effort basis. And they serve some kind of important function and they just keep getting.
jerry: Consolidated. And I don’t [00:15:00] think there’s a real appreciation for how pervasively, some of these things. Are being used. They do talk about in the recommendations about creating built in a better bill of material for software, which I think is good. But it’s still, that’s like coming at it the wrong way.
jerry: It seems to me like we need to be looking for hotspots and addressing those hotspots. And I just don’t, I’m not seeing that it’s concerning to me.
Andy: what do you mean by hotspots?
jerry: Hotspots in terms of potentially. Poorly managed or not. That’s not the right way to say it, but less well-managed open source packages that have become super ingrained.
jerry: In the it ecosystem like log4j like openssl has been in some of the other bash, And so on.
jerry: We see this come and go. But at the end of the day I don’t know that we have a good handle on where those things are. So we’re just going to continue to get [00:16:00] surprised when some enterprising researcher. Lifts up a rug that nobody’s looked under before and realizes, oh gosh, there’s this piece of code that was managed by
jerry: a teenager in the proverbial basement. And they’ve since moved on to college and it’s you. It’s not being maintained anymore. Any more, but it’s like being used by By everybody and their dog.
jerry: We don’t seem to be thinking about that problem, at least in that way.
Andy: Yeah, you said something early on in covering this too about how open sources less rigorous and their controls than commercial, but I think it’s very fair to say that. vast majority of commercial applications. Are reusing tons of open source. And their code, right? That.
Andy: The kind of odd implication there is that. Commercial entities write everything from the ground up when that’s not true. Now here’s the flip side. If I’ve got a well known, mature, vetted [00:17:00] package. That does its job well that I can include in my software package. I could potentially save myself a lot of bugs and.
Andy: And vulnerabilities because that package has been so well vetted. In theory, right?
jerry: A hundred percent.
Andy: like writing your own encryption algorithm, bad idea. There’s a whole. Whole litany of people who’ve, edited, ruined because they thought they knew better. And that’s a really hard problem to solve. So I think there’s value in having. Almost like engineering standards of this type of strength of concrete,
Andy: that is reused because it’s a known quantity as opposed to, Hey, we’re just going to invent some new concrete and give it a whirl. I see it a little bit like that. But I agree with you. I also wonder how often.
Andy: Dev shops can spare someone who his whole job is to dig deep into the ecosystem of all the packages they pull in. When they do their development and know the life cycle of those. To the level we’re [00:18:00] talking about versus, Hey, that’s a solved problem. I’d just pull it off the shelf and move on.
jerry: I think that is the very issue as I see it. That is the. Problem because I don’t think most companies have the ability to do that.
Andy: What do you thinking like a curated.
Andy: Market of open source tools that are well-maintained.
jerry: Think we’re headed in that direction. I don’t. I don’t love the idea. By any stretch. I’m not saying don’t mean to imply that I do. But. I don’t see a good alternative. And the reason is that, like you said, you want. As a, as the developer of a application, whether it’s open source or not.
jerry: You want to use? You don’t want to recreate something that’s already existing and you want to use something that’s reliable. I think that one of the problems is that. These smaller pieces of open source. Technology like I have a strong feeling that like when the, when log4j started out, they didn’t expect that they were [00:19:00] going to be in every fricking piece of commercial and open source software out there.
jerry: It just happened. It happened.
jerry: over time. And. And.
jerry: I just think there was little consideration on both sides of the equation for what was happening. It was just happening and nobody really was aware of it.
Andy: It’s not like the log4j team was like, gum, use me everywhere. And then, there’s a little bit of, Hey, I wrote this, it’s up to you. If you want to use it, that’s on you.
jerry: Yeah. It’s there. Caveat. Emptor.
Andy: so it’s.
Andy: Yeah, this is. I don’t know. It’s a tough problem. I don’t know. The software bill materials is your solve either. I know a lot of people are talking about it. I know that it helps, but.
jerry: It, I think it, it helps in so much as if you have a, a few as a. Manufacturer of software or even you as a consumer. Have a S bomb that goes all the way down, which by the way, is itself a. Pretty tricky. When something like log4j hits [00:20:00] it becomes much easier to look across your environment and say, yep, I got it there and there.
jerry: That’s what I have to go fix. By the way, like it’s. You’re also dependent on your close source. Commercial software providers. Also doing a. A similar kind of job. So I think there’s a coming set of standards and processes. That the industry is going to have to, to get to, because this problem isn’t going to go away. It’s going to continue to get worse.
jerry: And somebody is either going to Some enterprising government like Australia or India or the U S is going to stuff a. Solution, none of us would like that our throat, or we’re going to have to come up with something.
Andy: Yeah. You’re not wrong.
Andy: It’ll be interesting to see how it plays out.
Andy: Now that I think the genie’s out of the bottle, you got to assume some of these big cybercrime. Syndicates or whatever term you want to use are attempting to replicate this.
jerry: Oh a hundred percent. A hundred percent, they gotta be looking around saying, what is. [00:21:00] open source components exist in, pervasively and what would be easy ish.
jerry: For me to take over slash compromise so that I could, roll and roll up into as many. Environments as I can, like that would be. Super convenient as a, as an adversary.
jerry: So anyway, there’s lots more to come on that I do think we’re going to see lots of hyper-focus on.
jerry: Source code supply chain, open source. Coming. And I fear that it’s going to. Be largely misguided, at least for awhile.
Andy: Fair enough.
jerry: All right. The next story comes from bleeping. Computer in the, this is a fascinating one. Title is hackers impersonate cybersecurity firms in callback phishing attacks.
Andy: Clever people.
jerry: We have a story here about an adversary or maybe multiple adversaries, who it becomes super enterprising and they [00:22:00] are sending letters to unwitting. Employees at different companies. And I don’t know how well targeted this is. There’s really not a lot of discussion about that, but. In the example they cite they have a letter.
jerry: I think it comes. By way of email. On CrowdStrike letterhead. And it basically says, Hey, CrowdStrike and your employer have this. Have this contract in place, we’ve seen some anomalous activity. You have you and your company. Are beholden to different regulatory requirements and we have to move really fast. We need you to call this phone number and to schedule an assessment. And it. Unlike by the way, a lot of a lot of these things is pretty well written. I would like to think that if I got it. I would say. That’s BS, but like it is really well-written, there’s not, it’s not full of grammatical errors. That kind of makes sense.
jerry: And apparently if you follow the instructions, by the way, [00:23:00] It, the hypothesis is that it will lead to unsurprisingly a ransomware infection because they’ll install a remote access Trojan on your workstation. And then, use that, use that as a beachhead to get into your.
jerry: Your company’s network.
Andy: Yeah. I hate to say it, but another good reason why you shouldn’t let your employees just randomly install software.
Andy: And you have to assume. There’ll be some, this is where I struggled by the way with social engineering training is I really do believe, and it’s not a failure. It’s not a moral failure it’s not an intelligent failure. It’s a psychological weakness of how human beings. Brain’s work that.
Andy: These bad guys are exploiting and they will find some percentage in some certain circumstances. That will fall for these sorts of efforts. And you’ve got to be resilient against that. I don’t think you can train that risk away.
jerry: I, yeah, I would say that it’s [00:24:00] perilous to think that you can train it away, because then you start to think that when it happens, It’s the failure of the person. And actually think that’s the wrong way to think about it. If you have, Obviously. You want to do some level of training?
jerry: Just if for no other reason, you’re obligated to do that by many regulations and whatnot. But, also like you want people to understand. Like what to look for, it’s it helps in the long run, but at the end of the day, like you, we have to design our environments. To withstand that kind of.
jerry: Issue right.
jerry: if we’re. If our security is predicated on someone. Recognizing that a well-written email on CrowdStrike letterhead. Is is fake. Like we have problems.
Andy: Yeah. If you’re never going to be taken down by one error click on an employee.
Andy: That I think is a problem you need to solve.
jerry: Yeah. And that’s a failure on, on, [00:25:00] on our.
jerry: Like it and security side, not on the employee side.
jerry: So anyway, be on the lookout. Obviously this is a pretty, I hadn’t heard of this before. It makes total sense in hindsight, but something to be on the lookout for.
jerry: All right. The last story we have comes from cybersecurity. dive.com. One of my new new favorite websites, by the way. The good stuff on there. Title is Microsoft rollback on macro blocking in office sows confusion. So earlier in the year, Microsoft made a much heralded. Announcement. That they were going to be blocking.
jerry: Macros in Microsoft office from anything that was. Originated from the internet. And and that. Was born out by the way, by an apparent. But some researchers have said that. It’s much as two thirds. Of [00:26:00] the. Attacks involving macros has fallen away. So pretty effective control Microsoft last week.
jerry: Now it’s that they were reversing course and re enabling macros. I assume. Because CFO’s everywhere were in full meltdown that their fancy spreadsheets we’re no longer working and obviously we should assume that, the the attacks are going to be back on the upswing. And apparently this is a temporary reprieve. It’s a little unclear when Microsoft is gonna re enable it. But I have a strong feeling that a lot of.
jerry: Organizations have. Taking us taking a breather on this front because Microsoft solved it. For us and now we need to be back on, on the the defensive.
Andy: Yeah, I’m really curious what the conversation was like that Forced them to reverse course, like what broke. That was that big of a deal that was so imperative because this has been a [00:27:00] problem. For at least 15 years with Microsoft.
Andy: least. This was a pretty big win. And now it’s. Kinda get rolled back. So I was disappointed.
jerry: So there are. I think there’s some links in here. You can actually go back and re enable it through group policy settings. Obviously if if you’re so inclined, Probably a really good idea. As a, as an it industry, I think we’re worse off. For this change until they re enable it.
Andy: Yeah. This is without knowing all the reasons behind it. This feels like such a pure example of productivity versus security sort of trade off and playing out in real time.
jerry: Yeah. I can almost guarantee you this what’s going on.
jerry: So that yeah. That is a little concerning. Definitely. Be on the lookout.
Andy: Indeed. We’ll see what happens to be continued. Stay tuned.
jerry: To be continued. And that is [00:28:00] that is the story for tonight. Just one little bit of editorial. I spend a lot of time during the week reading. Different stories, all kinds of Google alerts set up for For different security stories and whatnot to help pick what we talk about on these podcasts.
jerry: And. It is amazing to me. How many. Stories that are Couched asnews are actually. Basically marketing pieces.
jerry: It’s I know that we’ve talked about this in the past, but it is alarming. I actually gotten to the point now where I dropped down to the end to see what they’re going to try to sell me before I get too invested in. The
Andy: I look at who wrote it. And if they’re like not a staff writer, if they’re like contributing writer from, chief marketing officer from blah, blah, blah, I’m like, Nope.
Andy: I very quickly just. Stop reading it. If it’s something written by an employee of a vendor or some variety. [00:29:00] And I don’t mean to be that harsh about it. It’s just.
Andy: There’s a bias there that they believe their own marketing. And their own dog food and they’re clearly pushing the problem. They know how to solve.
jerry: Yeah, they’re characterizing. The problem is. Something offerings can solve.
jerry: And, and I think it’s a. It’s certainly an understandable. Position, but I. I’m concerned that as a industry,
jerry: Where do we go to get actual best practices. Because if you’re, if everything you read is written by a security vendor who wants. The best practices are install crowdStrike install red Canary install. McAfee installed
Andy: you bring up an interesting. You bring up an interesting side point, which is. I’m seeing some movement in the cyber insurance industry that they’re basically saying. At the broadest level for those that are less sophisticated. These are the three EDRs. We want you [00:30:00] to have one of and if it’s not one of these three, you don’t get premium pricing.
jerry: Oh, that’s interesting.
Andy: And you’re like, wow. Especially because it’s such a blanket statement. And so many environments are different and I’m. I’m not passing judgment on the efficacy of those three vendors, which is why I’m not saying them. It’s more, that’s feels like a very.
Andy: Lack of nuanced opinion that, Very blunt instrument being applied there.
jerry: Yeah, and It also.
jerry: Ignores like a whole spectrum of other stuff that you should be doing in.
Andy: That’s just their EDR. Table-stakes right. And which is all coming very much from ransomware. They’re just getting their ass kicked the ransomware payouts. And so they’re like what is what will stop ransomware?
jerry: Fair enough. That’s a fair. That’s a fair point.
Andy: Back to your point about, So many marketing pieces being masquerading as InfoSec news, I think is very true. And on that note, I want to thank today’s sponsor of Bob’s budget, firewalls.
jerry: [00:31:00] We proudly have I think we’ve cleared 10 years of no No vendor sponsorship. No sponsorship of any kind, other than a donation.
Andy: Yes, which we appreciate.
jerry: All right. is the show for this week. Happy to have done two weeks in a row now. Got to make a habit of this.
Andy: I know this is great. I appreciate it.
Andy: all four listeners that we still have.
jerry: I moved to a commercial podcasting hosting platform. And so we get actually now get some metrics and we have about
jerry: About 10,000 ish.
jerry: Or so.
Andy: counting the inmates that are forced to listen as part of their correction.
jerry: No see see
jerry: think actually because That’s a one to many thing so there’s probably like one stream is forcing like maybe 500 people. To listen.
jerry: And then when they do crowd control, like that could be thousands of
Andy: That is true.
Andy: I was quite [00:32:00] entertained. And really proud of you when I found out that your voice. Was found to be one of the best tools to disperse crowds.
jerry: Hey, we all have to be good at something right.
Andy: It is up there with. Fire
jerry: Yeah. Yeah.
Andy: neck and neck. Better than tear gas. I, are you aware of this better
jerry: I was not aware that I had overtaken tear gas.
Andy: It’s impressive. My friend, you should be proud.
jerry: I, I am.
Andy: should be proud.
jerry: am. I’m going to go tell them.
Andy: All right.
jerry: Have a good one, everyone.
Andy: Alrighty. Bye.