Screaming in the Cloud

Chris Farris, Cloud Security Nerd at PrimeHarbor Technologies, LLC, joins Corey on Screaming in the Cloud to discuss his new project, breaches.cloud, and why he feels having a centralized location for cloud security breach information is so important. Corey and Chris also discuss what it means to dive into entrepreneurship, including both the benefits of not having to work within a corporate structure and the challenges that come with running your own business. Chris also reveals what led him to start breaches.cloud, and what he’s learned about some of the biggest cloud security breaches so far.

About Chris

Chris Farris is a highly experienced IT professional with a career spanning over 25 years. During this time, he has focused on various areas, including Linux, networking, and security. For the past eight years, he has been deeply involved in public-cloud and public-cloud security in media and entertainment, leveraging his expertise to build and evolve multiple cloud security programs.

Chris is passionate about enabling the broader security team’s objectives of secure design, incident response, and vulnerability management. He has developed cloud security standards and baselines to provide risk-based guidance to development and operations teams. As a practitioner, he has architected and implemented numerous serverless and traditional cloud applications, focusing on deployment, security, operations, and financial modeling.

He is one of the organizers of the fwd:cloudsec conference and presented at various AWS conferences and BSides events. Chris shares his insights on security and technology on social media platforms like Twitter, Mastodon and his website https://www.chrisfarris.com.

Links Referenced:

• fwd:cloudsec: https://fwdcloudsec.org/
• breaches.cloud: https://breaches.cloud
• Twitter: https://twitter.com/jcfarris
• Company Site: https://www.primeharbor.com

Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. My returning guest today is Chris Farris, now at PrimeHarbor, which is his own consultancy. Chris, welcome back. Last time we spoke, you were a Turbot, and now you’ve decided to go independent because you don’t like sleep anymore.

Chris: Yeah, I don’t like sleep.

Corey: [laugh]. It’s one of those things where when I went independent, at least in my case, everyone thought that it was, oh, I have this grand vision of what the world could be and how I could look at these things, and that’s going to just be great and awesome and everyone’s going to just be a better world for it. In my case, it was, no, just there was quite literally nothing else for me to do that didn’t feel like an exact reframing of what I’d already been doing for years. I’m a terrible employee and setting out on my own was important. It was the only way I found that I could wind up getting to a place of not worrying about getting fired all the time because that was my particular skill set. And I look back at it now, almost seven years in, and it’s one of those things where if I had known then what I know now, I never would have started.

Chris: Well, that was encouraging. Thank you [laugh].

Corey: Oh, of course. And in sincerity, it’s not one of those things where there’s any one thing that stops you, but it’s the, a lot of people get into the independent consulting dance because they want to do a thing and they’re very good at that thing and they love that thing. The problem is, when you’re independent, and at least starting out, I was spending over 70% of my time on things that were not billable, which included things like go and find new clients, go and talk to existing clients, the freaking accounting. One of the first hires I made was a fractional CFO, which changed my life. Up until that, my business partner and I were more or less dead reckoning of looking at the bank account and how much money is in there to determine if we could afford things. That’s a very unsophisticated way of navigating. It’s like driving by braille.

Chris: Yeah, I think I went into it mostly as a way to define my professional identity outside of my W-2 employer. I had built cloud security programs for two major media companies and felt like that was my identity: I was the cloud security person for these companies. And so, I was like, ehh, why don’t I just define myself as myself, rather than define myself as being part of a company that, in the media space, they are getting overwhelmed by change, and job security, job satisfaction, wasn’t really something that I could count on.

Corey: One of the weird things that I found—it’s counterintuitive—is that when you’re independent, you have gotten to a point where you have hit a point of sustainability, where you’re not doing the oh, I’m just going to go work for 40 billable hours a week for a client. It’s just like being an employee without a bunch of protections and extra steps. That doesn’t work super well. But now, at the point where I’m at where the largest client we have is a single-digit percentage of revenue, I can’t get fired anymore, without having a whole bunch of people suddenly turn on me because I’ve done something monstrous, in which case, I probably deserve not to have business anymore, or there’s something systemic in the macro environment, which given that I do the media side and I do the cost-cutting side, I work on the way up, I work on the way down, I’m questioning what that looks like in a scenario that doesn’t involve me hunting for food. But it’s counterintuitive to people who have been employees their whole life, like I was, where, oh, it’s risky and dangerous to go out on your own.

Chris: It’s risky and dangerous to be, you know, tied to a single, yeah, W-2 paycheck. So.

Corey: Yeah. The question I’d like to ask is, how many people need to be really pissed off before you have one of those conversations with HR that doesn’t involve giving you a cup of coffee? That’s the tell: when you don’t get coffee, it’s a bad conversation.

Chris: Actually, that you haven’t seen [unintelligible 00:04:25] coffee these days. You don’t want the cup of coffee, you know. That’s—

Corey: Even when they don’t give you the crappy percolator navy coffee, like, midnight hobo diner style, it’s still going to be a bad meeting because [unintelligible 00:04:37] pretend the coffee’s palatable.

Chris: Perhaps, yes. I like not having to deal with my own HR department. And I do agree that yeah, getting out of the W-2 space allows me to work on side projects that interests me or, you know, volunteer to do things like continuing the fwd:cloudsec, developing breaches.cloud, et cetera.

Corey: I’ll never forget, one of my last jobs I had a boss who walked past and saw me looking at Reddit and asked me if that was really the best use of my time. At first—it was in, I think, the sysadmin forum at the time, so yes, it was very much the best use of my time for the problem I was focusing on, but also, even if it wasn’t, I spent an inordinate amount of time on social media, just telling stories and building audiences, on some level. That’s the weird thing is that what counts as work versus what doesn’t count as work gets very squishy when you’re doing your own marketing.

Chris: True. And even when I was a W-2 employee, I spent a lot of time on Twitter because Twitter was an intel source for us. It was like, “Hey, who’s talking about the latest cloud security misconfigurations? Who’s talking about the latest data breach? What is Mandiant tweeting about?” It was, you know—I consider it part of my job to be on Twitter and watching things.

Corey: Oh, people ask me that. “So, you’re on Twitter an awful lot. Don’t you have a newsletter to write?” Like, yeah, where do you think that content comes from, buddy?

Chris: Exactly. Twitter and Mastodon. And Reddit now.

Corey: There’s a whole argument to be had about where to find various things. For me at least, because I’m only security adjacent, I was always trying to report the news that other people had, not make the news myself.

Chris: You don’t want to be the one making the news in security.

Corey: Speaking of, I’d like to talk a bit about what you just alluded to breaches.cloud. I don’t think I’ve seen that come across my desk yet, which tells me that it has not been making a big splash just yet.

Chris: I haven’t been really announcing it; it got published the other night and so basically, yeah, is this is sort of a inaugural marketing push for breaches.cloud. So, what we’re looking to do is document all the public cloud security breaches, what happened, why, and more importantly, what the companies did or didn’t do that led to the security incident or the security breach.

Corey: How are you slicing the difference between broad versus deep? And what I mean by that is, there are some companies where there are indictments and massive deep dives into everything that happens with timelines and blows-by-blows, and other times you wind up with the email that shows up one day of, “Security is very important to us. Now, listen to how we completely dropped the ball on it.” And it just makes the biggest description that they can get away with of what happened. Occasionally, you find out oh, it was an open S3 buckets, or they’ll allude to something that sounds like it. Does that count for inclusion? Does it not? How do you make those editorial decisions?

Chris: So, we haven’t yet built a page around just all of the recipients of the Bucket Negligence Award. We’re looking at the specific ones where there’s been something that’s happened that’s usually involving IAM credentials—oftentimes involving IAM credentials found in GitHub—and what led to that. So, in a lot of cases, if there’s a detailed company postmortem that they send their customers that said, “Hey, we goofed up, but complete transparency—” and then they hit all the bullet points of how they goofed up. Or in the case of certain others, like Uber, “Hey, we have court transcripts that we can go to,” or, “We have federal indictments,” or, “We have court transcripts, and federal indictments and FTC civil actions.” And so, we go through those trying to suss out what the company did or did not do that led to the breach. And really, the goal here is to be able to articulate as security practitioners, hey, don’t attach S3 full access to this role on EC2. That’s what got Capital One in trouble.

Corey: I have a lot of sympathy for the Capital One breach and I wish they would talk about it more than they do, for obvious reasons, just because it was not, someone showed up and made a very obvious dumb decision, like, “Oh, that was what that giant red screaming thing in the S3 console means.” It was a series of small misconfigurations that led to another one, to another one, to another one, and eventually gets to a point where a sophisticated attacker was able to chain them all together. And yes, it’s bad, yes, they’re a bank and the rest, but I look at that and it’s—that’s the sort of exploit that you look at and it’s okay, I see it. I absolutely see it. Someone was very clever, and a bunch of small things that didn’t rise to the obvious. But they got dragged and castigated as if they basically had a four-character password that they’d left on the back of the laptop on a Post-It note in an airport lounge when their CEO was traveling. Which is not the case.

Chris: Or all of the highlighting the fact that Paige Thompson was a former Amazon employee, making it seem like it was her insider abilities that lead to the incident, rather than she just knew that, hey, there’s a metadata service and it gives me creds if I ask it.

Corey: Right. That drove me nuts. There was no maleficence as an employee. And to be very direct, from what I understand of internal AWS controls, had there been, it would have been audited, flagged, caught, interdicted. I have talked to enough Amazonians that either a lot of them are lying to me very consistently despite not knowing each other, or they’re being honest when they say that you can’t get access to customer data using secret inside hacks.

Chris: Yeah. I have reasonably good faith in AWS and their ability to not touch customer data in most scenarios. And I’ve had cases that I’m not allowed to talk about where Amazon has gone and accessed customer data, and the amount of rigmarole and questions and drilling that I got as a customer to have them do that was pretty intense and somewhat, actually, annoying.

Corey: Oh, absolutely. And, on some level, it gets frustrating when it’s a, look, this is a test account. I have nothing of sensitive value in here. I want the thing that isn’t working to start working. Can I just give you a whole, like, admin-powered user account and we can move on past all of this? And their answer is always absolutely not.

Chris: Yes. Or, “Hey, can you put this in our bucket?” “No, we can’t even write to a public bucket or a bucket that, you know, they can share too.” So.

Corey: An Amazonian had to mail me a hard drive because they could not send anything out of S3 to me.

Chris: There you go.

Corey: So, then I wound up uploading it back to S3 with, you know, a Snowball Edge because there’s no overkill like massive overkill.

Chris: No, the [snowmobile 00:11:29] would have been the massive overkill. But depending on where you live, you know, you might not have been able to get a permit to park the snowmobile there.

Corey: They apparently require a loading dock. Same as with the outposts. I can’t fake having one of those on my front porch yet.

Chris: Ah. Well, there you go. I mean, you know it’s the right height though, and you don’t mind them ruining your lawn.

Corey: So, help me understand. It makes sense to me at least, on some level, why having a central repository of all the various cloud security breaches in one place that’s easy to reference is valuable. But what caused you to decide, you know, rather than saying it’d be nice to have, I’m going to go build that thing?

Chris: Yeah, so it was actually right before the last time we spoke, Nicholas Sharp was indicted. And there was like, hey, this person was indicted for, you know, this cloud security case. And I’m like, that name rings a bell, but I don’t remember who this person was. And so, I kind of realized that there’s so many of these things happening now that I forget who is who. And so, when a new piece of news comes along, I’m like, where did this come from and how does this fit into what my knowledge of cloud security is and cloud security cases?

So, I kind of realized that these are all running together in my mind. The Department of Justice only referenced ‘Company One,’ so it wasn’t clear to me if this even was a new cloud incident or one I already knew about. And so basically, I decided, okay, let’s build this. Breaches.cloud was available; I think I kind of got the idea from hackingthe.cloud.

And I had been working with some college students through the Collegiate Cyber Defense Competition, and I was like, “Hey, anybody want a spring research project that I will pay you for?” And so yeah, PrimeHarbor funded two college students to do quite a bit of the background research for me, I mentored them through, “Hey, so here’s what this means,” and, “Hey, have we noticed that all of these seem to relate to credentials found in GitHub? You know, maybe there’s a pattern here.” So, if you’re not yet scanning for secrets in GitHub, I recommend you start scanning for secrets in your GitHub, private and public repos.

Corey: Also, it makes sense to look at the history. Because, oh, I committed a secret. I’m going to go ahead and revert that commit and push that. That solves the problem, right?

Chris: No, no, it doesn’t. Yes, apparently, you can force push and delete an entire commit, but you really want to use a tool that’s going to go back through the commit history and dig through it because as we saw in the Uber incident, when—the second Uber incident, the one that led to the CSOs conviction—yeah, the two attackers, [unintelligible 00:14:09] stuffed a Uber employee’s personal GitHub account that they were also using for Uber work, and yeah, then they dug through all the source code and dug through the commit histories until they found a set of keys, and that’s what they used for the second Uber breach.

Corey: Awful when that hits. It’s one of those things where it’s just… [sigh], one thing leads to another leads to another. And on some level, I’m kind of amazed by the forensics that happen around all of these things. With the counterpoint, it is so… freakishly difficult, I think, for lack of a better term, just to be able to say what happened with any degree of certainty, so I can’t help but wonder in those dark nights when the creeping dread starts sinking in, how many things like this happen that we just never hear about because they don’t know?

Chris: Because they don’t turn on CloudTrail. Probably a number of them. Once the data gets out and shows up on the dark web, then people start knocking on doors. You know, Troy Hunt’s got a large collection of data breach stuff, and you know, when there’s a data breach, people will send him, “Hey, I found these passwords on the dark web,” and he loads them into Have I Been Pwned, and you know, [laugh] then the CSO finds out. So yeah, there’s probably a lot of this that happens in the quiet of night, but once it hits the dark web, I think that data starts becoming available and the victimized company finds out.

Corey: I am profoundly cynical, in case that was unclear. So, I’m wondering, on some level, what is the likelihood or commonality, I suppose, of people who are fundamentally just viewing security breach response from a perspective of step one, make sure my resume is always up to date. Because we talk about these business continuity plans and these DR approaches, but very often it feels like step one, secure your own mask before assisting others, as they always say on the flight. Where does personal preservation come in? And how does that compare with company preservation?

Chris: I think down at the [IaC 00:16:17] level, I don’t know of anybody who has not gotten a job because they had Equifax on their resume back in, what, 2017, 2018, right? Yes, the CSO, the CEO, the CIO probably all lost their jobs. And you know, now they’re scraping by book deals and speaking engagements.

Corey: And these things are always, to be clear, nuanced. It’s rare that this is always one person’s fault. If you’re a one-person company, okay, yeah, it’s kind of your fault, let’s be clear here, but there are controls and cost controls and audit trails—presumably—for all of these things, so it feels like that’s a relatively easy thing to talk around, that it was a process failure, not that one person sucked. “Well, didn’t you design and implement the process?” “Yes. But it turned out there were some holes in it and my team reported that those weren’t there and it turned out that they were and, well, live and learn.” It feels like that’s something that could be talked around.

Chris: It’s an investment failure. And again, you know, if we go back to Harry Truman, “The buck stops here,” you know, it’s the CEO who decides that, hey, we’re going to buy a corporate jet rather than buy a [SIIM 00:17:22]. And those are the choices that happen at the top level that define, do you have a capable security team, and more importantly, do you have a capable security culture such that your security team isn’t the only ones who are actually thinking about security?

Corey: That’s, I guess, a fair question. I saw a take on Twitter—which is always a weird thing—or maybe was Blue-ski or somewhere else recently, that if you don’t have a C-level executive responsible for security with security in their title, your company does not take security seriously. And I can see that past a certain point of scale, but as a one-person company, do you have a designated CSO?

Chris: As a one-person company and as a security company, I sort of do have a designated CSO. I also have, you know, the person who’s like, oh, I’m going to not put MFA on the root of this one thing because, while it’s an experiment and it’s a sandbox and whatever else, but I also know that that’s not where I’m going to be putting any customer data, so I can measure and evaluate the risk from both a security perspective and a business existential investment perspective. When you get to the larger the organization, the more detached the CEO gets from the risk and what the company is building and what the company is doing, is where you get into trouble. And lots of companies have C-level somebody who’s responsible for security. It’s called the CSO, but oftentimes, they report four levels down, or even more, from the chief executive who is actually the one making the investment decisions.

Corey: On some level, the oh yeah, that’s my responsibility, too, but it feels like it’s a trap that falls into. Like, well, the CTO is responsible for security at a publicly traded company. Like, well… that tends to not work anymore, past certain points of scale. Like when I started out independently, yes, I was the CSO. I was also the accountant. I was also the head of marketing. I was also the janitor. There’s a bunch of different roles; we all wear different hats at different times.

I’m also not a big fan of shaming that oh, yeah. This is a universal truth that applies to every company in existence. That’s also where I think Twitter started to go wrong where you would get called out whenever making an observation or witticism or whatnot because there was some vertex case to which it did not necessarily apply and then people would ‘well, actually,’ you to death.

Chris: Yeah. Well, and I think there’s a lot of us in the security community who are in the security one-percenters. We’re, “Hey, yes, I’m a cloud security person on a 15-person cloud security team, and here’s this awesome thing we’re doing.” And then you’ve got most of the other companies in this country that are probably below the security poverty line. They may or may not have a dedicated security person, they certainly don’t have a SIIM, they certainly don’t have anybody who’s monitoring their endpoints for malware attacks or anything else, and those are the companies that are getting hit all the time with, you know, a lot of this ransomware stuff. Healthcare is particularly vulnerable to that.

Corey: When you take a look across the industry, what is it that you’re doing now at PrimeHarbor that you feel has been an unmet need in the space? And let me be clear, as of this recording earlier today, we signed a contract with you for a project. There’s more to come on that in the future. So, this is me asking you to tell a story, not challenging, like, what do you actually do? This is not a refund request, let’s be very clear here. But what’s the unmet need that you saw?

Chris: I think the unmet need that I see is we don’t talk to our builder community. And when I say builder, I mean, developers, DevOps, sysadmins, whatever. AWS likes the term builder and I think it works. We don’t talk to our builder community about risk in a way that makes sense to them. So, we can say, “Hey, well, you know, we have this security policy and section 24601 says that all data’s classifications must be signed off by the data custodian,” and a developer is going to look at you with their head tilted, and be like, “Huh? What? I just need to get the sprint done.”

Whereas if we can articulate the risk—and one of the reasons I wanted to do breaches.cloud was to have that corpus of articulated risk around specific things—I can articulate the risk and say, “Hey, look, you know how easy it is for somebody to go in and enumerate an S3 bucket? And then once they’ve enumerated and guessed that S3 bucket exists, they list it, and oh, hey, look, now that they’ve listed it, they know all of the objects and all of the juicy PII that you just made public.” If you demonstrate that to them, then they’re going to be like, “Oh, I’m going to add the extra story point to this story to go figure out how to do CloudFront origin access identity.” And now you’ve solved, you know, one more security thing. And you’ve done in a way that not just giving a man a fish or closing the bucket for them, but now they know, hey, I should always use origin access identity. This is why I need to do this particular thing.

Corey: One of the challenges that I’ve seen in a variety of different sites that have tried to start cataloging different breaches and other collections of things happening in public is the discoverability or the library management problem. The most obvious example of this is, of course, the AWS console itself, where when it paginates things like, oh, there are 3000 things here, ten at a time, through various pages for it. Like, the marketplace is just a joke of discoverability. How do you wind up separating the stuff that is interesting and notable, rather than, well, this has about three sentences to it because that’s all the company would say?

Chris: So, I think even the ones where there’s three sentences, we may actually go ahead and add it to the repo, or we may just hold it as a draft, so that we know later on when, “Hey, look, here’s a federal indictment for Company Three. Oh, hey, look. Company Three was actually this breach announcement that we heard about three months ago,” or even three years ago. So like, you know, Chegg is a great example of, you know, one of those where, hey, you know, there was an incident, and they disclosed something, and then, years later, FTC comes along and starts banging them over the head. And in the FTC documentation, or in the FTC civil complaint, we got all sorts of useful data.

Like, not only were they using root API keys, every contractor and employee there was sharing the root API keys, so when they had a contractor who left, it was too hard to change the keys and share it with everybody, so they just didn’t do that. The contractor still had the keys, and that was one of the findings from the FTC against Chegg. Similar to that, Cisco didn’t turn off contractors’ access, and I think—this is pure speculation—I think the poor contractor one day logged into his Google Cloud Shell, cd’ed into a Terraform directory, ran ‘terraform destroy’, and rather than destroying what he thought he was destroying, it had the access keys back to Cisco WebEx and took down 400 EC2 instances that made up all of WebEx. These are the kinds of things that I think it’s worth capturing because the stories are going to come out over time.

Corey: What have you seen in your, I guess, so far, a limited history of curating this that—I guess, first what is it you’ve learned that you’ve started seeing as far as patterns go, as far as what warrants inclusion, what doesn’t, and of course, once you started launching and going a bit more public with it, I’m curious to hear what the response from companies is going to be.

Chris: So, I want to be very careful and clear that if I’m going to name somebody, that we’re sourcing something from the criminal justice system, that we’re not going to say, “Hey, everybody knows that it was Paige Thompson who was behind it.” No, no, here’s the indictment that said it was Paige Thompson that was, you know, indicted for this Capital One sort of thing. All the data that I’m using, it all comes from public sources, it’s all sited, so it’s not like, hey, some insider said, “Hey, this is what actually happened.” You know? I very much learned from the Ubiquiti case that I don’t want to be in the position of Brian Krebs, where it’s the attacker themselves who’s updating the site and telling us everything that went wrong, when in fact, it’s not because they’re in fact the perpetrator.

Corey: Yeah, there’s a lot of lessons to be learned. And fortunately, for what it’s s—at least it seems… mostly, that we’ve moved past the battle days of security researchers getting sued on a whim from large companies for saying embarrassing things about them. Of course, watch me be tempting fate and by the time this publishes, I’ll get sued by some company, probably Azure or whatnot, telling me that, “Okay, we’ve had enough of you saying bad things about our security.” It’s like, well, cool, but I also read the complaint before you file because your security is bad. Buh-dum-tss. I’m kidding. I’m kidding. Please don’t sue me.

Chris: So, you know, whether it’s slander or libel, depending on whether you’re reading this or hearing it, you know, truth is an actual defense, so I think Microsoft doesn’t have a case against you. I think for what we’re doing in breaches, you know—and one of the reasons that I’m going to be very clear on anybody who contributes—and just for the record, anybody is welcome to contribute. The GitHub repo that runs breaches.cloud is public and anybody can submit me a pull request and I will take their write-ups of incidents. But whatever it is, it has to be sourced.

One of the things that I’m looking to do shortly, is start soliciting sponsorships for breaches so that we can afford to go pull down the PACER documents. Because apparently in this country, while we have a right to a speedy trial, we don’t have a right to actually get the court transcripts for less than ten cents a page. And so, part of what we need to do next is download those—and once we’ve purchased them, we can make them public—download those, make them public, and let everybody see exactly what the transcript was from the Capital One incident, or the Joey Sullivan trial.

Corey: You’re absolutely right. It drives me nuts that I have to wind up budgeting money for PACER to pull up court records. And at ten cents a page, it hasn’t changed in decades, where it’s oh, this is the cost of providing that data. It’s, I’m not asking someone to walk to the back room and fax it to me. I want to be very clear here. It just feels like it’s one of those areas where the technology and government is not caught up and it’s—part of the problem is, of course, having no competition.

Chris: There is that. And I think I read somewhere that the ent—if you wanted to download the entire PACER, it would be, like, $100 million. Not that you would do that, but you know, it is the moneymaker for the judicial system, and you know, they do need to keep the lights on. Although I guess that’s what my taxes are for. But again, yes, they’re a monopoly; they can do that.

Corey: Wildly frustrating, isn’t it?

Chris: Yeah [sigh]… yeah, yeah, yeah. Yeah, I think there’s a lot of value in the court transcripts. I’ve held off on publishing the Capital One case because one, well, already there’s been a lot of ink spilled on it, and two, I think all the good detail is going to be in the trial transcripts from Paige Thompson’s trial.

Corey: So, I am curious what your take is on… well, let’s called the ‘FTX thing.’ I don’t even know how to describe it at this point. Is it a breach? Is it just maleficence? Is it 15,000 other things? But I noticed that it’s something that breaches.cloud does talk about a bit.

Chris: Yeah. So, that one was a fascinating one that came out because as I was starting this project, I heard you know, somebody who was tweeting was like, “Hey, they were storing all of the crypto private keys in AWS Secrets Manager.” And I was like, “Errr?” And so, I went back and I read John J. Ray III’s interim report to the creditors.

Now, John Ray is the man who was behind the cleaning up of Enron, and his comment was “FTX is the”—“Never in my career have I seen such a complete failure of corporate controls and such a complete absence of trustworthy information as occurred here.” And as part of his general, broad write-up, they went into, in-depth, a lot of the FTX AWS practices. Like, we talk about, hey, you know, your company should be multi-account. FTX was worse. They had three or four different companies all operating in the same AWS account.

They had their main company, FTX US, Alameda, all of them had crypto keys in Secrets Manager and there was no access control between any of those. And what ended up happening on the day that SBF left and Ray came in as CEO, the $400 million worth of crypto somehow disappeared out of FTX’s wallets.

Corey: I want to call this out because otherwise, I will get letters from the AWS PR spin doctors. Because on the surface of it, I don’t know that there’s necessarily a lot wrong with using Secrets Manager as the backing store for private keys. I do that with other things myself. The question is, what other controls are there? You can’t just slap it into Secrets Manager and, “Well, my job is done. Let’s go to lunch early today.”

There are challenges [laugh] around the access levels, there are—around who has access, who can audit these things, and what happens. Because most of the secrets I have in Secrets Manager are not the sort of thing that is, it is now a viable strategy to take that thing and abscond to a country with a non-extradition treaty for the rest of my life, but with private keys and crypto, there kind of is.

Chris: That’s it. It’s like, you know, hey, okay, the RDS database password is one thing, but $400 million in crypto is potentially another thing. Putting it in and Secrets Manager might have been the right answer, too. You get KMS customer-managed keys, you get full auditability with CloudTrail, everything else, but we didn’t hear any of that coming out of Ray’s report to the creditors. So again, the question is, did they even have CloudTrail turned on? He did explicitly say that FTX had not enabled GuardDuty.

Corey: On some level, even if GuardDuty doesn’t do anything for you, which in my case, it doesn’t, but I want to be clear, you should still enable it anyway because you’re going to get dragged when there’s inevitable breach because there’s always a breach somewhere, and then you get yelled at for not having turned on something that was called GuardDuty. You already sound negligent, just with that sentence alone. Same with Security Hub. Good name on AWS’s part if you’re trying to drive service adoption. Just by calling it the thing that responsible people would use, you will see adoption, even if people never configure or understand it.

Chris: Yeah, and then of course, hey, you had Security Hub turned on, but you ignore the 80,000 findings in it. Why did you ignore those 80,000 findings? I find Security Hub to probably be a little bit too much noise. And it’s not Security Hub, it’s ‘Compliance Hub.’ Everything—and I’m going to have a blog post coming out shortly—on this, everything that Security Hub looks at, it looks at it from a compliance perspective.

If you look at all of its scoring, it’s not how many things are wrong; it’s how many rules you are a hundred percent compliant to. It is not useful for anybody below that AWS security poverty line to really master or to really operationalize.

Corey: I really want to thank you for taking the time to catch up with me once again. Although now that I’m the client, I expect I can do this on demand, which is just going to be delightful. If people want to learn more, where can they find you?

Chris: So, they can find breaches.cloud at, well https://breaches.cloud. If you’re looking for me, I am either on Twitter, still, at @jcfarris, or you can find me and my consulting company, which is www.primeharbor.com.

Corey: And we will, of course, put links to all of that in the [show notes 00:33:57]. Thank you so much for taking the time to speak with me. As always, I appreciate it.

Chris: Oh, thank you for having me again.

Corey: Chris Farris, cloud security nerd at PrimeHarbor. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry, insulting comment that you’re also going to use as the storage back-end for your private keys.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.