Buzzcast

Buzzcast is a roundtable discussion about podcasting from the people at Buzzsprout. We'll cover current events and news, podcast strategy, tools we are using, and dip into the Customer Support mailbag to test our podcasting knowledge. If you want to stay up-to-date on what's working in podcasting, Buzzcast is the show for you.

https://buzzcast.buzzsprout.com/

subscribe
share





episode 46: The Day Podcasts Stopped [transcript]


Go behind the scenes of the first DDoS cyber attack in Buzzsprout's history and learn what we're doing to prevent outages like these moving forward.

Read the full DDoS Technical Post-Mortem Report on our Blog

Charities we supported:

  • Feeding Texas
  • Front Steps
  • Project Vesta


Special thanks to:

  • Jack Rhysider, host of "Darknet Diaries"
  • Jordan Blair, host of "Dreamful"


Review Buzzcast in Podchaser or Apple Podcasts to let us know what you think of the show.

Buzzsprout's Dynamic Content tool now allows you to save multiple clips in your Dynamic Content Library and track how many downloads each clip receives. Learn more on our New Features page.


share







 2021-02-26  31m
 
 
00:00  Priscilla
And then I saw
00:00
something about a hacker and I
00:03
was like, well, that's crazy an
00:03
attack? What?
00:06  John
I get a text from Kevin
00:06
that says, hey, alert the FBI.
00:14
This guy must be some kind of
00:14
joke, something going on. And
00:19
then Tom FaceTime me,
00:21  Kevin
I got that text and
00:21
started digging into everything
00:23
that was going on. And then I
00:23
realized, sorry, kids, it's time
00:26
to hop in the car, we need to
00:26
head back to Jacksonville.
00:44  Travis
On Sunday, February 21,
00:44
Buzzsprout was the victim of the
00:48
cyber attack. Now, none of our
00:48
customers podcasts or personal
00:52
information was ever at risk.
00:52
But it was still a big deal.
00:57
Normally, on the show, we talked
00:57
about what's going on in the
01:00
podcasting industry and dive
01:00
into new features that
01:02
Buzzsprout is rolling out to our
01:02
podcasters. But this week, we
01:05
wanted to do something
01:05
different. So in this episode,
01:08
we're going to take you behind
01:08
the scenes and share the real
01:11
stories of what it was like to
01:11
be inside Buzzsprout during a
01:15
tenuous 27 hour period, and
01:15
explain exactly what happened.
01:21
Don't worry, though, the story
01:21
has a happy ending. It all
01:24
started late Sunday morning,
01:24
when Tom one of the cofounders
01:28
of Buzzsprout got a notification
01:28
on his phone.
01:32  Tom
I think we started
01:32
receiving a notification about
01:35
1140. And it was definitely
01:35
something I hadn't seen before I
01:39
was at church with my family,
01:39
and just got up and left. And
01:43
they had no idea I didn't say a
01:43
word because I just I had to get
01:46
out and go look and see what was
01:46
going on, and was immediately in
01:50
touch with Brian traywick, who
01:50
does all of our operations. And
01:54
he was already on the case and
01:54
investigating what the issue
01:57
was,
01:58  Bryan
I was actually pressure
01:58
washing my house at the time.
02:02
And I got a notification on my
02:02
phone that the website was down.
02:07
So I immediately put everything
02:07
down, ran inside to my computer
02:11
started looking into what was
02:11
happening, and saw that traffic
02:16
was significantly higher than
02:16
normal, about 10 to 15 times
02:21
higher than normal levels of
02:21
traffic.
02:23  Priscilla
I had done some in
02:23
the morning cleared, set out
02:25
some of the problems that I
02:25
could get to and then was going
02:28
to get back on in the afternoon.
02:28
And I looked at my phone before
02:31
starting a project in the house
02:31
and saw someone had posted in
02:34
Basecamp. And Tom had said looks
02:34
like something's going on. like
02:38
there might be we might be
02:38
getting hacked. I called Megan.
02:41
And I was like, all right,
02:41
Megan, what's going on, as she
02:43
and I were kind of going through
02:43
the information we knew to kind
02:47
of craft a response to support
02:47
people and to everyone writing
02:51
in who didn't know what was
02:51
going on. And then we were also
02:54
trying to communicate with Tom
02:54
and with Brian to figure out eta
02:58
is how serious this is what's
02:58
compromised. If anything's
03:03
compromised, all of that kind of
03:03
stuff.
03:05  Alban
I went to a museum with
03:05
some friends. And I'm driving
03:07
back and Priscilla starts
03:07
texting me from support, like,
03:11
are you seeing this, and I get
03:11
back home, sat down at my chair
03:16
and did not move from about 230.
03:16
Until 1am. It was like time did
03:21
not really work.
03:22  Travis
From then on. As word
03:22
traveled quickly, everyone
03:26
started jumping online to see
03:26
what they could do. But just to
03:30
give you some context,
03:30
Buzzsprout rarely goes down.
03:34  Alban
If ever, you can be like,
03:34
okay, we're down for a minute.
03:37
That's a big deal. We're down
03:37
for five minutes. This is a bad
03:40
outage, you're down for an hour,
03:40
and you're like, Oh my gosh,
03:43
that probably only happens once
03:43
a year. And then by the time I
03:48
you know had been there and we
03:48
got back up after three hours,
03:52
you know, is the biggest outage
03:52
I think in Buzzsprout, probably
03:54
11 year history at that point.
03:56  Travis
Now at this point, the
03:56
entire ops team, support team
04:00
and marketing team are all hands
04:00
on deck. And it was a tough go
04:05
there for a few hours on Sunday,
04:05
when we were trading blow for
04:08
blow with the attacker to see
04:08
how we could get ahead.
04:12  Bryan
The first thing was put
04:12
our site in a mode and I'm under
04:15
attack mode to try to keep it
04:15
online despite an ongoing
04:20
attack.
04:20  Travis
That's Brian, our senior
04:20
Site Reliability engineer,
04:23  Bryan
you may be unable to
04:23
access the site at times, it
04:26
most likely was very slow,
04:26
because we were trying to cut
04:29
down on the amount of automated
04:29
traffic coming from the attack
04:31
that was hitting the site at a
04:31
time and as a result, some
04:34
legitimate traffic, quite a lot
04:34
of legitimate traffic when that
04:37
mode is enabled was also being
04:37
affected. And then the next
04:41
phase was to analyze the traffic
04:41
of the attack and to find
04:46
patterns that we could actively
04:46
block to prevent just the attack
04:51
and therefore allow as much
04:51
legitimate traffic as possible
04:54
through to the site.
04:55  Travis
I asked Megan what was
04:55
going on with the support team
04:58
during this time.
04:59  Megan
So You're kind of
04:59
starting to see it in patterns.
05:01
Because as the attack was
05:01
active, we would be shutting
05:04
down certain things and then re
05:04
activating certain things just
05:07
to kind of counteract all the
05:07
attacks. So you would start to
05:09
see a lot of things that are
05:09
like, I can't even log into my
05:11
Buzzsprout account. And so then
05:11
you'd have to be like, Okay,
05:14
look, we're getting hacked,
05:14
like, keep trying, you know,
05:16
we're doing our best. And then
05:16
once we were able to get in,
05:19
they couldn't click on anything,
05:19
because our servers were being
05:21
so overloaded. They couldn't
05:21
click on anything in the
05:23
account. So they would be able
05:23
to get in and then it was like
05:25
you're frozen. And just a lot of
05:25
people being concerned about
05:28
their content. Was it safe, a
05:28
lot of people concerned about
05:31
their personal information? Was
05:31
it safe, which Luckily, we were
05:33
able to reassure everyone that
05:33
this was not a data breach, and
05:35
they were okay with all their
05:35
information was fine. all the
05:38
episodes were okay. But I think
05:38
Sunday especially, it was a day
05:42
that a lot of podcasters put up
05:42
their episodes or scheduled
05:45
their episodes for the week. And
05:45
so there was just a lot of
05:48
anxiety around when were they
05:48
going to be able to upload these
05:50
episodes to keep their audiences
05:50
happy and aware, and what should
05:53
they be telling their audiences?
05:53
So trying to communicate all of
05:56
that?
05:56  Travis
Yeah, to say that our
05:56
customers noticed that the
06:00
website was down and that none
06:00
of the episodes were playing is
06:02
a bit of an understatement.
06:02
Here's how many emails we
06:05
normally answer in customer
06:05
support on a Sunday,
06:08  Megan
we usually do between
06:08
like 100 to 150. On Sundays, it
06:11
was at least over 800 that day,
06:11
a lot more than what we
06:14
typically see on a Sunday. And
06:14
between that over that Sunday,
06:18
and that Monday, both Priscilla
06:18
and I answered over 2000 emails,
06:21
which like a busy day in support
06:21
is 500 emails would be like a
06:25
really busy day. And so for it
06:25
to be like doubled. That was a
06:30
lot.
06:31  Travis
So at this point, early
06:31
Sunday afternoon, it's all hands
06:34
on deck fighting against this
06:34
cyber attack, which we pretty
06:37
quickly figured out was a
06:37
distributed denial of service
06:40
attack, or a DDoS. But what
06:40
exactly is a distributed denial
06:45
of service attack? And why are
06:45
networks like bus routes, so
06:49
susceptible to them?
06:51  Jack
What a denial of service
06:51
is, it's when somebody attacks
06:54
you in a way that your services
06:54
cannot work anymore.
06:58  Travis
That's Jack Rhysider.
06:58
He's the host of the Darknet
07:01
Diaries podcast, which focuses
07:01
on the world of cybercrime.
07:05  Jack
Now, this could be
07:05
flooding your internet
07:08
connectivity with so much
07:08
traffic, that legit person
07:11
trying to access the website or
07:11
podcast can't get to it. That's
07:15
a volumetric based DDoS attack.
07:15
But you can, you can have a DDoS
07:19
attack, that's just one packet
07:19
that can just come in and take
07:22
down your network. And you know,
07:22
if one packet takes down your
07:25
network, that's a denial of
07:25
service as well. But in your
07:28
particular case, what happened
07:28
was that you had a lot of
07:31
traffic coming to your systems
07:31
and servers that made it just
07:35
it's almost like a crowd of
07:35
people were filling up your
07:38
business with making it so that
07:38
nobody else can come into your
07:41
business. So yeah, it's kind of
07:41
like that. There's just so much
07:44
stuff in the way that you can't
07:44
get through legitimately.
07:48  Travis
Now, if this is the
07:48
first time that you're hearing
07:49
about a distributed denial of
07:49
service attack, and what it's
07:53
like, you're certainly not
07:53
alone. And you might wonder,
07:57
well, is this something that
07:57
happens frequently? Or does this
07:59
happen rarely at all?
08:00  Jack
It's common, because it's
08:00
simple and effective. So you can
08:04
get the tools necessary to do a
08:04
denial of service attack, in a
08:08
matter of hours, you can be all
08:08
ready to go, you can use your
08:12
own systems to do it, or you can
08:12
rent systems, people have set up
08:15
botnets and different things.
08:15
And then they rent out those
08:18
botnets for whatever people want
08:18
to do with them. And one of the
08:20
most common things is to launch
08:20
a denial of service attack. So
08:24
that means there's 1000s of
08:24
computers all trying to connect
08:27
to your system at once Sending
08:27
big payloads and packets. Yeah,
08:30
it's common just because it's
08:30
easy to do and effective because
08:34
you want your servers to be open
08:34
so that the whole world can
08:37
download your podcasts or your
08:37
or your content. So you don't
08:40
want to restrict it. And here,
08:40
you've got 1000s of servers all
08:44
trying to download stuff all at
08:44
the same time. And it's what is
08:48
what your systems have been
08:48
built to do is to accept this
08:50
kind of thing. So it's, it's
08:50
really difficult to defend
08:53
against because you you want
08:53
that door to be open. And what
08:57
they're doing is just kind of
08:57
blocking that doorway. So yeah,
09:00
it's it's common, it's it
09:00
happens quite frequently. It's
09:03
part of doing business online.
09:04  John
This is not a 15 year old
09:04
programmer, or kid who's just
09:08
hacking away.
09:09  Travis
That's john Buzzsprout,
09:09
VP of programming.
09:12  John
This is someone who
09:12
actually, I believe, understood
09:15
the podcast industry, and
09:15
understood user agents in
09:18
traffic behavior. So we believe
09:18
that sometimes he was masking
09:22
himself as Apple podcasts and
09:22
going after feeds, which are
09:25
also requested barely often, and
09:25
they're getting cycled. So the
09:30
IP addresses and the URLs that
09:30
they're attacking, constantly
09:33
change. You're looking for
09:33
patterns of behavior, and he's
09:36
constantly seeing that you are
09:36
stopping him. So he's changing
09:39
his pattern.
09:40  Travis
So in a nutshell, a
09:40
distributed denial of service
09:43
attack is when someone sends
09:43
your servers, your network, more
09:48
traffic, more page requests than
09:48
you can reasonably handle. And
09:52
you may be wondering, Well,
09:52
isn't that something that
09:55
Buzzsprout should be prepared
09:55
for? Like isn't that something
09:57
that we gameplan for getting
09:57
These spikes. Sure, but it's not
10:03
quite that simple.
10:04  Jack
People might be mad at you
10:04
for being so easily knocked
10:07
over. But it's a weird thing
10:07
where you could drive your car
10:11
through a restaurant. And it's
10:11
like, well, you know, how much
10:14
do you blame the restaurant for
10:14
easily being shut down because a
10:16
car got driven through it. And
10:16
it's one of those situations
10:19
where you can't really expect
10:19
like the entire massive amount
10:23
of load, because every day
10:23
you're taking inventory on how
10:26
much load or servers your
10:26
servers have, and how much do we
10:29
need for the future. And let's
10:29
build out for that. And you can
10:31
even expect some spikes because
10:31
there's some very exciting
10:35
episodes that you know, your top
10:35
shows or have put out or
10:38
something, right, so you can
10:38
kind of expect, alright, let's
10:40
let's prepare for twice as much
10:40
traffic or even three times as
10:43
much or four times as much, but
10:43
you don't expect for 30 times as
10:46
much, or 50 times as much. And
10:46
that's something that is really
10:50
difficult to defend against,
10:50
because it's such a massive
10:54
influx of traffic that it's just
10:54
very difficult to prepare for,
10:58
and plan for and reduce in the
10:58
moment as well.
11:01  Travis
What's the motivation
11:01
here? Like, why does this guy
11:05
want to shut down Buzzsprout?
11:05
Well, we checked Twitter and
11:09
found out
11:10  Tom
and somebody saw that there
11:10
was a message, somebody had sent
11:13
a request by Twitter asking for
11:13
money to stop the attack. We
11:17
already knew at that point that
11:17
we were being attacked. We just
11:21
didn't know what the reason
11:21
there's, there's various reasons
11:23
why somebody might do an attack
11:23
like that. We didn't know it was
11:26
going to be extortion.
11:27  Travis
That raises an
11:27
interesting question, do you pay
11:30
him to go away? Because at this
11:30
point, Buzzsprout had been down
11:35
for a couple of hours, and are
11:35
podcasters who are trying to
11:37
upload episodes, look at their
11:37
statistics, share their podcasts
11:41
with their listeners, were
11:41
unable to do so. So what do you
11:45
do at that point? What do you do
11:45
when you know that at least
11:49
hypothetically, you can pay this
11:49
person and make it go away?
11:57  Tom
We had posted immediately,
11:57
we posted on Twitter,
11:59
immediately posted on Facebook,
11:59
we wanted everybody to know that
12:02
we're aware there's an issue,
12:02
but then we find out there's a
12:04
ransom demand, you can make this
12:04
whole thing go away. So we know
12:07
that our customers are being
12:07
hurt as a result of this attack.
12:10
And we know, or at least, this
12:10
person is saying that if we send
12:14
him money, it can stop.
12:15  Kevin
You know, we're aware
12:15
that these things happen. And
12:17
we're aware that people try to
12:17
extort money from businesses,
12:20
especially if they can exert
12:20
some sort of control over them.
12:23
And so from the very beginning,
12:23
it was one of the first things
12:26
Tom said one of the first
12:26
messages he sent right after he
12:29
let us know that we were being
12:29
attacked was I think he sent a
12:32
ping that said, we don't
12:32
negotiate with terrorists. And
12:34
so I realized, okay, this is
12:34
what we're facing, we cannot
12:37
give in to criminals who are
12:37
trying to extort money out of
12:40
businesses, because it's just
12:40
going to further proliferate
12:42
criminal activity.
12:43  Travis
So there's a couple of
12:43
problems that arise when you
12:45
decide to pay the ransom to make
12:45
the attack stop. The first
12:48
problem is, you have no idea if
12:48
it's actually going to stop that
12:53
it is going to take that ransom
12:53
money you gave to him and use it
12:55
to fund further attacks to then
12:55
ask for more money, there's no
13:00
guarantee that it will actually
13:00
ever end. The second reason is
13:03
because even if he stops
13:03
attacking us, he will then take
13:07
the money we gave to him and
13:07
continue to negatively impact
13:10
other businesses. So we knew
13:10
that the right answer was not to
13:15
give into this person's demands,
13:15
not to give them any money, but
13:18
instead to just focus on shoring
13:18
up our infrastructure to fight
13:23
back. But that doesn't mean it
13:23
was an easy decision, even if it
13:27
was really clear what the right
13:27
answer was. Because we knew at
13:30
the end of the day, it wasn't
13:30
just us that were suffering. It
13:33
was our podcasters. But there
13:33
was one thing that really made a
13:37
difference that really gave us
13:37
the confidence to keep pushing
13:41
forward, knowing that we made
13:41
the right decision, even if it
13:45
meant that this attack was going
13:45
to go on for a little bit
13:48
longer.
13:49  Tom
I'm glad we landed where we
13:49
did, and I'm man, I'm so
13:51
encouraged by our customers,
13:51
they were all behind it. As soon
13:54
as we told him, we're like,
13:54
Look, guys, here's a situation.
13:57
There's a criminal and they're
13:57
demanding payment. We don't
14:01
think it's right to pay them. We
14:01
think if we pay them, it's just
14:03
going to result in more attacks
14:03
either on us or other people in
14:07
the podcasting industry or
14:07
anybody else that if we pay
14:10
them, they're just going to have
14:10
a target on their back. And so
14:13
we made the decision that we
14:13
were going to not pay and we
14:17
dragged it out as much as we
14:17
could. And we use that time to
14:19
go ahead and reinforce our
14:19
defenses and get ready for
14:22
another attack.
14:23  Travis
That's right, you guys,
14:23
our podcasters were with us 100%
14:28
behind the decision to not given
14:28
to this person's demands. Now,
14:33
at this point in the story, the
14:33
first attack had stopped, he was
14:37
waiting to see if we were going
14:37
to pay him and in the
14:39
background. We were making
14:39
decisions about how to shore up
14:43
our infrastructure to get
14:43
prepared for what we knew would
14:46
be a second attack that could
14:46
happen really, at any moment.
14:52
Now that we had a good look at
14:52
the exact type of attack we were
14:55
facing, we could gameplan and
14:55
implement a surgical defense.
15:00  Tom
The attack relented, we
15:00
didn't know when it was going to
15:02
start back up. But we also know
15:02
that Monday is the busiest time
15:08
for Buzzsprout. So Monday we
15:08
have a ton of activity on our
15:13
servers, not only are people
15:13
uploading episodes, but Monday's
15:16
are also when we send out a
15:16
weekly email that tells people
15:18
their stats from the previous
15:18
week. And those stats, emails
15:21
are really server intensive when
15:21
they run and generate all those
15:25
reports. And so we know, in the
15:25
back of our minds, we know that
15:29
the worst possible case is for
15:29
the attack to resume on Monday,
15:33
and we have to just prepare for
15:33
it. And so that's why we worked
15:38
through the evening,
15:38
implementing the best things
15:41
that we could find in terms of
15:41
preparing for that attack
15:45  Kevin
through some
15:45
communication that they were
15:46
pushing our way, we had good
15:46
reason to believe that they'd be
15:48
back, we didn't know exactly
15:48
when they gave some clues
15:51
unintentionally, and some of
15:51
their communications about what
15:55
part of the world that they
15:55
might be in. And so we thought
15:57
that they might be turning in
15:57
for the night. And so we figured
16:00
we would have an opportunity to
16:00
fortify a little bit before they
16:03
woke up, or we're waiting for us
16:03
to wake up and resume the
16:06
attack. And so we had an idea
16:06
that we might have a certain
16:09
amount of time. And so Tom and
16:09
his team, were putting a plan
16:11
together of what measures can we
16:11
put in place in the time that we
16:14
have to be able to defend
16:14
ourselves against the next
16:17
attack,
16:17  Bryan
it was only a matter of
16:17
time before he resumed the
16:19
attack, it could have been at
16:19
any moment. And we didn't know
16:22
how long we had. And so we
16:22
immediately started working to
16:25
protect our infrastructure as
16:25
best we could, to put us in a
16:29
better position where we
16:29
wouldn't be affected by some of
16:32
the worst aspects of the attack
16:32
and allow us to actively fight
16:36
the attack. The next time it
16:36
occurred more effectively.
16:39  Travis
Everything is up and
16:39
running. Our customers are
16:41
happy, but they don't know that
16:41
the quarter barbarians is out
16:44
there just waiting to attack. So
16:44
they're like, oh, everything's
16:48
fixed. But we know it's only
16:48
fixed until the next attack. Now
16:52
while the operations team and
16:52
the support team were working
16:54
diligently to put things in
16:54
place to help us be prepared for
16:58
the next attack, Kevin and Alban
16:58
were deciding how to best
17:02
communicate with our customers.
17:02
Because even though we had a
17:06
couple of 1000 reach out to us
17:06
in support, not everyone was
17:10
aware of what was going on. The
17:10
question on everyone's mind is,
17:14
well, how long is this going to
17:14
last? And when is the site going
17:16
to be back up?
17:17  Kevin
And that's not a question
17:17
that we could offer an answer
17:18
to. But what we could do is the
17:18
next best thing is give them
17:21
continuous updates. So we didn't
17:21
want to push an update, like
17:25
once an hour. Now we want to be
17:25
updating them like every 10 or
17:28
15 minutes. And anytime somebody
17:28
asked a question on a social
17:31
network, we want to try to be as
17:31
real time in our response as
17:33
possible.
17:34  Alban
Our company culture has
17:34
always been very open, telling
17:37
everything that we're
17:37
explaining, we actually assume
17:40
everyone will understand us the
17:40
way that we mean things we
17:42
understood. And there's positive
17:42
intent, we think we've built up
17:45
a lot of trust over the years.
17:45
So there wasn't ever really a
17:50
question not to send it, I think
17:50
this is really true. People want
17:54
to know that you actually are
17:54
going to tell them what's
17:57
happening, and you're honest
17:57
with them and treat them like
17:59
adults. If you treat everybody
17:59
like you can't handle the truth,
18:03
you don't really need to know
18:03
you're not going to take this
18:06
the right way you're going to
18:06
use this against me. If you act
18:09
like that with everyone, they
18:09
know that you're not telling
18:11
them the truth, you're keeping a
18:11
lot back. And when you do that
18:14
they move from this posture of
18:14
I'm giving you the benefit of
18:17
the doubt, to this posture of I
18:17
know you're trying to pull the
18:21
wool over my eyes. And now I
18:21
have to become a personal
18:24
investigative journalist to
18:24
figure out why my podcast is
18:26
broken. And so I think by going
18:26
a bit more on the openness side,
18:31
what we end up doing is we end
18:31
up getting a lot of good grace,
18:35
maybe I mean, we had so many
18:35
1000s of people reaching out on
18:40
Twitter, on Facebook, personal
18:40
emails that I still haven't
18:43
gotten to now 1000s of emails
18:43
that went into support, all of
18:48
them saying like, I can't
18:48
believe this guy is attacking
18:51
and extorting you, we're behind
18:51
you 100%. Even if my show is
18:55
down for two days, we're still
18:55
behind you.
18:58  Travis
So we had informed
18:58
everyone that had a Buzzsprout
19:00
podcast, what was going on, we
19:00
were doing everything that we
19:02
could to shore up our
19:02
infrastructure to get ready for
19:04
a second attack. And sure
19:04
enough, Monday morning, he was
19:09
back.
19:10  Alban
There were a lot of rough
19:10
patches night if we were never
19:13
down, but there were a lot of
19:13
like people running into bugs.
19:16
And by 9am, the guy was back
19:16
with bigger threats than ever.
19:21
And he was able to deliver on
19:21
them, at least in part, because
19:25
Buzzsprout went back down about
19:25
9am.
19:27  Tom
My thoughts Monday when
19:27
he attack resumed was how long
19:32
ow long can this person afford
19:32
to attack us? Is it going to be
19:36
an hour, two hours, four hours,
19:36
ix hou
19:38  Travis
so the attacker was back
19:38
in full force. But even though
19:41
he took down our site
19:41
temporarily, we were much better
19:44
prepared. The second time around
19:46  Tom
everything that we had done
19:46
Sunday night came to bear on the
19:50
attack that came on Monday, we
19:50
were able to use everything that
19:52
we did Sunday night to prepare
19:52
to help mitigate the attack on
19:56
Monday, and then around 2pm we
19:56
had effectively knocked down
20:00
enough of the traffic that the
20:00
extortionist didn't see as
20:04
worthwhile anymore. So you can
20:04
see the attack finally subsided.
20:08
And we found out later that they
20:08
moved on to some other
20:11
podcasting companies after they
20:11
stopped the attack on us
20:14  Bryan
around one to 2pm on
20:14
Monday, and we were working with
20:18
a cloud provider to actively
20:18
fight the attack. And they were
20:22
able to identify an aspect of
20:22
the attack that allowed us to
20:26
surgically block the DDoS attack
20:26
while allowing as much
20:30
legitimate traffic through as
20:30
possible. And pretty much as
20:34
soon as they put that block in
20:34
place, the attacker gave up
20:37  Travis
after the attacker
20:37
finally gave up, we were able to
20:40
quickly get everything back
20:40
online and get Buzzsprout fully
20:44
functional again. And while we
20:44
wouldn't necessarily have called
20:48
it a fun experience, there were
20:48
certainly some silver linings
20:52
that are invaluable. Now,
20:52
looking back, the thing that
20:57
really stuck out to Tom, the co
20:57
founder of Buzzsprout, was how
21:01
the team pulled together,
21:03  Tom
I was so encouraged, going
21:03
into that battle that I wasn't
21:07
alone, it wasn't just me and the
21:07
ops team and the support team,
21:11
everybody, we were all there for
21:11
the fight. Everybody on our
21:15
support team, all the
21:15
developers, everyone was
21:18
answering questions posting on
21:18
Twitter, interacting on
21:20
Facebook, on the code and the
21:20
ops side, we were doing
21:23
everything we could on the back
21:23
end, to squash as many fake
21:27
requests as we could to be able
21:27
to keep things running, helping
21:31
people, people who had critical
21:31
episodes that they needed to
21:33
upload, we were working with
21:33
them to get those episodes
21:36
uploaded and posted. So it was
21:36
it was a lot different, it was
21:40
much more encouraging. On
21:40
Monday, I felt like there was an
21:42
army of us fighting back. It's
21:42
still a lot of barbarians that
21:48
were fighting. But I felt like I
21:48
had a whole army with us and our
21:51
whole our whole team and all of
21:51
our customers all of our
21:54
podcasters. man it was it was
21:54
encouraging was painful, but it
21:58
was encouraging.
21:59  Travis
Another really encouragi
21:59
g thing that happened during thi
22:02
period was that severa
22:02
of our competitors,
22:05
t least competitors on paper,
22:05
reached out and offered their su
22:09
port to help us fig
22:10  Kevin
So Monday morning, while
22:10
we were in the throes of the
22:14
attack of another podcast host
22:14
Spreaker had been attacked by
22:18
the same we assume the same
22:18
criminal network within hours of
22:22
them attacking us again. And so
22:22
they were able to connect with
22:25
us they reached out proactively
22:25
and offered any support they
22:27
could and and so some members of
22:27
our technical team hopped on a
22:30
call with their technical team
22:30
and started exchanging emails
22:33
and documents. And when you're
22:33
fighting these attacks, there's,
22:35
again, I can't get into the
22:35
specifics of the tactics that we
22:38
use to mitigate the attack. But
22:38
the strategies that seemed most
22:42
effective for them, they were
22:42
able to share with us. And some
22:45
of the code that they use, they
22:45
were able to share with us. And
22:47
so that was amazing to see, like
22:47
a competitor in the space, come
22:51
to our aid in a time of need.
22:51
Like proactively, like that was
22:56
absolutely amazing. And so I
22:56
can't thank the spreaker team
22:59
enough. And the attacks didn't
22:59
stop with us like once we got on
23:03
the other side of the attack,
23:03
then pod bean was victimized and
23:05
so was captivate and so we tried
23:05
to repay that, like continued to
23:09
pay that forward to those teams
23:09
as
23:10  John
Podbean, Spreaker and I
23:10
ere talking today, as they w
23:14
re dealing with the attack
23:14
things that they could do to hel
23:16
mitigate it, Mark and Kieran
23:16
from Captivate are sharin
23:20
their information. So we're
23:20
aking all information, and we
23:23
want to start a little GitHub
23:23
repository. Because one thing
23:27
as been great about this experi
23:27
nce is while we're talkin
23:30
together over something that's
23:30
very stressful, very agoniz
23:33
ng, and frustrating. The podcas
23:33
community has been there so inc
23:38
usive, they're so great. And ev
23:38
n competing platforms are gettin
23:43
together and helping each o
23:43
her.
23:45  Travis
But at the end of the
23:45
day, the real MVP was you guys,
23:49  Alban
the amount of like,
23:49
understanding and kindness and
23:53
good wishes that people sent our
23:53
way was unbelievable. 1000s of
23:58
people were reaching out saying
23:58
you're doing the right thing.
24:01
We're behind you. I had somebody
24:01
write us and say I'm on the free
24:05
plan. But I'm telling you, as
24:05
soon as I can log in, I'm
24:07
upgrading, and you have a
24:07
customer for life. And I'm like,
24:11
you can't even log into the
24:11
website right now. Like, I don't
24:14
know what we've done to deserve
24:14
these kind of customers.
24:17  Priscilla
I just was blown away
24:17
by our community of podcasters.
24:22
I feel like, you know, right in
24:22
the beginning, obviously, there
24:26
is a lot of worry. And there's a
24:26
lot of unknown and especially if
24:30
your podcast is possibly at
24:30
risk, especially in the
24:34
beginning, when you just don't
24:34
know what's going on, that
24:35
frustration is totally
24:35
warranted. And I feel like
24:39
people would write in and be
24:39
like, what's going on, and we
24:41
would tell them, and they would
24:41
write back the kindness things
24:44
and be like, Oh, I'm seeing this
24:44
on Twitter. Oh my goodness.
24:46
Don't even worry about it. Thank
24:46
you so much. You guys are doing
24:49
a great job. Keep up the good
24:49
work. And I you know, as you're
24:52
sitting there in the support
24:52
inbox, going email, email,
24:55
email, email without looking up
24:55
and just typing as fast as your
24:58
fingers can type and then you
24:58
get an email from someone who's
25:01
like, Hey, I just wanted to
25:01
shoot you an email and let you
25:03
know that we're rooting for you.
25:03
And you got this and thank you
25:06
for everything. Or even when you
25:06
get an email that says, hey, I
25:10
have a question. But I know
25:10
you're in the middle of stuff.
25:11
So don't prioritize this. That
25:11
kind of stuff from our people.
25:16
It felt so good in the moment.
25:16
And even now looking back on it,
25:19
I'm like, oh, my goodness, it
25:19
just feels so good. And then you
25:22
just go, how did how did we luck
25:22
out with all of these people in
25:25
Buzzsprout? I just don't I don't
25:25
get it. What did we do to
25:29
deserve all these people? I
25:29
don't, I don't know.
25:32  Travis
So what happens next?
25:32
What is Buzzsprout doing to make
25:36
sure that if something like this
25:36
happens in the future, that we
25:39
are more prepared than we were
25:39
the first time,
25:42  Kevin
this is the first time in
25:42
the 11 plus years that we've
25:45
been operating Buzzsprout that
25:45
we were attacked with something
25:49
this sophisticated this severe
25:49
for this amount of time. And so
25:53
we've had, the team has done an
25:53
amazing job of putting operating
25:57
procedures in place for things
25:57
like this. This is a trial by
26:00
fire, right? Like, you can only
26:00
do so much to simulate the
26:04
scenario until a scenario hits.
26:04
And so what I was thrilled
26:08
about, although I didn't realize
26:08
it in the moment, of course,
26:10
because you're scrambling, like
26:10
nobody panicked. everybody
26:14
understood what was happening,
26:14
and what each of our individual
26:17
jobs were in that moment. Again,
26:17
it started on a Sunday. And
26:21
there wasn't any like nobody, I
26:21
don't think it had the thought
26:24
of well, it's Sunday, like, How
26:24
bad is this? No. I mean, it's
26:27
like our customers need us. And
26:27
our team needs us. And this is I
26:31
wouldn't use the word fun. But
26:31
it's an opportunity to rally
26:35
with your team and support
26:35
customers who have supported us
26:38
for years. And I hope that we've
26:38
done a good job over those years
26:41
of building that trust bank with
26:41
them that when they realized
26:44
that they went to log in on
26:44
Sunday, and things weren't
26:47
working the way that they would,
26:47
that they trusted, that we were
26:50
on it. And then we were going to
26:50
fight hard. Regardless of what
26:52
was coming at us, we're going to
26:52
try to do the right thing. And
26:54
we're going to be upfront and
26:54
transparent with them and do
26:56
everything we could and drop
26:56
everything to support our
26:58
customers.
26:59  Tom
So moving forward, we have
26:59
a action plan, we've experienced
27:03
it, we've gone through it, it's
27:03
it's something, it could happen
27:06
to anyone at any time. So we've
27:06
always known that we could be
27:09
the target of an attack like
27:09
this. So in some ways, it's a
27:13
little bit of a relief that
27:13
we've been through it because
27:15
we've always known it could
27:15
happen. Well, we've been through
27:18
it, we've learned a ton, we've
27:18
got a great infrastructure in
27:23
place to be able to mitigate it
27:23
in the future. I can't guarantee
27:27
that we won't have issues that
27:27
we couldn't be taken down for a
27:31
period of time as a result of
27:31
it. But I can say that we have
27:35
the best team with the most
27:35
experience in how to mitigate
27:39
this when it comes to Buzzsprout
27:39
podcasting,
27:41  Travis
being the victim of a
27:41
cyber attack, like what we
27:43
experienced this past week is
27:43
never something that you hope
27:46
happens to you. But at
27:46
Buzzsprout we're big believers
27:51
in turning negatives into
27:51
positives, which is why we're
27:55
going to do something special
27:55
with the money that the hacker
27:58
had hoped that we would be
27:58
sending to him.
28:00  Kevin
So now that we've had a
28:00
couple days to process and
28:02
recover, like what we want to do
28:02
on this side of that experience.
28:07
And so we've done two things.
28:07
One is we want to put our money
28:10
where our mouth is, and like we
28:10
have said that there's the
28:13
decision to not pay the ransom
28:13
had nothing to do with the
28:15
money. And so like, let's make
28:15
that real. And so there are
28:19
people in Texas right now,
28:19
there's people all over the
28:21
world. But there's people right
28:21
in our backyard in Texas, who
28:24
are just you know, they're
28:24
suffering from this weather
28:26
crisis that they were hit with
28:26
last week, there are people who
28:28
are still without power, the
28:28
food chain has been disrupted,
28:30
they don't have clean water. And
28:30
so we've taken the money that
28:33
the criminals demanded as ransom
28:33
holding our business captive.
28:36
And we've donated that to two
28:36
charities in Texas. One is
28:39
called feeding Texas and the
28:39
other one is called front steps.
28:42
And the other thing that we've
28:42
done is that we realized that
28:44
DDoS attacks require a lot of
28:44
energy to run. I mean, you're
28:47
spinning up 1000s of computers
28:47
for hours doing this one useless
28:51
task, and there is a negative
28:51
effect to that. And so we've
28:54
also made a donation to project
28:54
Vesta that will more than offset
28:58
the carbon footprint of the
28:58
attack. So the end result of the
29:01
attack is that podcast episodes
29:01
were delayed through the
29:04
Buzzsprout platform for a few
29:04
hours. But people in need are
29:08
now going to get food, water and
29:08
shelter and the global
29:11
environment is going to be
29:11
healthier overall.
29:14  Travis
Thanks Kevin for helping
29:14
me end this episode. on a high
29:16
note. If you'd like to read the
29:16
full DDoS technical post mortem,
29:21
you can click on the link in the
29:21
show notes to read it on our
29:23
blog. And we'll also leave links
29:23
to the three charities that we
29:27
donated to as a result of this
29:27
series of events in the show
29:30
notes as well. If you would like
29:30
to make a donation there to
29:33
special thanks to everyone on
29:33
the Buzzsprout team that gave
29:35
their time to share their
29:35
stories of what happened behind
29:38
the scenes. Jordan host of th
29:38
Dreamful podcast for our epic
29:42
pisode artwork. And thank you
29:42
or sticking with us through
29:46
hick and thin. We're so excited
29:46
o be able to continue to serve
29:49
ou and help you with your
29:49
odcast. Well that's it for this
29:52
eek. Thanks for listening. And
29:52
s always keep podcasting