Software Engineering Daily

Spotify is a streaming music company with more than 50 million users. Whenever a user listens to a song, Spotify records that event and uses it as input to learn more about the user’s preferences. Listening to a song is one type of event–there are hundreds of others. Opening the Spotify app, skipping a song, sharing a playlist with a friend–all of these are events that provide valuable insights to Spotify.

These are not the only types of events that Spotify cares about. There are also events that occur at the infrastructure level–for example a logging server that runs out of disk space. There are events that are relevant to all the users on Spotify–for example a new album release from Taylor Swift.

An “event” is an object that needs to be registered within a system. Since there are so many events on a platform like Spotify, delivering and processing them reliably requires significant investment.

Modern Internet companies are built by connecting cloud services, databases, and internal tools together. These different systems might respond to different events in different ways. Each system subscribes to the types of events that it wants to hear. Since there are so many events, and they might be received at uneven bursts, a modern architecture has a scalable queueing system to buffer events.

To put an event on the queue, the event producer “publishes” that event to the queue. The event is then received by each “subscriber.” That’s why queueing is often known as pub/sub–publish/subscribe.

Igor Maravic is an engineer with Spotify. In this episode, he explains why pub/sub is a key element of Spotify’s infrastructure–and he describes the migration that Spotify has made from Apache Kafka to Google Cloud Pubsub.

If you like this episode, we have done many other shows about cloud infrastructure. You can check out our back catalog by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that.

To build the kinds of things developers want to build today, they need better tools.  That’s why Amazon Web Services built Amazon Aurora. A relational database engine that’s compatible with MySQL and PostgreSQL, and provides up to five times the performance of standard MySQL—on the same hardware, at a tenth of the cost. Amazon Aurora from AWS can scale up to millions of transactions per minute. Automatically grow your storage up to 64 terabytes. And replicates data to three different Availability Zones. And you don’t have to manage a thing. There are no upfront charges, no commitments—you only pay for what you use. Check it out, at aurora.aws.

Amazon Redshift powers the analytics of your business–and Intermix.io powers the analytics of your Redshift. Intermix.io gives you the tools you need to analyze your Amazon Redshift performance and improve the toolchain of everyone downstream from your data warehouse. The team at Intermix has seen so many Redshift clusters, they are confident they can solve whatever performance issues you are having. Go to intermix.io/sedaily to get a free 30-day trial. Intermix collects all your Redshift logs and makes it easy to figure out what’s wrong so you can take action. All in a nice, intuitive dashboard. Go to intermix.io/sedaily to start your free 30-day trial.

Toptal is the best place to find reasonably priced, extremely talented software engineers to build your projects from scratch or scale your workforce. Get a free pair of Apple Airpods when you use Toptal.com/sedaily to work with an engineer for at least 20 hours.

If you want to start a podcast, check out Podsheets. Podsheets is a product we built to create and manage podcasts. We are podcasters ourselves–and we understand the difficulties of getting started. Podsheets makes it easy to post your episodes and distribute them to iTunes and Google Play with a single click. If you are curious about podcasting, but have no idea where to start, Podsheets will guide you through the process. With Software Engineering Daily, we have been producing 5 shows a week for 2 years. We understand recording, we understand how to produce your show and we understand how to get advertisers. We want to help you with this process. Check out Podsheets today. We will give you everything you need to create and manage your podcast–and if you have any questions or you get confused, you can always contact us directly for help. Podcasting is as easy as blogging–let us show you how to podcast, with Podsheets.