Summary
Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. Despite its widespread popularity, there are numerous accounts of the difficulty that operators face in keeping it reliable and performant, or trying to scale an installation. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine. In this episode he explains how they engineered a drop-in replacement for Kafka, replicating the numerous APIs, that can scale more easily and deliver consistently low latencies with a much lower hardware footprint. He also shares some of the areas of innovation that they have found to help foster the next wave of streaming applications while working within the constraints of the existing Kafka interfaces. This was a fascinating conversation with an energetic and enthusiastic engineer and founder about the challenges and opportunities in the realm of streaming data.
AnnouncementsVectorized
Free Download Trial
@vectorizedio Company Twitter Accn’t
Community Slack
Concord alternative to Flink
Apache Flink
FAANG == Facebook, Apple, Amazon, Netflix, and Google
Blackblaze
Raft
NATS
Pulsar
Open Messaging Specification
ScyllaDB
CockroachDB
MemSQL
WASM == Web Assembly
Debezium
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast