This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
https://www.dataengineeringpodcast.com
Building Real Time Applications On Streaming Data With Eventador
Summary
Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. Eventador is a managed platform designed to let you focus on using the data that you collect, without worrying about how to make it reliable. In this episode Eventador Founder and CEO Kenny Gorman describes how the platform is architected, the challenges inherent to managing reliable streams of data, the simplicity offered by a SQL interface, and the interesting projects that his customers have built on top of it. This was an interesting inside look at building a business on top of open source stream processing frameworks and how to reduce the burden on end users.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, a 40Gbit public network, fast object storage, and a brand new managed Kubernetes platform, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. And for your machine learning workloads, they’ve got dedicated CPU and GPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
- Your host is Tobias Macey and today I’m interviewing Kenny Gorman about the Eventador streaming SQL platform
- Introduction
- How did you get involved in the area of data management?
- Can you start by describing what the Eventador platform is and the story
- behind it?
- How has your experience at ObjectRocket influenced your approach to streaming SQL?
- How do the capabilities and developer experience of Eventador compare to other streaming SQL engines such as ksqlDB, Pulsar SQL, or Materialize?
- What are the main use cases that you are seeing people use for streaming SQL?
- How does it fit into an application architecture?
- What are some of the design changes in the different layers that are necessary to take advantage of the real time capabilities?
- Can you describe how the Eventador platform is architected?
- How has the system design evolved since you first began working on it?
- How has the overall landscape of streaming systems changed since you first began working on Eventador?
- If you were to start over today what would you do differently?
- What are some of the most interesting and challenging operational aspects of running your platform?
- What are some of the ways that you have modified or augmented the SQL dialect that you support?
- What is the tipping point for when SQL is insufficient for a given task and a user might want to leverage Flink?
- What is the workflow for developing and deploying different SQL jobs?
- How do you handle versioning of the queries and integration with the software development lifecycle?
- What are some data modeling considerations that users should be aware of?
- What are some of the sharp edges or design pitfalls that users should be aware of?
- What are some of the most interesting, innovative, or unexpected ways that you have seen your customers use your platform?
- What are some of the most interesting, unexpected, or challenging lessons that you have learned in the process of building and scaling Eventador?
- What do you have planned for the future of the platform?
- Blog
- @kennygorman on Twitter
- kgorman on Twitter
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
- Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
- Eventador
- Oracle DB
- Paypal
- EBay
- Semaphore
- MongoDB
- ObjectRocket
- RackSpace
- RethinkDB
- Apache Kafka
- Pulsar
- PostgreSQL Write-Ahead Log (WAL)
- ksqlDB
- Podcast Episode
- Pulsar SQL
- Materialize
- Podcast Episode
- PipelineDB
- Podcast Episode
- Apache Flink
- Podcast Episode
- Timely Dataflow
- FinTech == Financial Technology
- Anomaly Detection
- Network Security
- Materialized View
- Kubernetes
- Confluent Schema Registry
- Podcast Episode
- ANSI SQL
- Apache Calcite
- PostgreSQL
- User Defined Functions
- Change Data Capture
- Podcast Episode
- AWS Kinesis
- Uber AthenaX
- Netflix Keystone
- Ververica
- Rockset
- Podcast Episode
- Backpressure
- Keen.io
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast