Astronomer's Journey from Cloud Lock-in to Open Source Independence

Explore Astronomer's journey from cloud lock-in to open-source independence in this 24-minute Linux Foundation conference talk. Learn how the data engineering platform transitioned from AWS-managed services to open-source solutions running on Apache Mesos and DC/OS. Discover the challenges faced with proprietary components and the steps taken to adopt Apache Airflow, Apache Kafka, and Marathon. Gain insights into the company's engineering and product obstacles, infrastructure-as-code practices using Terraform, monitoring solutions with Prometheus, and future plans involving Kafka Connect, Amazon S3, and Kubernetes. Understand the benefits and considerations of moving away from cloud provider lock-in and embracing open-source technologies for greater flexibility and independence in data engineering workflows.

Introduction
Early priorities
V1 of Astronomer
Engineering obstacles
Product obstacles
Apache Airflow
Why Apache Airflow
mezzo scheduler
Airflow
Kinesis
Road to Kafka
Not all tools are created equal
Exploration
Terraform
Infrastructure is Code
Prometheus
Monitoring
Prometheus Monitoring
Kong
Kairos DB
Why Kairos DB
Whats next
Kafka Connect
Amazon S3
Kubernetes
Questions
Redundancy