Applying DevOps Practices in Data and ML Engineering

Explore how to apply DevOps practices in data and machine learning engineering in this comprehensive talk from the MLOps Community Meetup. Learn about the challenges of adopting DevOps principles in data engineering and discover how the open-source project Versatile Data Kit (VDK) addresses these issues. Follow along as Antoni Ivanov, Software Engineer at VMware, demonstrates creating and productionizing an end-to-end data pipeline efficiently. Gain insights into automating and abstracting the development process, managing the data journey, and extending functionality through practical examples and a step-by-step demo. Understand how VDK can transform data engineering towards a code-first, fully automated, and decentralized approach, potentially revolutionizing your data and ML workflows.

[] Introduction to Antoni Ivanov
[] Applying DevOps Practices in Data and ML Engineering
[] Agenda
[] DevOps Challenges
[] Versatile Data Kit
[] DevOps for Data as a Service
[] Components
[] VDK, SDK, and VDK Runtime Control Service
[] Automate and Abstract the Development Process
[] Quick Example: DevOps Plugin
[] Automate and Abstract the Data Journey
[] Quick Example: vdk-impala, vdk-trino
[] Data Journey: Ingestion
[] Ingestion Job
[] Demo
[] Step 1: Explore VDK's Functionalities
[] Step 2: Create a Data Job
[] Features in real-time
[] Working with streaming tools
[] VK as a training model
[] Step 3: Ingestion Job
[] Step 4: Processing Job
[] Step 5: Deploy
[] Step 6: Extend Anonymize
[] Step 6: Extend SQL Validation