Azure Data Engineer Associate (DP-203) Cert Prep: 4 Monitor and Optimize Data Storage and Data Processing

Learn about monitoring and optimizing data security and data processing on Azure and prepare to pass that domain of the Microsoft Azure Data Engineering (DP-203) exam.

Introduction

• Course introduction

1. Monitor Data Storage and Data Processing

• Implement logging used by Azure Monitor
• Configure monitoring services
• Measure performance of data movement
• Monitor data system/pipeline/cluster performance
• Measure query performance
• Schedule and monitor pipeline tests
• Interpret a Spark directed acyclic graph (DAG)

2. Optimize and Troubleshoot Data Storage and Data Processing

• Rewrite user-defined functions (UDFs)
• Handle skew in data and data spill
• Tune shuffle partitions/pipelines
• Optimize resource management
• Tune queries by using indexers and cache
• Troubleshoot a failed Spark job and pipeline run

Conclusion

• Summary and next steps