Chameleon: Expanding Open-Source Ambari for HPC
Offered By: Linux Foundation via YouTube
Course Description
Overview
Syllabus
Chameleon Expanding Open-Source Ambari for HPC
Motivation (Trend Perspective) HPC and Bigdata is converging
Motivation Application Perspective Genome analysis by Our scientist Some stages of data pipeline begin to support bigdata platform
Ambari Overview Apache Ambari is a 100% open source platform for provisioning managing and monitoring Hadoop clusters
Extension Points for custom service development Ambari view is a plugin that provides a way to connect custom functions to the web UI Ambari stack defines a set of everything needed to define services such as HDFS and YARN
lustrefs Management Service Lustre Kernel Installation function(LustrekernelUpdater)
Account Management Service Hadoop does not support strong authentication by default • Hadoop supports Kerberes for that, but, causes performance
Hadoop-on-Lustre architecture Comparison between HDFS and Lustre
Related works for Hadoop-on-Lustre Xyrates • MapReduce Job shows theoretical performance gains on an appropriately designed Lustre based HIPC cluster with Infinband network Seagate's lustrels plugin
2 Hadoop-on-Lustre execution environment Hadoop-on-Lustre execution environment • Works for diskless cluster backed by Lustre • Uses secure container configuration for multi-tenancy
Motivation Dynamic metrics management is required
YARN Application Monitoring Service Time-series data monitoring
3 TimeScaleDB Open-Source time-series database optimized for fast
3 Data management structure Alter Table
HPC Resources Monitoring Provides HPC monitoring information through web UI
Summary HPC and big data convergence makes the distinction between data analytics and computational science's ecosystem disappear. Chameleon is a Bigdata platform operation management system considering HPC environment Chameleon helps to merge Hadoop ecosystem and Lustres.
Taught by
Linux Foundation
Tags
Related Courses
High Performance ComputingGeorgia Institute of Technology via Udacity Введение в параллельное программирование с использованием OpenMP и MPI
Tomsk State University via Coursera High Performance Computing in the Cloud
Dublin City University via FutureLearn Production Machine Learning Systems
Google Cloud via Coursera LAFF-On Programming for High Performance
The University of Texas at Austin via edX