Best Practices When Accessing Big Data or Any Other Data

Explore best practices for accessing data on local or shared file systems in this ACCU 2019 conference talk. Learn from real customer examples how simple guidelines and proper tool usage can significantly improve code performance, reliability, scalability, and portability. Discover why data access methods are crucial, especially as datasets grow larger. Gain insights from Dr. Rosemary Francis, an expert in I/O profiling and high-performance computing, as she shares her experience working with scientific and high-performance computing customers. Understand the common challenges in I/O management and learn practical solutions to optimize applications and manage IT infrastructure effectively. Delve into topics such as the noisy neighbor problem, housekeeping and deletes, failed I/O, file system traversals, opens and closes, and storage comparisons. Benefit from a case study on tuning cancer pipelines at the Sanger Institute to see these principles applied in a real-world scenario.

Intro
Best practices when accessing Big Data ... or any other data!
Ellexus Ltd: The I/O Profiling Company
Ellexus enterprise products
The noisy neighbour problem
Housekeeping and deletes
Failed I/O and file system trawis
Opens and closes
What is normal?
Tuning cancer pipelines at the Sanger Institute
Profiling the cancer pipeline
Storage comparison
How long did this work take?