Hugging Face Workshops - Pretraining Language Models & CodeParrot
Offered By: HuggingFace via YouTube
Course Description
Overview
Explore pretraining language models and Hugging Face's CodeParrot in this live workshop led by Leandro and Merve. Dive into the intricacies of data transformation, batching, and deep speed techniques. Learn about CodeParrot's capabilities for SQL and its evaluation process. Tackle coding challenges, address issues with duplicates, and understand deduplication methods. Gain insights into logging, model training loops, clipping, and checkpoints. Discover the potential of crosslingual transfer in natural language processing.
Syllabus
Intro
CodeParrot Overview
Pretraining Language Models
Data Transformation
CodeParrot
CoParrot
Batching
Iter
Tensor
Deep Speed
BigQuery vs DataSets
CodeParrot for SQL
Evaluation of CodeParrot
Coding Challenges
Problems with Duplicates
Deduplication
Questions
Logging
Models
Training Loop
Clipping
Checkpoints
More Questions
Crosslingual Transfer
Taught by
Hugging Face
Related Courses
Natural Language ProcessingColumbia University via Coursera Natural Language Processing
Stanford University via Coursera Introduction to Natural Language Processing
University of Michigan via Coursera moocTLH: Nuevos retos en las tecnologías del lenguaje humano
Universidad de Alicante via Miríadax Natural Language Processing
Indian Institute of Technology, Kharagpur via Swayam