Computer Vision and NLP for Multi-Task Fashion Modeling

Explore a 30-minute conference talk from Strange Loop Conference on multi-task fashion modeling using computer vision and natural language processing. Dive into Shoprunner's approach to standardizing attributes for millions of products from various retailers. Learn about the challenges of predicting attributes using different data representations like images, product descriptions, titles, and brand names. Discover the multi-task learning ensemble developed by Shoprunner's Data Science team, combining custom multi-task CNNs for image processing and fine-tuned BERT models for text classification in PyTorch. Gain insights into attribute modeling techniques, including multi-task network architectures, training protocols, and ensemble approaches. Explore topics such as clothing detection, segmentation, and the implementation of BERT for text analysis. Understand how to expand attribute modeling to new categories and the concept of multi-dataset multi-task learning. Presented by Michael Sugimura, a Senior Data Scientist at Shoprunner specializing in computer vision applications for e-commerce.

Intro
Attribute Modeling at Shoprunner
Overview
Why Images and Text?: Jeans
Multi-Task Learning: Karate Kid Example
Multi-Task Learning: Shoprunner Example
Example Multi-Task Network
General Training Protocols
Image Model Architecture: Centercrops
Clothing Detection RED
Clothing Segmentation R&D
Text Model Architecture: BERT
BERT Implementation
Ensemble Architecture
Expanding to New Attributes
Franken Model Process
How do we Train New Attribute Task Heads?
Multi-Dataset Multi-Task Learning
Ongoing Work
Questions?
Contact Info