•1 min read•from Machine Learning
Educational PyTorch repo for distributed training from scratch: DP, FSDP, TP, FSDP+TP, and PP [P]
I put together a small educational repo that implements distributed training parallelism from scratch in PyTorch:
https://github.com/shreyansh26/pytorch-distributed-training-from-scratch
Instead of using high-level abstractions, the code writes the forward/backward logic and collectives explicitly so you can see the algorithm directly.
The model is intentionally just repeated 2-matmul MLP blocks on a synthetic task, so the communication patterns are the main thing being studied.
Built this mainly for people who want to map the math of distributed training to runnable code without digging through a large framework.
[link] [comments]
Want to read more?
Check out the full article on the original site
Tagged with
#rows.com
#no-code spreadsheet solutions
#natural language processing for spreadsheets
#generative AI for data analysis
#Excel alternatives for data analysis
#enterprise-level spreadsheet solutions
#large dataset processing
#cloud-based spreadsheet applications
#PyTorch
#distributed training
#parallelism
#FSDP
#TP
#2-matmul MLP
#collectives
#forward/backward logic
#communication patterns
#synthetic task
#educational repo
#algorithm