GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Gabriel Mongaras Gabriel Mongaras
8.89K subscribers
1,027 views
28

 Published On Mar 21, 2024

My notes: https://drive.google.com/file/d/1l2B4...

Paper: https://arxiv.org/abs/2403.03507


00:00 Intro
02:44 Intuition and proof of low rank
12:28 GaLore intuition
16:38 More GaLore intuition
21:20 GaLore algorithm
27:50 Algorithm analysis
33:00 Results

show more

Share/Embed