On the overlooked pitfalls of weight decay and how to mitigate them a gradient norm perspective. Additional information example for students. Inveeg split. Hudor Bauang.