Feat/sophia (!4) · Merge requests · deep-learning-nlp / token-tricksters · GitLab

This is an archived project. Repository and other project resources are read-only.

Lars Benedikt Kaesberg requested to merge feat/sophia into main Jun 17, 2023

Implemented the Sophia optimizer.
Added SophiaH (with the Hutchinson Estimator) as an option to the train loop.
Added gradient clipping to the train loop.
Added AMP Autocast to bfloat16.

Edited Aug 21, 2023 by Niklas Bauer