Include Lion Optimizer

[Haoxiang-Wang]

Google Brain releases a new optimizer, Lion (https://arxiv.org/abs/2302.06675) which shows superior performance than AdamW on a wide range of deep learning training tasks. They provide a PyTorch implementation at https://github.com/google/automl/tree/master/lion

@lucidrains also implemented a version at https://github.com/lucidrains/lion-pytorch

Could you include it in this codebase? @jettify

[zhaoguangyuan123]

I tried, but actually, it does not work well. It works well on that paper's dataset but not well on my dataset. AdamW is still the best one on my dataset.

[zhaoguangyuan123]

I tried, but actually, it does not work well. It works well on that paper's dataset but not well on my dataset. AdamW is still the best one on my dataset.

just add a data point from a user's perspective.