Adam

A widely used method for updating a network's weights during training. It gives each weight its own step size, based on running averages of its recent gradients. AdamW is a common variant with a different form of weight decay.