8-bit Optimizers - one of the tricks widely used to trian large model to save memory space in gpus. It does the job with good, and make it possible to train larger model in consumer level gpus (16gb, 24gb). It’s available in pytorch. Would you please inform if there’s an equivalent in tensorflow?
Yes, It is called the mixed precision training API. This API allows you to train your models using lower precision numbers, such as 8-bit integers, while still maintaining good accuracy. This can save a significant amount of memory, which can be important for training large models on consumer-level GPUs.you can then enable mixed precision training by setting the tf.config.optimizer.set_experimental_options use mixed_precision flag to True . Once you have enabled mixed precision training, you can then choose which variables to train in lower precision. You can do this by setting the tf.config.optimizer.experimental_mixed_precision_loss_scale flag to a value between 0 and 1. A value of 0 will train all variables in full precision, while a value of 1 will train all variables in lower precision.
Please see the following articles for more information:
To enable mixed precision training in TensorFlow, you can use the “tf.keras.mixed_precision” module. You can set the policy to “mixed_float16” to enable mixed precision training, which will automatically cast your weights and activations to float16 when appropriate.