How to use sample weight under MirroredStrategy mode

General Discussion
#1

Hello,
I have tried (input_batch, label_batch, sample_weight_batch) under MirroredStrategy mode，but sample_weight doesn’t work，because loss doesn’t change significantly under hight or low sample weight.

thanks in advance for any explanations/solutions/ideas

#3

@Xinwei_Gu try this structured approach to troubleshoot:

  1. Ensure your MirroredStrategy is correctly set up. Are all GPUs actively engaged, and is there no isolation? Reassess if your batch splitting aligns seamlessly with the distributed configuration.
  2. Delve into the intricacies of how sample_weight is integrated into your loss function. Is the scaling appropriate? Confirm that your loss function is effectively responsive to variations in sample_weight.
  3. While it may seem traditional, inserting print statements to monitor the values of sample_weight during training could unveil valuable insights.
#5

Thank your solutions.
I have tried put optimizer inside with mirrored_strategy.scope()，put compile outside with mirrored_strategy.scope(), the sample_weight does work. But I wonder the reason.

1 Like
#6

@Xinwei_Gu glad to hear that adjusting the optimizer and compilation scope worked for you! The reason behind this behavior could be related to how MirroredStrategy distributes the optimization process across GPUs.

Placing the optimizer inside mirrored_strategy.scope() ensures distributed optimization, while putting compile outside might be influencing how the loss and sample_weight are handled in this distributed setup.