Recently I’m working with binary classification problem by training the Neural Network using Keras. I’m observing some strangeness in the model’s performance. I got the ROC AUC is to be 0.82. But, for example while I’m trying to get the trained model’s predictions for my dataset, it barely distinguishes two classes from each other. To be more clear, I’m trying to discriminate the signal from the background processes using Particle Physics data. I’m assigning 1 to the signal samples and 0 to the background samples. Then I’m training the NN and once the training done, I’d like to get the model’s predictions for signal and background samples separately. But model distinguishes signal from background very poorly, although the ROC AUC is 0.82. I’ve checked many times whether there are any bug in my code, however everything seems fine. One guess is that I have sample weights for my processes, is it possible that sample weights I’m using can somehow affect the model? If so, how can I overcome this problem??
Any suggestions would be appreciated!