Performance of the Keras

Special Interest Groups Keras
,
#1

Hi all,

Recently I’m working with binary classification problem by training the Neural Network using Keras. I’m observing some strangeness in the model’s performance. I got the ROC AUC is to be 0.82. But, for example while I’m trying to get the trained model’s predictions for my dataset, it barely distinguishes two classes from each other. To be more clear, I’m trying to discriminate the signal from the background processes using Particle Physics data. I’m assigning 1 to the signal samples and 0 to the background samples. Then I’m training the NN and once the training done, I’d like to get the model’s predictions for signal and background samples separately. But model distinguishes signal from background very poorly, although the ROC AUC is 0.82. I’ve checked many times whether there are any bug in my code, however everything seems fine. One guess is that I have sample weights for my processes, is it possible that sample weights I’m using can somehow affect the model? If so, how can I overcome this problem??

Any suggestions would be appreciated!

Cheers,
Aizat

#3

Hi Aizat,

A couple of questions:

  1. By what measure is the model distinguishing signal from background poorly?
  2. Was the 0.82 ROC-AUC computed during training or on a test dataset not used in training?

Roger

#5

Hi Roger,

  1. Please find in URL a jpgDistribution showing the distribution of the trained NN predictions, a.k.a. scores for the signal and for the background samples. You can see on horizontal axes the trained NN model’s outputs, i.e. predictions, i.e. scores. Red and yellow lines are the background data, green and blue lines are the signal data. Initially, I assigned 1 to the signal data and 0 to the background data, so my expectation is that red and yellow lines massively tend to be around zero, green and blue lines should tend to be around 1. I assumed that if the ROC AUC is 0.82, then this distribution would be as I anticipated. Such distribution is crucial for me because using this histogram I need to obtain important value such as significance of the signal.
  2. ROC AUC I obtained for both training and test data, the results are almost the same.

Here the quick question: in Keras I get the trained NN’s output by invoking the method Predict(). Is that correct?

Cheers,
Aizat

#7

Yes, model.predict() will use the trained model to make predictions.

I may not be reading your score distribution chart correctly, but it doesn’t seem too surprising to me. The red and yellow (background) frequencies slope downward pretty steeply as the scores increase. The green and blue (signal) frequencies slope upward for the most part.

Roger