Signal classifiers do not generalize to real-world data

General Discussion
,
#1

I’ve created several models with different architectures (Dense layers, CNN, combinations) for binary classification, and many of them train and perform well on train, validate, test datasets. When it comes to classifying a window of a signal, they all fail miserably, and classify pretty much every window as positive. I also tried an autoencoder anomaly detector, but false positives and false negatives were very high (50%).

I realize this is a very general description, but can provide more detail, and would greatly appreciate any advice.

#2

Hi @John, This may be due to overfitting, did you apply regularization to your model. Thank You.

#3

Thanks Kiran, good call!

I’m using L1 regularization, but with a low value of 0.001.
I’ll iterate through some values and see how things go.
And perhaps try L2 also.

Thanks again!