ERROR: tensorflow model.compile() with imbalanced dataset

General Discussion
,
#1

hi community :slight_smile:

I am training a simple LSTM 3 layers network on a small dataset size of around 6K records, with 6 imbalanced classess.

My network is overfitting , I tried early stopping, different batch size, and learning rate scheduler . I am now exploring class_weight option but I am getting error when compiling. Here is my code:

Identifiying my network

from keras.layers import Embedding, Bidirectional, LSTM, GlobalMaxPooling1D, Dense
def get_model_text_lstm_embedding(input_shape=False):
model = Sequential()
model.add(
    Embedding(
        input_dim=len(vocab_list),
        output_dim=embedding_matrix.shape[1],
        input_length=train_features_indexed_padding.shape[1],
    )
)
model.add(Bidirectional(LSTM(units=128, return_sequences=True)))
model.add(GlobalMaxPooling1D())
model.add(Dense(units=HP.n_classes, activation="softmax"))
return model

Class Weights

from sklearn.utils.class_weight import compute_class_weight
train_classes = np.argmax(y_train_text, axis=1)
class_weights = compute_class_weight(class_weight="balanced", classes=np.unique(train_classes), y=train_classes)

Compling and Training

text_model_lstm_embedding.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
loss="categorical_crossentropy", metrics=["accuracy"], class_weight=class_weights)
history_text_model_lstm_embedding = text_model_lstm_embedding.fit(
x_train_text, y_train_text,validation_data=(x_test_text, y_test_text), 
callbacks=callback_f(nameof(text_model_lstm_embedding)),
epochs=HP.epochs,verbose=3,shuffle=True,batch_size=32)

`

The error message

TypeError: Invalid keyword argument(s) in compile(): ({‘class_weight’},). Valid keyword arguments include “cloning”, “experimental_run_tf_function”, “distribute”, “target_tensors”, or “sample_weight_mode”.

#3

Hi @Wael_AbuRezeq, I have few suggestions in your code, while calculating the class weights you have to get the all unique classes for which you have to use np.unique if you use argmax you only get the highest class number

import numpy as np
from sklearn.utils.class_weight import compute_class_weight
y = [1, 1, 1, 1, 0, 0]
c_weights=compute_class_weight(class_weight="balanced", classes=np.unique(y), y=y)

If you compute class_weights in this way you will get the class weights in an array, but while passing class_weights in the compile method it should be in dictionary format. For example class_weight = {0: weight_for_0, 1: weight_for_1, ...........}.

Thank You.

#5

hi Kiran,
thank you for your reply!

I tried it as a dictionary format, the error presists. I believe the error is more into the compatability of my tf compile with the keyword class_weight

TypeError: Invalid keyword argument(s) in compile(): ({‘class_weight’},). Valid keyword arguments include “cloning”, “experimental_run_tf_function”, “distribute”, “target_tensors”, or “sample_weight_mode”.

#8

Hi @Wael_AbuRezeq, Could you please stand alone code to reproduce the issue. Thank You.

#10

hi @Kiran,

Here is the code

from sklearn.utils.class_weight import compute_class_weight
train_classes = np.argmax(y_train_text, axis=1)
class_weights = compute_class_weight(class_weight = “balanced”, classes= np.unique(train_classes), y= train_classes)
class_weights_dict = dict(zip(np.unique(train_classes), class_weights))
class_weights_dict
text_model_lstm_embedding = get_model_text_lstm_embedding()

Compile the model with class weights

text_model_lstm_embedding.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),
loss= ‘categorical_crossentropy’ ,
class_weight=class_weights_dict
)

TypeError: Invalid keyword argument(s) in compile() : ({‘class_weight’},). Valid keyword arguments include “cloning”, “experimental_run_tf_function”, “distribute”, “target_tensors”, or “sample_weight_mode”.

#12

Hi @Wael_AbuRezeq, The class weights argument should be used in model.fit, but i can see that you are using that argument in the model.compile. For example,

model.fit(
    train_features,
    train_labels,
    epochs=10,
   # The class weights go here
    class_weight=class_weight)

Thank You.

#14

Thanks a million :slight_smile: it finally worked!

Just one correction, the class_weight gets the input as dictionary (as you originally stated but you used the array version in your response