How improve neural network

General Discussion
, ,
#1

Hi.
I have a test dataset with 285 columns.
As I understand it is mean that I need 284 input as X and 1 column for Y as result.
I have only 40 output options, so NN has 40 output shape.

This is my code in python:

def get_x(dataset):
    X = dataset.iloc[:,1:285].values
    return X

def get_y(dataset):
    y = dataset.iloc[:,285:286].values
    return y

def make_nn():
    output = 40  
    model = Sequential()
    model.add(Dense(284, input_dim=284, activation='softmax'))
    model.add(Dense(284, activation='softmax'))

    model.add(Dense(output, activation='softmax'))
    for i in range(250):
        model.add(Dense(284, activation='softmax'))
    model.add(Dense(40, activation='softmax'))
    model.summary()

    model.compile(loss='categorical_crossentropy', optimizer='adamax', metrics=['accuracy'])
    return model

def train_nn(dataset, model):
    x=get_x(dataset)
    y=get_y(dataset)

    X_train,X_test,y_train,y_test = train_test_split(x,y,test_size = 0.1,random_state = 0)
    output = 40 
    y_train_cat = keras.utils.to_categorical(y_train, output)
    y_test_cat = keras.utils.to_categorical(y_test, output)

    model.fit(X_train, y_train_cat ,validation_data = (X_test,y_test_cat), epochs=1000, batch_size=6068)



dataset=get_dataset('test.csv')
model=make_nn()
for i in range(50):
    train_nn(dataset, model)
    print("Save model")
    model.save('model/model.test.h5')

But when I run it, I see information like this:

Epoch 1/10
2/2 [==============================] - 31s 8s/step - loss: 3.6861 - accuracy: 0.0721 - val_loss: 3.6820 - val_accuracy: 0.0597

I haven’t seen that accuracy exceeded 0,14. As I understand, it is too small number.
Loss was also not less than 2ю
What am I doing wrong? Any advice, please?
Also, what is better: big batch_size or small?

#2

No sure where’s the issue but why are you adding 250 Dense layers to your model?

#4

I tested with different number of layers.
Practices show that as much layers I have, as smaller loss I have.

#6

You might be overfitting the training data using that many amount of layers, try smaller networks and increase progressively their capacity. Always check the gap between train and validation losses during training :slight_smile:

2 Likes
#8

You can check several things to improve model accuracy:

  • Make sure that all 40 classes are distributed proportionally between the train and validation sets. If some classes are absent or underrepresented in the train set, there is no way the model learns how to identify them. Use ‘stratify’ argument in the ‘train_test_split’ and pass target values there.
  • Normalize input data before passing it to the model (for example use MinMaxScaler from sklearn) or use Keras Normalization layer as the first layer in the model.
  • Decrease the batch size. Normally it should be somewhere between 32 and 512 depending on the total number of samples in your dataset.
  • In classification models usually only the last dense layer has softmax activation. All previous layers could have relu or something else. Softmax converts the vector into probabilities, which is done after all other transformations.
  • Decrease the number of dense layers. Number of units in each layers does not have to be equal to the number of features. Usually some multiple of 2 is used as the number of units and this number decreases with every next dense layer like: 256, 128, 64…

You train the model for 1000 epochs without using a callback. If after this many epochs the model does not show reasonable accuracy, repeating the training 50 times in a loop using the same model and the same input data would not help.
If you want to train several models and select the best run, you should initialize the model at every iteration from scratch (move your make_nn() function into the loop) and probably use different subsets of your data at every training run or at least different random states in splitting the data. For this you can move get_dataset() into the loop and and pass i as a random state.

2 Likes