Performance with and without tfdf

General Discussion
, ,
#1

Hi, I made a “simple” python script with embedding layer, skip connection… for tabular data, with usual API function for a multi classification target and I have tried to connect tfdf.keras.GradientBoostedTreesModel to one of the last layer (before the softmax) to see if it can improves the result. It works, and when you compare the Neural Network alone and the NN + Gradient BoostedTrees, you get always a better logloss result. But when you don’t import tensorflow_decision_forests as tfdf the NN gets better results than in the first case, even comparing to NN+Gradient. I made several tests and the difference is quite significant and reproducible. Weird ? it seems that tf 2.5 get worse resuts comparing to 2.4.1 (script without tfdf) ?

#2

@Mathieu might be able to help here

#3

Thank you very much.
@Mathieu without importing tfdf but only tf (version 2.5.0) I get the same issue for the NN model, tf 2.4.1 gives much better result (based upon many tries) in the log_loss, with the same script. So it does not come from tfdf but tf version 2.5.0.
I guess we can not have a workaround to use tfdf with tf 2.4.1 ?

For you information, the script imports :
import tensorflow as tf
import tensorflow_addons as tfa
import tensorflow.keras.backend as K
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras import activations,callbacks
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import categorical_crossentropy
from tensorflow.keras import initializers
from keras.models import Model

#4

@Mathieu, After further investigations, it appears when the pip installation is running that tfdf is using a nightly version of KERAS ( keras-nightly 2.5.0.dev2021032900). I didn’t succeed in using keras==2.4.3 which gives also (as 2.4.1) good results for NN only…which is not the case with tf & Keras 2.5. Again, I’m afraid we can not use tfdf with Keras 2.4.3 ? :sob:

#5

Hi Laurent,

About your observations, this sounds like one of the following:

  1. A regression in the TF2.4.1->TF2.5.0 code.
  2. A change of implementation (e.g. changing a random seed) that will affect individual results, but should not impact global distributions.
  3. A change in API e.g. the default value / logic of a method has been changed.

Case 2 can be validated or eliminated by replicating the experiment multiple times: If the results are 50% of the time better, case 2 is likely true. If the results are always worse, case 2 is likely false.

For case 1 and 3, it might be interesting to create and share a minimal colab using public data that illustrates the problem. If the complexity of the model is small, it might be possible to detect case 3. Otherwise, this is the kind of colab that should be reported as an issue to TF. However, there is no guarantee of a quick resolution.

A few things have changed internally between TF2.4.1 and TF2.5.0, and we tried not to have TF-DF for both versions. It is possible to recompile TF-DF for TF2.4.1 with a few tweaks, but it is probably not interesting in your case. Sorry I cannot help you more.

Cheers,
M.

1 Like