Compatibility of the CRF layers in TF-addons and TF2.6

Special Interest Groups Addons
#1

Hi everyone,

I am trying to use the CRFModelWrapper method following the tutorial as addons/layers_crf.ipynb at add_crf_tutorial · howl-anderson/addons · GitHub to implement a Bi-LSTM -CRF neural-network for a multi-classes time-series NER problem, and it works in TF 2.7 in my PC. However, when I used the same code and same data runing in TF 2.6 environment, it pops out some errors(as shown below) regarding to the tfa-CRF layer. So, could someone please let me know if the addons CRF layer is only compatible for the TF 2.7 version or this is because I made some mistakes? Thank you very much.

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\training.py", line 1134, in fit
    data_handler = data_adapter.get_data_handler(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 1383, in get_data_handler
    return DataHandler(*args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 1138, in __init__
    self._adapter = adapter_cls(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 917, in __init__
    super(KerasSequenceAdapter, self).__init__(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 801, in __init__
    model.distribute_strategy.run(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1286, in run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 2849, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 3632, in _call_for_each_replica
    return fn(*args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 597, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 802, in <lambda>
    lambda x: model(x, training=False), args=(concrete_x,))
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\base_layer.py", line 1037, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "<input>", line 47, in call
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\base_layer.py", line 1037, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow_addons\layers\crf.py", line 131, in call
    raise NotImplementedError(
NotImplementedError: Currently, CRF layer do not support left padding
#2

/cc @XiaoquanKong can you check this?

#3

I found this might be the incorrect data type I assigned to the input layer of the base model(dtype=int32 → dtype = float32, as shown below). After the modification, it works in both TF2.6 and TF2.7. (Still don’t know the reason why the results in the TF2.6 and TF2.7 are different if the input data type are wrongly assigned)

# Build the model
def build_embedding_bilstm_crf_model(
    vocab_size: int, embed_dims: int, lstm_unit: int
) -> tf.keras.Model:
    x = tf.keras.layers.Input(shape=(None,), dtype=tf.int32, name="x")
    y = tf.keras.layers.Embedding(vocab_size, embed_dims, mask_zero=True)(x)
    y = tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(lstm_unit, return_sequences=True)
    )(y)

    return tf.keras.Model(
        inputs=x, outputs=y
    )


base_model = build_embedding_bilstm_crf_model(VOCAB_SIZE, 32, 64)
#5

@Bhack Thank you for your reminder, I will handle this.

#6

Hi @dada_Lai, can you provide me with some code and data to reproduce this behavior on my computer? Or can you repeat this experiment (use both TF2.6 and TF2.7) a few times on your own computer using the old code? I guess there may be randomness in this behavior.

#7

Hi @XiaoquanKong,

I run the old code (set the dtype = tf.int32) several times on my PC and all got the same error messages. I tried to run it on Google Colab with and without the GPU and they all get the same errors in TF2.6.0.(But I couldn’t run the GPU version properly on Google Colab even I set the dtype = tf.float32 because it cannot find the cudnn module) Please find My code and the testing data through the links and let me know if they cannot work properly.

About the other related question, I run the same code with the same testing data provided above on Google Colab, and it only shows a few minor warnings when saving:

WARNING:absl:Found untraced functions such as dense_1_layer_call_and_return_conditional_losses, dense_1_layer_call_fn, dense_1_layer_call_fn, dense_1_layer_call_and_return_conditional_losses, dense_1_layer_call_and_return_conditional_losses while saving (showing 5 of 15). These functions will not be directly callable after loading.

I am wondering if you ever test the code with tensorflow GPU version (i.e. tensorflow-gpu=2.6.0) because this is the main difference between my tests running on Google Colab (tensorflow=2.6.0) and on my PC (tensorflow-gpu=2.6.0). So far I couldn’t run the code with tensorflow GPU version on Google Colab and will let you know once I fix it.

Thank you very much.