Auto comment generation project - model training loop crash

General Discussion
,
#1

Hello,

I posted this issue on the project and TF github, and was told I’d have more success here.

I am trying to train a model in an automatic comment generation project and the training loop throws an exception.
This error has occured with 3 combinations of python/tensorflow versions, which I note below as (1), (2) and (3), as well as on both a conda env in Powershell and in the WSL Ubuntu command line.
It also happens on 2 different machines, both with the official (GitHub - tech-srl/code2seq: Code for the model presented in the paper: "code2seq: Generating Sequences from Structured Representations of Code") and unofficial (GitHub - Kolkir/code2seq: Code for the model presented in the paper: "code2seq: Generating Sequences from Structured Representations of Code") versions of the project.

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Not sure if the project devs did, I assume not.
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 x64, Ubuntu 20.04
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 1.12 (1), 2.8.0 (2), 2.8.0 (3)
  • Python version: 3.6.5 (1), 3.9.7 (2), 3.8.10 (3)
  • Bazel version (if compiling from source): N/A
  • GCC/Compiler version (if compiling from source): N/A
  • CUDA/cuDNN version: N/A
  • GPU model and memory: RTX 2080 SUPER 8GB, but N/A

Describe the current behavior

Running the training loop with bash train.sh both in Powershell (conda) and Ubuntu WSL command line :

Traceback (most recent call last):
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 407, in _flatten_module
    leaves = nest.flatten_with_tuple_paths(
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\util\nest.py", line 1698, in flatten_with_tuple_paths
    flatten(structure, expand_composites=expand_composites)))
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\util\nest.py", line 451, in flatten
    return _pywrap_utils.Flatten(structure, expand_composites)
TypeError: '<' not supported between instances of 'WhileBodyFuncGraph' and 'FuncGraph'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\gbaulard\Documents\GitHub\code2seq\code2seq.py", line 29, in <module>
    model.train()
  File "C:\Users\gbaulard\Documents\GitHub\code2seq\modelrunner.py", line 129, in train
    gradients = tape.gradient(loss, self.model.trainable_variables)
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 171, in trainable_variables
    return tuple(
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 449, in _flatten_module
    for subvalue in subvalues:
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 449, in _flatten_module
    for subvalue in subvalues:
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 449, in _flatten_module
    for subvalue in subvalues:
  File "C:\Users\gbaulard\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 410, in _flatten_module
    six.raise_from(
  File "<string>", line 3, in raise_from
ValueError: Error processing property '_dropout_mask_cache' of <ContextValueCache at 0x17309fd0df0>

Code snippet being triggered in module.py of tensorflow :

try:
      leaves = nest.flatten_with_tuple_paths(
          prop, expand_composites=expand_composites)
    except Exception as cause:  # pylint: disable=broad-except
      six.raise_from(
          ValueError(
              "Error processing property {!r} of {!r}".format(key, prop)),
          cause)

Describe the expected behavior

The training loop should terminate without errors.

Standalone code to reproduce the issue

Here are the project githubs:

Here’s a link with the minimal amount of files to launch the training : https://we.tl/t-yfoK0m3Gkx
Run this in a terminal at the root with python 3.6+, tensorflow 2.1+ installed.
python -u code2seq.py --data data/java-small/java-small --test data/java-small/java-small.val.c2s --save_path models/java-small-model/model