Cannot register 2 metrics with the same name: /tensorflow/api/keras/optimizers

We are starting to see this error in some of the unit tests, within recent docker containers we are creating.

2021-07-14 02:13:34.052676: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/api/keras/optimizers

see this in the tip of develop branch (which is a fork of the upstream/master branch) and in the tip of our r2.6 branch (which is a fork of the upstream/r2.6 branch)

see this in about 10+ of the unit tests…these ones

  21134://tensorflow/python/compiler/xla:xla_test_gpu                            FAILED in 3 out of 3 in 3.6s                                                                                                                                                      
  21139://tensorflow/python/keras/benchmarks:eager_microbenchmarks_test_gpu      FAILED in 3 out of 3 in 3.6s                                                                                                                                                      
  21144://tensorflow/python/keras/benchmarks:model_components_benchmarks_test_gpu FAILED in 3 out of 3 in 3.5s                                                                                                                                                     
  21149://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:antirectifier_benchmark_test_gpu FAILED in 3 out of 3 in 3.4s                                                                                                                               
  21154://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:bidirectional_lstm_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s                                                                                                                          
  21159://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:cifar10_cnn_benchmark_test_gpu FAILED in 3 out of 3 in 3.6s                                                                                                                                 
  21164://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_conv_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s                                                                                                                                  
  21169://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_conv_custom_training_benchmark_test_gpu FAILED in 3 out of 3 in 3.7s                                                                                                                  
  21174://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_hierarchical_rnn_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s                                                                                                                      
  21179://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_irnn_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s                                                                                                                                  
  21184://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:reuters_mlp_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s                                                                                                                                 
  21189://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:text_classification_transformer_benchmark_test_gpu FAILED in 3 out of 3 in 3.7s                                                                                                             
  21194://tensorflow/python/ops/numpy_ops:np_interop_test_gpu                    FAILED in 3 out of 3 in 29.1s         

Anyone else running into this?

Any insight as to what might be causing this?

The docker containers we build, use this script to install all the pip packages

which in turn seems to install the keras-nightly package

is that still the correct thing to do? and is that co-related to the error we are getting?

thanks

On 2.6 branch is

Or not?

pinning the version numbers worked…thank you @Bhack

Hello Bhack,
I saw your reply to Deven_Desai. I’m having a similar problem.
I have installed tensorflow on Jetson Nano of Nvidia. The installation went well but when I load a tensorflow model stored on hdf5 file, I’m getting this error:

2021-10-10 16:39:06.985798: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/api/keras/optimizers
Traceback (most recent call last):
File “benchmarkObjectDetection.py”, line 25, in
nn_loaded_model = tf.keras.models.load_model(filepath_for_model, compile = False) # IMPORTANT! the compile = False input is crucial for loading the model as the custom loss function is unknown in the compilation phase; we can only use the model for prediction this way.
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/lazy_loader.py”, line 62, in getattr
module = self._load()
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/lazy_loader.py”, line 45, in _load
module = importlib.import_module(self.name)
File “/usr/lib/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.6/dist-packages/keras/init.py”, line 25, in
from keras import models
File “/usr/local/lib/python3.6/dist-packages/keras/models.py”, line 20, in
from keras import metrics as metrics_module
File “/usr/local/lib/python3.6/dist-packages/keras/metrics.py”, line 26, in
from keras import activations
File “/usr/local/lib/python3.6/dist-packages/keras/activations.py”, line 20, in
from keras.layers import advanced_activations
File “/usr/local/lib/python3.6/dist-packages/keras/layers/init.py”, line 23, in
from keras.engine.input_layer import Input
File “/usr/local/lib/python3.6/dist-packages/keras/engine/input_layer.py”, line 21, in
from keras.engine import base_layer
File “/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py”, line 43, in
from keras.mixed_precision import loss_scale_optimizer
File “/usr/local/lib/python3.6/dist-packages/keras/mixed_precision/loss_scale_optimizer.py”, line 18, in
from keras import optimizers
File “/usr/local/lib/python3.6/dist-packages/keras/optimizers.py”, line 26, in
from keras.optimizer_v2 import adadelta as adadelta_v2
File “/usr/local/lib/python3.6/dist-packages/keras/optimizer_v2/adadelta.py”, line 22, in
from keras.optimizer_v2 import optimizer_v2
File “/usr/local/lib/python3.6/dist-packages/keras/optimizer_v2/optimizer_v2.py”, line 37, in
“/tensorflow/api/keras/optimizers”, “keras optimizer usage”, “method”)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py”, line 361, in init
len(labels), name, description, *labels)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py”, line 135, in init
self._metric = self._metric_methods[self._label_length].create(*args)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Another metric with the same name already exists.

I would appreciate very much any help on this problem.
Thanks!

Can list with pip what TensorFlow and Keras versions you have installed?

Thank you very much Bhack!
This is what I get:

Name: tensorflow
Version: 2.6.0+nv21.9

Name: keras
Version: 2.7.0rc0

Thank you!
Avner

Can you try to install and test It in a venv:

Hi,
Can you please explain what you think is the problem and why the installation within a venv should help? It is just that I have installed tensorflow as per the instruction on Nvidia site for Jetsons…and I’m not sure how the installation in venv will affect the performance. BTW, when I run a simple tensorflow calculation, the code runs with no issue.
Thanks.

It just to check that the env Is clean.

E.g. see TF 2.6 with Keras 2.7rc0

Hi,
I followed the instruction on the link you sent me. After setting up the venv I’m trying to install tensroflow using pip but I’m getting the following message:

avner@avner-desktop:~$ source ./venv/bin/activate
(venv) avner@avner-desktop:~$ pip install tensorflow
ERROR: Could not find a version that satisfies the requirement tensorflow from versions: none)
ERROR: No matching distribution found for tensorflow

I also tried to specify versions of tensorflow but was not successful.
Any suggestions? I’m running on Jetson with ubunto 18.04.

Thanks.

Jest but the venv you need to follow the Jetson steps Set up the Virtual Environment:

https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/index.html#install_multiple_versions_tensorflow

Thanks Bhack.
The virtual environment doesn’t help. However, I reinstalled tensorflow with different version 2.5.0 (instead of 2.6.0) and with Nvidia TensorFlow container 21.07 and it works great!

Instead of using (doesn’t work for me, by default installing tf 2.6.0 version and no container):
“sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v46 tensorflow”

I used (runs excellent):

“sudo pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v46 tensorflow==2.5.0+nv21.07”

Thanks a lot for you help!

Cause you need to always check the NVIDIA TensorFlow Container Versions e.g. if you check for TF 2.6.0 the minimum required is 21.09:

https://docs.nvidia.com/deeplearning/frameworks/tensorflow-release-notes/rel_21-09.html#rel_21-09

Correct. However, the installation without the container is not working for me.
Thanks.

Without container you need to double check yourself the required python, CUDA, cudnn versions etc. for the specific TF version.