How to get reproducibility in Tensorflow C++ using GPU

General Discussion
, , ,
#1

How do I get reproduciblility when using GPU with TensorFlow C ++?
I tried the following methods and was able to get reproducibility by running only the CPU, but not by running the GPU.

  • What I tried

    • Fixed random seeding in TensorFlow library
      • Replace the return value of the non-deterministic function “device_random” with a fixed value
    • Specify Cuda to use reproducible algorithms
      • Set the environment variable TF_CUDNN_DETERMINISTIC=1

  • Development environment

    • Windows10
    • TensorFlow 1.15 C++ (Build from code)
    • CUDA 10.2
    • cudNN 7.5

Can you help me solve this problem?