TensorFrames Equivalent in TF 2.0

General Discussion
#1

Hi All,

We are trying to use tensorflow over spark for numerical computation and further visualize in tensorboard for calculation transperancy.

For TF1.0 - found this cool library - GitHub - databricks/tensorframes: [DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark
which actually allows us to define a spark dataframe column as a tensor and apply operations on top of that.

However, it’s deprecated now. The Github page suggests to use Pandas UDF which does not work with @tf.function decorator and hence we cannot visualize stuff with tensorboard.

In summary, we are interested in the below:

  1. Tensorflow with apache spark.
  2. Manipulate Spark dataframe as Tensors.
  3. Visualize the operations in tensorboard.

Appreciate any leads on this.

Thanks.

#3

Tensorflow >=2.0 with apache spark references

  1. Distributed TensorFlow on Apache Spark 3.0

  2. TensorFlowOnSpark