Performing data wrangling on tf.data.Dataset

General Discussion
, ,
#1

Are the tf.data.experimental.CsvDataset available in (https://methods)) enough to perform data wrangling and transformation usually done using libraries like scipy, numpy and pandas? . for example, i have been trying to perform an equivalent of the below line of pandas code using dataset methods but couldn’t find the suitable one for it. Looking forward to your reply.

sum_of_features = {}
for col in features:
    sum_of_features[col] = sum_ofs[col].groupby(["year", "month", "day", "second"]).sum(col)
#3

Do you mean this page?
https://www.tensorflow.org/api_docs/python/tf/data/experimental/CsvDataset

It refers to a “richer” API here:
https://www.tensorflow.org/api_docs/python/tf/data/experimental/make_csv_dataset

And has a tutorial here:

#5

yes, i meant tf.data.experimental.CsvDataset  |  TensorFlow Core v2.8.0. can the APIs be used for achieving the data wrangling code i posted above? i haven’t found the right API for doing such

#7

Apparently TF has its own “Pandas” called “Tensorflow Transform”. I haven’t used it.

#9

You can do feature engineering (some folks call it wrangling, others say that wrangling is different) in both TensorFlow Transform and Keras Preprocessing Layers. Transform is a more scalable solution and is well-integrated into TFX, so if your dataset is large or you need to do a lot of processing it’s a better choice. KPL is of course more tightly integrated into Keras.