How can I run StatisticsGen with custom generator (TFDV)?

Hi, all.
I have a question about TFDV (tensorflow data validation) and tfx.

Let me explain my case first. I want to pass custom generators and calculate custom statistics without re-writing StatisticsGen (tfx.v1.components.StatisticsGen  |  TFX  |  TensorFlow).
So I checked StatisticsGen can get the stats_option parameter which has the type Optional[tfdv.StatsOptions] and tfdv.StatsOptions can get custom generators via the generators parameter (tfdv.StatsOptions  |  TFX  |  TensorFlow).

But as we can see below links, we cannot pass the generators parameter to StatisticsGen’s executor, since StatisticsGen cannot serialize custom generators in tfdv.StatsOptions.

In this situation, how can I pass custom generators to StatisticsGen and run with them?

Thank you :slight_smile:

Hi - Thanks for posting your question. Unfortunately, we do not currently have a way to specify custom stats generators in the StatisticsGen component.

1 Like