Writing tensorboard logs to s3 bucket using keras.callbacks

General Discussion
,
#1

I have a TensorBoard server started with a s3 like minio bucket in kubeflow, and would like to write tensorboard logs to this s3 bucket.

While calling tf.keras.callbacks.TensorBoard(log_dir="s3://minio-server:port/my-public-bucket"), i got the following error
tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented

how can i write the keras training log for tensorboard to s3 like minio bucket with keras.callbacks?
I really appreciate if anyone can share some code snippets.

#3

I don’t have experience using minio so I’m not sure how much help I will be. But some thoughts:

  • Have you installed tensorflow-io? It’s the companion package that handles much of the io and protocols.
  • Have you tried using another protocol like “s3e” or “s3a”? I don’t recall what protocols are supported with tensorflow-io, but you might experiment with ‘s3e://bucket_name/path/to/file’.
  • What version of tensorflow? Can you experiment with other versions?

Sorry I don’t have anything more definitive. Good luck. And let us know what worked.

cheers,
Dennis

#5

@dennisobrien Thanks for mention tensorflow-io, I used a cpu TF 2.9.1 image and tensorflow-io==0.26.0, according to version compatibility it should work.


import tensorflow_io
...
log_dir = "s3e://minio_ip:port/public_bucket/"
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir)
...

and still got

tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3e' not implemented (file: 's3e://xxx.xxx.xxx.107:xxx97/kf-tensor-board/train') [Op:CreateSummaryFileWriter]

Just want to double check, do i need to wrap the s3/minio path with some tensorflow_io writter?

Thanks for mention s3e, which seems to be the case for tensorflow-io==0.17 (File system scheme 's3' not implemented · Issue #40302 · tensorflow/tensorflow · GitHub)