Convert to tfrecord and write directly to an AWS S3 Bucket

Lu_Bin_Liu · February 11, 2022, 7:59pm

I have some images that I would like to convert to TFRecord format and load them into an S3 Bucket on AWS. However, existing solutions I found requires saving the TFRecords locally first, then uploading the saved local files to S3. I’m wondering if there’s a way to convert to TFRecords and save the output as .tfrecord directly to an S3 bucket.

Bhack · February 11, 2022, 9:12pm

I think that if you check point 6 probably it could work also with an s3 url:

Lu_Bin_Liu · February 11, 2022, 10:55pm

Unfortunately I got an File system scheme 's3' not implemented error when I tried that. The filename I used with tf.io.TFRecordWriter was s3://BUCKET-NAME/record-name.tfrecord.

Bhack · February 11, 2022, 11:29pm

Have you tried to install TF.io?

See:

github.com/tensorflow/tensorflow

File system scheme 's3' not implemented

opened 04:33AM - 09 Jun 20 UTC

closed 07:05PM - 05 Aug 21 UTC

kumaradi08

stat:awaiting response type:bug stalled comp:apis TF 2.2

**System information** - Have I written custom code (as opposed to using a stoc…k example script provided in TensorFlow): No - OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 - Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No - TensorFlow installed from (source or binary): Binary - TensorFlow version (use command below): v2.2.0-rc4-8-g2b96f3662b 2.2.0 - Python version: 3.6.5 - Bazel version (if compiling from source): NA - GCC/Compiler version (if compiling from source): NA - CUDA/cuDNN version: NA - GPU model and memory: NA **Describe the current behavior** tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented **Describe the expected behavior** Should be able to connect the s3 filesystem. **Standalone code to reproduce the issue** from tensorflow.python.lib.io import file_io print file_io.stat('s3://bucketname/path/') **Other info / logs** Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Lu_Bin_Liu · February 12, 2022, 12:35am

I attempted the above with with tf.io 0.17 installed and got the same error (also tried with s3e and got the same).

Bhack · February 12, 2022, 11:53am

@mihaimaruseac as you have replied on the mentioned ticket do you know something more?

Lu_Bin_Liu · February 12, 2022, 4:30pm

For additional information, I’m using tensorflow 2.4.4, with OS Windows 10. For some reason tf.io.read_file works, even without installing tf.io, but other methods like tf.io.gfile.GFile' returns the same s3 not implemented` error