tensorflow.python.framework.errors_impl.FailedPreconditionError: Corrupted response from the server null

Hello,

I’m working on training a neural network with data from elasticsearch cluster. I’ve followed this tutorial : Streaming structured data from Elasticsearch using Tensorflow-IO  |  TensorFlow I/O
but i’m getting an error.

python3.9 main.py
Type Age Breed1 Gender Color1 Color2 MaturitySize FurLength Vaccinated Sterilized Health Fee Description PhotoAmt AdoptionSpeed
0 Cat 3 Tabby Male Black White Small Short No No Healthy 100 Nibble is a 3+ month old ball of cuteness. He … 1 2
1 Cat 1 Domestic Medium Hair Male Black Brown Medium Medium Not Sure Not Sure Healthy 0 I just found it alone yesterday near my apartm… 2 0
2 Dog 1 Mixed Breed Male Brown White Medium Medium Yes No Healthy 0 Their pregnant mother was dumped by her irresp… 7 3
3 Dog 4 Mixed Breed Female Black Brown Medium Short Yes No Healthy 150 Good guard dog, very alert, active, obedience … 8 2
4 Dog 1 Mixed Breed Male Black No Color Medium Short No No Healthy 0 This handsome yet cute boy is up for adoption… 3 2
11537 14
Number of training samples: 8075
Number of testing sample: 3462
/home/cdt/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:208: ElasticsearchWarning: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See Set up minimal security for Elasticsearch | Elasticsearch Guide [7.13] | Elastic to enable security.
warnings.warn(message, category=ElasticsearchWarning)
deleting the ‘train’ index.
Response from server: {‘acknowledged’: True}
creating the ‘train’ index.
Response from server: {‘acknowledged’: True, ‘shards_acknowledged’: True, ‘index’: ‘train’}
bulk index the data
/home/cdt/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:208: ElasticsearchWarning: [types removal] Specifying types in bulk requests is deprecated.
warnings.warn(message, category=ElasticsearchWarning)
Errors: False, Num of records indexed: 8075
deleting the ‘test’ index.
Response from server: {‘acknowledged’: True}
creating the ‘test’ index.
Response from server: {‘acknowledged’: True, ‘shards_acknowledged’: True, ‘index’: ‘test’}
bulk index the data
Errors: False, Num of records indexed: 3462
Connection successful: http://172.16.238.221:9200/_cluster/health
Connection successful: http://172.16.238.221:9200/_cluster/health
{‘Type’: b’Cat’, ‘Age’: 2, ‘Breed1’: b’Domestic Short Hair’, ‘Gender’: b’Male’, ‘Color1’: b’Black’, ‘Color2’: b’White’, ‘MaturitySize’: b’Small’, ‘FurLength’: b’Short’, ‘Vaccinated’: b’No’, ‘Sterilized’: b’No’, ‘Health’: b’Healthy’, ‘Fee’: 0, ‘PhotoAmt’: 1}
Traceback (most recent call last):
File “/home/cdt/stream/elk/main.py”, line 149, in
File “/home/cdt/stream/elk/main.py”, line 108, in get_normalization_layer
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_preprocessing_layer.py”, line 242, in adapt
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py”, line 889, in call
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py”, line 917, in _call
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/eager/function.py”, line 3023, in call
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/eager/function.py”, line 1960, in _call_flat
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/eager/function.py”, line 591, in call
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/eager/execute.py”, line 59, in quick_execute
tensorflow.python.framework.errors_impl.FailedPreconditionError: Corrupted response from the server null
[[{{node IO>ElasticsearchReadableNext}}]]
[[IteratorGetNext]] [Op:__inference_adapt_step_829]

Function call stack:
adapt_step

Can you help me

import os
os.environ[‘TF_CPP_MIN_LOG_LEVEL’] = ‘3’
import tensorflow_io as tfio
import tensorflow as tf
from tensorflow.keras import layers

dataset = tfio.experimental.elasticsearch.ElasticsearchIODataset(
** nodes=[“XXX.XXX.XXX.XXX:9200”],**
** index=“test”**
** )**

dataset = dataset.enumerate(start=0)
for element in dataset.as_numpy_iterator():
** print(element)**

there are 3462 records, and this code works until 3379

(3379, {‘Type’: b’Cat’, ‘Age’: 4, ‘Breed1’: b’Domestic Medium Hair’, ‘Gender’: b’Female’, ‘Color1’: b’Black’, ‘Color2’: b’No Color’, ‘MaturitySize’: b’Medium’, ‘FurLength’: b’Medium’, ‘Vaccinated’: b’No’, ‘Sterilized’: b’No’, ‘Health’: b’Healthy’, ‘Fee’: 0, ‘PhotoAmt’: 2, ‘target’: 1})
Traceback (most recent call last):
File “/home/cdt/temp/elk.py”, line 13, in
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/data/ops/dataset_ops.py”, line 4194, in next
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py”, line 761, in next
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py”, line 744, in _next_internal
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/ops/gen_dataset_ops.py”, line 2728, in iterator_get_next
File “/home/cdt/.local/lib/python3.9/site-packages/tensorflow/python/framework/ops.py”, line 6897, in raise_from_not_ok_status
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.FailedPreconditionError: Corrupted response from the server null
[[{{node IO>ElasticsearchReadableNext}}]] [Op:IteratorGetNext]