How to change seq length in BERT preprocessor from TF Hub

General Discussion
,
#1

I am following this example to use BERT for sentiment classification.

text_input = tf.keras.layers.Input(shape=(), dtype=tf.string)
preprocessor = hub.KerasLayer(
    "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
encoder_inputs = preprocessor(text_input)
encoder = hub.KerasLayer(
    "https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4",
    trainable=True)
outputs = encoder(encoder_inputs)
pooled_output = outputs["pooled_output"]      # [batch_size, 768].
sequence_output = outputs["sequence_output"]  # [batch_size, seq_length, 768].
embedding_model = tf.keras.Model(text_input, pooled_output)sentences = tf.constant(["(your text here)"])print(embedding_model(sentences))

The sequence length by default seems to 128 from looking at the output shape from encoder_inputs. However, I’m not sure how to change this? Ideally I’d like to use to a larger sequence length.

There’s an example of modifying sequence length from the preprocessor page, but I’m not sure how to incorporate this into the functional model definition I have above? I would greatly appreciate any help with this.

#3

Hi Kay,

If you look into this tutorial: Solve GLUE tasks using BERT on TPU  |  Text  |  TensorFlow

You’ll get this exactly information to help you. To be more exact, in the make_bert_preprocess_model function