AoC 2022 in TensorFlow - Day 6. Efficient solution with dataset interleave

Show and Tell
#1

Hey folks,

while I’m having fun trying to solve the various puzzles of the Advent of Code 2022 in pure TensorFlow, I’m going on with the write-ups about my solutions.

This time, the article is about the day 6 problem and how I solved it, efficiently and in a few lines, thanks to the tf.data.Dataset.interleave method, that’s the super-hero of the data transformation IMHO.

Any feedback is welcome!

1 Like
#3

You can see the complete solutions in folder 6 in the dedicated GitHub repository (in the 2022 folder): GitHub - galeone/tf-aoc.

Solving problem 6 allowed us to use a very powerful feature of tf.data.Dataset: interleave. In a few lines, this method allows us to define a complete, highly parallel, and efficient data transformation pipeline, that allows us to transform and group data gathered from different datasets. The expressive power of this method, moreover, allowed us to solve the problem in a very elegant way IMHO.

If you missed the article about the previous days’ solutions, here’s a handy list

:+1: @pgaleone

3 Likes
#4

Hey I have been following your series for a while now, and I really appreciate it, as it brings out unheard and less used abilities of tensorflow.

1 Like
#6

Thank you for the feedback! FYI I’m going on with the articles and the puzzles :smiley:

1 Like
#8

I see that you use a scan() method for your dataset processing in day 1. Care to throw more light on how it functions? I looked it up on the documentation but it was not explicitly clear and lacked enough examples to make sense of it.

#9

The scan method is neat. It allows you to iterate over the elements of a dataset and at the same time carry over a state.

Just think about a simple dataset that produces the values 1, 2, 3, 4, 5.

With the map function, you can loop over every single element of this dataset and apply a transformation.

dataset.map(lambda x: x*2) # 2, 4, 6, 8, 10

With scan instead, you can carry over some information from the previous iterations.

So, for example, if you want to sum up the previous element, you can use the scan method.

initial_state = tf.constant(0)

def scan_fun(old_state, input_element):
    new_state = input_element
    output_element = input_element + old_state
    return new_state, output_element

dataset.scan(initial_state, scan_func) # 1 +0 , 2 + 1, 3 + 2, ...

So you carry over the next iteration, the old_state, and every time you iterate over a new input_element you can generate a new new_state (that becomes the old_state input for the next iteration), and produce (as output) the output_element.

I used scan in several solutions (which is super helpful), I suggest you read all the articles and search inside the code how I used it. I hope it’s helpful for you to understand a little bit more about how to use this great feature.

1 Like
#10

This was very helpful. Thank you

1 Like