Understanding Implementations of layers related to RNN (e.g. LSTM, GRU etc)

General Discussion
#1

I am not that good with RNNs and want to understand it in depth. Also, not just how to create RNN based models and which models are good, I also want to understand how tensorflow has implemented LSTM and GRU like layers.

Any materials (Books, Blogs, Videos etc.) would be appreciated for learning RNNs.
Please, also suggest the part of the code which can be starting point to understand implementation of RNN based layers in tf.

2 Likes
#2

Personally, in TF/Keras I suggest you to make a first pass on

4 Likes
#3
  1. Dive into Deep Learning (http://d2l.ai/)
    When you’re starting out, just the mathematics could be intimidating and just the code could be too shallow to learn. In my case, this book was the perfect mixture of code and maths and it has all of its code available as colab notebook.
  2. Andrew Ng Deep Learning Specialization
    It is probably the best beginners course in Deep Learning and if you can’t pay for coursera, you could watch the videos on youtube.

‘Learn ML’ section of tensorflow has a very good set of educational resources which can also act as a roadmap for you

2 Likes
#4

@sangam Thanks for resources. Dive to deeplearning is something that I wanted. I already have gone through andrew ng courses and it is not that I am complete beginner. I just take time working with RNNs and want to understand by looking at some real life implementations for learning.

2 Likes
#5

The “Understanding LSTM Networks” by Chris Olah is an excellent resource on LSTM internals. For applications and code samples I found tutorials at Machine Learning Mastery very useful. I suspect some of the code might not follow the TF 2 APIs, something to keep in mind.

2 Likes
#6

@manik_galkissa , Thanks so much for the resource. Can you please provide a link as well?

1 Like
#7

Sorry , seems like I’m not allowed to post links just yet. But the top results with the above keywords should point you in the right direction.

2 Likes
#8

https://colah.github.io/posts/2015-08-Understanding-LSTMs/

2 Likes
#9

I suggest also to take a look at this cheatsheet

2 Likes
#10

For high-level theory and more practical implementation of RNN & LSTM with TensorFlow, two books:

or Course 3 of TensorFlow Developer Program - DeepLearning.AI

3 Likes
#11

I forgot to add this - Lecture 2 Intro to Deep Learning MIT - Deep Sequence Modeling. - As you wanted to understand RNN &LSTMs in-depth, and implement them with TF, this would most likely be a quick (but still theoretical & practical ) option

3 Likes