By Considering these Chart of Complete Neural Networks Types:
And also here is a Commonly used Neural Network types in Keras I found, which I take note also for its most Practice Cases (You can correct me if I am wrong):
1 DNN (Deep Neural Network) → Most Practice Cases: Time series + predicting plain response y variable (flexible on numeric integer categoric
2 CNN (Convolution Neural Network) → Most Practice Cases: Time Series + Image Classification/Computer Vision
3 LSTM (Long Short-Term Memory) → Most Practice Cases: Time Series + NLP, → Recommended use for a long set of training data (More complex structure than GRU → LSTM has three gates (namely input, output and forget gates)
4 RNN (Recurrent Neural Network) → Most Practice Cases: Time Series + NLP, RNN → Recommended use for Sequence Data (faster training, computationally less expensive
5 GRU (Gated Recurrent Unit → Most Practice Cases: Time Series + NLP GRU → Recommended use for a shorter set of training data (Less complex structure than LSTM GRU has two gates (reset and update gates)
6 Auto Encoders (AE) → Most Practice Cases: Image Classification/Computer Vision (Noising & Denoising), Auto Encoders basically are a set of CNN, Pooling2D, and CNN-Transpose
Finally my Question:
Are there any types of Neural Network in above chart which structure of Network are currently not possible to build by Keras component?
if there’s any Network which aren’t possible, could you point me what types, and why?
Are there any more Commonly used Neural Network aside from what I notes above? Appreciate it if theres any improvisation added to it
Appreciate any effort put into this question, Thank You!
I’d associate GRU, RNN and LSTM with sequences instead of time series. It’s broader and easier to remember. That’s why NLP is in the same group (sequence of characters/words)
As far as I know, Keras can build all the Neural Networks presented and if the layer you need is not available directly, you can customize or create your own with the same API.
The only detail on your image that I could spot is the Support Vector Machine (SVM). I don’t know if that can be considered a Neural Network and I don’t know if that can be built using Keras.
Just to play a little bit with the concept we accepted a tutorial
if the layer you need is not available directly, you can customize or create your own with the same API.
I think this is the main point.
More in general, to extend the evaluation over that specific cheat-sheet, you could need to write a custom op inside a brand new layer and the compositional performance of you python code for the TF ops is too slow to practically train a model.
In that case, as Keras is a standalone python-only project, you need to add your own custom ops/kernel in TF, in TF Addons or to maintain it in your own repository.
This could create a management overhead in the case you will need more general model portability (TFlite, TPU, other GPUs) etc.
We can always collaborate with the XLA/MLIR team to better improve the compiler stack lowering driven by new model requests (IMHO this process could be driven by TF model garden and TF Addons with a better interplay with the compiler stack team).
Thanks for the feedback! so basically to create the custom op (operators) it is recommended to do it in C++? and then test the op in Python, according to the OP Guide?
The problem is that sometime, when you combine different TF native ops in the implementation, the performance could be too low (memory/flops) to reach an efficient training/inference as some part of your specific codepath could be not explicitly optimized by the TF compiler stack (e.g. missing ops fusion, etc…).
Generally uou can try to benchmark your specific implementation using also tf.function but if you canno’t directly interact/contribute to the compiler stack to improve that specific codepath you will probably need to write C++/CUDA custom op/kernel to support your python custom layer.