Keras-cv, keras-nlp, keras applications, models garden

Bhack · January 25, 2022, 2:29pm

Can someone give a general overview about the models request we are collecting on Keras-cv?
What kind of relationship we will have between this, models garden, tf.keras.applications namespace and “marginally” TFHUB?

I would be really nice to disambiguate a little bit this topic to avoid duplication, fragmentation and confusion about the contribution path in the TF ecosystem and to optimize the external contributors resources.

We have already have some historically pinned tickets about models garden community requests and help wanted requests at:

github.com/tensorflow/models

📄 Community requests: New paper implementations

opened 07:14AM - 06 Jun 20 UTC

jaeyounkim

type:support models:official

This issue contains **all open requests for paper implementations requested by t…he community**. We cannot guarantee that we can fulfill community requests for specific paper implementations. If you'd like to contribute, **please add a comment to the relevant GitHub issue to express your interest in providing your paper implementation**. Awesome external contributors will be nominated for [Google Open Source Peer Bonus](https://opensource.google/docs/growing/peer-bonus/). Please also see our [contribution guidelines](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution) and [paper selection criteria](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution#model-selection). ## Computer Vision | Paper | Conference | GitHub issue | Note | --------|------------|--------------|------| | ResNeXt: [Aggregated Residual Transformations for Deep Neural Networks](https://arxiv.org/abs/1611.05431) | CVPR 2017 | #6752 | | | DenseNet: [Densely Connected Convolutional Networks](https://arxiv.org/abs/1608.06993) | CVPR 2017 | #8278 | | | [Density estimation using Real NVP](https://arxiv.org/abs/1605.08803) | ICLR 2017 | #7848 | Need to migrate [TF 1 code](https://github.com/tensorflow/models/tree/master/research/real_nvp) to TF 2 | | [Spatiotemporal Contrastive Video Representation Learning](https://arxiv.org/abs/2008.03800) | CVPR 2021 | #9993 | In progress (Internally) |

github.com/tensorflow/models

[Help wanted] Research paper implementations (Project Tracker)

opened 03:32AM - 21 Jun 20 UTC

jaeyounkim

help wanted:paper implementation

# Help wanted: Research paper code and models This issue contains a list of t…he research papers we want to implement in TensorFlow 2 with help from the community. If you'd like to contribute, please **add a comment to the relevant GitHub issue** or **create a new issue** to express your interest in providing your paper implementation. Awesome external contributors will be nominated for [Google Open Source Peer Bonus](https://opensource.google/docs/growing/peer-bonus/). Please also see our [contribution guidelines](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution) and [paper selection criteria](https://github.com/tensorflow/models/wiki/Research-paper-code-contribution#model-selection). ## Computer Vision | Paper | GitHub issue | Status | |-------|--------------|--------| | FCOS: Fully Convolutional One-Stage Object Detection | #10275 | In progress | | DarkPose: [Distribution Aware Coordinate Representation for Human Pose Estimation](https://arxiv.org/abs/1910.06278) | #8713 | In progress | | MoCo: [Momentum Contrast for Unsupervised Visual Representation Learning](https://arxiv.org/abs/1911.05722) | #8708 | Need contribution | | YOLOv4 [Optimal Speed and Accuracy of Object Detection](https://arxiv.org/abs/2004.10934) | N/A | [In progress](https://github.com/tensorflow/models/tree/master/official/vision/beta/projects/yolo) | ## Natural Language Processing | Paper | GitHub issue | Status | |-------|--------------|--------| | RoBERTa: [A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) | #8704 | Need contribution | | RoFormer: Enhanced Transformer with Rotary Position Embedding | N/A | In progress | | Longformer: The Long-Document Transformer | N/A | In progress | | BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension | N/A | In progress | ### Benchmark datasets | Dataset | GitHub issue(s) | Status | |----------|------------------|--------| ## Speech Recognition | Paper | Conference | GitHub issue | Status | |-------|------------|--------------|--------| | Deep Speech 2: [End-to-End Speech Recognition in English and Mandarin](https://arxiv.org/abs/1512.02595) | ICML 2016 | #8702 | In progress |

Useful for the context

See also other community members comments like @sebastian-sz :

Or our thread at:

github.com/keras-team/keras

Updating the ResNet-* weights

opened 01:41PM - 12 Dec 21 UTC

closed 06:02AM - 11 Apr 22 UTC

sayakpaul

type:feature Contributions welcome

If you open a GitHub issue, here is our policy: It must be a bug, a feature r…equest, or a significant problem with the documentation (for small docs fixes please send a PR instead). The form below must be filled out. **Here's why we have that policy:**. Keras developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow. **System information**. TensorFlow version (you are using): 2.7.0 Are you willing to contribute it (Yes/No) : Currently no **Describe the feature and the current behavior/state**. ResNets are arguably one of the most influential architectures in deep learning. Today, they are used in different capacities. For example, sometimes they act as strong baselines, sometimes they are used as backbones. Since their inception, their performance on ImageNet-1k, in particular, has improved quite a lot. I think it's time the ResNets under `tf.keras.applications` were updated to facilitate these changes. **Will this change the current api? How?** ResNet-RS (https://arxiv.org/abs/2103.07579) introduces slight architectural changes to the vanilla ResNet architecture (https://arxiv.org/abs/1512.03385). So, yes, there will be changes to the current implementation of ResNets (among other things) we have under `tf.keras.applications`. We could call it `tf.keras.applications.ResNet50RS`, for example. Following summarizes the performance benefits that ResNet-RS introduces to the final ImageNet-1k performance (measured on the `val` set): ![image](https://user-images.githubusercontent.com/22957388/145714574-b057a95e-4d6c-48ba-b7ea-67e953e83f29.png) <sub><a href=https://github.com/tensorflow/tpu/tree/master/models/official/resnet/resnet_rs#imagenet-checkpoints>Source</a></sub> **Who will benefit from this feature?** Keras users that use ResNets from `tf.keras.applications` for building downstream applications. **[Contributing](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md)** - Do you want to contribute a PR? (yes/no): Currently no - If yes, please read [this page](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md) for instructions - Briefly describe your candidate solution(if contributing):

/cc @thea @yarri-oss @lgusm @Luke_Wood @Jaehong_Kim @Scott_Zhu

Scott_Zhu · January 25, 2022, 6:33pm

Thanks Bhack for the question.

We will have some readme/contribution guild available on the keras-cv github project to provide more details about what’s the difference between keras-cv/model garden/keras.applications/tf-hub.

Bhack · January 25, 2022, 7:31pm

P.s. I suppose that we could extend this topic as is also on keras-nlp or more generally to any other multi-modal/omni network that is hard to constrain in a specific cv/nlp/etc. domain.

Bhack · January 27, 2022, 12:35pm

Standardized / preferred way to implement blocks and models.

opened 10:41AM - 27 Jan 22 UTC

closed 06:17AM - 28 Jan 22 UTC

sebastian-sz

Given the models requirements are being gathered in the [discussion](https://git…hub.com/keras-team/keras-cv/discussions/52), is there a preferred way to implement them? There are multiple ways, to implement blocks and models: 1. `keras.applications` way -> Models and blocks are functional [Example](https://github.com/keras-team/keras/blob/2c48a3b38b6b6139be2da501982fd2f61d7d48fe/keras/applications/resnet.py#L440-L459). 2. Model Garden way -> Model is [functional](https://github.com/tensorflow/models/blob/c8a402fc7fc0cc391a9e3ac56fb7b3ea6f9d202e/official/vision/beta/modeling/backbones/resnet.py#L99), but blocks are [layer subclasses](https://github.com/tensorflow/models/blob/c8a402fc7fc0cc391a9e3ac56fb7b3ea6f9d202e/official/vision/beta/modeling/layers/nn_blocks.py#L56). (even though the model is a direct subclass, it does not override call method). 3. Blocks are layer subclasses, and models are model subclasses of `keras.layers.Layer` and `keras.Model` respectively. Both implement `call` method. Each way has it's own benefits and drawbacks. Is one of the above preferred? Or maybe something entirely different?

Bhack · January 27, 2022, 10:33pm

Keras-cv Now we have:

I will add some inline comments