Grouped transposed convolution?

pascal · May 26, 2021, 9:05pm

I wish we would have working grouped transposed convolutions. We waited for a long time to have (standard) grouped convolution in TF when it had been available in other frameworks for quite a while. We don’t need to set some kind of tradition over this.

Bhack · May 26, 2021, 10:35pm

We have a ticket at

github.com/tensorflow/tensorflow

Groups parameter of Conv2d and Conv2dTranpose ('deconvolution') not Working ?

opened 03:47AM - 27 Nov 20 UTC

NguyenThaiHoc1

stat:contributions welcome type:bug comp:keras TF 2.3 TF 2.7

Hello Authors I'm a user of tensorflow. Someday i have use Deconvolution and …i know it called "Conv2dTranpose" in tensorflow. But I want to reduce parameters of model by groups. I found groups property at Pytorch and I see it in Conv2D class (Conv2dTranspose inheritance Conv2d). But when i use i get result which i don't want This is my code ` import tensorflow as tf from tensorflow.keras.layers import Conv2D, Conv2DTranspose, BatchNormalization, ReLU, MaxPool2D model = tf.keras.Sequential([ Conv2D(512, kernel_size=(3, 3), strides=(1, 1), padding='same', use_bias=False, name='conv'), BatchNormalization(), ReLU(), Conv2DTranspose(256, kernel_size=(4, 4), strides=(2, 2), padding='same', use_bias=False, groups=256, kernel_initializer='he_normal'), BatchNormalization(), ReLU() ]) model.build((32, 256, 192, 3)) model.summary() ` This is summary when i received > Layer (type) Output Shape Param # conv (Conv2D) (32, 256, 192, 512) 13824 batch_normalization (BatchNo (32, 256, 192, 512) 2048 re_lu (ReLU) (32, 256, 192, 512) 0 conv2d_transpose (Conv2DTran (32, 512, 384, 256) 2097152 batch_normalization_1 (Batch (32, 512, 384, 256) 1024 re_lu_1 (ReLU) (32, 512, 384, 256) 0 Total params: 2,114,048 Trainable params: 2,112,512 Non-trainable params: 1,536 I think conv2d_transpose will: 2097152 / 256(groups) = 8192 params ? Sorry for my writer is not good Thanks you for reading Thai Hoc

pascal · May 27, 2021, 1:14pm

@Bhack Exactly. We used to have a ticket on grouped convolution too, and it took more than 2.5 years to close. (Issue #3332 on GitHub, can’t post link here). This one is 6 months old already (it could have been filed much earlier tbh) and I am just concerned history will repeat.

PyTorch has it since at least v0.2 (July 2017, about the time “Attention is all you need” was published to get an idea of how long in DL terms). My understanding is that even jax can do it using conv_general_dilated and some paracetamol.

Bhack · May 27, 2021, 1:41pm

Yes but the grouped conv was contributed by the community in the end with:

github.com/tensorflow/tensorflow

Add support for cudnn's group convolution.

tensorflow:master ← ppwwyyxx:master

opened 11:15PM - 17 Feb 19 UTC

ppwwyyxx

+251 -65

This PR enables group convolution in cudnn, a feature that's highly desired for …many years (#3332, https://github.com/tensorflow/tensorflow/issues/12052#issuecomment-320465264, #11662, #10482). With this PR, now it's allowed to call `tf.nn.conv2d(inputs, filters)`, where the depth of `inputs` is not necessarily equal to `filters.shape[2]`, but be a multiple of `filters.shape[2]`. The core of this PR is only two lines of code (https://github.com/tensorflow/tensorflow/issues/3332#issuecomment-464308902) which removes the shape check. Then I added some extra checks and tests. This benchmark script: ```python import tensorflow as tf import time import os N = 64 C = 256 G = 32 H, W = 64, 64 print("N, C, H, W:", [N, C, H, W]) def benchmark_all(use_loop, format): shape4d = [N, C, H, W] if format == 'NCHW' else [N, H, W, C] tf.reset_default_graph() input = tf.get_variable('input', shape=shape4d, dtype=tf.float32) filter = tf.get_variable('filter', shape=[3, 3, C // G, C], dtype=tf.float32) if use_loop: inputs = tf.split(input, G, axis=1 if format == 'NCHW' else 3) filters = tf.split(filter, G, axis=3) output = tf.concat( [tf.nn.conv2d(i, f, strides=[1,1,1,1], padding='SAME', data_format=format) for i, f in zip(inputs, filters)], axis=1 if format == 'NCHW' else 3) else: output = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME', data_format=format) forward_op = output.op cost = tf.reduce_sum(output) backward_op = tf.train.GradientDescentOptimizer(0.1).minimize(cost) def benchmark(op, nr_iter=200, nr_warmup=10): for k in range(nr_warmup): op.run() start = time.perf_counter() for k in range(nr_iter): op.run() end = time.perf_counter() itr_per_sec = nr_iter * 1. / (end - start) return itr_per_sec sess = tf.Session() with sess.as_default(): sess.run(tf.global_variables_initializer()) spd_forward = benchmark(forward_op) print("Loop={}, Format={}, Forward: {} itr/s".format(use_loop, format, spd_forward)) spd_backward = benchmark(backward_op) print("Loop={}, Format={}, Backward: {} itr/s".format(use_loop, format, spd_backward)) formats = ['NHWC', 'NCHW'] for format in formats: for use_loop in [True, False]: benchmark_all(use_loop, format) ``` Executed on V100, cuda10, cudnn 7.4.2, it prints: ``` N, C, H, W: [64, 256, 64, 64] Loop=True, Format=NHWC, Forward: 65.49446747235214 itr/s Loop=True, Format=NHWC, Backward: 32.26484275606916 itr/s Loop=False, Format=NHWC, Forward: 117.40288830454352 itr/s Loop=False, Format=NHWC, Backward: 50.051492362319074 itr/s Loop=True, Format=NCHW, Forward: 98.8428390274372 itr/s Loop=True, Format=NCHW, Backward: 35.672312085388455 itr/s Loop=False, Format=NCHW, Forward: 152.24726060851506 itr/s Loop=False, Format=NCHW, Backward: 56.21414524041962 itr/s ``` which shows around 50~80% speed up over a naive loop-based implementation.

I could ask to @thea to check if it the transposed group conv ticket could be labeled as contribution welcome or it is already the internal roadmap.

Also that specific ticket has not so much community upvotes.

pascal · May 27, 2021, 2:21pm

Thanks, I didn’t know this feature was eventually implemented by a contribution from the community. I am even more concerned about the implementation of this one now.

I get your remark about community upvotes. At the same time depthwise/grouped (transposed) convolutions are so ubiquitous, it is hard for me to understand the reluctance.

thea · May 27, 2021, 5:25pm

@Bhack Exactly. We used to have a ticket on grouped convolution too, and it took more than 2.5 years to close. (Issue #3332 on GitHub, can’t post link here). This one is 6 months old already (it could have been filed much earlier tbh) and I am just concerned history will repeat.

We’re currently working through our issue backlog. Ultimately implementing changes comes down to bandwidth and the volume of requests in the community. It might be more difficult to implement than it looks like on the surface.

Thanks for flagging this thread. I think it’s important to also ensure that further discussion validating this bug happens on the bug thread – that way our triagers and maintainers have full context in a single place. I’ll leave this thread open for now in case maintainers would like to respond here, but may close it in the future to ensure that relevant details and use cases are added to the bug.

I could ask to @thea to check if it the transposed group conv ticket could be labeled as contribution welcome or it is already the internal roadmap.

@Bhack Can you add a mention on the bug that it might be a good candidate for contributions welcome so that can be looked at as part of the ongoing issue grooming work?