Customize structural pruning rate in each layer

Shawn_Zhuang · March 16, 2022, 2:23pm

Hi, is it possible to specify the pruning rate in each layer and ensure there is really no value instead of setting to 0.
for example, in the following easy model, the shape of weight of the first Conv2D is (3,3,1,32), I want to prune to (3,3,1,10) according to the L1 magnitude. Is it possible?

input_shape = (28, 28, 1)
num_classes = 10
model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation="softmax"),
    ]
)

model.summary()

I also try to manually run layer.set_weights, but I was informed that

Layer conv2d weight shape (3, 3, 1, 32) is not compatible with provided weight shape (3, 3, 1, 10)

Thanks for any advice.

Bhack · March 16, 2022, 3:11pm

Are you looking for something like:

github.com/tensorflow/model-optimization

Structural (filter) pruning for convolutional layers

opened 12:34PM - 08 Jun 21 UTC

marcelroed

feature request technique:pruning

**System information** - TensorFlow version (you are using): 2.5.0 - Are you… willing to contribute it (Yes/No): Yes **Motivation** Deciding on where to have high filter/channel counts in convnets can be difficult, and smarter reductions in these numbers can lead to faster inference time across all devices. Pruning is currently not very useful on GPU, since sparse operations are much slower than dense operations, so it would be useful to have a method of pruning that results in a reduced dense representation. The current implementation I have (that isn't finished) doesn't require many additional components, since it works similarly to block sparsity and can reuse much of this code. **Describe the feature** Add an option to `prune_low_magnitude` for "filter pruning" (alternatively "structural pruning") that restricts pruning of supported layers to blocks of the weights at a time. For convolutional layers these blocks represent the output channels of the layer. In addition, an option is added to `strip_pruning` to restructure the layers that have been pruned in this manner, with fewer output channels than the original layers. The change in shape needs to be propagated forwards to future layers. **Describe how the feature helps achieve the use case** With these two additions, models can be pruned in a way that is meaningful when running on GPU, saving memory and compute. It is also possible to find a reasonable layout for the number of output channels in each layer without hyperparameter tuning. This feature makes pruning useful on GPU, where it currently is not so useful. **Describe how existing APIs don't satisfy your use case** Using `tfmot.python.core.sparsity.keras.prune.prune_low_magnitude` on a convolutional layer will consider each element of the weights variable on its own, and very rarely leads to pruning that can be useful for reducing inference time on the GPU. In addition, `tfmot.python.core.sparsity.keras.prune.strip_pruning` will always leave weights with zeros in them, even if a reduction in the size of the layer would be beneficial. If the outputs of a filter in the kernel of a convolutional layer are all zero, `strip_pruning` will leave restructuring as a step for the runtime.

Shawn_Zhuang · March 16, 2022, 4:20pm

yes. So it is still an to-be-done feature? Sad.

Bhack · March 16, 2022, 4:41pm

You could try to upvote and add a comment there for an update or in the case you want to contribute this feature to the ecosystem with a PR.

Rino_Lee · May 13, 2022, 12:06pm

Structural pruning for Conv2D is available now for the input-channel dimension. Yapısal budama kullanan seyrek ağırlıklar | TensorFlow Model Optimization

However, I think what you want is actual reduce of dimension in Conv2D weight, which is not a scope of sparsity toolkit of TF for now.