Weight tensors after applying Structural pruning

General Discussion
,
#1

Hi,

In the above example, after applying structural pruning to some layers. How the shape of the tensor will change? weights related to one node or neuron will be completely replaced with zero after pruning? If not, is there any method to do like that?

Ex: Consider the layer “prune_low_magnitude_structural_pruning_dense”
shape before pruning : (3136, 1024)
after pruning(non zero weights shape ): (1568 , 1024) but this is contribution of all the neurons (i.e., no input or output neuron weights is completely zero).

Weight tensor after structural pruning:

     [[-0.029, -0.        ,  0.        , ...,  0.        , -0.        , -0.029],
     [ 0.0293, -0.        , -0.0372, ...,  0.0229, -0.022,  0.027],
     [-0.        , -0.0265 ,  0.0193, ..., -0.0375,     -0.        ,  0.        ],
      ...,
     [-0.        ,  0.        ,  0.0294, ..., -0.0153, 0.        ,  0.        ]], dtype=float32)

Expected weight tensor after structural pruning:

     [[-0., -0.        ,  0.        , ...,  0.        , -0.        , -0.],
     [ 0.0293, -0.026  , -0.0372 ...,  0.0229 -0.022,  0.027=],
     [-0.        , -0.,  0., ..., -  0.,     -0.        ,  0.        ],
      ...,
     [-0.0264  ,  0.0342,  0.0294, ..., -0.0153, 0.013        ,  0.0213 ]], dtype=float32)
    
      Is there any way to like this? plz suggest if any.

Thanks

#3

The shape of weights stay the same. Only some values of weights will be nullified according to this rule

For example, Conv2D layer weights in TensorFlow Lite have the structure [channel_out, height, width, channel_in] and Dense layer weights have the structure [channel_out, channel_in]. The sparsity pattern is applied to the weights in the last dimension: channel_in.

#5

Hello, any updates on the topic ? I am facing the same issue. Thank you