I’m wondering what’s the Tensorflow way of storing and performing Tensor manipulation (i.e. Tensor multiplication, apply_gradient) of trainable weights in a custom layer that could potentially be sparse based on user specification. I’ve looked into
but I’m not sure if it’s possible to apply these as ways to store and update trainable weights.
June 27, 2021, 10:37am
Have you already take a look at:
07:57PM - 21 Jan 17 UTC
04:40PM - 16 Jun 17 UTC
I have a normal feed-forward network that produces a vector v. The elements of v
… are then used as the non-zero entries of a sparse matrix M (assume the coordinates are predefined). The sparse matrix is then multiplied by a dense vector and a loss is defined on the resulting scalar. I want to back-propagate the loss w.r.t. the weights of the network, which entails going through the sparse matrix.
This seems like a perfectly reasonable use-case for a sparse matrix, but it appears that such functionality is not supported. Indeed, even calling tf.gradients(M,[v]) produces an error:
> AttributeError: 'SparseTensor' object has no attribute 'value_index'
Am I doing something wrong or am I correct in presuming that this functionality doesn't (yet?) exist? If the latter, then is there a work-around for this particular use-case short of rewriting all of the sparse tensor operations with gradients defined?