I’m wondering what’s the Tensorflow way of storing and performing Tensor manipulation (i.e. Tensor multiplication, apply_gradient) of trainable weights in a custom layer that could potentially be sparse based on user specification. I’ve looked into
https://www.tensorflow.org/api_docs/python/tf/boolean_mask
and
https://www.tensorflow.org/api_docs/python/tf/RaggedTensor
but I’m not sure if it’s possible to apply these as ways to store and update trainable weights.
Bhack
June 27, 2021, 10:37am
#2
Have you already take a look at:
opened 07:57PM - 21 Jan 17 UTC
closed 04:40PM - 16 Jun 17 UTC
stat:awaiting tensorflower
I have a normal feed-forward network that produces a vector v. The elements of v… are then used as the non-zero entries of a sparse matrix M (assume the coordinates are predefined). The sparse matrix is then multiplied by a dense vector and a loss is defined on the resulting scalar. I want to back-propagate the loss w.r.t. the weights of the network, which entails going through the sparse matrix.
This seems like a perfectly reasonable use-case for a sparse matrix, but it appears that such functionality is not supported. Indeed, even calling tf.gradients(M,[v]) produces an error:
> AttributeError: 'SparseTensor' object has no attribute 'value_index'
Am I doing something wrong or am I correct in presuming that this functionality doesn't (yet?) exist? If the latter, then is there a work-around for this particular use-case short of rewriting all of the sparse tensor operations with gradients defined?