Trying to understand Tensorflow Internals, would appreciate any insight on Tensor data

General Discussion
, ,
#1

I have been spending a lot of time trying to understand the TensorFlow internals for a personal project by looking at source code and messing around in GDB. I know that Tensors are stored in the tensorflow::Tensor class which contains a shape, element count and a pointer to a tensorflow::TensorBuffer. TensorBuffer has a reference count for the actual data in the tensor and a pointer to the data. I have been able to find tensorflow::Tensors c++ data structures when messing around in GDB and am able to access their underlying buffers. However, I am struggling to find the data structures that manage the tensorflow::Tensors and how they are kept track of internally in a model.

For example, how do I associate a trainable variable in a model to its underlying in memory buffer?

When looking at how model.predict() works I know that the inputs to the execute function (TFE_Py_Execute) in quick_execute in tensorflow/python/eager/execute.py are passed in as tf.Tensor. When looking at the C++ code for TFE_Py_Execute in GDB the inputs are passed into that function as TFE_InputTensorHandles and an individual TFE_TensorHandle can be accessed with inputs->at(n) where n is any number below input size. Casting an individual TFE_TensorHandle to a tensorflow::TensorHandle in gdb lets me access the methods of the TensorHandle class for each TensorHandle.

However, there is no method that lets me pull out the data buffer for the TensorHandle and I seem to be missing some connection as when I go to access the data_ field of the TensorHandle which should contain a LocalTensorHandleData it is filled with info that I was not expecting. Where as a typical tensorflow::Tensor has a shape, element count, and pointer to a TensorBuffer which subsequently contains a pointer to a in memory buffer filled with tensor elements, the data_ for these TensorHandles is filled with a bunch of strings such as device string, localhost, a couple byte sections with 2c in them etc. I am not sure of what type this is and why this isnt just an in memory buffer.

So I would really like a clarification on what I am missing to more easily access the underlying tensorflow::Tensor data structure and its buffer as well as how I can maybe get from a model and its trainable variables to its underlying buffer.

Thanks!

1 Like
#3

I recently started implementing custom ops, and I agree TF C++ is a lot harder to grapple with that the Python side, and the current state of the docs does not help at all.

That said, grep or Search in VS Code is your ally: searching class TensorHandle in the TF repo, you can pull up tensorflow/cc/experimental/base/public/tensorhandle.h and see that the Tensor Resolve(Status* status) from which you could then get the buffer?

#5

Yup, Greping is what Ive been doing for about a month now :frowning:

That tensorflow::experimental::tensorhandle class is very interesting, but unfortunately it looks to be something a bit different from the TensorHandles returned in execution. There is a Resolve method in the tensor_handle located in core/common_runtime/eager/tensor_handle but it doesnt unwrap to a Tensor, instead unwraps to an AbstractTensorInterface which is something I havent looked into yet.

An Ideal scenario for me would be a path from the tf.Variables for model variables in the model dictionary to the actual c++ Tensors themselves if pointers are followed but probably isnt a thing :frowning:

#6

it seems you can call tensor_handle.Resolve().Data() to get a void *. From the AbstractTensorInterfaceyou could also usedown_cast`? There are examples such as

Status TF_TensorToTensor(const TF_Tensor* src, Tensor* dst) {
  return tensorflow::down_cast<const tensorflow::TensorInterface*>(src->tensor)
      ->ToTensor(dst);
}

which might help.

#7

Gotcha, that makes sense. The biggest issue though is that this still doesnt clear up the problem I described in the initial post where execution for lets say the predict function of a model starts in python at execute.py, it seems to pass in all of its model variables into TFE_Py_Execute as inputs which get read as TFE_TensorHandleInputs in C space and added to the operation. But those tensorhandles dont have a data_ field containing a Tensor, instead they contain debug information.

I guess my main point is, am I missing that the context contains the handles to the tensors with Data? Because the inputs to the execute function definitely dont. And if so, how do I pull those TensorHandles from the context (EagerContext not OpKernelContext)