Building tensorflow 2.10 GPU from source

General Discussion
#1

Hi all,

I try to build TensorFlow 2.10 GPU version from source, here is my setup:
gcc version 11.2.0
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

But in building the GPU variant I get this error after some time:

external/com_google_absl/absl/base/casts.h(164): error: type name is not allowed

external/com_google_absl/absl/base/casts.h(164): error: identifier "__builtin_bit_cast" is undefined
          detected during instantiation of "Dest absl::lts_20220623::bit_cast<Dest,Source,<unnamed>>(const Source &) [with Dest=uint16_t, Source=int16_t, <unnamed>=0]"

external/com_google_absl/absl/base/internal/endian.h(143): here

external/com_google_absl/absl/base/internal/endian.h(143): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/casts.h(164): error: identifier "__builtin_bit_cast" is undefined
          detected during instantiation of "Dest absl::lts_20220623::bit_cast<Dest,Source,<unnamed>>(const Source &) [with Dest=uint32_t, Source=int32_t, <unnamed>=0]"
external/com_google_absl/absl/base/internal/endian.h(146): here

external/com_google_absl/absl/base/internal/endian.h(146): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/casts.h(164): error: identifier "__builtin_bit_cast" is undefined
          detected during instantiation of "Dest absl::lts_20220623::bit_cast<Dest,Source,<unnamed>>(const Source &) [with Dest=uint64_t, Source=int64_t, <unnamed>=0]"
external/com_google_absl/absl/base/internal/endian.h(149): here

external/com_google_absl/absl/base/internal/endian.h(149): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(153): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(156): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(159): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(233): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(236): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(239): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(243): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(246): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/internal/endian.h(249): error: no instance of function template "absl::lts_20220623::bit_cast" matches the argument list
            argument types are: (<error-type>)

external/com_google_absl/absl/base/casts.h(164): error: identifier "__builtin_bit_cast" is undefined
          detected during:
            instantiation of "Dest absl::lts_20220623::bit_cast<Dest,Source,<unnamed>>(const Source &) [with Dest=const void *, Source=void (*)(int64_t, int64_t, int64_t
, const int64_t *, const int64_t *, const int64_t *, int64_t *), <unnamed>=0]"
./tensorflow/core/util/gpu_kernel_helper.h(108): here
            instantiation of "tensorflow::Status tensorflow::GpuLaunchKernel(void (*)(Ts...), dim3, dim3, size_t, gpuStream_t, Args...) [with Ts=<int64_t, int64_t, int64
_t, const int64_t *, const int64_t *, const int64_t *, int64_t *>, Args=<tensorflow::int64, tensorflow::int64, tensorflow::int64, std::conditional_t<true, int64_t *, con
st int64_t *>, std::conditional_t<true, int64_t *, const int64_t *>, std::conditional_t<false, const int64_t *, const int64_t *>, std::conditional_t<true, int64_t *, con
st int64_t *>>]"
tensorflow/core/kernels/reshape_util_gpu.cu.cc(96): here

17 errors detected in the compilation of "tensorflow/core/kernels/reshape_util_gpu.cu.cc".
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 3247.325s, Critical Path: 413.40s
INFO: 27322 processes: 11188 internal, 16134 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

It looks like a problem with abseil libs but I don’t find any related issue on github or anywhere else. But this abseil source is downloaded while building Tensorflow, right? How do I get rid of this error?

Thanks,
Babar

#3

One investigation.

The error occurs here:

return GpuLaunchKernel(ReshapeSparseTensorKernel<int64_t>, config.block_count,
                         config.thread_per_block, 0, device.stream(), nnz,
                         /*input_rank=*/input_rank,
                         /*output_rank=*/output_rank,
                         /*input_shape=*/input_shape_gpu.data(),
                         /*output_shape=*/output_shape_gpu.data(),
                         /*input_indices=*/input_indices.data(),
                         /*output_indices=*/output_indices.data());

The function GpuLaunchKernel calls the absl::bit_cast function here like this:

template <typename... Ts, typename... Args>
Status GpuLaunchKernel(void (*function)(Ts...), dim3 grid_dim, dim3 block_dim,
                       size_t shared_memory_size_bytes, gpuStream_t stream,
                       Args... arguments) {
  static_assert(detail::NoneIsReference<Ts...>(),
                "Kernels with reference arguments have undefined behaviour.");
#if GOOGLE_CUDA
  auto func_ptr = absl::bit_cast<const void*>(function);

Some reason this call fails? I tried making simple program with the same function:

#include <iostream>
#include <string>
#include <vector>

#include "absl/base/casts.h"

void testfunc() {
  std::cout << "In test" << std::endl;
}

int main() {
  auto testptr = absl::bit_cast<const void*>(&testfunc);
  std::cout << "Finished" << std::endl;

  return 0;
}

but this function compiles with success.

What is wrong then?