Data type in tensor: i32 vs f32

Special Interest Groups TF.js
,
#1

I use TensorFlow.js. Tensor data have the same type (dtype). In my case study, all data are integers with discrete distribution (typically, one of my input feature varies between 1 and 20).
My feeling (not certain at all) is that choosing i32 is “natural” and computations at training time are more efficient.
My question is: has the choice between i32 and f32 impact (beyond efficiency) on the model accuracy at training/prediction time? Since most of my input data are discrete with fixed variation intervals, I have the bad feeling that using Gauss distribution for normalization is irrelevant and/or i32 (to the dtriment of f32) is the good choice?
Sorry about this naive question…

#3

Your question is quite insightful and touches on several important aspects of machine learning model development, particularly in the context of TensorFlow.js and data types. Let’s unpack the various elements of your query to provide a comprehensive answer.

Data Types: i32 vs. f32

In TensorFlow.js (as in most machine learning frameworks), i32 represents 32-bit integer data, while f32 represents 32-bit floating-point data. The choice between these two data types can indeed impact both the efficiency and the behavior of your machine learning models, though perhaps not always in the way one might initially think.

Efficiency

  • Computational Efficiency: Generally, integer operations can be faster and more memory-efficient than floating-point operations, especially on devices optimized for integer arithmetic. However, the degree of this efficiency gain can vary widely depending on the specific hardware and the nature of the computations being performed.
  • Memory Usage: Using i32 can also reduce memory consumption, as you’re using the full precision of the storage for meaningful data, without the overhead that comes with representing floating-point numbers.

Impact on Model Behavior and Accuracy

  • Gradient-Based Optimization: Most machine learning models, especially those involving deep learning, rely on gradient-based optimization techniques (like stochastic gradient descent). These techniques inherently depend on the calculation of derivatives, which implies a continuous space. Even if your input data are integers, the model’s internal computations, including weights and activations, often benefit from the flexibility of floating-point representation (f32), which allows for fractional adjustments during learning.
  • Normalization and Scaling: You mentioned concerns about normalization, particularly using a Gaussian distribution. While it’s true that normalization is often applied to data that follow or approximately follow a Gaussian distribution, the goal of normalization (making the data have a mean of 0 and a standard deviation of 1, for instance) is to standardize the scale of the features, improving the convergence behavior of the training process. This is beneficial regardless of the underlying distribution of the features and is generally applicable to both i32 and f32 data types. However, the actual normalization computations would typically require floating-point arithmetic to avoid losing the nuance in the data distribution.
  • Quantization: In some cases, especially in deployment scenarios where efficiency is critical, models are quantized to use lower-precision arithmetic (like integer operations), but this is usually done after training a model in floating-point precision to ensure that the training process can fully exploit the continuous nature of the optimization landscape.

Practical Recommendation

Given your case study, where the features are discrete and have a limited range, starting with f32 for model variables and computations is generally advisable for the reasons outlined above, particularly to preserve the fidelity of gradient-based optimization. If efficiency becomes a critical concern, especially in deployment, you might explore quantization techniques to convert the model to use i32 or lower-precision arithmetic, bearing in mind that this can introduce some degree of approximation and potentially affect model accuracy.

Conclusion

While using i32 for input data that are inherently integer might seem natural and more efficient, the broader context of machine learning model development and training usually benefits from the flexibility and nuance offered by f32. This doesn’t preclude the use of i32 for certain operations or in post-training optimization, but during the training phase, f32 is typically preferred to ensure the model can learn effectively.

#5

Hi Tim,
Many many thanks for your long answer. No surprise about efficiency. More confused on the rest… In fact, I got “poor”/“unsatisfactory” (without being “wrong”) results at prediction time (using f32). I train my model with 10.000 records. In fact, my output feature (expected prediction) is also made of discrete/sparse values (between 60000 and 80000). Prediction results are quite far from reality. Expectations in some cases have to be close to min (60000) and max (80000) while all predictions remain around the mean, eg, 70000 ± 5000.

Again, I feel that I have to better control hyper-parameters including normailzation, which can be something else that Gauss? Thanks for your help.