Is the TensoFlow Decision Forests prediction probability calibrated?

General Discussion
,
#1

I would like to understand whether the TensoFlow Decision Forests prediction probability is calibrated?

If not what are the ways of calibrating it.

Thank you in advance.

#3

Hi,

TF-DF models are not calibrated by default, and automated calibration is (currently) not available in the open source version. The amount of calibration/non-calibration of a model is dependent on the learning algorithm (e.g. RF or GBT) and the hyper-parameters. For example, the classification Random Forest with default hyper-parameter is certainly not calibrated.

A solution is to train a calibration model on top of the model. For example, (1) you can learn an isotonic regression with the Pava algorithm, or (2) train a piece-wise linear function with Gradient descent using TF-Latice.

Cheers,

#4

More on calibration using TF-Lattice here. The idea is that once you trained the TF-DF model, one connects its output to tfl.layers.PWLCalibration with monotonicity=1 (increasing). And retrain this composed model (the TF-DF model will be frozen, only the calibration will be updated).

Hopefully we’ll add automatic calibration in the near future.