Lowering for tf raw_ops ImageProjectiveTransform

uday · March 22, 2022, 6:09pm

I wanted to know what the policy/approach to support lowering for certain useful TF raw ops is: these currently aren’t in the MLIR TF dialect, and several higher-level abstractions are de-abstracted through these. As an example, the tensorflow addons package’s tfa.image.translate lowers through the general “projective transformation” op (tf.ImageProjectiveTransformationV3), which can model combinations of rotation, scaling, skewing, translations, etc.) and is extremely useful to support for further optimization and code generation. I’ve added a lowering for this op from TF to lower level TF ops for a typical case (commit link below):

https://www.tensorflow.org/api_docs/python/tf/raw_ops/ImageProjectiveTransformV3

Without such a lowering, the op otherwise fails conversion beyond the MLIR TF dialect. I’m assuming TF/MLIR is open to contributions to lowering such ops?

Bhack · March 22, 2022, 6:15pm

Is this also related to:

github.com/tensorflow/tensorflow

XLA support for ImageProjectiveTransformV3

opened 05:14PM - 10 Mar 22 UTC

bhack

stat:awaiting tensorflower type:feature comp:xla

**System information** - TensorFlow version (you are using): master - Are you… willing to contribute it (Yes/No): I don't know, only if I have a clear contribution path **Describe the feature and the current behavior/state.** XLA support for `ImageProjectiveTransformV3` **Will this change the current api? How?** No **Who will benefit with this feature?** Performance on image preprocessing **Any Other info.** ```python import tensorflow as tf import numpy as np @tf.function(jit_compile=True) def transform(): inp = np.arange(15).reshape((1, 5, 3, 1)).astype(np.float32) transform_matrix = np.asarray([[1., 0., 0., 0., 1., -1., 0., 0.]]) output_shape = tf.shape(inp)[1:3] tf.raw_ops.ImageProjectiveTransformV3( images=inp, output_shape=output_shape, fill_value=0.0, transforms=transform_matrix, fill_mode="constant", interpolation="bilinear") transform() ``` ``` on XLA_CPU_JIT: ImageProjectiveTransformV3 (No registered 'ImageProjectiveTransformV3' OpKernel for XLA_CPU_JIT devices compatible with node {{node ImageProjectiveTransformV3}}){{node ImageProjectiveTransformV3}} ``` Extra: Please also note that the CPU/GPU TF2XLA supported ops tables are probably outdated (2018): https://github.com/tensorflow/tensorflow/issues/14798#issuecomment-1047796247

Bhack · March 22, 2022, 6:18pm

We are also talking about this in some augmentation/preprocessing layers performance tickets:

github.com/keras-team/keras

Scipy affine transform

opened 04:37PM - 10 Mar 22 UTC

bhack

type:feature

**System information**. TensorFlow version (you are using): master Are you …willing to contribute it (Yes/No) : I need more detail **Describe the feature and the current behavior/state**. I think that we need to cover core image processing transformation with TF native ops. Currently a core transformation in preprocessing still rely on numpy/scipy impl. https://github.com/keras-team/keras/blob/master/keras/preprocessing/image.py#L2622 Describe the feature clearly here. Be sure to convey here why the requested feature is needed. Any brief description about the use-case would help. **Will this change the current api? How?** **Who will benefit from this feature?** **[Contributing](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md)** - Do you want to contribute a PR? (yes/no): - If yes, please read [this page](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md) for instructions - Briefly describe your candidate solution(if contributing):

github.com/keras-team/keras-cv

[Canary test] GridMask: Vectorized function call on the batch dimension

keras-team:master ← bhack:patch-2

opened 02:31PM - 22 Feb 22 UTC

bhack

+1 -1

As we have discussed in https://github.com/keras-team/keras-cv/pull/143#issuecom…ment-1047215737 this is just a canary (failing) test (check the CI): ``` ValueError: Input "maxval" of op 'RandomUniformInt' expected to be loop invariant. ``` As I've mentioned in the thread we really need to understand if we want to have randomness inside the batch or between the batches and what kind of impact we have between the computing overhead, contributing speed/code readability and network convergence. Also I don't know if @joker-eph or @qlzh727 could expose us a little bit the pro and cons of `jit_compile` a function vs using the `vectorized_map` or if they are orthogonal. With many CV transformations we cannot compile the function as the underline `tf.raw_ops.ImageProjectiveTransformV3` op isn't supported by XLA. /cc @chjort

uday · March 22, 2022, 6:31pm

It’s the same op but my post isn’t about XLA proper or the TF → XLA support (yes, this isn’t supported on the TF → XLA path as well).

Bhack · March 22, 2022, 6:44pm

Yes sorry, it Is super hard to understand every day, for the avg contributor or end user, when and where MLIR is involved or not in a compiler path: