Keras torch backend only using CPU

I am following High-performance image generation using Stable Diffusion in KerasCV on a Windows 11 PC, with Torch 2.1.0 installed with CUDA 11.8 by Conda. Keras version is 3.1.1.

My GPU is visible for Torch, for example print(torch.cuda.get_device_name(0)), but running the first image batch example always uses CPU. To be sure of using Torch I set

import os
os.environ["KERAS_BACKEND"] = "torch"
import keras

in the code.

Here is the output from starting the script:

2024-03-20 21:39:16.583862: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 
2024-03-20 21:39:17.555294: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 
NVIDIA GeForce RTX 2080 SUPER
By using this model checkpoint, you acknowledge that its usage is subject to the terms of the CreativeML Open RAIL-M license at https://raw.githubusercontent.com/CompVis/stable-diffusion/main/LICENSE
2024-03-20 21:39:20.187392: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From C:\Users\USERNAME\miniconda3\envs\keras3\Lib\site-packages\keras\src\backend\tensorflow\core.py:188: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

It seems like it is using TF backend?

EDIT: at the first runs I did not specify backend. Am I right to assume that this lead to TF being backend for this specific Conda env, and the solution would be to delete the env and create a new?

So the problem was me importing both keras and keras-cv in the code snippet, and keras_cv import statement was above the backend initialization call, like

import keras_cv
import os
os.environ["KERAS_BACKEND"] = "torch"
import keras

So yeah: attention was all I needed.