TF.js: Beginner's question about tensors and multiplication

General Discussion
, ,
#1

Hello TF-fans!

I am new to TF and Machine Learning and trying to understand the basics. For this I am using the JavaScript API.

I have a vector of weight w and a vector of returns R where each element in R is a random variable, and each weight w_i is the corresponding weight of its random variable R_i.

My question is, wow do I multiply each weight in w with each random variable in R? I.e.,

image

This is my code:

const assetsReturns = [
  {
    name: "AAPL",
    R: [
      0.37, 0.58, 0.66, -0.02, 0.11, -0.18, -0.07, 0.59, 0.22, 0.19, -0.33,
      -0.16, 0.25, 0.38, 0.06, 0.2, 0.14, 0.54, -0.04, -0.33, 0.03,
    ],
  },
  {
    name: "TSLA",
    R: [
      0.29, -0.1, 0.43, 0.22, 0.25, 0.16, -0.7, 0.4, -0.35, 0.58, 0.1, -0.25,
      -0.02, 0.1, 0.38, 0.2, 0.01, 0.04, -0.54, -0.62, -0.67,
    ],
  },
];

// Initialize portfolio asset weights
function initializeWeightVector(assets) {
  let arr: number[] = [];
  for (let i = 0; i < assets.length; i++) {
    arr.push(Math.random());
  }
  // normalize asset weights
  let sumOfWeights = arr.reduce((acc, w) => acc + w, 0);
  arr.map((w) => (w /= sumOfWeights));
  console.log(arr);
  return tf.tensor(arr);
}

let w = initializeWeightVector(assetsReturns);
// w = tensor([[ 0.5625502154236828, 0.5801456476968492 ]]);

let R = tf.tensor(assetsReturns.map((asset) => asset.R)); 
/**
 * R = tensor(
  [
  0.37, 0.58, 0.66, -0.02, 0.11, -0.18, -0.07, 0.59, 0.22, 0.19, -0.33,
  -0.16, 0.25, 0.38, 0.06, 0.2, 0.14, 0.54, -0.04, -0.33, 0.03,
],
[
  0.29, -0.1, 0.43, 0.22, 0.25, 0.16, -0.7, 0.4, -0.35, 0.58, 0.1, -0.25,
  -0.02, 0.1, 0.38, 0.2, 0.01, 0.04, -0.54, -0.62, -0.67,
],
)
 */
console.log(R.shape[0]); //2
console.log(w.shape[0]); //2
console.log(R.shape[1]); //21
console.log(w.shape[1]); //undefined

Thank you for your help!

#3

Hi @LongBear ,

welcome to the Forum. For elementwise multiplications, you can use the tf.mul() operation.
The api reference also provides operations for matrices e.g: tf.transpose() and tf.dot(),
like in your equation. (Tipp: Try the Edit/Run Buttons below the code cell as an playground).

If you change the dimensions of R to [2, 21] and w to [2, 1], I think the following example is almost equal.

w = tf.randomNormal([2,1]);
R = tf.randomNormal([2,5]);   // here just 5 instead of 21

w.print(true);
R.print(true);

tf.sum( w.mul(R) ).print();           
tf.sum( w.transpose().dot(R) ).print();

outputs:

Tensor
  dtype: float32
  rank: 2
  shape: [2,1]
  values:
    [[1.4114478 ],
     [-0.6962689]]
Tensor
  dtype: float32
  rank: 2
  shape: [2,5]
  values:
    [[1.0965234 , -1.1874419, -1.9267293, 0.4351287 , -0.8840594],
     [-0.8211503, 1.0060594 , -1.090449 , -0.4771225, 0.3255759 ]]

Tensor
    -2.7454302310943604
Tensor
    -2.7454302310943604

Depending on your needs/targets, you can also try to initialise a basic Layer like tf.layers.dense() , as your weights.
It’s then ‘easier’ to combine multiple layers and create a more complex model …

Looking forward,
Dennis

#4

Hi @Dennis and thank you welcome and answer to my question.

Initializing a basic layer with weight might be a good option for creating a more complex model.

Ultimately, my goal is to minimize this matrix.

where q is a pre-determined scalar value.

I hope this will be straightforward in TF.

#5

What would be the advantage of creating a more complex model in TensorFlow compared to writing a simple Monte Carlo function in JavaScript to minimize the above matrix? @Dennis

#6

In your above example w tensor outputs a tensor whose sum is not 1. How can I generate a random tensor of numbers, sampled from a normal distribution whose sum is 1?

Tensor
  dtype: float32
  rank: 2
  shape: [2,1]
  values:
    [[1.4114478 ],
     [-0.6962689]]
#8

@LongBear one way could be by dividing w with their own sum(w) e.g:

const w = tf.randomNormal([2,1]);
w.print();

const w_mm = w.div(tf.sum(w));	
w_mm.print();
tf.sum(w_mm).print();

*************
Outputs:
Tensor
    [[0.1052495 ],
     [-0.5519487]]
Tensor
    [[-0.2356161],
     [1.2356161 ]]
Tensor
    1
#10

A simple function is always preferable compared to a complex model.
In cases you would need more parameters/weights a layer model might be more comfortable in adjusting …

#12

Feel free to have a look to the optimizer.minimize() . There’s a nice code example here.
Basically you can straightforward plug-in your equation into y.

const w = tf.randomNormal([2,1]).variable();
const R = tf.randomNormal([2,5]);
const q = tf.scalar(0.9);
//y
const f = x => w.transpose().dot(tf.sum(w, 1)).sub(q.mul(x.transpose().dot(w)));

f(R).print();

Outputs:
Tensor
    [[10.5561895],
     [8.068594  ],
     [9.1572447 ],
     [8.1347704 ],
     [5.8570642 ]]
#14

Thanks a lot @Dennis and aplogize for the late reply. I was actually trying to read up on how the optimizer works but I haven’t fully been able to convert it into a working example.

This is where I am at the moment:

let f = () => tf.matMul(tf.matMul(w.transpose(), tf.matMul(R.transpose(), R)), w);
let num = 1000;
for (let i = 0; i < num; i++) {
  console.log(`start of training num ${i}`);
  optimizer.minimize(f, true);
}

However, I ran into two challenges. First one is that the output of the function is a Tensor and not a Scalar, which is of course a requirement to be able to minimize the function.

Output:

Tensor
     [[0.0181985],]

I tried a bunch of things to convert it to a Scalar, for example flatten() or wrapping it in tf.scalar() but without success.

Secondly, once I figure out how to conver the function outpput to a scalar value, I suspect I will also need to figure how to make the weight vector w trainable. As you know, I initialized it like this:

function initializeRandomUniformWeights(n: number) {
  let w = tf.randomUniform([n,1]);
  return tf.div(w, tf.sum(w));
}

It should have the same shape as the R. But perhaps the correct way is to somehow initialize a Tensor of tf.variables instead of random numbers. I am not sure yet.

TensorFlow is a unique API and requires some time to fully master I think. The transformations seem to be the hardest, as I am finding it somewhat challenging to figure out simple things like how to convert a tensor to a scalar.

#16

@LongBear you can make them variable/trainable by adding .variable() to the end e.g:

// let w = tf.randomUniform([n,1]);
let w = tf.randomUniform([n,1]).variable(); 
//.variable() makes them trainable now

By using e.g: tf.sum(f) or tf.mean(f) you could also receive a tensor scalar.
Can you test the following:

//optimizer.minimize(f, true);
optimizer.minimize(tf.sum(f), true);

Hope that helps in a nutshell.

Edit: Just remembered of the .asScalar() function … maybe helpful as well

a = tf.tensor([[-0.1]]);
a.print();

b = a.asScalar();
b.print();

Outputs:
Tensor
     [[-0.1],]
Tensor
    -0.10000000149011612
#20

I cannot thank you enough @Dennis. Without your help, I think I would still be scratching my head at this point, trying to find the answers in the API documentation.

Calling variable() works just fine. Conveniently, making the weight matrix trainable seems to make the elements trainable.

Using tf.sum() over the evaluated expression works fine to create a scalar. But for readability, I prefer your suggestion of calling asScalar().

With the following code it seems the weight vector is getting trained and the expression minimized as expected:

function initRandomWeights(n: number) {
  let rawDist = tf.randomUniform([n, 1]);
  let scaledDist = tf.div(rawDist, tf.sum(rawDist));
  return tf.variable(scaledDist);
}
let w = initRaofomWeights(21);

function train(learningRate: number, epochs: number): void {
  const optimizer = tf.train.sgd(learningRate);
  //function with matrix expression to be optimized
  let f = () => tf.matMul(tf.matMul(w.transpose(), tf.matMul(R.transpose(), R)), w).asScalar();
  for (let i = 0; i < epochs; i++) {
    console.log(`Start of training epoch ${i}`);
    optimizer.minimize(f, true, [w]);
    let variance = f().dataSync()[0];
    let weights = w.dataSync()[0]; //get weights to vizualize evolution of parameter training
    console.log(
      `
      Variance: ${variance}\n
      Weights: ${weights}\n
      R: ${R.dataSync()[0]}\n
      `
    );
  }
}

train(0.01, 2);

I read about TensorFlow where it is possible to visualize the training parameters and the evolution of the training.

Do you know if using TensorBoard with the tfjs package without GPU acceleration is possible? You see, I am using an experimental runtime. This runtime should be able to run Node packages. However, despite being able to install tfjs-node and tfjs-node-gpu, it throws an error whenever I try to import the packages. I am convinced from the error that this is not related to TF but my runtime.

Not being able to use tfjs-node means I cannot benefit from typings that would be available with tfjs-node and, unfortunately, I have not been able to find a d.ts file elsewhere. Having typings would make my task of learning TF a bit easier.

#22

@LongBear TipTop, you’re welcome. Give me a float64 of it, thank more the API :wink:
About the tfjs-node|-gpu question, I’m not precisely sure … you may want to create a new post.
Looking forward