Maths and Logic with Tensor Pytorch

Now that you know how to create tensors, let's dive into what you can actually do with them. The fun starts with basic arithmetic operations and how tensors interact with simple scalars.

Vectorization

Let's begin with some fundamental operations:

As shown, arithmetic operations like addition, subtraction, multiplication, division, and exponentiation apply element-wise to the tensor. The result of each operation is a new tensor, which means you can chain operations together, following the usual operator precedence rules.

But what about operations between two tensors? They behave just as you would expect:

Notice that all tensors in these examples share the same shape. But what happens if we try to perform operations on tensors of different shapes?

Here's a hint: it doesn't go well. The following example throws a runtime error on purpose:

In general, you can not perform binary operations on tensors with different shapes, even if they contain the same number of elements. This concept is crucial to understanding tensor operations and avoiding common pitfalls in your PyTorch projects.

Note: For vectorization two tensors must have the same shape.

Tensor Broadcasting

The exception to the same-shapes rule is tensor broadcasting. Here’s an example:

What is the trick here? How did we multiply a 2x4 tensor by a 1x4 tensor?

Broadcasting allows you to perform operations between tensors that have compatible shapes. In the example above, the one-row, four-column tensor is multiplied by both rows of the two-row, four-column tensor.

This operation is essential in deep learning, especially when multiplying learning weights by a batch of input tensors, applying the operation to each instance in the batch separately, and returning a tensor of identical shape.

The rules for broadcasting are:

1. Each tensor must have at least one dimension—no empty tensors.

2. When comparing dimension sizes of the two tensors from last to first:

• Each dimension must be equal, or

• One of the dimensions must be of size 1, or

• The dimension does not exist in one of the tensors

Tensors of identical shape, of course, are trivially “broadcastable,” as seen earlier.

More Maths with Tensor

PyTorch tensors have over three hundred operations that can be performed on them.

Altering Tensors in Place

When you perform operations on tensors, they usually create new tensors. For example, if you have c = a * b (where a and b are tensors), c will be a new tensor stored in a different memory location from a and b.
However, sometimes you may want to modify a tensor directly, especially if you don't need to keep the original values. PyTorch provides in-place operations for this purpose. These functions have an underscore (_) at the end of their names.

Here's an example:

These in-place arithmetic functions are methods of the torch.Tensor object, not the torch module.
For example, a.add_(b) modifies a directly.
Another way to place the result of a computation in an existing tensor is by using the out argument.
Many PyTorch functions and methods, including tensor creation methods, have this argument.

Here's an example:

Using in-place operations and the out argument helps manage memory more efficiently by reusing existing tensors. This can be especially important in deep learning applications where memory usage is critical.

Copying Tensor:

When you assign a tensor to a variable in Python, you are not creating a new copy of the tensor; you are just creating a new reference to the same tensor. For example:

In this case, modifying a also changes b because both variables point to the same tensor. But what if you need an actual copy? That's where the clone() method comes in:

Using clone(), b is a separate tensor with the same data as a, but changes to a don't affect b.

Important Note on clone() and Autograd

If your source tensor has autograd enabled, the cloned tensor will also have autograd enabled. This can be useful if both the original tensor and its clone are involved in a model's forward pass and need to track gradients for learning. However, there are situations where you might want to disable autograd for the clone to save on performance. For this, you can use the detach() method.

The detach() method effectively removes the tensor from its computation history, allowing you to perform operations without tracking gradients, which can improve performance in certain cases.