1. Tensor Basics: From Data to PyTorch’s Core Structure¶

In PyTorch, tensors are the fundamental structure for storing data, similar to NumPy arrays but with GPU acceleration and automatic differentiation support. Understanding tensors is the first step to mastering PyTorch.

1.1 Creating Tensors¶

• From Lists/Numbers: Directly convert Python lists to tensors using torch.tensor().

import torch

# Create a 1D tensor
x = torch.tensor([1, 2, 3])
print(x)  # Output: tensor([1, 2, 3])
print(x.shape)  # Output: torch.Size([3]) (shape is 3)

• From NumPy Arrays: Convert NumPy arrays to tensors with torch.from_numpy() (and vice versa with .numpy()).

import numpy as np

a = np.array([[1, 2], [3, 4]])  # NumPy array
b = torch.from_numpy(a)         # Convert to PyTorch tensor
print(b)  # Output: tensor([[1, 2], [3, 4]], dtype=torch.int64)

• Using PyTorch Built-in Functions: Quickly generate tensors filled with zeros, ones, or random values.

# 2x3 tensor of zeros
zeros = torch.zeros(2, 3)
# 3x2 tensor of ones
ones = torch.ones(3, 2)
# 1x4 tensor with uniform random numbers
rand = torch.rand(1, 4)
# 1x4 tensor with standard normal distribution
randn = torch.randn(1, 4)

2. Tensor Dimension Transformations: Core Operations for Shape Adjustment¶

In deep learning, data dimensions (shapes) often need adjustment. PyTorch provides multiple tools for shape manipulation:

2.1 reshape(): Flexible Shape Adjustment¶

reshape(new_shape) changes the tensor’s shape to new_shape, but the total number of elements must remain the same.
Use case: Adjust shape safely regardless of whether the original tensor is contiguous.

# Create a 2x3 tensor (6 elements)
x = torch.rand(2, 3)
print("Original shape:", x.shape)  # Output: torch.Size([2, 3])

# Reshape to 3x2 (3*2=6 elements)
y = x.reshape(3, 2)
print("After reshape:", y.shape)  # Output: torch.Size([3, 2])

2.2 view(): Efficient Adjustment for Contiguous Storage¶

view(new_shape) adjusts the shape only if the original tensor is contiguous (e.g., not modified by transpose).
Use case: Faster than reshape() when the tensor is contiguous.

# 2x3 tensor reshaped to 3x2 (contiguous storage)
x = torch.rand(2, 3)
y = x.view(3, 2)  # Same effect as reshape(3,2)
print(y.shape)  # Output: torch.Size([3, 2])

2.3 squeeze(): Remove Dimensions of Size 1¶

squeeze(dim=None) removes all dimensions with size 1 (a “compression” operation).
Specify a dimension to remove with dim=index.

# Original shape: (1, 2, 1, 3) (multiple dimensions of size 1)
x = torch.rand(1, 2, 1, 3)
print("Original shape:", x.shape)  # Output: torch.Size([1, 2, 1, 3])

# Remove all dimensions of size 1
y = x.squeeze()
print("After squeeze:", y.shape)  # Output: torch.Size([2, 3])

# Remove only the 0th dimension (size 1)
z = x.squeeze(0)
print("After squeezing dim 0:", z.shape)  # Output: torch.Size([2, 1, 3])

2.4 unsqueeze(): Add a Dimension of Size 1¶

unsqueeze(dim) inserts a new dimension of size 1 at the specified position (an “expansion” operation).
Use case: Fixing input dimensions for models (e.g., channel dimension for convolutional layers).

# Original shape: (2, 3)
x = torch.rand(2, 3)
print("Original shape:", x.shape)  # Output: torch.Size([2, 3])

# Insert dimension 1 at position 0 → (1, 2, 3)
y = x.unsqueeze(0)
print("After unsqueeze:", y.shape)  # Output: torch.Size([1, 2, 3])

# Insert dimension 1 at position 2 → (2, 3, 1)
z = x.unsqueeze(2)
print("After unsqueeze dim 2:", z.shape)  # Output: torch.Size([2, 3, 1])

2.5 transpose() and permute(): Swap Dimensions¶

• transpose(dim0, dim1): Swaps two dimensions and returns a new tensor.
• permute(dims): Swaps multiple dimensions (parameter is a tuple of dimension indices) for greater flexibility.

# Original shape: (2, 3)
x = torch.rand(2, 3)
print("Original shape:", x.shape)  # Output: torch.Size([2, 3])

# Swap dim 0 and dim 1 → (3, 2)
y = x.transpose(0, 1)
print("After transpose:", y.shape)  # Output: torch.Size([3, 2])

# Original shape: (1, 2, 3)
x = torch.rand(1, 2, 3)
# Permute dimensions to (2, 0, 1)
y = x.permute(2, 0, 1)
print("After permute:", y.shape)  # Output: torch.Size([3, 1, 2])

3. Common Tensor Operations: From Arithmetic to Aggregation¶

Beyond dimension transformations, tensors support essential operations for model computations:

3.1 Basic Arithmetic Operations¶

Element-wise operations (+, -, *, /) are supported, with shape unchanged unless broadcasting occurs.

x = torch.tensor([1, 2, 3])
y = torch.tensor([4, 5, 6])
print("x + y:", x + y)    # Output: tensor([5, 7, 9])
print("x * y:", x * y)    # Output: tensor([4, 10, 18])
print("x / y:", x / y)    # Output: tensor([0.2500, 0.4000, 0.5000])

3.2 Matrix Multiplication: matmul()¶

Matrix multiplication (not element-wise) is used in deep learning, implemented via matmul().

# 2x3 matrix × 3x4 matrix → 2x4 matrix
x = torch.rand(2, 3)
y = torch.rand(3, 4)
z = torch.matmul(x, y)
print("After matmul:", z.shape)  # Output: torch.Size([2, 4])

3.3 Broadcasting Mechanism¶

PyTorch automatically extends dimensions when two tensors have mismatched shapes but satisfy conditions (similar to NumPy broadcasting).
Rule: Align dimensions from right to left; dimensions of size 1 are expanded to match the other tensor.

# Shape (2,1) + Shape (1,3) → Auto-expand to (2,3)
a = torch.tensor([[1], [2]])  # Shape (2,1)
b = torch.tensor([[3, 4, 5]])  # Shape (1,3)
c = a + b
print("c shape:", c.shape)  # Output: torch.Size([2, 3])
print("c content:\n", c)
# Result:
# [[4, 5, 6],
#  [5, 6, 7]]

3.4 Aggregation Operations: sum(), mean(), max()¶

Tensors support summation, mean, and max/min operations along specific dimensions (critical for loss calculation).

x = torch.tensor([[1, 2], [3, 4]])
print("Total sum:", x.sum())          # Output: 10 (sum all elements)
print("Sum by row:", x.sum(dim=1))  # Output: tensor([3, 7]) (sum each row)
print("Mean:", x.mean())              # Output: 2.5 (mean of all elements)
print("Max by column:", x.max(dim=0))  # Output: (tensor([3, 4]), tensor([1, 1]))

4. Summary and Exercises¶

Tensors are the core of PyTorch, and mastering dimension transformations and operations is foundational for building models. Practice with these exercises:

1. Create a tensor of shape (2, 2, 1), use squeeze() to remove extra dimensions, then unsqueeze() to restore the original shape.
2. Test addition of tensors with different dimensions and observe how broadcasting automatically expands dimensions.
3. Use permute() to transform a tensor of shape (1, 2, 3) into (3, 2, 1).

By working through these, you’ll master tensor “shape language” and lay the groundwork for building neural networks and deep learning models!