Sequences of different lengths
Real sequences have different lengths, but a batch must be one tensor, so shorter sequences are padded with zeros at the end. A recurrent network would then read the padding too, and its last state would describe zeros.
import torch
from torch import nn
from torch.nn.utils.rnn import (
pad_sequence, pack_padded_sequence)
seqs = [torch.ones(3, 1), torch.ones(5, 1)]
x = pad_sequence(seqs, batch_first=True)
lengths = torch.tensor([3, 5])
lstm = nn.LSTM(1, 4, batch_first=True)
packed = pack_padded_sequence(
x, lengths, batch_first=True,
enforce_sorted=False)
out, (h, c) = lstm(packed)
print(x.shape, h.shape)pack_padded_sequence tells the network each length, so every sequence stops at its own end and h holds the state after its last real step. enforce_sorted=False lets the batch come in any order.
An LSTM carries two states: h, its output, and c, a cell that gates decide to keep, overwrite or forget.
Your task
Write pad_batch(seqs) taking a list of (length, features) tensors and returning (x, lengths): the zero-padded batch (batch, longest, features) and a long tensor of lengths. Write LastValue(features, hidden), an nn.Module with self.lstm = nn.LSTM(features, hidden, batch_first=True) and self.head = nn.Linear(hidden, 1), whose forward(x, lengths) packs the batch and returns the head applied to h[-1], shape (batch, 1).
The explanation opens here once every check passes.