PythiloLesson 24 of 30 · progress kept in this browser

Sequences of different lengths

Real sequences have different lengths, but a batch must be one tensor, so shorter sequences are padded with zeros at the end. A recurrent network would then read the padding too, and its last state would describe zeros.

import torch
from torch import nn
from torch.nn.utils.rnn import (
    pad_sequence, pack_padded_sequence)

seqs = [torch.ones(3, 1), torch.ones(5, 1)]
x = pad_sequence(seqs, batch_first=True)
lengths = torch.tensor([3, 5])
lstm = nn.LSTM(1, 4, batch_first=True)
packed = pack_padded_sequence(
    x, lengths, batch_first=True,
    enforce_sorted=False)
out, (h, c) = lstm(packed)
print(x.shape, h.shape)

pack_padded_sequence tells the network each length, so every sequence stops at its own end and h holds the state after its last real step. enforce_sorted=False lets the batch come in any order.

An LSTM carries two states: h, its output, and c, a cell that gates decide to keep, overwrite or forget.

Your task

Write pad_batch(seqs) taking a list of (length, features) tensors and returning (x, lengths): the zero-padded batch (batch, longest, features) and a long tensor of lengths. Write LastValue(features, hidden), an nn.Module with self.lstm = nn.LSTM(features, hidden, batch_first=True) and self.head = nn.Linear(hidden, 1), whose forward(x, lengths) packs the batch and returns the head applied to h[-1], shape (batch, 1).

This lesson uses torch, which Python in this browser does not include. Read it here, and run and check it in the Pythilo app on iPhone or iPad.

Next: Attention