PythiloLesson 16 of 18 · progress kept in this browser

Recurrent networks

A recurrent network reads a sequence one step at a time and carries a hidden state forward, a summary of everything it has seen so far. The same weights handle every step, so one model works for sequences of any length.

import torch
from torch import nn

gru = nn.GRU(input_size=1, hidden_size=8,
             batch_first=True)
# 4 sequences, 10 steps, 1 feature each
x = torch.randn(4, 10, 1)
out, h = gru(x)
print(out.shape)  # state after each step
print(h.shape)  # only the last state

With batch_first=True the input is (batch, steps, features); without it PyTorch expects the steps first. out holds the hidden state after every step and h only the last one, shaped (layers, batch, hidden), so h[-1] is one summary per sequence. A linear layer on it classifies the whole sequence.

nn.GRU and nn.LSTM have gates that decide what to keep and what to forget, which lets them remember over far more steps than a plain nn.RNN.

Your task

Write sign_data(n, length, seed) returning (x, y): after torch.manual_seed(seed), n sequences of shape (length, 1) uniform from -1 to 1, labelled 1 (long) when their sum is positive, else 0. Write SignClassifier(nn.Module), an nn.GRU(1, 16) with an nn.Linear(16, 2) on its last hidden state, taking (batch, steps, 1) input, and train_rnn(x, y, steps=100) returning one trained with Adam at lr=0.03 and cross-entropy.

This lesson uses torch, which Python in this browser does not include. Read it here, and run and check it in the Pythilo app on iPhone or iPad.

Next: Attention