Everything in a session shares one context window: the instructions, every earlier prompt and reply (the transcript), tool descriptions, and the reply still to come. Apple's on-device model has about 4,096 tokens in all. When a session goes past it, Foundation Models throws an error for exceeding the context window instead of answering.

A token is a piece of a word. Without the model's own tokenizer, about four characters of English per token is a usable estimate; round up, so the estimate errs on the safe side.

import math

def estimate(text):
    return math.ceil(len(text) / 4)

print(estimate("Summarise this note."))

When a conversation grows, keep the instructions, reserve room for the reply, and drop the oldest turns first. The newest turns are what the user is talking about.

Write estimate_tokens(text) returning ceil(len(text) / 4). Write fit_transcript(instructions, turns, limit, reply_budget) returning the longest run of the newest turns (a list, oldest first, kept in order) such that the instructions, those turns and reply_budget together are at most limit; raise ValueError when the instructions and the reply alone do not fit.

The explanation opens here once every check passes.