PythiloLesson 4 of 30 · progress kept in this browser

Bytes, str and Unicode

A str is a sequence of Unicode code points. A bytes object is a sequence of numbers from 0 to 255. Files, networks and hashes work in bytes, so text is encoded on the way out and decoded on the way in.

s = "café"
b = s.encode("utf-8")
print(len(s), len(b))  # 4 5
print(b)               # b'caf\xc3\xa9'
print(b[0], list(b[3:]))
print(b.decode("utf-8"), ord("é"))

UTF-8 uses one byte for ASCII and up to four for other characters, so the length in characters and the length in bytes differ. Decoding with the wrong codec raises UnicodeDecodeError or, worse, quietly produces the wrong text. errors="replace" puts the marker U+FFFD in place of bad bytes.

Your task

Write utf8_size(text) returning how many bytes text takes in UTF-8. Write safe_decode(data) decoding UTF-8 bytes with each invalid byte replaced by U+FFFD instead of raising. Write same_text(a, b) telling whether two strings match after NFC normalization and case folding.

Next: Classes