PythiloLesson 4 of 12 · progress kept in this browser

Queues for producers and consumers

A queue.Queue is a thread-safe pipe. Producers put items in, consumers get them out, and the queue does all the locking. get() blocks until an item arrives, so idle workers simply wait.

import queue
import threading

jobs = queue.Queue()
done = []

def worker():
    while True:
        item = jobs.get()
        if item is None:  # the stop signal
            break
        done.append(item * 10)

t = threading.Thread(target=worker)
t.start()
for n in [1, 2, 3]:
    jobs.put(n)
jobs.put(None)
t.join()
print(done)  # [10, 20, 30]

A worker cannot tell "nothing yet" from "nothing ever", so the producer sends a sentinel, here None, when it has finished. With several workers, send one sentinel per worker: each sentinel stops exactly one of them.

Queue(maxsize=10) makes put block while the queue is full, which keeps a fast producer from running far ahead of slow consumers.

Your task

Write process(items, handle, workers=3): start workers threads that take items from a queue.Queue and call handle(item). Return the results in the order of items, and make sure every worker thread has stopped before you return.

This lesson uses threads, which Python in this browser does not include. Read it here, and run and check it in the Pythilo app on iPhone or iPad.