:class:~tornado.queues.Queue example - a concurrent web spider

.. currentmodule:: tornado.queues

Tornado's tornado.queues module (and the very similar Queue classes in asyncio) implements an asynchronous producer / consumer pattern for coroutines, analogous to the pattern implemented for threads by the Python standard library's queue module.

A coroutine that yields Queue.get pauses until there is an item in the queue. If the queue has a maximum size set, a coroutine that yields Queue.put pauses until there is room for another item.

A ~Queue maintains a count of unfinished tasks, which begins at zero. ~Queue.put increments the count; ~Queue.task_done decrements it.

In the web-spider example here, the queue begins containing only base_url. When a worker fetches a page it parses the links and puts new ones in the queue, then calls ~Queue.task_done to decrement the counter once. Eventually, a worker fetches a page whose URLs have all been seen before, and there is also no work left in the queue. Thus that worker's call to ~Queue.task_done decrements the counter to zero. The main coroutine, which is waiting for ~Queue.join, is unpaused and finishes.

.. literalinclude:: ../../demos/webspider/webspider.py