Python Queues

Thanks to my job, I found myself needing to set up a way to have tasks go into and out of a Python script.  Especially when I’m trying to make something that we’ll be sending tasks to over a longer period, I need to have something a bit more solid.

Python queues work pretty simply.  You create a queue, then have a worker pull tasks off the top of the queue.  The worker works the tasks, then notifies the queue the task is complete, and the tasks pulls another queue.  At any point in this process, you can add more tasks.

Python queues are documented elsewhere, but I feel like a better explanation can help.

from Queue import Queue
from threading import Thread

This will import the queueing and threading libraries.  Queues are designed to work with multiple workers, and blocking the main thread (which contains the queue) means that we can’t effectively add things.

def worker(q):
  while True:
    print q.get()
    q.task_done()

This is the simplest worker we can make – it literally gets the oldest active task (queues are FIFO – first in, first out) and prints its contents, marks it done, and repeats.

q = Queue(maxsize=0)
num_threads = 10

Here, we are telling the Queue to accept any number of tasks, and setting our number of threads to 10, meaning we can have up to 10 workers.

for i in range(num_threads):
  worker = Thread(target=worker, args=(q,))
  worker.setDaemon(True)
  worker.start()

This will spin up the appropriate number of worker threads – it literally just loops through and creates 10 worker threads, oddly simple.  We call a new Thread on each one, creating a new worker and passing in the queue object “q”, and tell it to start.

Since the workers will loop on their own, we don’t need to do anything else.

for x in range(100):
  q.put(x)

In our simple example, this will allow us to put the numbers 1-100 into our queue “q”, and the workers will pull them and run them!

All told, this is a really, really simple script, but it can unlock so many things.

Leave a Reply

Your email address will not be published. Required fields are marked *