Queue.Queue vs. collections.deque

python thread-safety queue
miracle2k picture miracle2k · Apr 4, 2009 · Viewed 66.2k times · Source

I need a queue which multiple threads can put stuff into, and multiple threads may read from.

Python has at least two queue classes, Queue.Queue and collections.deque, with the former seemingly using the latter internally. Both claim to be thread-safe in the documentation.

However, the Queue docs also state:

collections.deque is an alternative implementation of unbounded queues with fast atomic append() and popleft() operations that do not require locking.

Which I guess I don't quite unterstand: Does this mean deque isn't fully thread-safe after all?

If it is, I may not fully understand the difference between the two classes. I can see that Queue adds blocking functionality. On the other hand, it loses some deque features like support for the in-operator.

Accessing the internal deque object directly, is

x in Queue().deque

thread-safe?

Also, why does Queue employ a mutex for it's operations when deque is thread-safe already?

Answer

Keith Gaughan picture Keith Gaughan · Apr 4, 2009

Queue.Queue and collections.deque serve different purposes. Queue.Queue is intended for allowing different threads to communicate using queued messages/data, whereas collections.deque is simply intended as a datastructure. That's why Queue.Queue has methods like put_nowait(), get_nowait(), and join(), whereas collections.deque doesn't. Queue.Queue isn't intended to be used as a collection, which is why it lacks the likes of the in operator.

It boils down to this: if you have multiple threads and you want them to be able to communicate without the need for locks, you're looking for Queue.Queue; if you just want a queue or a double-ended queue as a datastructure, use collections.deque.

Finally, accessing and manipulating the internal deque of a Queue.Queue is playing with fire - you really don't want to be doing that.

Related questions

Thread Safety in Python's dictionary

I have a class which holds a dictionary class OrderBook: orders = {'Restaurant1': None, 'Restaurant2': None, 'Restaurant3': None, 'Restaurant4': None} @staticmethod def addOrder(restaurant_name, orders): OrderBook.orders[restaurant_name] = orders And I am running 4 threads (one …

Read More
Are global variables thread-safe in Flask? How do I share data between requests?

In my application, the state of a common object is changed by making requests, and the response depends on the state. class SomeObj(): def __init__(self, param): self.param = param def query(self): self.param += 1 return self.param global_obj = …

Read More
Are Python built-in containers thread-safe?

I would like to know if the Python built-in containers (list, vector, set...) are thread-safe? Or do I need to implement a locking/unlocking environment for my shared variable?

Read More