'and' (boolean) vs '&' (bitwise) - Why difference in behavior with lists vs numpy arrays?

Question

What explains the difference in behavior of boolean and bitwise operations on lists vs NumPy arrays?

I'm confused about the appropriate use of & vs and in Python, illustrated in the following examples.

mylist1 = [True,  True,  True, False,  True]
mylist2 = [False, True, False,  True, False]

>>> len(mylist1) == len(mylist2)
True

# ---- Example 1 ----
>>> mylist1 and mylist2
[False, True, False, True, False]
# I would have expected [False, True, False, False, False]

# ---- Example 2 ----
>>> mylist1 & mylist2
TypeError: unsupported operand type(s) for &: 'list' and 'list'
# Why not just like example 1?

>>> import numpy as np

# ---- Example 3 ----
>>> np.array(mylist1) and np.array(mylist2)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
# Why not just like Example 4?

# ---- Example 4 ----
>>> np.array(mylist1) & np.array(mylist2)
array([False,  True, False, False, False], dtype=bool)
# This is the output I was expecting!

This answer and this answer helped me understand that and is a boolean operation but & is a bitwise operation.

I read about bitwise operations to better understand the concept, but I am struggling to use that information to make sense of my above 4 examples.

Example 4 led me to my desired output, so that is fine, but I am still confused about when/how/why I should use and vs &. Why do lists and NumPy arrays behave differently with these operators?

Can anyone help me understand the difference between boolean and bitwise operations to explain why they handle lists and NumPy arrays differently?

Answer 1

and tests whether both expressions are logically True while & (when used with True/False values) tests if both are True.

In Python, empty built-in objects are typically treated as logically False while non-empty built-ins are logically True. This facilitates the common use case where you want to do something if a list is empty and something else if the list is not. Note that this means that the list [False] is logically True:

>>> if [False]:
...    print 'True'
...
True

So in Example 1, the first list is non-empty and therefore logically True, so the truth value of the and is the same as that of the second list. (In our case, the second list is non-empty and therefore logically True, but identifying that would require an unnecessary step of calculation.)

For example 2, lists cannot meaningfully be combined in a bitwise fashion because they can contain arbitrary unlike elements. Things that can be combined bitwise include: Trues and Falses, integers.

NumPy objects, by contrast, support vectorized calculations. That is, they let you perform the same operations on multiple pieces of data.

Example 3 fails because NumPy arrays (of length > 1) have no truth value as this prevents vector-based logic confusion.

Example 4 is simply a vectorized bit and operation.

Bottom Line

• If you are not dealing with arrays and are not performing math manipulations of integers, you probably want and.

• If you have vectors of truth values that you wish to combine, use numpy with &.