How does cassandra find the node that contains the data?

Question

I've read quite a few articles and a lot of question/answers on SO about Cassandra but I still can't figure out how Cassandra decides which node(s) to go to when it's reading the data.

First, some assumptions about an imaginary cluster:

1. Replication Strategy = simple
2. Using Random Partitioner
3. Cluster of 10 nodes
4. Replication Factor of 5

Here's my understanding of how writes work based on various Datastax articles and other blog posts I've read:

• Client sends the data to a random node
• The "random" node is decided based on the MD5 hash of the primary key.

• Data is written to the commit_log and memtable and then propagated 4 times (with RF = 5).

• The 4 next nodes in the ring are then selected and data is persisted in them.

So far, so good.

Now the question is, when the client sends a read request (say with CL = 3) to the cluster, how does Cassandra know which nodes (5 out of 10 as the worst case scenario) it needs to contact to get this data? Surely it's not going to all 10 nodes as that would be inefficient.

Am I correct in assuming that Cassandra will again, do an MD5 hash of the primary key (of the request) and choose the node according to that and then walks the ring?

Also, how does the network topology case work? if I have multiple data centers, how does Cassandra know which nodes in each DC/Rack contain the data? From what I understand, only the first node is obvious (since the hash of the primary key has resulted in that node explicitly).

Sorry if the question is not very clear and please add a comment if you need more details about my question.

Many thanks,

Answer 1

Client sends the data to a random node

It might seem that way, but there is actually a non-random way that your driver picks a node to talk to. This node is called a "coordinator node" and is typically chosen based-on having the least (closest) "network distance." Client requests can really be sent to any node, and at first they will be sent to the nodes which your driver knows about. But once it connects and understands the topology of your cluster, it may change to a "closer" coordinator.

The nodes in your cluster exchange topology information with each other using the Gossip Protocol. The gossiper runs every second, and ensures that all nodes are kept current with data from whichever Snitch you have configured. The snitch keeps track of which data centers and racks each node belongs to.

In this way, the coordinator node also has data about which nodes are responsible for each token range. You can see this information by running a nodetool ring from the command line. Although if you are using vnodes, that will be trickier to ascertain, as data on all 256 (default) virtual nodes will quickly flash by on the screen.

So let's say that I have a table that I'm using to keep track of ship crew members by their first name, and let's assume that I want to look-up Malcolm Reynolds. Running this query:

SELECT token(firstname),firstname, id, lastname 
FROM usersbyfirstname  WHERE firstname='Mal';

...returns this row:

 token(firstname)     | firstname | id | lastname
----------------------+-----------+----+-----------
  4016264465811926804 |       Mal |  2 |  Reynolds

By running a nodetool ring I can see which node is responsible for this token:

192.168.1.22  rack1       Up     Normal  348.31 KB   3976595151390728557                         
192.168.1.22  rack1       Up     Normal  348.31 KB   4142666302960897745                         

Or even easier, I can use nodetool getendpoints to see this data:

$ nodetool getendpoints stackoverflow usersbyfirstname Mal
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
192.168.1.22

For more information, check out some of the items linked above, or try running nodetool gossipinfo.