Difference between spring batch remote chunking and remote partitioning
What is the difference between spring batch remote chunking and remote partitioning?
I can not understand the difference between remote chunking and remote partitioning in spring batch. Could anybody please explain?
Answer
Remote Partitioning
Partitioning is a master/slave step configuration that allows for partitions of data to be processed in parallel. Each partition is described via some metadata. For example, if you were processing a database table, partition 1 may be ids 0-100, partition 2 being 101-200, etc. For Spring Batch, a master step uses a Partitioner to generate ExecutionContexts that contain the metadata for each partition. These ExecutionContexts are distributed to slave step for processing by a PartitionHandler (for remote partitioning, the MessageChannelPartitionHandler is typically used). The slaves execute their step and return the resulting statuses for aggregation by the master.
Things to note about remote partitioning:
- Input and output are local to the slaves. For example, if the input is a file, the slaves need access to the file.
- Slaves need access to the JobRepository. Slaves are fully defined Spring Batch steps and so they need JobRepository access.
Remote Chunking
Remote chunking is similar to remote partitioning in that it is a master/slave configuration. However with remote chunking, the data is read at by the master and sent over the wire to the slave for processing. Once the processing is done, the result of the ItemProcessor is returned to the master for writing.
Things to note about remote chunking:
- All I/O is done by the master.
- The slaves handle processing only and therefore do not need JobRepository access.
- Remote chunking is more I/O intensive than remote partitioning since the actual data is sent over the wire instead of metadata describing it.
I did a talk on scaling Spring Batch and do a demonstration of remote partitioning that you can watch here: http://www.youtube.com/watch?v=CYTj5YT7CZU