How would I split a stream in Apache Storm?

james picture james · Nov 6, 2013 · Viewed 11.6k times · Source

I am not understanding how I would split a stream in Apache Storm. For example, I have bolt A that after some computation has somevalue1, somevalue2, and somevalue3. It wants to send somevalue1 to bolt B, somevalue2 to bolt C, and somevalue1,somevalue2 to bolt D. How would I do this in Storm? What grouping would I use and what would my topology look like? Thank you in advance for your help.

Answer

zenbeni picture zenbeni · Nov 8, 2013

You can use different streams if your case needs that, it is not really splitting, but you will have a lot of flexibility, you could use it for content based routing from a bolt for instance:

You declare the stream in the bolt:

@Override
public void declareOutputFields(final OutputFieldsDeclarer outputFieldsDeclarer) {
    outputFieldsDeclarer.declareStream("stream1", new Fields("field1"));
    outputFieldsDeclarer.declareStream("stream2", new Fields("field1"));
}

You emit from the bolt on the chosen stream:

collector.emit("stream1", new Values("field1Value"));

You listen to the correct stream through the topology

builder.setBolt("myBolt1", new MyBolt1()).shuffleGrouping("boltWithStreams", "stream1");
builder.setBolt("myBolt2", new MyBolt2()).shuffleGrouping("boltWithStreams", "stream2");