How to log progress of tasks in Talend Open Studio?
I have some sample jobs that migrate data from one database to another and I would like to have some information about the current progress, like the one you have when the job is run interactively from the application itself (I export and run it from command line). I use flowMeter and statsCatcher but everything i got is the overall time and the overall number of records passed (e.g. 4657 sec, 50.000.000 rows). Is there any solution to get a decent log ?
Answer
Your solution is about adding a conditional clause to logging. Something true one row every, let's say, 50000. This condition using a sequence should work:
Numeric.sequence("log_seq",1,1) % 50000 == 0
You can use the custom component bcLogBack to basically output your log using an sl4j facade stack. The component has an option called "Conditional logging" to send the message only when the condition evaluate to true.
Alternatively, if you don't like the idea of install a custom component, you can end your subjob using the standard tLogRow (or tWarn, tDie or whatever) prefixed by a tFilter with the same expression as advanced condition. This way you'll let the stream pass (and the log message to be triggered) just one time every 50000. Here's a very basic job diagram
//---->tMySqlOutput--->tFilter-----//filter--->tWarn (or tLogRow)