How to do loop in pentaho for getting file names?

pentaho kettle
Cynosure picture Cynosure · Dec 7, 2012 · Viewed 15.3k times · Source

I have 100 000 files. I want to get the name of those file names and have to put in database, I have to do like this

  1. get 10 files name's;
  2. update/insert names into database; and
  3. move those 10 files to another directory; and loop these three steps till no files are found.

Is this possible?

Answer

Milos picture Milos · Dec 10, 2012

I'm attaching a working example (I tested it with ~400 text files on kettle 4.3.).

transformation.ktr job.kjb

Both transformation and job contain detailed notes on what to set and where.

  • Transformation.ktr It reads first 10 filenames from given source folder, creates destination filepath for file moving. It outputs filenames to insert/update (I used dummy step as a placeholder) and uses "Copy rows to resultset" to output needed source and destination paths for file moving.
  • job.kjb All the looping is done in this job. It executes "transformation.ktr" (which does insert/update for 10 files), and then moves those 10 files to destination folder. After that, it checks whether there's any more files in source folder. If there is, process is repeated, if not, it declares success.

Related questions

Pentaho: How to dynamically add Field (= Column) to OutputRow?

I would like to dynamically add fields (or a new columns) to the resulting output row in Kettle. After spending hours reading through froum posts and he not so well done scripting documentation, I wondered if Stackoverflow would be of …

Read More
Use JSON Input step to process uneven data

I'm trying to process the following with an JSON Input step: {"address":[ {"AddressId":"1_1","Street":"A Street"}, {"AddressId":"1_101","Street":"Another Street"}, {"AddressId":"1_102","Street":"One more street", "Locality":"Buenos Aires"}, {"AddressId":"1_102","Locality":"New York"} ]} However this seems not to be possible: Json …

Read More
Pentaho Kettle: how to pass variable from transformation to another transformation inside job

I have two transformations in the job. In the first trasnformation - I get details about the file. Now I would like to pass this information to the second transformation, I have set variable in the settings parameters of the …

Read More