How can I limit Parallel.ForEach?

c# .net asynchronous parallel.foreach
eugeneK picture eugeneK · Feb 15, 2012 · Viewed 162.3k times · Source

I have a Parallel.ForEach() async loop with which I download some webpages. My bandwidth is limited so I can download only x pages per time but Parallel.ForEach executes whole list of desired webpages.

Is there a way to limit thread number or any other limiter while running Parallel.ForEach?

Demo code:

Parallel.ForEach(listOfWebpages, webpage => {
  Download(webpage);
});

The real task has nothing to do with webpages, so creative web crawling solutions won't help.

Answer

Nicholas Butler picture Nicholas Butler · Feb 15, 2012

You can specify a MaxDegreeOfParallelism in a ParallelOptions parameter:

Parallel.ForEach(
    listOfWebpages,
    new ParallelOptions { MaxDegreeOfParallelism = 4 },
    webpage => { Download(webpage); }
);

MSDN: Parallel.ForEach

MSDN: ParallelOptions.MaxDegreeOfParallelism

Related questions

How and when to use ‘async’ and ‘await’

From my understanding one of the main things that async and await do is to make code easy to write and read - but is using them equal to spawning background threads to perform long duration logic? I'm currently trying …

Read More
Running multiple async tasks and waiting for them all to complete

I need to run multiple async tasks in a console application, and wait for them all to complete before further processing. There's many articles out there, but I seem to get more confused the more I read. I've read and …

Read More
Can't specify the 'async' modifier on the 'Main' method of a console app

I am new to asynchronous programming with the async modifier. I am trying to figure out how to make sure that my Main method of a console application actually runs asynchronously. class Program { static void Main(string[] args) { Bootstrapper bs = …

Read More