How can I make `await …` work with `yield return` (i.e. inside an iterator method)?

Question

I have existing code that looks similar to:

IEnumerable<SomeClass> GetStuff()
{
    using (SqlConnection conn = new SqlConnection(connectionString))
    using (SqlCommand cmd = new SqlCommand(sql, conn)
    {
        conn.Open();
        SqlDataReader reader = cmd.ExecuteReader();
        while (reader.Read())
        {
            SomeClass someClass = f(reader); // create instance based on returned row
            yield return someClass;
        }
    } 
}

It seems I could benefit by using reader.ReadAsync(). However, if I just modify the one line:

        while (await reader.ReadAsync())

the compiler informs me that await can only be used in methods marked with async, and suggests I modify the method signature to be:

async Task<IEnumerable<SomeClass>> GetStuff()

However, doing that makes GetStuff() unusable because:

The body of GetStuff() cannot be an iterator block because Task<IEnumerable<SomeClass>> is not an iterator interface type.

I'm sure I am missing a key concept with the async programming model.

Questions:

• Can I use ReadAsync() in my iterator? How?
• How can I think about the async paradigm differently so that I understand how it works in this type of situation?

Answer 1

The problem is what you're asking doesn't actually make much sense. IEnumerable<T> is a synchronous interface, and returning Task<IEnumerable<T>> isn't going to help you much, because some thread would have to block waiting for each item, no matter what.

What you actually want to return is some asynchronous alternative to IEnumerable<T>: something like IObservable<T>, dataflow block from TPL Dataflow or IAsyncEnumerable<T>, which is planned to be added to C# 8.0/.Net Core 3.0. (And in the meantime, there are some libraries that contain it.)

Using TPL Dataflow, one way to do this would be:

ISourceBlock<SomeClass> GetStuff() {
    var block = new BufferBlock<SomeClass>();

    Task.Run(async () =>
    {
        using (SqlConnection conn = new SqlConnection(connectionString))
        using (SqlCommand cmd = new SqlCommand(sql, conn))
        {
            await conn.OpenAsync();
            SqlDataReader reader = await cmd.ExecuteReaderAsync();
            while (await reader.ReadAsync())
            {
                SomeClass someClass;
                // Create an instance of SomeClass based on row returned.
                block.Post(someClass);
            }
            block.Complete();
        } 
    });

    return block;
}

You'll probably want to add error handling to the above code, but otherwise, it should work and it will be completely asynchronous.

The rest of your code would then consume items from the returned block also asynchronously, probably using ActionBlock.