MongoWaitQueueFullException: The wait queue for acquiring a connection to server is full

Question

Sometimes when inserting a small bunch of different document (synchronously), I get the following exception (see full stack trace further down):

MongoDB.Driver.MongoWaitQueueFullException: The wait queue for acquiring a connection to server xyz.mongolab.com:54128 is full.

I am using singleton MongoDatabase instance (and thus a single connection) between all my repositories. Essentially, I am doing something like this (with no more than 20 documents in each collection):

Context.Collection<ClientDocument>("clients").InsertMany(clients);
Context.Collection<VendorDocument>("vendors").InsertMany(vendors);
Context.Collection<SaleDocument>("sales").InsertOne(sale);

Below is the singleton context:

public class MongoContext
{
    public IMongoDatabase Database { get; }

    public MongoContext(IOptions<MongoSettings> settings)
    {
        var url = MongoUrl.Create(settings.Value.EndpointUri);

        var client = new MongoClient(new MongoClientSettings()
        {
            Server = url.Server
        });

        Database = client.GetDatabase(url.DatabaseName);
    }

    public IMongoCollection<TDocument> Collection<TDocument>(string collection)
        where TDocument : IDocument
    {
        return Database.GetCollection<TDocument>(collection);
    }
}

Something similar was filed on MongoDB's Jira (https://jira.mongodb.org/browse/CSHARP-1144) but these cases are dealing with huge bulk inserts (and often asynchronously).

I don't see the need to increase MaxConnectionPoolSize or WaitQueueSize with such small inserts.

What could be the cause of this?

I am using MongoDB 3.0.7 hosted in mLabs. Our application is hosted in Azure (as a Web App) and I am using the C# 2.2.3 SDK.

MongoDB.Driver.MongoWaitQueueFullException: The wait queue for acquiring a connection to server xyz.mongolab.com:54128 is full. at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.CheckingOutConnection() at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnection(CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.ClusterableServer.GetChannel(CancellationToken cancellationToken) at MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken cancellationToken) at MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken) at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation'1 operation, CancellationToken cancellationToken) at MongoDB.Driver.MongoCollectionImpl'1.ExecuteWriteOperation[TResult](IWriteOperation`1 operation, CancellationToken cancellationToken) at MongoDB.Driver.MongoCollectionImpl'1.BulkWrite(IEnumerable'1 requests, BulkWriteOptions options, CancellationToken cancellationToken) at MongoDB.Driver.MongoCollectionBase'1.InsertOne(TDocument document, InsertOneOptions options, CancellationToken cancellationToken)

EDIT:

If I set MaxConnectionPoolSize to 500 and WaitQueueSize to 2000, then I get the following exception:

MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions 191.235.xxx.xxx:54128

Instantiating MongoClient:

var client = new MongoClient(new MongoClientSettings()
{
    Server = url.Server,
    Credentials = credentials,
    MaxConnectionPoolSize = 500,
    WaitQueueSize = 2000
});

I raised this problem initially here. This led to me trying to figure out why on earth I have so many connections. This led to this post (questioning if Insert/InsertBulk could be a cause). Regardless, I still need to fix the original MongoWaitQueueFullException problem.

Answer 1

A long term way to solve your problem might be to introduce a throttling mechanism, to ensure you aren't exceeding your max number of connections. Fortunately, with a Semaphore this is pretty easy to implement.

public class ConnectionThrottlingPipeline : IConnectionThrottlingPipeline
{
    private readonly Semaphore openConnectionSemaphore;

    public ConnectionThrottlingPipeline( IMongoClient client )
    {
        //Only grabbing half the available connections to hedge against collisions.
        //If you send every operation through here
        //you should be able to use the entire connection pool.
        openConnectionSemaphore = new Semaphore( client.Settings.MaxConnectionPoolSize / 2,
            client.Settings.MaxConnectionPoolSize / 2 );
    }

    public async Task<T> AddRequest<T>( Task<T> task )
    {
        openConnectionSemaphore.WaitOne();
        try
        {
            var result = await task;
            return result;
        }
        finally
        {
            openConnectionSemaphore.Release();
        }
    }
}

If you send all your requests to the database through this throttling pipeline you should never hit the limit.

In your case sending the operations through the pipeline might look like this (the big change you'd have to make would be making your database calls asynchronous) :

await connectionThrottlingPipeline.AddRequest( 
   Context.Collection<ClientDocument>("clients").InsertManyAsync(clients))