gRPC: What is the recommended way to shut down an asynchronous server in C++?

Question

I have a gRPC server that hosts two asynchronous services ("Master" and "Worker"), and I would like to implement graceful shutdown for the server. Each service has its own grpc::CompletionQueue.

There appear to be two Shutdown() methods that might be relevant: grpc::CompletionQueue::Shutdown() and grpc::Server::Shutdown(), but it's not clear from the documentation which ones should be used.

What is a good pattern for shutting down an asynchronous service?

Answer 1

TL;DR: You must call both grpc::Server::Shutdown() and grpc::CompletionQueue::Shutdown() (for each completion queue used in the service) to shut down cleanly.

1. If you call cq_->Shutdown(), the only observable effect is that subsequent calls to Service::AsyncService::RequestFoo() (the generated method for the corresponding Foo RPC) fail with an assertion. From reading the documentation of the corresponding C API method (grpc_completion_queue_shutdown()), it appears that it is illegal to add new work to the queue—i.e. by calling RequestFoo()—so I added an is_shutdown_ member to my service wrapper classes (protected by a mutex) so that no enqueue attempts are made after cq_->Shutdown() is called. However, after doing this, the completion queue blocks indefinitely in cq_->Next(). None of the enqueued tags complete (with an error or otherwise).

2. If instead you call server_->Shutdown(), all of the enqueued tags complete immediately (with ok == false). However, the completion queue continues to block indefinitely in cq_->Next().

Calling both cq_->Shutdown() (for each defined completion queue) and server_->Shutdown() results in a clean shutdown.

One caveat: if you use grpc::ServerContext::AsyncNotifyWhenDone() to register a tag for call cancellation, these will not be returned by cq_->Next() if the server shuts down before the initial request is received for that call. You will need to be cautious with the memory management of the corresponding tag structure, if you want to avoid memory leaks.