What is the reason for two separate but similar servers? What are the differences? Can I run both in docker? Do both support same things, like all authentication types?
Kestrel vs HTTP.sys - I've highlighted the fundamental differences below.
(The words are Microsoft's and I've only edited it for brevity and clarity. See the sources linked at the bottom).
Kestrel previously always required the use of a reverse proxy with edge deployments (exposed to traffic from the Internet) for security reasons. With Kestrel in ASP.Net Core 2.x this is no longer the case. Take a look at the documentation for more information. Kestrel Web Server Documentation
Weblistener was renamed HTTP.sys in ASP.NET Core 2.0
HTTP.sys is windows-only HTTP / Web Server for ASP.NET Core that allows you to expose the server directly to the Internet without needing to use IIS. HTTP.sys is built on top of Http.Sys ( the same mature technology that also powers IIS' HTTP Listener) as is as such very feature rich and provides protection against various attacks.
Kestrel on the other hand, is a cross-platform web server for ASP.NET Core that is designed to be run behind a proxy (for example IIS or Nginx) and should not be deployed directly facing the Internet. Kestrel is relatively new and does not have a full complement of defenses against attacks. It's also not as feature rich as HTTP.sys and comes with timeout limits, size limits and concurrent user limits.
In essence, the choice comes down to your web application's Deployment scenario.