Why am I getting a generic constraint violation at runtime?

Question

I'm getting the following exception while trying to create a new instance of a class that heavily relies on generics:

new TestServer(8888);

System.TypeLoadException

GenericArguments[0], 'TOutPacket', on     
'Library.Net.Relay`4[TInPacket,TOutPacket,TCryptograph,TEndian]' 
violates the constraint of type parameter 'TInPacket'.

at System.RuntimeTypeHandle.Instantiate(RuntimeTypeHandle handle, IntPtr* pInst, Int32 numGenericArgs, ObjectHandleOnStack type)
at System.RuntimeTypeHandle.Instantiate(Type[] inst)
at System.RuntimeType.MakeGenericType(Type[] instantiation)

I'm puzzled as to why this happens. Aren't generic constraints checked at compile time?

My googling brought me to the conclusion that this has something to do with either of these causes, or (sometimes?) both:

• The order in which the generic constraints (where) are defined in the classes;
• The use of the self-referencing generic pattern (conter-intuitive but very legal, see Eric Lippert's blog post)

One thing I am not ready to sacrifice is the self-referencing pattern. I absolutely need it for a specific purpose.

However, I'd like some help to point out where and why this problem occurs. As the library is massive and makes huge generic patterns, I think it would be best to progressively give code bits on request.

Upon request, declarations again. But I'd like to stress the fact that I would rather know generally why an exception like this can occur and then proceed to fix it myself in my specific code rather than find a specific fix, for posterity. Also, it will be much longer for anyone analyzing the code to answer than to give a general explanation as to why generic type constraints can be violated at runtime.

Implementation declarations:

class TestServer : Server<TestServer, TestClient, ServerPacket.In, ServerPacket.Out, BlankCryptograph, LittleEndianBitConverter>

class TestClient : AwareClient<TestOperationCode, TestServer, TestClient, ServerPacket.In, ServerPacket.Out, BlankCryptograph, LittleEndianBitConverter>

class ServerPacket
{
    public abstract class In : AwarePacket<TestOperationCode, TestServer, TestClient, ServerPacket.In, ServerPacket.Out, BlankCryptograph, LittleEndianBitConverter>.In
    public class Out : OperationPacket<TestOperationCode, LittleEndianBitConverter>.Out
}

public enum TestOperationCode : byte

Library declarations:

public abstract class Server<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian> : IDisposable
    where TServer : Server<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TClient : Client<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TInPacket : Packet<TEndian>.In
    where TOutPacket : Packet<TEndian>.Out
    where TCryptograph : Cryptograph, new()
    where TEndian : EndianBitConverter, new()

public abstract class Relay<TInPacket, TOutPacket, TCryptograph, TEndian> : IDisposable
    where TInPacket : Packet<TEndian>.In
    where TOutPacket : Packet<TEndian>.Out
    where TCryptograph : Cryptograph, new()
    where TEndian : EndianBitConverter, new()

public abstract class Client<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian> : Relay<TInPacket, TOutPacket, TCryptograph, TEndian>, IDisposable
    where TServer : Server<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TClient : Client<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TInPacket : Packet<TEndian>.In
    where TOutPacket : Packet<TEndian>.Out
    where TCryptograph : Cryptograph, new()
    where TEndian : EndianBitConverter, new()

public abstract class Packet<TEndian> : ByteBuffer<TEndian>, IDisposable 
    where TEndian : EndianBitConverter, new()
{
    public abstract class In : Packet<TEndian>
    public abstract class Out : Packet<TEndian>
}

public class OperationPacket<TOperationCode, TEndian> 
    where TEndian : EndianBitConverter, new()
{
    public class In : Packet<TEndian>.In
    public class Out : Packet<TEndian>.Out
}

public abstract class AwareClient<TOperationCode, TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian> : Client<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>, IDisposable
    where TCryptograph : Cryptograph, new()
    where TInPacket : AwarePacket<TOperationCode, TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>.In
    where TOutPacket : Packet<TEndian>.Out
    where TServer : Server<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TClient : AwareClient<TOperationCode, TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TEndian : EndianBitConverter, new()

public class AwarePacket<TOperationCode, TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TCryptograph : Cryptograph, new()
    where TInPacket : AwarePacket<TOperationCode, TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>.In
    where TOutPacket : Packet<TEndian>.Out
    where TServer : Server<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TClient : AwareClient<TOperationCode, TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
    where TEndian : EndianBitConverter, new()
{
    public abstract class In : OperationPacket<TOperationCode, TEndian>.In
}

As noted in the comments, the simplest way to get help on this question for me would be to minimize the code to a small and reproducible example in which the bug is still present. However, this is both hard and long for me, and has the high chances of making the bug a heisenbug, as it occurs from complexity.

I tried to isolate it to the following, but I don't get the bug when I do:

// Equivalent of library
class A<TA, TB, TI, TO> // Client
    where TA : A<TA, TB, TI, TO>
    where TB : B<TA, TB, TI, TO>
    where TI : I
    where TO : O
{ }

class B<TA, TB, TI, TO> // Server
    where TA : A<TA, TB, TI, TO>
    where TB : B<TA, TB, TI, TO>
    where TI : I
    where TO : O
{ }

class I { } // Input packet

class O { } // Output packet

// Equivalent of Aware

class Ii<TA, TB, TI, TO> : I { } // Aware input packet

class Ai<TA, TB, TI, TO> : A<TA, TB, TI, TO> // Aware capable client
    where TA : Ai<TA, TB, TI, TO>
    where TB : B<TA, TB, TI, TO>
    where TI : Ii<TA, TB, TI, TO>
    where TO : O
{ }

// Equivalent of implementation

class XI : Ii<XA, XB, XI, XO> { }
class XO : O { }

class XA : Ai<XA, XB, XI, XO> { }
class XB : B<XA, XB, XI, XO> { }

class Program
{
    static void Main(string[] args)
    {
        new XB(); // Works, so bad isolation
    }
}

Gory Details

1. Analyzing the exception tells us that TOutPacket violates TInPacket on Relay<TInPacket, TOutPacket, TCryptograph, Tendian>.
2. The instance of Relay we have is TestClient, which implements AwareClient, which implements Client, which implements Relay.
   • AwareClient is used in conjunction with AwarePacket so that both ends are aware of which type of client receives which type of packets.
3. Therefore, we know that TOutPacket in TestClient violates TInPacket in TestClient.
4. The class implementing TOutPacket is ServerPacket.Out, which is a derivative of OperationPacket. This type is relatively simple in terms of generics, as it only provides an enum type and an endian type, making no cross-reference to other classes. Conclusion: The problem is not (most likely) not in this declaration by itself.
5. The class implementing TInPacket is ServerPacket.In, which is a derivative of AwarePacket. This type is much more complex than TOutPacket, since it cross-references generics to be aware (AwarePacket) of the client that received it. It is probably in this generic mess that the problem occurs.

Then, many hypotheses can fuse. At this point, what I read is correct and accepted by the compiler, but there is evidently something wrong there.

Can you help me find out why I'm getting a generic constraint violation at runtime with my code?

Answer 1

Solution:

So, after some finagling with the generic parameters and constraints, I think I've finally found the issue/solution, and I hope I'm not celebrating too early.

First things first, I still think this is a bug (or at least a quirk) with how the dynamic runtime is trying to invoke the constructor of TestServer. It could also be a compiler bug, that is, if it is against the standard to convert a typed class to a dynamic (then I suppose back again) instead of casting it to its expected type.

By that, I mean that this code:

 TestServer test = new TestServer(GetPort());

turns into the Binder.InvokeConstructor below, doing a whole bunch of extra casting and looks nothing like the code you would expect it to be (the code below generated after an int cast would be expected)

On to the solution, it all has to do with the order of the generic arguments. As far as I know, there's nothing in the standard that has a say in what order you should put your generics in. The code works when you instantiate the class with a normal int. Take a look at how Server and Client have their arguments ordered:

 Client<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>
 Server<TServer, TClient, TInPacket, TOutPacket, TCryptograph, TEndian>

Exactly the same. If remove all the other classes from TestClient, and make TestClient's constraints only work with the base Client and Server class, everything works as expected, no exceptions. I've found the issue is with AwareClient and AwarePacket and the addition of TOperationCode

If you remove TOperationCode and on the abstract classes and the inheriting classes, the code again, works as expected. This is undesirable, as you probably want that generic argument in your class. I've found that moving it to the end of the arguments solves the problem.

 AwareClient<TOperationCode, TServer, TClient, 
             TInPacket, TOutPacket, TCryptograph, TEndian>
 AwarePacket<TOperationCode, TServer, TClient, TInPacket, 
             TOutPacket, TCryptograph, TEndian>

becomes

 AwareClient<TServer, TClient, TInPacket, TOutPacket, 
                   TCryptograph, TEndian, TOperationCode>
 AwarePacket<TServer, TClient, TInPacket, TOutPacket, 
                   TCryptograph, TEndian, TOperationCode>

Of course, you have to make a few more changes with the order of the generic constraints to get it to compile, but that seems to solve your issue.

That said, my gut feeling tells me this is a bug in the clr. Now, it's not just as simple as having 2 classes with the generic arguments out of order, or one that inherits from the other with an added argument. I'm working on trying to reproduce this with an simpler example, but so far, this one case is the only one I've been able to get an exception with.

---

EDIT(s)/My Discovery Process

If you remove the constraints in the Relay<TInPacket, TOutPacket, TCryptograph, TEndian> class, the exceptions are not thrown.

I think what I find more interesting is that the exceptions are only thrown the first time you try to create the TestClient, at least on my machine (these are still FirstChanceExceptions that are apparently handled by the internal runtime, they are unhandled by user code).

Doing this:

new TestServer(GetPort());
new TestServer(GetPort());
new TestServer(GetPort());

does not result in the same call via the dynamic method, but rather the compiler makes three separate CallSite classes internally, three separate declarations. This makes sense from an implementation standpoint. What I find especially interesting, though, is that even though, from what I can see, their code is not shared (who knows if it is internally), the exceptions are only being thrown the on the first call to the constructor.

I wish I had the ability to debug this, but Symbol Servers won't download the source for the dynamic builders, and the locals window is not very helpful. I'm hoping someone from Microsoft can help to answer this mystery.

---

I think I have it, but I'm not sure. I would definitely need an expert on C# dynamics to confirm this.

So, I did a few tests to figure out why it would fail with an explicit cast vs a implicit cast when passing it to TestServer constructor.

This is the main code for your version as compiled:

private static void Main(string[] args)
{
    if (<Main>o__SiteContainer0.<>p__Site1 == null)
    {
        <Main>o__SiteContainer0.<>p__Site1 = 
        CallSite<Func<CallSite, Type, object, TestServer>>.Create(
        Binder.InvokeConstructor(CSharpBinderFlags.None, typeof(Program),
        new CSharpArgumentInfo[] {
            CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.IsStaticType | 
            CSharpArgumentInfoFlags.UseCompileTimeType, null), 
            CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) }));
    }

    TestServer server = <Main>o__SiteContainer0.<>p__Site1.Target.Invoke(
    <Main>o__SiteContainer0.<>p__Site1, typeof(TestServer), GetPort());
    Console.ReadLine();
}

Essentially, what is happening is that the RuntimeBinder has created a function is trying to create, not the int to pass to GetPort(), but instead a new TestServer, dynamically invoking its constructor.

Look at the difference when you cast it to an int and pass it to the constructor:

private static void Main(string[] args)
{
    if (<Main>o__SiteContainer0.<>p__Site1 == null)
    {
        <Main>o__SiteContainer0.<>p__Site1 = 
        CallSite<Func<CallSite, object, int>>.Create(Binder.Convert(
        CSharpBinderFlags.ConvertExplicit, typeof(int), typeof(Program)));
    }

    TestServer server = new TestServer(
    <Main>o__SiteContainer0.<>p__Site1.Target.Invoke(
    <Main>o__SiteContainer0.<>p__Site1, GetPort()));

    Console.ReadLine();
}

Notice instead of it creating a InvokeConstructor binding, it creates a Convert binding, with an Explicit flag. Instead of trying to dynamically invoke the constructor, it invokes a function that converts the dynamic to the TestServer constructor, thus passing it an actual int instead of a generic object.

I guess my point is that, there is definitely nothing wrong with your generics (aside from the fact they're fairly illegible and IMO overused), but rather an issue with how the compiler is trying to dynamically invoke the constructor.

Furthermore, it looks like it has nothing to do with actually passing along the int to the constructor. I removed the constructor from TestClient and made this CallSite, (essentially the same as the erroring one minus the int parameter)

        var lawl = CallSite<Func<CallSite, Type, TestServer>>.Create(
        Binder.InvokeConstructor(CSharpBinderFlags.None, typeof(Program), 
        new CSharpArgumentInfo[] { 
            CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.IsStaticType | 
            CSharpArgumentInfoFlags.UseCompileTimeType, null), 
            CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) }));

        TestServer lol = lawl.Target.Invoke(lawl, typeof(TestServer));

And the same TypeLoadException, GenericArguments[0], 'TOutPacket', on 'ConsoleApplication1.Relay`4[TInPacket,TOutPacket,TCryptograph,TEndian]' violates the constraint of type parameter 'TInPacket'. occurred. Apparently the runtime is having a hard time invoking constructors on your generic type.

Seems like this might be a bug...

---

If you enable .NET Source browsing and enabling breakpoints on any thrown exception, you will catch the TypeLoadException. and can view the entire .net stack trace. Also, it is you can reproduce it with WinDbg.