Calling certain native functions from custom hosted dotnet causes error

[Linx145]

Description

I'm writing a game engine that relies on running the hosted dotnet via hostfxr and the like. In lieu of pinvoke, I enable the scripting runtime to call native engine functions by passing function pointers of the native function to the C# runtime, via regular reverse-pinvoke as stated in the custom dotnet runtime host guide.

In doing so, I've stumbled upon a very strange possible bug where certain native functions get random values as arguments, as compared to what the C# assembly had called the function pointer with. All function pointers on C# side are marked as unmanaged [cdecl], and on the C++ side they are cdecl too. (Though this doesn't seem to matter at all.)

Upon further investigation, it appears that when calling certain function pointers, C# pushes input arguments to the wrong register (I believe it's RCX) while C++ expects it to be in a different register. This may seem to be a calling convention issue, but it only ever happens with a certain, apparently random set of functions (around 10 functions cause this issue amongst the 200+ passed as function pointers to native), which is probably going to be an issue when reproducing this bug. All other 190+ functions call using this same method just fine. Perhaps most baffling to me is the current workaround for the issue, which is just changing the name of the return type.

This issue happens on both native executables compiled with MSVC and clang, so I assume it's an issue with dotnet itself and not the C++ compiler.

Reproduction Steps

C++ end:

//.hpp
struct Actor
{
    u32 ID;
    u32 generation;
}

Actor Actor_NewActor(u32 world)
{
    printf("%u\n", world);
    return Actor();
}

C# end:

//autogenerated native function pointer imports + calls
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static unsafe EternityWorks.Actor Actor_NewActor(uint world) {
    var _result = ((delegate* unmanaged[Cdecl]<uint, EternityWorks.Actor>)functionPointers[8])(world);
    return _result;
}
...
public static void Initialize()
{
    Actor_NewActor(1);
}

Expected behavior

Native code Actor_NewActor prints 1

Actual behavior

Native code Actor_NewActor prints a random integer

Regression?

The bug has been here since .net 8.0, persists in .net 9.0

Known Workarounds

Changing the name of the output type of the function seems to fix it, somehow. For example, Actor_New(uint32_t worldID) returns an Actor, which is two uint32s. Changing that to a return a NativeActor, which is the exact same thing (2 uint32s), fixes the bug.

Configuration

Windows 10, .net 8.0/9.0, x64

Other information

I'll try to get a minimal reproducible build in a separate repo, but my project is currently quite massive and from the fickle nature of the fix, there's no guarantee that the exact same configuration of functions, operations etc will cause the same issue if it is in another codebase.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

P/Invoke are sensitive to the exact definition of structs. In this case, the exact definition of Actor or NativeActor matters, including attributes and field layout.

Additionally, you can use decompilation tools like https://github.com/EgorBo/Disasmo to find the difference of assembly code for the P/Invoke.