P/Invoke Generation Pipeline

The P/Invoke source generator is responsible for finding all methods marked with LibraryImportAttribute and generating code for their implementations (stubs) and corresponding P/Invokes that will be called by the stubs. For every method, the steps are:

1. Process the symbols and metadata for the method, its parameters, and its return type.
2. Determine the marshalling generators that will be responsible for generating the stub code for each parameter and return
3. Generate the stub code
4. Generate the corresponding P/Invoke
5. Add the generated source to the compilation.

The pipeline uses the Roslyn Syntax APIs to create the generated code. This imposes some structure for the marshalling generators and allows for easier inspection or modification (if desired) of the generated code.

Symbol and metadata processing

The generator processes the method's LibraryImportAttribute data, the method's parameter and return types, and the metadata on them (e.g. LCIDConversionAttribute, MarshalAsAttribute, struct marshalling attributes). This information is used to determine the corresponding native type for each managed parameter/return type and how they will be marshalled.

A TypePositionInfo is created for each type that needs to be marshalled. For each parameter and return type, this captures the managed type, managed and native positions (return or index in parameter list), and marshalling information.

The marshalling information is represented by various subclasses of MarshallingInfo and represents all user-defined marshalling information for the specific parameter or return type. These classes are intended to simply capture any specified marshalling information, not interpret what that information means in terms of marshalling behaviour; that is handled when determining the marshalling generator for each TypePositionInfo.

The processing step also includes handling any implicit parameter/return types that are required for the P/Invoke, but not part of the managed method signature; for example, a method with PreserveSig=false requires an HRESULT return type and potentially an out parameter matching the managed method's return type.

PreserveSig=false

The below signature indicates that the native function returns an HRESULT, but has no other return value (out parameter).

[LibraryImport("Lib", PreserveSig = false)]
static partial void Method();

Processing the above signature would create a TypePositionInfo for the HRESULT return type for native call, with properties indicating that it is in the native return position and has no managed position. The actual P/Invoke would be:

[DllImport("Lib", EntryPoint = "Method")]
static partial int Method__PInvoke__();

The below signature indicates that the native function returns an HRESULT and also has an out parameter to be used as the managed return value.

[LibraryImport("Lib", PreserveSig = false)]
[return: MarshalAs(UnmanagedType.U1)]
static partial bool MethodWithReturn();

Processing the above signature would create a TypePositionInfo for the HRESULT return type for native call, with properties indicating that it is in the native return position and has no managed position. The TypePositionInfo representing the bool return on the managed method would have properties indicating it is the last parameter for the native call and is in the managed return position. The actual P/Invoke would be:

[DllImport("Lib", EntryPoint = "MethodWithReturn")]
static partial int MethodWithReturn__PInvoke__(byte* retVal);

Marshalling generators

Each parameter and return for the method is handled by an IMarshallingGenerator instance. The processed information for each parameter and return type is used to determine the appropriate marshalling generator for handling that type. Support for different types can be added in the form of new implementations of IMarshallingGenerator.

The marshalling generators are responsible for generating the code for each stage of the stub. They are intended to be stateless, such that they are given all the data ([TypePositionInfo][src_TypePositionInfo]) for which they need to generate marshalling code and the context (StubCodeContext) under which that code should be generated.

Stub code generation

Generation of the stub code happens in stages. The marshalling generator for each parameter and return is called to generate code for each stage of the stub. The statements and syntax provided by each marshalling generator for each stage combine to form the full stub implementation.

The stub code generator itself will handle some initial setup and variable declarations:

• Assign out parameters to default
• Declare variable for managed representation of return value
• Declare variables for native representation of parameters and return value (if necessary)

Stages

1. Setup: initialization that happens before marshalling any data
   • If the method has a non-void return, call Generate on the marshalling generator for the return
   • Call Generate on the marshalling generator for every parameter
2. Marshal: conversion of managed to native data
   • Call Generate on the marshalling generator for every parameter
3. Pin: data pinning in preparation for calling the generated P/Invoke
   • Call Generate on the marshalling generator for every parameter
   • Ignore any statements that are not fixed statements
4. PinnedMarshal: conversion of managed to native data
   • Call Generate on the marshalling generator for every parameter
5. Invoke: call to the generated P/Invoke
   • Call AsArgument on the marshalling generator for every parameter
   • Create invocation statement that calls the generated P/Invoke
6. NotifyForSuccessfulInvoke: Notify a marshaller that all stages through the "Invoke" stage were successful.
   • Used to keep alive any objects who's native representation won't keep them alive across the call.
   • Call Generate on the marshalling generator for every parameter.
7. UnmarshalCapture: capture any native out parameters to avoid memory leaks if exceptions are thrown during Unmarshal.
   • If the method has a non-void return, call Generate on the marshalling generator for the return
   • Call Generate on the marshalling generator for every parameter
8. Unmarshal: conversion of native to managed data
   • If the method has a non-void return, call Generate on the marshalling generator for the return
   • Call Generate on the marshalling generator for every parameter
9. GuaranteedUnmarshal: conversion of native to managed data even when an exception is thrown
   • Call Generate on the marshalling generator for every parameter.
   • If this stage has any statements, put them in an if statement where the condition represents whether the call succeeded
10. CleanupCallerAllocated: free any resources allocated by the caller
    • Call Generate on the marshalling generator for every parameter
11. CleanupCalleeAllocated: if the native method succeeded, free any resources allocated by the callee (out parameters and return values)
    • Call Generate on the marshalling generator for every parameter
    • If this stage has any statements, put them in an if statement where the condition represents whether the call succeeded

Generated P/Invoke structure (if no code is generated for GuaranteedUnmarshal and Cleanup, the try-finally is omitted):

<< Variable Declarations >>
<< Setup >>
try
{
    << Marshal >>
    << Pin >> (fixed)
    {
        << Pinned Marshal >>
        << Invoke >>
    }
    << Notify For Successful Invoke >>
    << Unmarshal Capture >>
    << Unmarshal >>
}
finally
{
    << GuaranteedUnmarshal >>
    << CleanupCalleeAllocated >>
    << CleanupCallerAllocated >>
}

Stub conditional features

Some marshalling optimizations are only available in specific scenarios. Generally, there are 4 basic marshalling contexts:

• P/Invoke
• Reverse P/Invoke
• User-defined structure marshalling
• Non-blittable array marshalling

This experiment generally is currently only focusing on two of the concepts: P/Invoke and non-blittable array marshalling (in the context of a P/Invoke).

There are three features for specialized marshalling features that may only be available in some contexts:

• Pinning to marshal data without copying (the fixed statement)
• Stack allocation across the native context (using the stackalloc keyword or https://github.com/dotnet/runtime/issues/25423)
• Storing additional temporary state in extra local variables

Support for these features is indicated in code by the abstract SingleFrameSpansNativeContext and AdditionalTemporaryStateLivesAcrossStages properties on the StubCodeContext type. The SingleFrameSpansNativeContext property represents whether or not both pinning and stack-allocation are supported. These concepts are combined because we cannot safely support a conditional-stackalloc style API (such as https://github.com/dotnet/runtime/issues/52065) and safely get a pointer to data without also being able to pin.

The various scenarios mentioned above have different levels of support for these specialized features:

To help enable developers to use the full model described in the Struct Marshalling design, we declare that in contexts where AdditionalTemporaryStateLivesAcrossStages is false, developers can still assume that state declared in the Setup phase is valid in any phase, but any side effects in code emitted in a phase other than Setup will not be guaranteed to be visible in other phases. This enables developers to still use the identifiers declared in the Setup phase in their other phases, but they'll need to take care to design their generators to handle these rules.

SetLastError=true

The stub code generation also handles SetLastError=true behaviour. This configuration indicates that system error code (errno on Unix, GetLastError on Windows) should be stored after the native invocation, such that it can be retrieved using Marshal.GetLastWin32Error.

This means that, rather than simply invoke the native method, the generated stub will:

1. Clear the system error by setting it to 0
2. Invoke the native method
3. Get the system error
4. Set the stored error for the P/Invoke (accessible via Marshal.GetLastWin32Error)

A core requirement of this functionality is that the P/Invoke called in (2) is blittable (the purpose of the P/Invoke source generator), such that there will be no additional operations (e.g unmarshalling) after the invocation that could change the system error that is retrieved in (3). Similarly, (3) must not involve any operations before getting the system error that could change the system error. This also relies on the runtime itself handling preserving the last error (see BEGIN/END_PRESERVE_LAST_ERROR macros) during JIT and P/Invoke resolution.

Clearing the system error (1) is necessary because the native method may not set the error at all on success and the system error would retain its value from a previous operation. The developer should be able to check Marshal.GetLastWin32Error after a P/Inovke to determine success or failure, so the stub explicitly clears the error before the native invocation, such that the last error will indicate success if the native call does not change it.

P/Invoke

The P/Invoke called by the stub is created based on the user's original declaration of the stub. The signature is generated using the syntax returned by AsNativeType and AsParameter of the marshalling generators for the return and parameters. Any marshalling attributes on the return and parameters of the managed method - MarshalAsAttribute, InAttribute, OutAttribute - are dropped.

The fields of the DllImportAttribute are set based on the fields of LibraryImportAttribute as follows:

Examples

Explicit EntryPoint:

// Original declaration
[LibraryImport("Lib")]
static partial void Method(out int i);

// Generated P/Invoke
[DllImport("Lib", EntryPoint = "Method")]
static partial void Method__PInvoke__(int* i);

Passed through:

// Original declaration
[LibraryImport("Lib", EntryPoint = "EntryPoint", CharSet = CharSet.Unicode)]
static partial int Method(string s);

// Generated P/Invoke
[DllImport("Lib",  EntryPoint = "EntryPoint", CharSet = CharSet.Unicode)]
static partial int Method__PInvoke__(ushort* s);

Handled by generated source (dropped from DllImport):

// Original declaration
[LibraryImport("Lib", SetLastError = true)]
[return: [MarshalAs(UnmanagedType.U1)]
static partial bool Method([In][MarshasAs(UnmanagedType.LPWStr)] string s);

// Generated P/Invoke
[DllImport("Lib", EntryPoint = "Method")]
static partial byte Method__PInvoke__(ushort* s);