Source Generators need to be able to call other source generators

[YairHalberstadt]

There are plans to push for source generators to be used for all sorts of scenarios that are currently done at runtime. For example:

1. The runtime is looking into using SourceGenerators to generate interop stubs: https://github.com/dotnet/runtime/blob/master/docs/design/features/source-generator-pinvokes.md

2. There's a push to get serialization frameworks to use Source Generators to help with trimming among other scenarios: https://devblogs.microsoft.com/dotnet/app-trimming-in-net-5/

I'm sure there are many more - for example Asp.Net is definitely thinking about how it can use Source Generators to wire things up.

However at the moment source generators can't use other source generators. What this means is that it will be impossible for a source generator to generate pinvoke code, asp.net code, or code using serialization, since the relevant source generators which are meant to process these wont see the generated files (although they can always use the runtime/reflection option as a fallback). This seems like a pretty critical limitation.

In an ideal world it would be possible to create a graph of exactly which source generators rely on which other ones. However in practice this means creating potentially as many compilations as there are source generators which may lead to serious performance problems: https://discord.com/channels/732297728826277939/735233259763400715/762740254921719848. There's also no obvious, reliable way to specify this graph.

More pragmatically I think it would be sufficient to run source generators in 2 passes. By default all source generators run in the second pass, and cannot use other source generators. However a SourceGenerator can opt in to running in the first pass. This will mean files it generates will be seen by the second pass. So either your generator can use other generators, or it can be used by other generators, but not both.

That way low level source generators which expect to be used by other source generators will make sure not to rely on any other source generator (even if that leads to a lot of extra work). Other source generators can then freely use these if they want.

[jasonmalinowski]

We've so far intentionally tried to avoid this because of all of the problems you've mentioned. 😄

By default all source generators run in the second pass, and cannot use other source generators. However a SourceGenerator can opt in to running in the first pass.

I worry about this a bit though: a serialization library needs to run "after" other any other thing trying to generate a type because it might have to serialize that type. So that implicitly means anything generating a type has to be in the first pass, not second pass. And the challenge here is if somebody writes a generator and didn't opt themselves into the first pass, then the user is now broken when they try to use both...

[YairHalberstadt]

Surely a serialisation library wouldn't have to generate serialisization code for everything - that would be super inefficient. It would have to be opt in only.

Also only a code generator generating exchange types is likely to require serialisation of those types.

[jasonmalinowski]

I think the expectation is some serializers can "opt in" by simply seeing uses: if you see some code calling Serializer.Serialize<MyType> then it knows it needs to generate the logic for serializing MyType; if MyType came from another generator then it'll need to see that type.

[YairHalberstadt]

Well in that case the Json Serializer will give an error because it can't see MyType.

There could be an option in the csproj to change which pass a generator runs in from the default. In the relatively rare case where A depends on B depends on C you would be stuck, but I think worrying about that would be letting perfect be the enemy of good.

[stefanloerwald]

I'm convinced sequencing is something absolutely crucial, also w.r.t. breaking changes: for example, blazor/razor currently uses MSBuild-based source generation (a razor file is converted to a csharp file which then gets compiled normally). This is something that currently doesn't make use of roslyn-SGs, but in the future might very well do so. However, we can currently write source generators that use the MSBuild-SG-generated code, e.g. to inspect parameters of components (razor files are components, which translate to classes with properties annotated with the [Parameter] attribute). If the generator were now to be replaced by a roslyn-SG, that now breaks every roslyn-SG that previously relied on the generated code. Requiring every SG to look through razor files too would duplicate the effort (and would be error-prone), so the only good choice for this problem is to have a clear sequence "razor-SG" before "my-SG".

My gut feeling says that generating a dependency graph (enforced to be cycle-free) should suffice to get rid of most problems.

[YairHalberstadt]

To give an idea how this would work:

I have 5 generators:

1. Json generator
2. Interop generator
3. Generator A (depends on json)
4. Generator B (depends on interop)
5. Generator C (depends on json and interop)

Then we do as follows

1. Create an initial compilation InitialComp
2. Run generators A, B and C with the initial compilation, to generate AFiles, BFiles and CFiles.
3. Create a compilation InitialACComp consisting of InitialComp+ AFiles + CFiles
4. Run json generator with InitialACComp, to generate JsonFiles
5. Create a compilation InitialBCComp consisting of InitialComp+ BFiles + CFiles
6. Run interop generator with InitialBCComp to generate InteropFiles
7. Create a final compilation consisting of Initial + AFiles + BFiles + CFiles + JsonFiles + InteropFiles

Here a total of 4 compilations are needed.
As you can see the maximum number of compilations is equivalent to the number of generators + 1, since in the worst case you need a new compilation for every generator.

[stefanloerwald]

To give an idea how this would work:

I have 5 generators:

1. Json generator
2. Interop generator
3. Generator A (depends on json)
4. Generator B (depends on interop)
5. Generator C (depends on json and interop)

Then we do as follows

1. Create an initial compilation InitialComp
2. Run generators A, B and C with the initial compilation, to generate AFiles, BFiles and CFiles.

Why would A, B, and C run before their dependency completed?

If I understand your dependencies right you have

IOp ---> B
     \
       > C
     /
Json ---> A 

or did you mean it the exact other way round?

In my head the terminology "A depends on the JSON-SG" would mean "A should only run once it can see the output of the JSON-SG", which would make runnign the A-SG before the JSON-SG rather pointless.

[YairHalberstadt]

I meant the other way round. The way I'm thinking of it is, in most cases a source generator will generate code that it will want another generator to consume. The second generator is agnostic to where the code it sees comes from. Therefore if a depends on b, a runs first, and then b fixes the code generated by a.

However I guess you would want to be able to specify it both ways - this generator runs before generator X, and after generator y.

[stefanloerwald]

Agreed. Maybe something similar to what MSBuild does with AfterTargets/BeforeTargets?

[svick]

The micro-optimzer in me thinks this could be improved as follows:

1. Create an initial compilation InitialComp.
2. Run generators A, B and C with the initial compilation, to generate AFiles, BFiles and CFiles.
3. Create a compilation InitialABCCComp consisting of InitialComp+ AFiles + BFiles + CFiles.
4. Run json generator and interop generator with InitialABCComp, to generate JsonFiles and InteropFiles.
5. Create a final compilation consisting of Initial + AFiles + BFiles + CFiles + JsonFiles + InteropFiles.

This still satisfies all dependencies and results in fewer compilations being created. (The actual number depends on the length of the critical path in the graph.)

Though maybe doing it this way is not a good idea, because it means which set of files each generator sees would be kind of unpredictable.

[dallmair]

Just faced this in a custom source generator, which generates a custom JsonSerializerContext and expected it to be picked up by the STJ source generator -- wrote the code although I knew source generators operate on the user's compilation unit and cannot see each other's generated source.

So definitely 👍 for some solution in this space.

[domportera]

im trying to do the exact same thing and hitting this wall over and over. did you ever come up with a solution?

[msedi]

If this would be possible, would it then also be possible to let source generators modify the code and not just add code?

[msedi]

It should be a general type which means I need to combine all possible combinations of all possible units that can exist. You can see a list of these quantities here: wiki. With the list multiple permutations/combinations are possible and it would be almost impossible to precreate all of these.

I understand that this is a special case and that there should not be any dialect as @333fred said. I'm just trying to find a native way to describe units which makes keeps code readable.

[CyrusNajmabadi]

which means I need to combine all possible combinations of all possible units that can exis

You only need to define extensions for cases you actually end up using. Normally this would fall out from the math operations you are doing. In the rarer cases where you are defining an initial value, you can just have extensions to help there.

[msedi]

We are in a physics department ;-) so I cannot forsee what units we have, we are dealing with all (at least too many) of them ;-)
But I looked at other solutions from Python, C++, etc. all of them need to solve it that way lacking a better idea (or language support).

[CyrusNajmabadi]

Add when you run into something new you need. You'd have to do that anyway with a new dialect of the language.

[dowhilefor]

Also stumbled upon this. Implementing a sort of RPC framework. For that im putting the arguments of the methods into MemoryPack classes and add some boiler plate using a source generator. Unfortunately the MemoryPack source generator does not see my generated Argument files and therefore it can't generate it. Now i'm contemplating if i should write a manualy code parser as a pre-build step, that generates the files, adds it to the prj and then the MemoryPack generator can do its work.

[mwadams]

I have just hit this issue with the source generator in https://github.com/corvus-dotnet/corvus.jsonschema

This emits code for serialization/validation using JSON Schema files as the basis.

When you run the command line tool, the code it emits uses the System.Text.RegularExpressions SourceGenerator.

This will not compile, so we have to fall back on runtime creation of Regular Expressions, which (I assume) will negatively impact start-up time.

[thomhurst]

I'm the author of TUnit, a testing framework that uses source generators.

Recently a user tried source-generating their tests, but then of course TUnit also uses source generators, so it can't actually see their tests to discover and register them.

So a good example where it'd be nice to allow source generators to see generated code.

[CyrusNajmabadi]

@thomhurst No design has been presented that seems like it can adequately solve this.