.NET 9 SDK introduces hang in dotnet format when run on Azure Pipelines in combination with NB.GV

[AArnott]

Describe the bug

dotnet format --verify-no-changes --no-restore just hangs with no output when run in this configuration:

• Azure Pipelines (local is fine)
• .NET 9 SDK (.NET 8 SDK is fine)
• With the Nerdbank.GitVersioning nuget package referenced

Working fine with .NET 8 SDK
Hanging (across 2 attempts) with .NET 9 SDK

@nietras reported the same issue with their own repo, who isolated the repro to requiring both .NET 9 SDK and the Nerdbank.GitVersioning package.

In my case, the hang is on a linux build agent. For @nietras I believe it was a Windows agent. Both of us run this tool only on one agent in the pipeline.
 

To Reproduce

1. Fork the repo at https://github.com/dotnet/Nerdbank.Streams.git
2. Run it under Azure Pipelines on this commit: 09b404915c9a6d4566023e3bdebbbf8b08eebf42

Exceptions (if any)

Further technical details

@nietras also shared the diagnostic output:

D:\agent\_work\_tool\dotnet\dotnet.exe format --verify-no-changes -v diagnostic
  The dotnet runtime version is '9.0.0'.
  The dotnet CLI version is '9.0.100'.
  Using MSBuild.exe located in 'D:\agent\_work\_tool/dotnet\sdk\9.0.100\'.
  Formatting code files in workspace 'D:\agent\_work\210\s\Test.sln'.
  Loading workspace.
    Determining projects to restore...
  Restored D:\agent\_work\210\s\src\XYZ\XYZ.csproj (in 198 ms)

`dotnet --info` output

 MSBuild version:   17.12.7+5b8665660

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  22.04
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /opt/hostedtoolcache/dotnet/sdk/9.0.100/

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      9.0.0
  Architecture: x64
  Commit:       9d5a6a9aa4

.NET SDKs installed:
  9.0.100 [/opt/hostedtoolcache/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.0 [/opt/hostedtoolcache/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.11 [/opt/hostedtoolcache/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0 [/opt/hostedtoolcache/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_ROOT       [/opt/hostedtoolcache/dotnet]

global.json file:
  /home/vsts/work/1/s/global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download
Mono JIT compiler version 6.12.0.200 (tarball Tue Jul 11 21:37:50 UTC 2023)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(610)
	Suspend:       hybrid
	GC:            sgen (concurrent by default)

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[AArnott]

Nerdbank.GitVersioning is a build-related nuget package that adds msbuild targets to projects. But I wouldn't expect it to ever hang (it never has before this), nor do I expect it to do anything special in a dotnet format context.

[KalleOlaviNiemitalo]

Perhaps strace could show whether it's blocked in a system call on the Linux agent, e.g. trying to write something to a full pipe.

I wonder if the MSBuild terminal logger is causing the hang somehow, even though Azure Pipelines should not be detected as an "interactive" terminal session. Could try disabling it with the environment variable.

[AArnott]

Thanks for the suggestion, @KalleOlaviNiemitalo. It didn't work though.

[AArnott]

This workaround has proven effective, but hacky.
TLDR: make NB.GV a conditional dependency, and set an env variable to turn it off around the dotnet format command invocation.

[nietras]

@AArnott thank you for reporting the issue here. I am indeed seeing the same on Windows on in-house build servers with the following info.

D:\agent\_work\_tool\dotnet\dotnet.exe --info
.NET SDK:
 Version:           9.0.100
 Commit:            59db016f11
 Workload version:  9.0.100-manifests.3068a692
 MSBuild version:   17.12.7+5b8665660

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19045
 OS Platform: Windows
 RID:         win-x64
 Base Path:   D:\agent\_work\_tool\dotnet\sdk\9.0.100\

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      9.0.0
  Architecture: x64
  Commit:       9d5a6a9aa4

It is very easy to reproduce by creating a small csproj that references Nerdbank.GitVersioning

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Nerdbank.GitVersioning" Version="3.6.143" 
                      PrivateAssets="all" Condition="!Exists('packages.config')" />
  </ItemGroup>

</Project>

with global.json:

{
  "sdk": {
    "version": "9.0.100",
    "rollForward": "latestPatch",
    "allowPrerelease": false
  }
}

and then defining a pipeline azure-pipeline.yml like:

trigger:
- main

pool:
  name: Default

jobs:
- job: Build_and_Test
  steps:
  - checkout: self
    clean: true
    persistCredentials: true
    fetchDepth: 0
  
  - task: NuGetToolInstaller@1
    displayName: 'Use NuGet 6.4'
    inputs:
      versionSpec: 6.4
  
  - task: NuGetAuthenticate@1
    displayName: 'NuGet Authenticate'
    inputs:
      forceReinstallCredentialProvider: true
  
  - task: UseDotNet@2
    displayName: 'Use .NET - global.json'
    inputs:
      useGlobalJson: true

  - task: DotNetCoreCLI@2
    displayName: 'dotnet --info'
    inputs:
      command: custom
      custom: '--info'

  - task: DotNetCoreCLI@2
    displayName: 'dotnet format verify-no-changes'
    inputs:
      command: custom
      custom: format
      arguments: '--verify-no-changes -v diagnostic'

which will then hang with no other output than:

The dotnet runtime version is '9.0.0'.
  The dotnet CLI version is '9.0.100'.
  Using MSBuild.exe located in 'D:\agent\_work\_tool/dotnet\sdk\9.0.100\'.
  Formatting code files in workspace 'D:\agent\_work\96\s\Test.sln'.
  Loading workspace.
    Determining projects to restore...
  Restored D:\agent\_work\96\s\src\Some\Some.csproj (in 130 ms).