restarted. Azure DevOps can't recover from restarts. #3879

lewing · 2024-08-26T19:12:06Z

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

Feature changes/additions
Bug fixes
Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Pull Request
903560	dotnet/runtime	browser-wasm linux Release LibraryTests
903535	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110883
903299	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110967
903231	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#107118
903228	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110966
903196	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110965
903065	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110956
903059	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110945
903007	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110953
902892	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110948
902803	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110780
902797	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110943
902737	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
902691	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110884
902689	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110883
902619	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
902599	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110915
902315	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110915
902303	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
902217	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110915
902009	dotnet/runtime	browser-wasm linux Release LibraryTests
901938	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110884
901709	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110894
901605	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110203
901524	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110888
901282	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
900922	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110876
900965	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#101356
900916	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110873
900732	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110862
900696	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110687
900456	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110688
900045	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110676
900030	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110842
899946	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#101356
899723	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108732
899430	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#102704
899369	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#105004
899348	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110824
899023	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110755
896525	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110750
898987	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110816
898507	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110781
898483	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110471
898303	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
898050	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110790
897907	dotnet/runtime	browser-wasm linux Release LibraryTests
897902	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#101356
897549	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110119
897802	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110790
897739	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110688
897575	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110690
897529	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110781
897468	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110448
897455	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110778
897306	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110739
896944	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110627
896801	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#101356
896640	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110676
896405	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110744
896347	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110741
896179	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110528
896173	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110308
895948	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110725
895925	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110647
895814	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110721
895768	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110690
895502	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110648
895456	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110567
895372	dotnet/runtime	browser-wasm linux Release LibraryTests
895068	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110701
895056	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110477
895020	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110322
894999	dotnet/runtime	browser-wasm linux Release LibraryTests
894986	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#104999
894781	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110197
894700	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110690
894601	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110687
894524	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110641
894279	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110622
894251	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#110635
894208	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110655
894108	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110669
894054	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110654
893805	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110654
893766	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110611
893752	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110567
893661	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110084
893658	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#109087
893639	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110648
893629	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110647
892381	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110606
892609	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#110618
893374	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110641
893334	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
893203	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110476
893149	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110591
893010	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110630
893004	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110477
892926	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#110627
Displaying 100 of 231 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	24	231

The text was updated successfully, but these errors were encountered:

am11 · 2024-10-07T05:54:23Z

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

am11 · 2024-10-07T12:57:20Z

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

steveisok · 2024-10-07T23:32:59Z

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

ericstj · 2024-10-10T17:47:41Z

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857
https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923
https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.

snakex64 · 2024-10-30T15:07:32Z

This seems to be happening consistently in PR #109320 . Should I just try re-running a third time or is it never gonna work?

lewing added First Responder Known Build Error labels Aug 26, 2024

guihkx mentioned this issue Aug 26, 2024

"Error: 'NoneType' object is not subscriptable" on Channel.from_directory DevilXD/TwitchDropsMiner#547

Closed

jkotas mentioned this issue Aug 26, 2024

Fix alloc-dealloc mismatch dotnet/runtime#106957

Merged

github-staff deleted a comment from masooddahmedd Sep 10, 2024

Maoni0 mentioned this issue Sep 16, 2024

instrumentation (mostly) for DATAS dotnet/runtime#107853

Merged

lewing mentioned this issue Sep 17, 2024

Fix RegexOptions.Compiled|IgnoreCase perf when dynamic code isn't supported dotnet/runtime#107874

Merged

CarnaViire mentioned this issue Sep 18, 2024

WebSocket Feedback Follow-up dotnet/runtime#107662

Merged

am11 mentioned this issue Oct 7, 2024

Disable linux-x64 dev-innerloop dotnet/runtime#108581

Merged

am11 mentioned this issue Oct 7, 2024

Move more helpers from DomainAssembly to Assembly dotnet/runtime#108568

Merged

This was referenced Oct 7, 2024

[main] Update dependencies from 7 repositories dotnet/runtime#108385

Merged

[main] Update dependencies from dotnet/arcade dotnet/runtime#108323

Merged

[main] Update dependencies from dotnet/source-build-reference-packages dotnet/runtime#108593

Merged

am11 mentioned this issue Oct 8, 2024

[RISC-V] Fix some minor errors dotnet/runtime#107179

Merged

AaronRobinsonMSFT mentioned this issue Nov 25, 2024

Remove Helper Method Frames (HMF) from Reflection dotnet/runtime#109996

Merged

ilonatommy mentioned this issue Nov 28, 2024

[MT][Firefox] Disable multithreading tests on Firefox dotnet/runtime#110155

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restarted. Azure DevOps can't recover from restarts. #3879

restarted. Azure DevOps can't recover from restarts. #3879

lewing commented Aug 26, 2024 •

edited by build-analysis bot

Loading

am11 commented Oct 7, 2024

am11 commented Oct 7, 2024

steveisok commented Oct 7, 2024

ericstj commented Oct 10, 2024

snakex64 commented Oct 30, 2024 •

edited

Loading

restarted. Azure DevOps can't recover from restarts. #3879

restarted. Azure DevOps can't recover from restarts. #3879

Comments

lewing commented Aug 26, 2024 • edited by build-analysis bot Loading

Build

Build leg reported

Pull Request

Known issue core information

Release Note Category

Release Note Description

Additional information about the issue reported

Known issue validation

Report

Summary

am11 commented Oct 7, 2024

am11 commented Oct 7, 2024

steveisok commented Oct 7, 2024

ericstj commented Oct 10, 2024

snakex64 commented Oct 30, 2024 • edited Loading

lewing commented Aug 26, 2024 •

edited by build-analysis bot

Loading

snakex64 commented Oct 30, 2024 •

edited

Loading