Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: Disk Space Warnings in Azure DevOps Pipeline Agent on CoreOS 41+ with ComposeFS #5067

Open
1 of 4 tasks
sebb0 opened this issue Dec 18, 2024 · 1 comment
Open
1 of 4 tasks

Comments

@sebb0
Copy link

sebb0 commented Dec 18, 2024

What happened?

Description

When running the Azure DevOps pipeline agent on CoreOS 41 or later, disk space checks result in warnings due to the use of composefs as the root filesystem. Since composefs is designed to always appear as fully utilized (100%), the agent's check incorrectly reports insufficient disk space on /.

Steps to Reproduce

  1. Deploy a CoreOS 41+ host with composefs as the root filesystem.
  2. Install the Azure DevOps pipeline agent.
  3. Execute any pipeline job that triggers the disk space check.

Actual Behavior

A warning message is displayed in the timeline:

##[warning]Free disk space on / is lower than 5%; Currently used: 100.00%

This warning appears for every job, cluttering the pipeline timeline and making it harder to identify genuine issues.

Expected Behavior

The disk space check should correctly evaluate the volume where the working directory resides, rather than relying on the root filesystem (/).

Additional Context

  • composefs was introduced as the default root filesystem in CoreOS 41. Its design always shows 100% utilization for the root filesystem (link), which causes the disk space check to fail incorrectly. This spams the timeline with warnings, making it harder to identify genuine issues.
  • In general, checking free disk space on the root filesystem (/) is unreliable, especially when the working directory resides on a separate mounted volume. The current approach does not account for the actual volume used by the working directory, leading to misleading results in many possible configurations.

Possible Solution

Update the disk space check logic to evaluate the actual working directory's mount point or volume instead of the root filesystem (/).

Versions

v3.248.0 / Fedora CoreOS 41.20241109.3.0

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

@aleksandrlevochkin
Copy link
Contributor

Hi @sebb0, thanks for the detailed report on this issue, we will take a look. By the way, I think you can disable the warnings by setting the agent knob AZP_ENABLE_RESOURCE_UTILIZATION_WARNINGS to false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants