Rulesets cannot be set for all repos when the org does not have an enterprise subscription

[cachedout]

Problem Description

When the GitHub organization does not have an enterprise subscription, it is not possible to set rulesets for all repos in the organization.

What is actually happening

What happens is that safe-settings queries https://api.github.com/orgs/<my_org>/rulesets and the API returns a 403 with a message that reads: Upgrade to GitHub Enterprise to enable this feature.. This prevents the use of safe-settings to apply rulesets to all individual repos in the org.

What is the expected behavior

It is expected that safe-settings should allow a user to set a ruleset for all repos in an organization by controlling rulesets on all repos in the org instead of via the org-level rulesets.

Error output, if available

  |  
-- | --


{"level":50,"time":1699959656065,"pid":26,"hostname":"safe-settings-demo-99cb5dcf8-hwg58","name":"probot","name":"probot","name":"event","id":"13bc6ffc-82dd-11ee-92eb-11a42222cbb3","name":"HttpError","status":403,"response":{"url":"https://api.github.com/orgs/[REDACTED]/rulesets","status":403,"headers":{"access-control-allow-origin":"*","access-control-expose-headers":"ETag,
 Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, 
X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, 
X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, 
X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id,
 Deprecation, 
Sunset","connection":"close","content-encoding":"gzip","content-security-policy":"default-src
 'none'","content-type":"application/json; charset=utf-8","date":"Tue, 
14 Nov 2023 11:00:56 GMT","referrer-policy":"origin-when-cross-origin, 
strict-origin-when-cross-origin","server":"GitHub.com","strict-transport-security":"max-age=31536000;
 includeSubdomains; 
preload","transfer-encoding":"chunked","vary":"Accept-Encoding, Accept, 
X-Requested-With","x-accepted-github-permissions":"organization_administration=write","x-content-type-options":"nosniff","x-frame-options":"deny","x-github-api-version-selected":"2022-11-28","x-github-media-type":"github.v3;
 
format=json","x-github-request-id":"B62A:3EA2:4C2CA2:9E0057:65535367","x-ratelimit-limit":"5000","x-ratelimit-remaining":"4899","x-ratelimit-reset":"1699962859","x-ratelimit-resource":"core","x-ratelimit-used":"101","x-xss-protection":"0"},"data":{"message":"Upgrade
 to GitHub Enterprise to enable this 
feature.","documentation_url":"https://docs.github.com/rest/orgs/rules#create-an-organization-repository-ruleset"}},"request":{"method":"POST","url":"https://api.github.com/orgs/[REDACTED]/rulesets","headers":{"accept":"application/vnd.github.v3+json","user-agent":"probot/12.3.1
 octokit-core.js/3.6.0 Node.js/16.20.2 (linux; 
x64)","x-github-api-version":"2022-11-28","authorization":"token 
[REDACTED]","content-type":"application/json; 
charset=utf-8"},"body":"{\"name\":\"Enforce pull 
requests\",\"target\":\"branch\",\"enforcement\":\"active\",\"conditions\":{\"ref_name\":{\"include\":[\"~DEFAULT_BRANCH\"],\"exclude\":[\"refs/heads/oldmaster\"]},\"repository_name\":{\"include\":[\"<mark>[REDACTED]</mark>\"],\"exclude\":[\"test\",\"test1\"],\"protected\":true}},\"rules\":[{\"type\":\"pull_request\",\"parameters\":{\"dismiss_stale_reviews_on_push\":true,\"require_code_owner_review\":true,\"require_last_push_approval\":true,\"required_approving_review_count\":1,\"required_review_thread_resolution\":true}}]}","request":{"retryCount":1}},"stack":"HttpError:
 Upgrade to GitHub Enterprise to enable this feature.\n    at 
/opt/safe-settings/node_modules/probot/node_modules/@octokit/request/dist-node/index.js:86:21\n
    at runMicrotasks (&lt;anonymous&gt;)\n    at 
processTicksAndRejections (node:internal/process/task_queues:96:5)\n    
at async sendRequestWithRetries 
(/opt/safe-settings/node_modules/octokit-auth-probot/node_modules/@octokit/auth-app/dist-node/index.js:398:12)\n
    at async Job.doExecute 
(/opt/safe-settings/node_modules/bottleneck/light.js:405:18)","type":"Error","msg":"Upgrade
 to GitHub Enterprise to enable this feature."}

Context

This may well be the intended behavior and if so, it would be good to make this clear in the documentation.

Are you using the hosted instance of probot/settings or running your own?

Self-hosting

If running your own instance, are you using it with github.com or GitHub Enterprise?

github.com

Version of probot/settings

Probot v12.3.1 (Node.js: v16.20.2)

Version of GitHub Enterprise

N/A

[raleonardo]

also having this issue with GitHub Team subscription

[paddyroddy]

I have found this too. A bit concerning as I suspect GitHub might full deprecate classic branch protection at some point.

[anderssonjohan]

@cachedout @raleonardo @paddyroddy I also just started trying out rulesets and have so far only used it in repo specific settings files. The behavior you describe is actually mentioned in the README. Maybe it's not super clear, but the rulesets defined in the settings.yml are applied as org level rulesets only. If you want repo level rulesets you have to use suborgs or repo level settings files.
Recent commit with changes of the relevant section in the readme: 1540810

You should be able to use repo level rulesets by defining a suborg that includes the desired repositories by using a glob pattern.
Bare in mind a repo can only be part of a single suborg, so the final solution varies depending on your config structure.

If you don't have any suborgs or repo level override files, then you should be able to do something like this:

File: admin/.github/suborgs/a-descriptive-filename.yml

# "Sub organization" for trying out repo level rulesets

suborgrepos:
  - *
 
rulesets:
  # demo ruleset with some random settings
  - name: my-repo-level-ruleset
    target: branch
    enforcement: active
    conditions:
      ref_name:
        include:
          - "refs/heads/main"
        exclude: 
          - "refs/heads/[0-9]*.[0-9x]*.x"
    bypass:
      actors:
        users:
          - "dependabot"
    current_user_can_bypass: never
    description: |
      Repository-level ruleset applied to a glob pattern of repos.
    rules:
      - type: required_linear_history
      - type: required_signatures
      - type: deletion
      - type: update
      - type: creation
      - type: non_fast_forward
      - type: pull_request
        parameters:
          required_approving_review_count: 1
          dismiss_stale_reviews_on_push: true
          require_code_owner_review: true
          require_last_push_approval: true
          required_review_thread_resolution: true
          automatic_copilot_code_review_enabled: true
      - type: required_status_checks
        parameters:
          strict_required_status_checks_policy: true
          do_not_enforce_on_create: true
          required_status_checks:
            - context: ci
            - context: lint

Disclaimer: I don't know how this will work if you have defined the same ruleset in both the suborg and a repo level settings file. With branch protections you could have it defined on the suborg level and then just override the required status checks. I'm not sure if you can do that with the rulesets and the rules (or if you have to define the complete required_status_checks rule in every repo settings file).

EDIT: FWIW, suborg rulesets are also mentioned here #597 (comment)

[paddyroddy]

@anderssonjohan thanks! Will give this a go today

[paddyroddy]

Nice that worked well https://github.com/UCL-MIRSG/.github/blob/main/safe-settings/suborgs/rulesets.yaml