canonical · swetha1654 · Dec 11, 2024 · Dec 2, 2024 · Dec 2, 2024 · Dec 2, 2024
@@ -0,0 +1,96 @@
+# Copyright 2024 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+name: Allure Report Generation
+
+on: [workflow_call]
+
+jobs:
+  allure-report:
+    name: Publish Allure report
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    if: always() && !cancelled()
+    steps:
+      - name: Download Allure
+        # Following instructions from  https://allurereport.org/docs/install-for-linux/#install-from-a-deb-package
+        run: gh release download --repo allure-framework/allure2 --pattern 'allure_*.deb'
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: Install Allure
+        run: |
+          sudo apt-get update
+          sudo apt-get install ./allure_*.deb -y
+      - name: Checkout GitHub pages branch
+        uses: actions/checkout@v4
+        with:
+          ref: gh-pages
+          path: repo/
+      - name: Download default test results
+        uses: actions/download-artifact@v4
+        with:
+          path: allure-collection-default-results/
+          pattern: allure-default-results*
+          merge-multiple: true
+      - name: Download actual test results
+        uses: actions/download-artifact@v4
+        with:
+          path: allure-results/
+          pattern: allure-results*
+          merge-multiple: true
+      - name: Install CLI
+        run: pipx install git+https://github.com/canonical/operator-workflows@ISD-2620-allure#subdirectory=python/cli
+      - name: Combine Allure default results & actual results
+        # For every test: if actual result available, use that. Otherwise, use default result
+        # So that, if actual result not available, Allure report will show "unknown"/"failed" test result
+        # instead of omitting the test
+        run: allure-add-default-for-missing-results --allure-results-dir=allure-results --allure-collection-default-results-dir=allure-collection-default-results
+      - name: Load test report history
+        run: |
+          if [[ -d repo/_latest/history/ ]]
+          then
+            echo 'Loading history'
+            cp -r repo/_latest/history/ allure-results/
+          fi
+      - name: Create executor.json
+        shell: python
+        run: |
+          # Reverse engineered from https://github.com/simple-elf/allure-report-action/blob/eca283b643d577c69b8e4f048dd6cd8eb8457cfd/entrypoint.sh
+          import json
+
+          DATA = {
+              "name": "GitHub Actions",
+              "type": "github",
+              "buildOrder": ${{ github.run_number }},  # TODO future improvement: use run ID
+              "buildName": "Run ${{ github.run_id }}",
+              "buildUrl": "https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}",
+              "reportUrl": "../${{ github.run_number }}/",
+          }
+          with open("allure-results/executor.json", "w") as file:
+              json.dump(DATA, file)
+      - name: Generate Allure report
+        run: allure generate
+      - name: Create index.html
+        shell: python
+        run: |
+          DATA = f"""<!DOCTYPE html>
+          <meta charset="utf-8">
+          <meta http-equiv="cache-control" content="no-cache">
+          <meta http-equiv="refresh" content="0; url=${{ github.run_number }}">
+          """
+          with open("repo/index.html", "w") as file:
+              file.write(DATA)
+      - name: Update GitHub pages branch
+        working-directory: repo/
+        # TODO future improvement: commit message
+        run: |
+          mkdir '${{ github.run_number }}'
+          rm -f _latest
+          ln -s '${{ github.run_number }}' _latest
+          cp -r ../allure-report/. _latest/
+          git add .
+          git config user.name "GitHub Actions"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git commit -m "Allure report ${{ github.run_number }}"
+          # Uses token set in checkout step
+          git push origin gh-pages
diff --git a/.github/workflows/integration_test_allure.md b/.github/workflows/integration_test_allure.md
@@ -0,0 +1,64 @@
+# Allure Report Integration
+
+[Allure Report](https://allurereport.org/) for [integration_test_run.yaml](https://github.com/canonical/operator-workflows?tab=readme-ov-file#integration-test-workflow-canonicaloperator-workflowsgithubworkflowsintegration_testyamlmain)
+
+In order to integrate Allure with your repository, perform the following actions:
+
+## 1. Adding allure-pytest and pytest collection plugin
+
+Please add the following into the `requirements.txt` that is called by the integration test -
+
+```
+allure-pytest>=2.8.18
+```
+
+Add the following line under the dependencies (`deps`) in the integration section inside `tox.ini` -
+
+```
+git+https://github.com/canonical/operator-workflows@main\#subdirectory=python/pytest_plugins/allure_pytest_collection_report
+```
+
+## 2. Calling the allure-workflow
+
+Add the following lines at the end of the workflow that runs the integrations tests by calling the reuable workflow [integration-test.yaml](https://github.com/canonical/operator-workflows/blob/main/.github/workflows/integration_test.yaml):
+
+```
+  allure-report:
+    if: always() && !cancelled()
+    needs:
+      - [list of jobs with tests you would like to visualize]
+    uses: canonical/operator-workflows/.github/workflows/allure_report.yaml@main
+```
+
+Here's an [example for the above](https://github.com/canonical/github-runner-operator/pull/412).
+
+**NOTE:** If the workflow is being called inside a matrix with the same test modules run with different parameters, the allure report will only display the results of the last combination.
+
+## 3. Changing branch permissions
+
+**NOTE:** For this, you would require admin access to the repository.
+
+- Go to the repository's **Settings > Branches** and next to Branch protection rules, select **Add rule**
+- Enter the branch name **gh-pages** and select **Allow force pushes** and click **Save changes**
+
+## 4. Github pages branch
+
+- Create `gh-pages` branch:
+
+```
+# For first run, manually create branch with no history 
+ # (e.g. 
+ # git checkout --orphan gh-pages
+ # git rm -rf . 
+ # touch .nojekyll 
+ # git add .nojekyll 
+ # git commit -m "Initial commit" 
+ # git push origin gh-pages
+ # ) 
+ ```
+
+ - Enable gh pages publishing at ** Settings > Pages ** and set branch name as `gh-pages`:
+
+<img width="816" alt="image" src="https://github.com/user-attachments/assets/346c04fc-0daa-40bc-92b5-93b0ea639f94">
+
+ [Example PR for steps 1 & 2](https://github.com/canonical/github-runner-operator/pull/412/files#)
@@ -171,6 +171,36 @@ jobs:
         )) || inputs.runs-on
       }}
     steps:
+      - uses: actions/[email protected]
+      - name: Integration tests variable setting
+        run: |
+          allure_artifact_suffix=$(uuidgen)
+          series=""
+          if [ ! -z ${{ matrix.series }} ]; then
+            series="--series ${{ matrix.series }}"
+            allure_artifact_suffix="$allure_artifact_suffix"-${{ matrix.series }}
+          fi
+          echo "SERIES=$series" >> $GITHUB_ENV
+          module=""
+          if [ ! -z ${{ matrix.modules }} ]; then
+            module="-k ${{ matrix.modules }}"
+            allure_artifact_suffix="$allure_artifact_suffix"-${{ matrix.modules }}
+          fi
+          echo "MODULE=$module" >> $GITHUB_ENV
+          echo "ALLURE_ARTIFACT_SUFFIX=$allure_artifact_suffix" >> $GITHUB_ENV
+      - name: Install tox
+        run: |
+          pipx install tox 
+      - name: Collect tests for Allure
+        working-directory: ${{ inputs.working-directory }}
+        run: |
+          tox -e ${{ inputs.test-tox-env }} -- --keep-models ${{ env.SERIES }} ${{ env.MODULE }} --allure-collection-dir=allure-default ${{ inputs.extra-arguments }} ${{ secrets.INTEGRATION_TEST_ARGS }}
+      - name: Upload Default Allure results
+        timeout-minutes: 3
+        uses: actions/upload-artifact@v4
+        with:
+          name: allure-default-results-${{ env.ALLURE_ARTIFACT_SUFFIX }}
+          path: ${{ inputs.working-directory }}allure-default/
       - name: Setup operator environment
         uses: charmed-kubernetes/actions-operator@main
         with:
@@ -266,14 +296,12 @@ jobs:
           VAULT_APPROLE_SECRET_ID: ${{ secrets.VAULT_APPROLE_SECRET_ID }}
 
       - run: sudo apt install skopeo -y
-
       - name: Plan Integration
         uses: canonical/operator-workflows/internal/plan-integration@main
         id: plan-integration
         with:
           plan: ${{ inputs.plan }}
-
-      - name: Integration tests variable setting
+      - name: Charm name setting
         working-directory: ${{ inputs.working-directory }}/${{ inputs.charm-directory }}
         run: |
           CHARM_NAME="$([ -f metadata.yaml ] && yq '.name' metadata.yaml || echo UNKNOWN)"
@@ -284,26 +312,23 @@ jobs:
 
           args="${{ steps.plan-integration.outputs.args }}"
           echo "ARGS=$args" >> $GITHUB_ENV
-          series=""
-          if [ ! -z ${{ matrix.series }} ]; then
-            series="--series ${{ matrix.series }}"
-          fi
-          echo "SERIES=$series" >> $GITHUB_ENV
-          module=""
-          if [ ! -z ${{ matrix.modules }} ]; then
-            module="-k ${{ matrix.modules }}"
-          fi
-          echo "MODULE=$module" >> $GITHUB_ENV
       - name: Run k8s integration tests
         working-directory: ${{ inputs.working-directory }}
         if: ${{ inputs.provider == 'microk8s' }}
         run: |
-          tox -e ${{ inputs.test-tox-env }} -- --model testing --keep-models ${{ env.SERIES }} ${{ env.MODULE }} ${{ env.ARGS }} ${{ inputs.extra-arguments }} ${{ secrets.INTEGRATION_TEST_ARGS }}
+          tox -e ${{ inputs.test-tox-env }} -- --model testing --keep-models ${{ env.SERIES }} ${{ env.MODULE }} ${{ env.ARGS }} --alluredir=allure-results ${{ inputs.extra-arguments }} ${{ secrets.INTEGRATION_TEST_ARGS }} 
       - name: Run lxd integration tests
         working-directory: ${{ inputs.working-directory }}
         if: ${{ inputs.provider == 'lxd' }}
         run: |
-          tox -e ${{ inputs.test-tox-env }} -- --keep-models ${{ env.SERIES }} ${{ env.MODULE }} ${{ env.ARGS }} ${{ inputs.extra-arguments }} ${{ secrets.INTEGRATION_TEST_ARGS }}
+          tox -e ${{ inputs.test-tox-env }} -- --keep-models ${{ env.SERIES }} ${{ env.MODULE }} ${{ env.ARGS }} --alluredir=allure-results ${{ inputs.extra-arguments }} ${{ secrets.INTEGRATION_TEST_ARGS }}
+      - name: Upload Allure results
+        timeout-minutes: 3
+        if: always() && !cancelled()
+        uses: actions/upload-artifact@v4
+        with:
+          name: allure-results-${{ env.ALLURE_ARTIFACT_SUFFIX }}
+          path: ${{ inputs.working-directory }}allure-results/
       - name: Tmate debugging session (self-hosted)
         if: ${{ failure() && (inputs.tmate-debug || runner.debug) && inputs.self-hosted-runner }}
         uses: canonical/action-tmate@main

@@ -103,6 +103,14 @@ jobs:
       integration-test-workflow-file: workflow_test.yaml
       working-directory: tests/workflows/integration/test-upload-charm/
       workflow-run-id: ${{ github.run_id }}
+  allure-report:
+    if: always() && !cancelled()
+    needs:
+      - integration
+      - integration-juju3
+      - integration-artifact
+      - integration-self-hosted
+    uses: ./.github/workflows/allure_report.yaml
   check:
     runs-on: ubuntu-latest
     if: always() && !cancelled()
@@ -119,6 +127,7 @@ jobs:
       - integration-craft
       - publish
       - publish-artifact
+      - allure-report
     steps:
       - run: |
           [ '${{ needs.simple.result }}' = 'success' ] || (echo simple failed && false)
@@ -132,3 +141,5 @@ jobs:
           [ '${{ needs.integration-craft.result }}' = 'success' ] || (echo integration-craft failed && false)
           [ '${{ needs.publish.result }}' != 'failure' ] || (echo publish failed && false)
           [ '${{ needs.publish-artifact.result }}' != 'failure' ] || (echo publish failed && false)
+          [ '${{ needs.allure-report.result }}' != 'failure' ] || (echo allure-report failed && false)
+
diff --git a/.licenserc.yaml b/.licenserc.yaml
@@ -32,4 +32,5 @@ header:
     - 'dist/**'
     - 'internal/**'
     - 'tests/workflows/integration/**'
+    - 'python/**/**/pyproject.toml'
   comment: on-failure
diff --git a/python/cli/README.md b/python/cli/README.md
@@ -0,0 +1,3 @@
+The `allure_add_default_for_missing_results` cli combines the Allure default results & actual results before generating the report. 
+
+Currently, if the result of a test is not available (for reasons such as setup failure), allure omits the test. This CLI ensures that for every test, if actual result is available, it will use that. Otherwise, it uses the default result `unknown`.
diff --git a/python/cli/allure_add_default_for_missing_results/__init__.py b/python/cli/allure_add_default_for_missing_results/__init__.py
@@ -0,0 +1,2 @@
+# Copyright 2024 Canonical Ltd.
+# See LICENSE file for licensing details.
diff --git a/python/cli/allure_add_default_for_missing_results/allure_add_default_for_missing_results.py b/python/cli/allure_add_default_for_missing_results/allure_add_default_for_missing_results.py
@@ -0,0 +1,51 @@
+# Copyright 2024 Canonical Ltd.
+# See LICENSE file for licensing details.
+import argparse
+import dataclasses
+import json
+import pathlib
+
+
+@dataclasses.dataclass(frozen=True)
+class Result:
+    test_case_id: str
+    path: pathlib.Path
+
+    def __eq__(self, other):
+        if not isinstance(other, type(self)):
+            return False
+        return self.test_case_id == other.test_case_id
+
+
+def main():
+    """Combine Allure default results & actual results
+
+    For every test: if actual result available, use that. Otherwise, use default result
+
+    So that, if actual result not available, Allure report will show "unknown"/"failed" test result
+    instead of omitting the test
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--allure-results-dir", required=True)
+    parser.add_argument("--allure-collection-default-results-dir", required=True)
+    args = parser.parse_args()
+
+    actual_results = pathlib.Path(args.allure_results_dir)
+    default_results = pathlib.Path(args.allure_collection_default_results_dir)
+
+    results: dict[pathlib.Path, set[Result]] = {
+        actual_results: set(),
+        default_results: set(),
+    }
+    for directory, results_ in results.items():
+        for path in directory.glob("*-result.json"):
+            with path.open("r") as file:
+                id_ = json.load(file)["testCaseId"]
+            results_.add(Result(id_, path))
+
+    actual_results.mkdir(exist_ok=True)
+
+    missing_results = results[default_results] - results[actual_results]
+    for default_result in missing_results:
+        # Move to `actual_results` directory
+        default_result.path.rename(actual_results / default_result.path.name)
diff --git a/python/cli/pyproject.toml b/python/cli/pyproject.toml
@@ -0,0 +1,20 @@
+[tool.poetry]
+name = "allure_add_default_for_missing_results"
+version = "0.1.0"
+description = ""
+license = "Apache-2.0"
+authors = ["Carl Csaposs <[email protected]>"]
+readme = "README.md"
+
+[tool.poetry.scripts]
+allure-add-default-for-missing-results = "allure_add_default_for_missing_results.allure_add_default_for_missing_results:main"
+
+[tool.poetry.dependencies]
+python = "^3.10"
+pyyaml = "^6.0.1"
+requests = "^2.31.0"
+
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
diff --git a/python/pytest_plugins/allure_pytest_collection_report/README.md b/python/pytest_plugins/allure_pytest_collection_report/README.md
@@ -0,0 +1 @@
+The `allure-pytest` plugin marks all the test statuses as unknown during collection time.
diff --git a/...ytest_plugins/allure_pytest_collection_report/allure_pytest_collection_report/__init__.py b/...ytest_plugins/allure_pytest_collection_report/allure_pytest_collection_report/__init__.py
@@ -0,0 +1,2 @@
+# Copyright 2024 Canonical Ltd.
+# See LICENSE file for licensing details.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		The `allure_add_default_for_missing_results` cli combines the Allure default results & actual results before generating the report.

		Currently, if the result of a test is not available (for reasons such as setup failure), allure omits the test. This CLI ensures that for every test, if actual result is available, it will use that. Otherwise, it uses the default result `unknown`.
swetha1654 marked this conversation as resolved. Show resolved Hide resolved
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		# Copyright 2024 Canonical Ltd.
		# See LICENSE file for licensing details.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		The `allure-pytest` plugin marks all the test statuses as unknown during collection time.