qa-report-app-admin.md

title = "QA Report App Administration"
toc = true
date = "2022-10-07"

QA Report App is the Apertis tool for reporting and viewing the results of both manual and automated (via LAVA) QA testing.

Primary Configuration File

QA Report App's configuration is handled via the file secrets/config.yaml.

General Options

Resource Links and virtual location

QA Report App needs to be able to generate links to the test cases and images:

app-url: https://qa.apertis.org
image-root: https://images.apertis.org

app-url should be the URL where QA Report App is running at, and image-root should point to the web location where images downloads are available at.

Additionally, in cases where the app needs to be mounted under a specific URL prefix (its virtual location), the url-prefix option can be set:

url-prefix: /URL-PREFIX

Where the prefix must start with a leading slash (/). Setting this option will cause the prefix to be automatically added to all URLs generated by the application.

Behind the scenes this will cause the WSGI SCRIPT_NAME to be set to this prefix and the PATH_INFO of any request which is prefixed with this value to be stripped of the prefix.

Debug Logging

Debug logging from the service can be enabled via:

debug: true

Open ID Connect user login

To activate authentication, the following configuration options must be set:

openid:
  client-id: CLIENT-ID
  client-secret: CLIENT-SECRET
  well-known-url: WELL-KNOWN-URL
  always-require-login: BOOL  # (optional)

Both client-id and client-secret are created by the OpenID Connect identity provider. It should also provide the well-known-url.

E.g.: For using a Gitlab instance as OpenID Connect identity provider, an application must be added on gitlab, with the openid scope and the https://<qa-report-app-server>/openid_callback callback URL. For Gitlab, the well-known-url value is https://<gitlab-instance>/.well-known/openid-configuration.

always-require-login is an optional boolean value. When set to true, the website will require users to be logged in to access any page. This is set to false by default.

Authentication groups

In addition to the login options described above, a set of authentication groups can be defined for a more fine grained control on which users/groups are allowed to login and their respective permissions:

auth-groups:
  - name: GROUP-1
  - name: GROUP-2
    extra-perms:
    - submit-reports
    - tag-images

Each group can optionally configure a set of extra permissions (defaults to empty). The supported values for extra-perms are:

• submit-reports: users in this group can submit manual test reports
• tag-images: users in this group can tag images

If auth-groups is omitted or empty (default), any user with the right credentials will be able to login, submit reports and tag images.

LAVA Authentication

QA Report App uses shared secrets to authenticate legitimate LAVA callback submissions, which can be set in the config file:

lava-callback-tokens:
  - TOKEN-1
  - TOKEN-2
  - ...

Additionally, they can be set in the environment.

In order to obtain a new token, you need to create a new token in the LAVA UI (API → Authentication Tokens) for the user submitting the notify callback. The token's name should match the name set in the notify→callback→token property of the LAVA job definition YAML.

Filter displayed releases

The display configuration section can be used to alter the minimal version to be displayed, as well as the number of results displayed for each release on the main page.

This makes it easier to go through the results.

The section is optional and can be configured as follows:

display:
  min-version: MIN-VERSION
  series-results:
    dailies: RESULT-COUNT
    releases: RESULT-COUNT
    weeklies: RESULT-COUNT

MIN-VERSION is a string value that specifies the minimal version to be shown. E.g.: 'v2021'. All results for versions inferior to MIN-VERSION will not be shown on the main page.

Each RESULT-COUNT is an integer value that specifies the number of results to show for each release. E.g.: At the start of year 2023, with

display:
  min-version: 'v2022'
  series-results:
    dailies: 7
    releases: 2
    weeklies: 4

the displayed results will be:

• for releases: show results for v2024dev0.0, v2023.0 & v2023pre.0 (last 2 release results in the v2023 series) and v2022.3 & v2022.2
• for dailies: show last 7 results for each of v2024, v2023 and v2022 daily builds
• for weeklies: show last 4 results for each of v2024, v2023 and v2022 weekly builds

Defining Supported Platforms and Deployments

QA Report App is designed to take in test case results from a given platform that the test cases were run on, such as fixedfunction-armhf-uboot-public or minimal-armhf-uboot-internal, along with a deployment describing how the OS was installed onto the platform (e.g. apt, ostree, lxc, nfs).

The support platforms and deployments can be configured via the images section of the configuration file:

images:
  architectures:
    ...
  variants:
    ...
  deployments:
    ...
  platform-groups:
    ...

Defining Variants and Architectures

Image platforms as described contain both variants (different versions of the same distribution) and architectures. For example, given the following platforms:

• fixedfunction-armhf-uboot-public
• tiny-lxc-amd64-amd64-uefi-internal

Each of these starts with a variant (fixedfunction, tiny-lxc) that is immediately followed by the architecture (armhf, amd64). The supported variants and architectures should be defined at the top of the images section of the config file:

images:
  architectures:
    - amd64
    - arm64
  variants:
    - fixedfunction
    - tiny-lxc

Defining Deployments

images:
  deployments:
    user-selectable:
      NAME-1: DESCRIPTION 1
    lava-only:
      - NAME-2
      - NAME-3

Deployments that can be selected by the user when manually submitting test results go in the user-selectable section, with each deployment name mapped to a human-readable description. For instance, we may have:

user-selectable:
  ostree: OSTree

In this case, ostree, is the deployment name, and OSTree is the human-readable description that will be shown in the UI.

If, instead, a deployment should not appear for manual test result submission, but instead only be available for automatically reported LAVA results, the deployment names should be placed in the lava-only section, e.g.:

lava-only:
  - lxc
  - nfs

No human-readable name is required, since these are never shown explicitly in the UI.

Defining Platforms

images:
  platform-groups:
    - platforms:
        PLATFORM-1: DESCRIPTION 1
        PLATFORM-2: DESCRIPTION 2
      deployments: [DEPLOYMENT-1, DEPLOYMENT-2]
      supported-by:
        - first:
            release: FIRST-RELEASE
            version: FIRST-VERSION
          last:
            release: LAST-RELEASE
            version: LAST-VERSION
    - ...

Image platforms, such as fixedfunction-armhf-uboot-public, are defined in groups called "platform groups". Each group contains:

• A list of platforms defined in this group.
• A list of deployments supported by the given platforms.
• (Optional) A list of release/version ranges that support the given platforms.

The purpose of placing platforms into these groups is to be able to define multiple platforms that support the same deployments same conditions in one place, reducing repition.

Example of some basic platform groups:

platform-groups:
  - platforms:
      tiny-lxc-armhf-uboot-public: Tiny LXC ARM
      tiny-lxc-arm64-uboot-public: Tiny LXC ARM64
      tiny-lxc-amd64-uefi-public: Tiny LXC AMD64
    deployments: [lxc]

  - platforms:
      nfsroot-armhf-uboot-public: Nfsroot ARM
      nfsroot-arm64-uboot-public: Nfsroot ARM64
      nfsroot-amd64-uefi-public: Nfsroot AMD64
    deployments: [nfs]

Platform Support Ranges

supported-by:
  - first:
      release: FIRST-RELEASE
      version: FIRST-VERSION
    last:
      release: LAST-RELEASE
      version: LAST-VERSION
  - first:
      release: FIRST-RELEASE
      version: FIRST-VERSION
    last:
      release: LAST-RELEASE
      version: LAST-VERSION

The supported-by key defines a list of ranges of releases that support the platforms in the current group.

The individual ranges inside have two keys, at least one of which must be given:

• first: the first / oldest version that supports this platform. If not given, it is assumed that the platform has been supported since the very first release.
• last defines the last / newest version that supports this platform. If not given, it is assumed that the platform is still supported by the very latest release.

Each of these endpoints of the range in turn contains release and version, to set the release and version of the given endpoint. The version is optional and, if not given, all versions in the release are counted. For example, this platform group is valid for all releases from v2021 to v2022 version 20220624:

supported-by:
  - first:
      release: v2021
    last:
      release: v2022
      version: '20220624'

The end ranges can be defined to be exclusive instead of inclusive, via:

supported-by:
  - first:
      release: v2021
      inclusive: false

This would normally mean "every release since and including v2021 is supported", but setting inclusive: false changes it to mean "every release since and not including v2021".

Multiple support ranges can be given:

supported-by:
  # Supported from v2019 to v2021 and from v2022dev0 to v2022dev2.
  - first:
      release: v2019
    last:
      release: v2021
  - first:
      release: v2022dev0
    last:
      release: v2022dev2

Some more examples:

# Supported from every release since, and including, v2022dev3, 20210910:
supported-by:
  - first:
      release: v2022dev3
      version: '20210910'

# Supported from every version *after* v2021, 20210910 up until, and including,
# v2022.
supported-by:
  - first:
      release: v2021
      version: '20210910'
      inclusive: false
    last:
      release: v2022

Changing the Test Cases Source

Test cases are, by default, pulled from the Apertis test cases Git repository. The individual YAML files describing test cases are then taken from every branch starting with apertis/, with the text afterwards being treated as the release (e.g. given apertis/v2020, the release would be v2020). If you want to change the URL and the branch prefix (the apertis/), set test-cases-repository:

test-cases-repository:
  url: THE-URL
  branch-prefix: THE-BRANCH-PREFIX

where:

• URL is a URL pointing to a Git repository containing the test cases.
• BRANCH-PREFIX is the prefix

Note that the portion of the branch name after the branch prefix is treated as the release/suite.

Bug Tracker Integration

Phabricator

bug-tracking: phabricator
phabricator-url: https://PHRABRICATOR-URL/
phabricator-token: PHABRICATOR-TOKEN
phabricator-tasks:
  space: S2
  filter-tag: PRIMARY-TAG
  tags:
    - ADDITIONAL-TAG-1
    - ADDITIONAL-TAG-2
    - ...

This sets up integration with Phabricator, for automated issue creation. PHABRICATOR-TOKEN is a Phabricator Conduit API token, which can be obtained via the following steps:

• Click your profile picture in Phabricator on the top-right -> Settings.
• Select Conduit API Tokens in the left sidebar.
• Click the Generate Token button.

The phabricator-tasks section controls the behavior of tasks that are automatically created by QA Report App from LAVA failures.

space is the Phabricator space that should have visibility into the created tickets.

filter-tag is a Phabricator tag should identify all tasks automatically filed; new tasks filed will have this tag assigned, and existing ones will be searched using this tag. tags defines any additional tags that should be added when opening a new ticket (but these are not used for any other purpose).

GitLab

bug-tracking: gitlab
gitlab-url: https://GITLAB-URL/
gitlab-token: GITLAB-TOKEN
gitlab-project: GITLAB/PROJECT
gitlab-issues:
  filter-label: PRIMARY-LABEL
  labels:
    - ADDITIONAL-LABEL-1
    - ADDITIONAL-LABEL-2
    - ...

This sets up integration with GitLab issues, for automated issue creation. GITLAB-TOKEN should be a [GitLab access token] https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token) (use the api scope), and GITLAB/PROJECT should be the GitLab project where issues are stored (e.g. infrastructure/apertis-issues).

The gitlab-issues section controls the behavior of issues that are automatically created by QA Report App from LAVA failures. filter-label is a GitLab label that should identify all issues automatically filed; new issues filed will have this label assigned, and existing ones will be searched using this label. labels defines any additional labels that should be added when opening a new issue (but these are not used for any other purpose).

Note that gitlab-url is additionally used for configuring OAuth logins and thus may be set even if GitLab issues integration is not set up.

Environment for docker-compose

In addition to the standalone configuration file, a separate file .env, providing environment variables to docker-compose, can be created in the qa-report-app directory. These variables are used to configure hosting aspects, rather than the behavior of the application itself.

The following variables are available to be set, as well as their default values:

# The port the application will be exposed on.
QA_REPORT_APP_PORT=28080
# The user ID and group ID the application and database will run as. In
# particular, this results in the database files also being owned by this user
# and group.
RUN_USER=1000:1000
# If lava-callback-tokens is not set in the config file, it should be set here.
LAVA_CALLBACK_TOKENS=["TOKEN-1", "TOKEN-2"]