GitLab

Include a new test case based on previous one to take into account LAVA
jobs that use branch instead of revision.
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Test cases can use branch instead of revision to allow LAVA to use a
shallow clone and reducing the overlay size. Add its support to allow
the app the parse these LAVA test definitions.
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In order to be consistent across the whole document fix the format used
in the deployment section.
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Since in some cases it is needed to do testing with real data add steps
to import the database from production.
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Fix the command line use to launch application in development mode.
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

In order to allow git fetch to properly update a bare repository use
mirror, which implies bare and also maps refs.
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

To allow easy monitor the submitted jobs in another *gitlab* job dump
add the ability to dump the ids to a json file. This can then be used as
an artifact to pass on to a monitoring job.
Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>

The `qa-report-app` container does not allow to run with setting GitLab
authentication, which doesn't allow easy development.

Add a parameter to `qa-report-app` to disable authentication, and configure
the development container, i.e. docker-compose.override.yml, to disable
authentication and connection to Phabricator.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Retrieval of test cases by release are now done using remote branches, we
can avoid to check out the worktree and directly work on bare repository.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Even if its public and encode by Lava in job data, it's better to remove
the token from job_data for tests.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

The branches needs to be updated or removed before updating the testcases
directory.
Simplify the code by using pygit2.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Current version generates the testcases from default branches, i.e. the
latest release. As the test cases diverge from release to release, some of
the previous versions tests are not available.

This commit generates the list of all tests for each version.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

`target` images have been renamed to `hmi` during the v2022 development 
cycle, let's make sure this is taken into account.
Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

The recent changes in the testcases caused the results for minimal and
internal images to ber marked `N/A` on older releases like v2020 and
v2020, due to the app not really supporting diverging releases since it
only loads the testcase definitions from the apertis-test-cases HEAD.

To work around the issue, add some hacks until diverging releases are
properly supported, see https://phabricator.apertis.org/T8245.

This also reverts commit 86bf12d6

 "Update test screenshot"
which actually caught the issue but was mistakenly interpreted as the
desired outcome.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

During the process of adding support for fixed function image a bug wasi
introduced that caused only fixed function images reports where showed.

Fix the different conditions used to render the report between different
releases.
Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

This reflects the change in our infrastructure and the name of the
`qa-report-app` container.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

The minimal test are no more applicable as they have been replaced by
fixedfunction one, so the test image needs to be updated.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Comment added to bug task doesn't indicate the release version for which
the failure occurs, people need to open the links to the LAVA logs to find
whether the bug affect a release (like v2021dev2 vs. v2020 vs. etc.).

It should also indicate the board tag used in the test, like `cbg-0` or
`cbg-1`, to make it easier to identify board specific issues.

And it should have links to the involved images, like it is being done
when creating the task.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

The image URL is already provided and we don't have to create it manually.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

`PyYAML.load()` can create arbitrary Python objects, a malicious actor
could exploit this to run arbitrary code.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

With the move from Werkzeug 0.14 to 1.0 as part of the switch to
Bullseye the sytax to enable the ProxyFix middleware has changed
and the app was raising an exception:

    Traceback (most recent call last):
      File "/app/manage.py", line 26, in <module>
        from webhook import app, db
      File "/app/webhook.py", line 58, in <module>
        from werkzeug.contrib.fixers import ProxyFix
    ModuleNotFoundError: No module named 'werkzeug.contrib'

Use the new syntax as already documented in a comment.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

After the update to LAVA 2021.05 the `end_time` field of the callback
data got mis-populated as `None`. Since it's not a valid date value it
ends up causing an exception in the handler:

        lava_end_time         = get_timeobj(job.data['end_time']),
      File "/app/save.py", line 32, in get_timeobj
        return datetime.strptime(time_str_fixed_tz, '%Y-%m-%d %H:%M:%S.%f%z')
      File "/usr/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
        tt, fraction, gmtoff_fraction = _strptime(data_string, format)
      File "/usr/lib/python3.7/_strptime.py", line 359, in _strptime
        (data_string, format))
    ValueError: time data 'Nne' does not match format '%Y-%m-%d %H:%M:%S.%f%z'

The issue has bee reported upstream:
https://git.lavasoftware.org/lava/lava/-/issues/493



In the meantime, handle the error and carry on so that submitted test
results can be stored again.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Buster only has Flask 1.0.2-3, while Flask-Dance from PyPI (it's not in
Debian) has started requiring 1.0.4.

This causes Flask 2.0.1 to be retrieved from PyPI which breaks since
we only installed Werkzeug 1.x from Debian:

    $ pip3 install Flask-Dance Flask-Testing needle
    Collecting Flask-Dance
      Downloading https://files.pythonhosted.org/packages/87/b5/aee85ebf30f384e27aa1794fe51fee96416ee77b0402e768564bd8feebd2/Flask_Dance-5.0.0-py2.py3-none-any.whl (58kB)
    Collecting Flask-Testing
      Downloading https://files.pythonhosted.org/packages/a2/70/5611137c59b576ac36e9e8226f01cd138d4cd08688d5aad9eadfdaf6f57e/Flask-Testing-0.8.1.tar.gz (45kB)
    Collecting needle
      Downloading https://files.pythonhosted.org/packages/8b/ed/520d29ad333f38f323b897374cc5e7f91093b9613504dee7df7814e8081a/needle-0.5.0-py2.py3-none-any.whl (45kB)
    Requirement already satisfied: Werkzeug<2.1 in /usr/lib/python3/dist-packages (from Flask-Dance) (0.14.1)
    Requirement already satisfied: urlobject in /usr/lib/python3/dist-packages (from Flask-Dance) (2.4.0)
    Collecting Flask>=1.0.4 (from Flask-Dance)
      Downloading https://files.pythonhosted.org/packages/54/4f/1b294c1a4ab7b2ad5ca5fc4a9a65a22ef1ac48be126289d97668852d4ab3/Flask-2.0.1-py3-none-any.whl

 (94kB)
    ...
    $ su testuser -c 'pytest-3 --verbose -ra --showlocals --junitxml=apertis-qa-report.junit.xml'
    ...
    pluggy.manager.PluginValidationError: Plugin 'pytest_flask_dance' could not be loaded: (Werkzeug 0.14.1 (/usr/lib/python3/dist-packages), Requirement.parse('Werkzeug>=2.0'), {'Flask'})!

Bullseye has Flask 1.1.2-2 which should make everything happy for now.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Apertis is in the process of opening GitLab registratio to the public:
the assumption that all the GitLab users can be trusted is thus no
longer valid.

Let administrators configure a set of groups: if the logging in user is
a member of *any* of those let the login complete, otherwise fail and
ignore the attempt.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather than requiring the broad scope needed to call the regular GitLab
API to get their username, switch to the `openid` scope and call
the dedicated OAuth2 userinfo endpoint, see
https://docs.gitlab.com/ee/integration/openid_connect_provider.html

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

See https://docs.gitlab.com/ee/user/application_security/sast/

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Somehow Selenium/Chromium are now using different margins in the saved
page, so let's simply refresh the baseline. Such is the pain with visual
regressions tests.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use a lightweight runner to tag the Docker images so it has a much
reduced latency as it does not have to spin up a (expensive) VM
from scratch.

Other jobs are likely too heavy for the lightweight runner.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Sometimes test job fail for a multitude of reasons, so it's useful to be
able to re-trigger them and see the updated results in the report.

Unfortunately the report was not using any kind of consistent ordering,
so it may have picked results from older jobs even if a new one had been
completed for the same release/buildid/suites.

Tweak the report to pick jobs by the time the app gets notified about
them, so that when a re-triggered job gets submitted its results
override in the report the results from previous jobs.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Provide an overview of the results for each run by showing some
aggregated counters by deployment type, listing the number of
executed testcases, how many of them are automated vs. manual,
and pass/skip/fail/incomplete counters.

Do this with a single SQL query returning a nested JSON structure
with all the data in it.

This moves some complexity from Python code to a relatively large but
not overly complex SQL query.

As a shortcut, the SQL query uses the Postgres-specific `DISTINCT ON`
syntax to keep things simple and fast.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Flask's debug mode gets really unhappy if it can't resolve usernames,
so let's add libnss-unknown to prevent it from failing.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Enable the debug mode when using the development override.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Use a `docker-compose.override.yml` file (which is read by
`docker-compose` by default) to localy build the image and bind mount
the current folder with the live application code rather than using the
official production images and bind mounting only the secrets folder.

By using `docker-compose.override.yml` running `docker-compose up` is
still enough to set up the application.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

For some reason `docker-compose.yaml` had CRLF line ending. Change it to
use unix line endings to be consistent with everything else.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When available, we look for the checkout information to find the commit
id for each testcase. However, if the job fails in the early stages,
before being able to check anything out, then such information is not
available, so we should handle gracefully the case instead of hitting
an exception:

    Exception on / [POST]
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/flask/app.py", line 2292, in wsgi_app
        response = self.full_dispatch_request()
      File "/usr/lib/python3/dist-packages/flask/app.py", line 1815, in full_dispatch_request
        rv = self.handle_user_exception(e)
      File "/usr/lib/python3/dist-packages/flask/app.py", line 1718, in handle_user_exception
        reraise(exc_type, exc_value, tb)
      File "/usr/lib/python3/dist-packages/flask/_compat.py", line 35, in reraise
        raise value
      File "/usr/lib/python3/dist-packages/flask/app.py", line 1813, in full_dispatch_request
        rv = self.dispatch_request()
      File "/usr/lib/python3/dist-packages/flask/app.py", line 1799, in dispatch_request
        return self.view_functions[rule.endpoint](**req.view_args)
      File "./webhook.py", line 203, in post_results
        return process_job(job_data)
      File "./webhook.py", line 248, in process_job
        jobs.setdefault(job_id, TestJob(job_data)).update(job_data)
      File "/app/testobjects.py", line 216, in __init__
        self.update(rawdata)
      File "/app/testobjects.py", line 224, in update
        data = lava_jobdata_extract(rawdata)
      File "/app/testobjects.py", line 185, in lava_jobdata_extract
        jobdata.update(lava_jobdata_extract_results(data))
      File "/app/testobjects.py", line 150, in lava_jobdata_extract_results
        action, extra = actions[t['repository'], t['path']]
    KeyError: ('https://gitlab.apertis.org/tests/apertis-test-cases.git

', 'test-cases/sanity-check.yaml')

In turn, this exception causes jobs not being marked as completed,
and thus we never manage to update Phabricator since the batch is stuck
with pending jobs forever.
Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The image name already contains the full name of the image including
extension `img.gz` both for Apertis and Tiny containers images.
With this extension Phabricator tasks are generated with incorrect
names and paths to images, so need to remove that suffix.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Regenerate screenshots due updated Tiny containers job data.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>