Update the documentation to describe using short-lived tokens

We now have the mechanisms in place to use short-lived tokens within
CI pipelines. There are some complexities here, especially around
their use within test-cases. I've also skipped describing in any
detail the issues around the Lava runner.

content/guides/lava-apertis-testing.md

@@ -452,6 +452,101 @@ basically checks that the system actually works, and `add-repo`, which
 isn't actually a test, and is used to add repositories to
 `/etc/apt/sources.list` on certain devices.
 
+### Using short-lived CI tokens
+
+Gitlab provides a short-lived token called `CI_JOB_TOKEN` which can be
+used to give access to the contents of internal and private
+repositories during CI runs. From `apertis/v2023dev3` we can make use
+of this token, using a different approach to job submission to the one
+described in the previous sections. That is, so far in this document,
+we've run `lava-submit.py` to batch upload the jobs generated by
+`generate-jobs.py` to LAVA. If we do the same thing in our CI
+pipeline, then the CI job will terminate shortly after the jobs are
+uploaded, invalidating our job token.
+
+{{% notice warning %}}
+Do not expose `CI_JOB_TOKEN` to the wider public by submitting
+publicly visible jobs. You should submit jobs with tokens in them as
+`private`.  You should also [reduce the privileges of job
+tokens](https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#configure-the-job-token-scope-limit)
+when using `CI_JOB_TOKEN` in LAVA jobs.
+{{% /notice %}}
+
+For this reason, instead of using `lava-submit.py`, we use a different
+tool, `generate-test-pipeline.py`, from the same repository when
+running CI tests. This makes a dynamic Gitlab pipeline to run the
+generated jobs. Each LAVA job will have its own Gitlab job to track
+it, and that means there is a short-lived token available that will
+remain valid for as long as the LAVA job
+runs. `generate-test-pipeline.py` can be found
+[here](https://gitlab.apertis.org/tests/apertis-test-cases/-/blob/apertis/v2023dev3/generate-test-pipeline.py).
+
+There are two different places you might want to use such tokens with
+LAVA, and they require slightly different approaches.  To use a
+short-lived token to gain access to a repository from a LAVA job
+description, for example to obtain test files from a private
+repository, the repository URL needs to be altered to show where to
+substitute the token.
+For example:
+
+       https://gitlab-ci-token:{{ '{{job.CI_JOB_TOKEN}}' }}@gitlab.apertis.org/tests/apertis-test-cases.git
+
+The odd appearance is because two rounds of templating are occurring:
+we escape the template for the job token so that `generate-jobs.py`
+will preserve it. When our dynamic pipeline runs, the LAVA runner
+will substitute its own value for `CI_JOB_TOKEN`.
+
+To use a short-lived token from within a test-case, we need to do two
+things. First, we need to add a parameter to the test's group template
+with the full URL for the repository we wish to include. The group
+templates form part of the job definition, and so we can modify the
+URL in exactly the same way as before.
+
+Secondly, we need to replace the repository URL in the test case with
+the new parameter. You cannot use templating within test cases
+themselves, you must setup a parameter or environment variable in the 
+job definition that the test case can use. Parameters are preferable
+because they can be used in the `install` section of a test.
+
+Putting things together, let's look at a section of a group template
+that:
+- Pulls test case files from `apertis-test-cases` using a short-lived
+  token.
+- Sets up a parameter which contains the URL to clone `glib-gio-fs`
+  using a short-lived token as authentication. We can use this
+  parameter in a test case to obtain our test data.
+
+```yaml
+  - test:
+      timeout:
+        minutes: 180
+      namespace: system
+      name: {{group}}-tests
+      definitions:
+{%- for test_name in tests %}
+        - repository: https://gitlab-ci-token:{{ '{{job.CI_JOB_TOKEN}}' }}@gitlab.apertis.org/tests/apertis-test-cases.git
+          branch: 'apertis/v2023dev3'
+          history: False
+          from: git
+          name: {{test_name}}
+          path: test-cases/{{test_name}}.yaml
+          parameters:
+            EXAMPLE_REPO_URL: |-
+              https://gitlab-ci-token:{{ '{{job.CI_JOB_TOKEN}}' }}@gitlab.apertis.org/tests/glib-gio-fs.git
+{%- endfor -%}
+```
+
+We could then amend our test-case in `apertis-test-cases` to use the
+parameter like this (note that there is no `$` when substituting the
+parameter in an `install` section):
+
+```yaml
+install:
+  git-repos:
+    - url: EXAMPLE_REPO_URL
+      branch: 'apertis/v2023dev3'
+```
+
 ### Non-public jobs
 
 These instructions are written to submit LAVA jobs for **ONLY PUBLIC**