https://phabricator.apertis.org/T8509



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Since the sdk jobs had to deal with multiple artifacts/branches (Sdk,
Sdk.Debug and Sdk.Sources), both the build and publish jobs got
completely duplicated from the platform jobs.

Refactor things so that it's sufficient to customize a variable and
any duplication is avoided.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Publishing right now fails with:

    Updating runtime/org.apertis.mildenhall.Platform/x86_64/v2021 b03a4670776c73c72416be7d8009edc4af09999d43247b20fef823d9cf81b37b to b45aae6c96b1698fbd7a58839ee31b6b5a07cdae62f14cd56aa40b5c39a5f09a
    Traceback (most recent call last):
      File "/usr/bin/ostree-push", line 588, in <module>
        exit(push_main())
      File "/usr/bin/ostree-push", line 561, in push_main
        return pusher.run()
      File "/usr/bin/ostree-push", line 426, in run
        needed = self.needed_commits(revs[0], revs[1], commits)
      File "/usr/bin/ostree-push", line 361, in needed_commits
        'commit %s' %(remote, local))
    __main__.PushException: Remote commit b03a4670776c73c72416be7d8009edc4af09999d43247b20fef823d9cf81b37b not descendent of commit b45aae6c96b1698fbd7a58839ee31b6b5a07cdae62f14cd56aa40b5c39a5f09a
    Fix publishing by ensuring generated commits are child of the current

This is because we currently commit to the OSTree repository *before*
pulling the remote HEAD and as a result the generated commits have the
null parent.

Fix that by ensuring we pull the metadata of each current HEAD before
generating the new commits with flatdeb.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The base step generates a plain sysroot tarball and does not involve
ostree at all. To save some confusion, do not pass the ostree
repository path as an argument.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ariel D'Alessandro authored 3 years ago and Emanuele Aina committed 3 years ago

In some cases, pulling a remote repo with --mirror option is not
creating refs properly, so the GitLab CI pipeline fails when the script
tries to reference the target branch.

In the past, before switching to GitLab CI, the local ostree repo was
pushed to remote using rsync. This required a complete mirror of the
remote repo to be pushed, so the initial pull was run with --mirror
option enabled.

There's no need to pull mirroring the complete remote repo as the GitLab
CI pipeline is now ensuring the remote repo is initialized before
pushing the new commit/branch, i.e. running:

  * ssh ${upload_host} ostree init ...
  * ostree-push ...

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The checkout of the remote commits fails with:

    error: opening repo: opendir(/builds/infrastructure/apertis-flatdeb-mildenhall/repo): No such file or directory
    No remote branch runtime/org.apertis.mildenhall.Platform/x86_64/v2021 found

Fix the typo that caused the use of a literal `repo` instead of the
`$repo` variable.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The artifacts here are pretty big and they get pushed to the public
publishing website anyway, there's no need to keep them around for long.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The detection of the target release is quite broken:

1. `APERTIS_RELEASE` is set to v2021 as a CI variable
2. `APERTIS_RELEASE` is blindly overridden by trying to derive it
   from `$CI_COMMIT_BRANCH`
3. if autodetection fails, for instance because a wip branch does not
   contain the release id in its name, APERTIS_RELEASE is left empty
4. if APERTIS_RELEASE is empty it defaults to `$APERTIS_RELEASE_DEFAULT`
   but the latter is not set anywhere so APERTIS_RELEASE will remain
   emtpy

Drop the autodetection stuff and rely on the already hardcoded release
identifier. We're not going to update this for newer releases anyway,
since Mildenhall has been dropped in v2022.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Remove curl call and use ostree refs instead.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

The URL has an extra `public/` component that woekrd only by mistake.

This has been fixed a while ago on the server, so let's restore
functionality on this side as well.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

Arnaud Ferraris authored 3 years ago and Emanuele Aina committed 3 years ago

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 3 years ago and Emanuele Aina committed 3 years ago

Similarly to what has been done in apertis-flatdeb-demo, the CI pipeline
has been extended to move repository publishing away from the build
stages. It now builds the repos for all supported architectures, while
automatically generating the required files based on the branch name.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Arnaud Ferraris authored 3 years ago and Emanuele Aina committed 3 years ago

The suite file is a simple declarative file not likely to evolve from
one release to the next. It can therefore be automatically generated
using a simple shell script. This will make updating it for new apertis
releases easier and will allow CI to generate the proper files based on
the branch name.

Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>

Build fails when installing runtime while a runtime or an app is pushed
to the repository. We can't use rsync to upload a runtime or an app.

This download commit metadata from the repository before build, then upload
using `ostree-push` instead of `rsync`. The repository is then updated
using `flatpak build-update-repo`.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This recipe has been move to its own branch as it is not release specific.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Move runtime and SDK to v2021pre.
The suite recipe name should be in sync with the Apertis version.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This fix runtime installation path in recipe used to build
applications, and get the correct version.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

New flatdeb-builder docker image already embbed packages request to build
and upload the Flatpak runtime and apps.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

New flatdeb-builder docker image uses latest debos version and does not
require kvm to be run.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This new docker image already embbed the apertis-flatdeb tool.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Split the creation of base, platform and sdk runtime as process fails to
access to network on 2nd call to debos, with the following errors:
  Error: Os { code: 98, kind: AddrInUse, message: "Address already in use" }
  Core dump limits :
  	soft - NONE
  	hard - NONE
  Checking that ptrace can change system call numbers...OK
  Checking syscall emulation patch for ptrace...OK
  Checking advanced syscall emulation patch for ptrace...OK
  Checking environment variables for a tempdir.../scratch
  Checking if /scratch is on tmpfs...OK
  Checking PROT_EXEC mmap in /scratch...OK
  Adding 3395584 bytes to physical memory to account for exec-shield gap
  Event loop failed: Connection timed out
  …
  2020/09/09 07:04:39 apt-install | E: Failed to fetch https://repositories.apertis.org/apertis/pool/hmi/m/mildenhall/mildenhall_0.2020.1bv2021dev1b1_all.deb

  Temporary failure resolving 'repositories.apertis.org'
  2020/09/09 07:04:39 apt-install | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
  2020/09/09 07:04:39 apt-install | Container root failed with error code 100.
  2020/09/09 07:04:39 apt-install |
  2020/09/09 07:04:39 apt-install | ERROR:flatdeb.apt-install:Command '['systemd-nspawn', '--directory=/scratch/root', '--as-pid2', '--tmpfs=/run/lock', '--register=no', 'env', 'DEBIAN_FRONTEND=noninteractive', 'apt-get', '-y', '--no-install-recommends', 'install', 'libc6:amd64', 'libstdc++6:amd64', 'libsystemd0', 'mildenhall']' returned non-zero exit status 100.
  2020/09/09 07:04:39 Action `run` failed at stage Run, error: exit status 1

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This will prepare split of build-runtime into build-{base,runtime,sdk}.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Sample application `cpp-library-example` includes opencv and needs `cmake`
and `libjpeg-dev` in the SDK to build.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Current app build failed with following error:
```
$ rsync -e "ssh -oStrictHostKeyChecking=no -i $ARCHIVE_SECRET_FILE" -aP ~/.cache/flatdeb/ostree-repo/. "$upload_host:$UPLOAD_ROOT"
sending incremental file list
rsync: change_dir "/root/.cache/flatdeb/ostree-repo" failed: No such file or directory (2)
```
This commit create a temporary repo directory used by `flatpak-builder`.
Set correct rights so the repository is accessible after upload.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Uploading runtime reset flatpak summary and hide other uploaded bundles.
Running `flatpak build-update-repo` fixes it.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This recipe should be used in Mildenhall projects to automatically build
the flatpak bundle.
This install the runtime and SDK from
`https://images.apertis.org/public/flatpak/mildenhall`

, then build the
bundle using flatpak-recipe.yaml found at project's root.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Embed runtime upload to `images.apertis.org` in `build-runtime` job
instead of a separated job which consume runners and time.
Main runtime is uploaded to `/srv/images/public/flatpak/mildenhall`
MRs are uploaded to `/srv/images/test/flatpak/mildenhall/$CI_COMMIT_REF_SLUG`

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

`helloworld-app` needs `gobject-introspection` and `hotdoc-0.8` to build.
Add them to the flatpak org.apertis.mildenhall.Sdk.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Artifacts failed to upload in gitlab-ci due to its size.
This should be fixed later.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

`v2021dev2-package-builder` is only available from
`registry.gitlab.apertis.org`

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

The runtime and SDK are uploaded to
https://images.apertis.org/public/flatpack/mildenhall



Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Apertis development repository branch is added in runtime recipe, and not
in suite recipe, using pre_apt_script to avoid conflicts between GPLv2 and
GPLv3 packages.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>