This is failing for unknown reasons, but appears to work on other Git
repositories, so just disable it for now.

apertis-issues#114



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Add some proper documentation:
* add a short general introduction
* describe the mapping to OBS projects, which has been a common topic of
  discussion recently
* describe the extension points

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The first CI job tries to do the rebase, then proceeds to create merge
requests if all went well.

In case there is an issue with the rebase, the first CI job will stop
and give instructions on how to manually rebase the kernel.
Once the developer is done, the pushed branch will be picked up by the
second CI job to finish the work and create merge requests.

Signed-off-by: Detlev Casanova <detlev.casanova@collabora.com>

Git attributes for Git LFS are rarely without a reason in the upstream
tarballs, so assume they make sense if they’re in .gitattributes,
and copy them over into .git/info/attributes after an override to make
sure they still work and files managed by Git LFS can still be checked
out.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Emanuele Aina authored 2 years ago and Andre Moreira Magalhaes committed 2 years ago

The rework of `rules` in the recent "Build sources only once" commit
causes the `build-source` job to be triggered when pushing tags.

Add a global `workflow` rule to completely avoid triggering pipelines on
tags in any case.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Commit 477e59b5 "Reset the git repostiory after adding gitattributes"
ensured on the `pull-mirror` job that the checked out contents match
the current gitattributes configuration.

This extends such guarantee to all jobs, avoiding situations like:

1. the upstream `.gitattributes` enables the `ident` filter on a file
2. on checkout by the CI, `$Id$` placeholders are replaced
3. we defuse the `.gitattributes`, but it is too late since the working
   copy is already dirty
4. jobs like `pull-updates` fail due to the unexpected changes

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ensure that when we push to pristine-lfs-source we also always push
the matching release tag.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Rather surprisingly, `git push origin --follow-tags pristine-lfs-source`
ends up pushing the release tag as well.

Let's pass `--follow-tags` only when appropriate to avoid confusing
behaviors.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Simplify the pipeline by:
1. computing if this is a release build in the preparation stage
2. building the sources only once
3. avoiding the _build-{snapshot|release} directory dance

This saves some resources: on big packages building the sources can
take a non-trivial amount of time and doing it twice in a single run is
not the greatest idea ever.

For instance, `texlive-extra` needs more than 30 minutes for each
*source* build and with this we build it only once.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `ci-buildpackage` tool uses `$RELEASE` to decide whether it should
tweak the changelog version to ensure development builds sort lower than
the actual release builds.

When doing so, it hardcodes `apertis` as the target distribution. Rather
than passing even more environment variables, just set the postexport
hook explicitly.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Because, for many repositories which carry Carriage Return Line Feeds,
aka CR;LF, just adding the .gitattributes is not enough. For it to make
effect for the already cloned repository, a `git reset --hard` is needed

Consider the below:

https://gitlab-apertispro.boschdevcloud.com/pkg/bnd/-/jobs/933233



1. The repository is cloned (with CRLF files)
2. A .gitattributes is added to ignore CRLF files
3. Changes aren't effective immediate

The above pipeline job failed because of the order of events. This
proposed MR just does a `git reset --hard` effectively bringing the
`.gitattributes` into action.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

This adds support for using the OBS runner to upload, monitor, and
clean up after packages. This can be controlled via two new variables:

- `OBS_RUNNER=auto/enabled/disabled`: Enables/disables the runner,
  `auto` will only enable it on builds for v2023*.
- `OBS_RUNNER_TAG=`: Customize the tag used by the runner, useful for
  testing via the runner's test instance (`obs-runner-test`).

As the runner cannot run any shell, code detecting package / build
information has been moved to the preparation job, and code selecting
which .dsc file to use has been moved to the jobs generating the .dsc
files.

In addition, the tests have been expanded to run through *both* code
paths, to ensure neither of them break from new changes. (This *does*
result in significantly longer CI run times, as the entire testing flow
is run twice now.)

https://phabricator.apertis.org/T7895



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

https://phabricator.apertis.org/T7895



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

If CI/OBS are under too much load, the tests will sometimes take a bit
longer than an hour, so bump the timeout to avoid problems.

https://phabricator.apertis.org/T7895



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

With v2022 having entered RC, we should now not land anything directly
onto the main branch. All updates/fixes should flow through -updates and
-security henceforth

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Since we are releasing v2022 RC we entered a hard freeze, from this point,
changes should land to either updates or security.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

Emanuele Aina authored 3 years ago and Andre Moreira Magalhaes committed 3 years ago

GitLab can show a textual description for variables that can be set when
manually triggering a pipeline, see
https://docs.gitlab.com/ee/ci/yaml/index.html#variablesdescription



Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Otherwise, this will also run these jobs on tags, except none of the
dependencies are running there, so it just results in failed pipelines.

https://phabricator.apertis.org/T7890



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This MR documents the Extra APT Repository feature with some example
screenshots.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

It is desirable to inject extra apt repository to pull in packages from.
This is useful in scenarios where packages from one repository could
have (build) dependencies from other repositories.

This feature expects 3 environment variables, from which it constructs
the extra apt repository information

* EXTRA_REPO_BASE_URL: https://repositories-apertispro.boschdevcloud.com
* EXTRA_REPO_VERSION: unset (we want to derive from branch name here)
* EXTRA_REPO_COMPONENTS: "hmi target development sdk"

The base url also supports password protected repositories, in the form
of: https://username:password@hostname/



The repository version may be specified from the environment variable.
In the absence of the value, it is determined from the current branch
name, like: `apertis/v2022`, which will result in the release name as:
`v2022`

Multiple components can be specified, separated by space.

These environment variables can be fed to the CI build environment
throuh the CI variable settings, on a per repository basis, or for
entire repository groups

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This will preserve the project until the corresponding branches are
deleted or the MR is merged, as well as being more reliable in
performing the cleanup tasks.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Ryan Gonzalez authored 3 years ago and Emanuele Aina committed 3 years ago

ci-package-builder's extensive use of `rules:` means that every push to
an MR branch results in *two* pipelines: one outside the MR context
(i.e. for the branch itself) and one in the MR's context:

https://docs.gitlab.com/ee/ci/jobs/job_control.html#avoid-duplicate-pipelines



In order to work around this, `run_on_mr_xor_branch_rule` is used, which
will only run a pipeline iff either a) there is no MR, or b) it is
running inside the MR context. That way, pipelines will run against
plain branches normally, but once an MR is opened, only the MR pipelines
will be run. (Running them in MR context is preferred because then we
can use the target branch name to figure out what release is being
targeted by the MR.)

However, if:

- An MR is merged against a release branch, e.g. `apertis/v2023dev1`
- The release pipeline encounters an error, or it cannot be run yet due
  to CI load.
- A new MR is opened to backport the changes, e.g. `apertis/v2023dev1`
  -> `apertis/v2022`.
- The original release pipeline for `apertis/v2023dev1` is now run or
  re-run.

Because an MR is open with `apertis/v2023dev1`,
`run_on_mr_xor_branch_rule` will now prevent the release pipeline from
running at all.

!228 was intended to fix this, but instead it resulted in backport MRs
uploading their CI results to the release repos (!231). Instead, we can
just change `run_on_mr_xor_branch_rule` to exclude blocking release
branches, so their pipelines can run normally.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

The current code falls back to `$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME`
when figuring out if a branched OBS project should be the target of
the upload:

    if [[ ${CI_COMMIT_BRANCH:-$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME} != $OSNAME/* ]];
    then
      # upload to branched OBS project
    fi

The big bug is that a MR for a backport going straight from
`apertis/v2023dev1` to `apertis/v2022` will skip the body of that `if`
and will actually upload to the production `apertis:v2022:*` OBS
project as soon the MR is **created** rather than when it is merged.

On a MR we should always upload to a branched project: to do that, we
can rely on the fact that `$CI_COMMIT_BRANCH` is not set in merge
request context and simply drop the fallback.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ryan Gonzalez authored 3 years ago and Andre Moreira Magalhaes committed 3 years ago

https://phabricator.apertis.org/T7890



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

https://phabricator.apertis.org/T7890



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

When creating a backport MR, for instance from `apertis/v2023dev1` to
`apertis/v2022`, the HEAD will have a release tag created when it landed
on the apertis/v2023dev1` branch.

The presence of the tag causes the `build-source` job to exit early, but
that meant that no `_build` artifacts directory is populated.

The `upload` job will not have anything to upload to OBS and will fail.

Extract the tagged source from the `pristine-lfs-source` branch in the
`build-source` job as well to have it ready for `upload`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The comment in 'after_script' is actually no longer valid, because it now
uploads the cleanup-project-branch artifact whether or not the build
succeeded. As a result, the cleanup job still runs and attempts to remove the
branch, even though it was already removed in the after_script.

https://phabricator.apertis.org/T7890



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

This also removes snapshots, since running builds without releasing can
now be done by simply using the second branch.

https://phabricator.apertis.org/T7890



Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

Thanks to @refi64 for making me discover that `osc branch` already had
this functionality built-in. D'oh.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Emanuele Aina authored 3 years ago and Ryan Gonzalez committed 3 years ago

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

The `pkg/dash` repository got fixed and now it got tagged properly,
the workaround is no longer needed.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Ritesh Raj Sarraf authored 3 years ago and Emanuele Aina committed 3 years ago

Commit 5802517d "pull-updates: Push all local refs atomically" broke
pushing multiple refs when pulling updates (that ism when pulling a
new upstream version) due to excessive quoting.

    $ git push --follow-tags --atomic ${CI_AUTH_PROJECT_URL} "$BRANCHES"
    fatal: invalid refspec 'debian/buster
    debian/buster-security
    pristine-lfs
    upstream/buster'

Let `$BRANCH` get expanded to multiple arguments rather than passing all
of them as a single argument.

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Avoid ending up in a inconsistent state by ensuring that we push all the
local refs (the `debian/*` branches, the `upstream/*` branches, their
tags and the `pristine-lfs` branch) at the same time.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Being able to copy and paste the actual command being run makes
reproducing issues locally a bit easier.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

No need to tweak the destination branch every time we branch a new
release, just target the default branch.

This is not 100% the same thing: packages that got dropped at some point
will have the default branch set to a different release (for instance,
`apertis/v2021`).

In those cases, either:
1. no `debian/bullseye` branch exist, so this is not an issue
2. if we want to land the `debian/bullseye` updates somewhere, trying
   with a non-existent branch name is not very useful anyway

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>