To import a new package from Debian the current process required several
manual steps. This commit adds the --push-remote option to allow
creating and pushing the imported package to apertis remote gitlab
instance.

The following steps are now automatized:
* package folder is created and import process is performed in there.
* remote repo is created if it doesn't exist.
* remote branches and tags are pushed.
* gitlab settings are applied to remote repo.
* CI pipeline is triggered on all downstream branches.

Signed-off-by: Ariel D'Alessandro <ariel.dalessandro@collabora.com>

With the new application framework, the application bundle is now
implemented using Flatpak, so drop this functionality from ADE.
Keep the sysroot management functionality as well as the
possibility of compile standard applications like dash.

Signed-off-by: Walter Lozano <walter.lozano@collabora.com>

These tools are moved from
`infrastructure/apertis-docker-images/package-source-builder`.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

This tool is moved from `infrastructure/packaging-tools`.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Running `ade sysroot install --release v2021dev3 --arch amd64` results
in `ade` complaining with:

    usage: ade sysroot install [-h] [--verbose] [--distro DISTRO]
                               [--release RELEASE] [--arch {armhf,arm64}]
                               [--url URL] [--user USER] [--password PASSWORD]
                               [--netrc NETRC] [--file FILE] [--force]
    ade sysroot install: error: argument --arch: invalid choice: 'amd64' (choose from 'armhf', 'arm64')

This means that developers attempting to target amd64 targets aren't
going to be able to follow the ADE documentation and we can't easily
target different releases using ADE when using an amd64 target, so
sync the list of allowable values with the main list of the kind of
sysroots we support.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

When no .netrc file is found, ade resorts to priting an error message.
This can be annoying when users want to use a remote server that doesn't
need auth.

This change demotes the error message to be displayed only in
verbose mode

Closes: https://phabricator.apertis.org/T6301



Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

urllib.parse.unquote() request a string as first parameter.
Check that parsed_url.password exists before calling it.

Fixes: APERTIS-5756

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

Passwords could be having special characters, in which case they need to
be percent encoded as per the RFC-3986 section 2.1. in the URL.

For such fields, where the password may be percent encoded, it also
needs to be decoded. That is what this change does. It takes the
password which may be percent encoded and provides with an ascii
equivalent string

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

If username and password are not passed on command line, nor in url, try
to find them from .netrc file.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

urllib does not support to set the username and password in the URL.
The URL needs to be parsed to extract them before calling urlopen or
urlretrieve.
The username and password found in the URL are used unless they are
already set on command line.

Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>

At the moment `ade` enforces a NN.NN versioning schema for sysroot with N
strictly being a digit.

Now that we switched to release versions like v2019dev0, v2019pre, v2019.0 this
is no longer appropriate.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

Apertis: T5626

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

A gdbserver may be running left over from a previous debug session.
It will still be listening on the TCP port, preventing a new instance
from starting, so we need to kill it.

However, if we allocate a PTY when spawning it, it will die once the
session ends.

For some reason, with get_pty=True, the gdbserver’s output is coming
from stdout, not stderr.

Apertis: T4463

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>

When the debugger thread starts, it spawns a gdbserver and
waits forever. Use the Channel from paramiko to monitor the process
return value and quit when it's terminated.
Discard any output from gdbserver as it can stall the ssh link in extreme
cases.
Additionnaly "while True: pass" loops are busy loops. Relax those.
Do that for non-interactive ade too.

Apertis: https://phabricator.apertis.org/T4462



Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D7431

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D7430

sysroot tarball packs `./etc/image_version`, ade fails
to identify it as `etc/image_version`, therefore, we
need to append `./` to the file path.

Signed-off-by: Héctor Orón Martínez <hector.oron@collabora.co.uk>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D7384

It is normal and expected for agent entry points to not have an icon.

Signed-off-by: Simon McVittie <smcv@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D6858

`ade configure` and `ade build` have `--native` option, but `install`
and `uninstall` commands doesn't. To install/uninstall a bundle which
is built with `--native`, user should use `ade install --simulator`
because two options actually do same thing.
For the consistency, the option, `--native`, is added.

Reviewed-by: Emanuele Aina <emanuele.aina@collabora.co.uk>
Signed-off-by: Justin Kim <justin.kim@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6807

When unparsed arguments were picked out, they were appended
to args. This is problematic if the wrapped command takes
command-line options with values of the form "--foo bar" (as
opposed to --foo=bar), because they would be re-ordered to "bar --foo",
likely causing a syntax error. It is obviously also a problem if
order is significant for the wrapped command. Prepend the unparsed
arguments to args instead of appending.

Signed-off-by: Simon McVittie <smcv@collabora.com>
Reviewed-by: André Magalhães <andre.magalhaes@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D6798

Picking out the known arguments and leaving unknown arguments is
dangerous, because typos and prefix-matching can have unintended
results. To discourage this, warn if any unknown arguments were
found. These are all considered correct, and are equivalent
(except for order):

    ade configure --native -- VARIABLE=1 --with-option
    ade configure --native VARIABLE=1 --with-option
    ade configure --native -- --with-option VARIABLE=1

but this will now provoke a warning for --with-option:

    ade configure --native --with-option VARIABLE=1

Signed-off-by: Simon McVittie <smcv@collabora.com>
Reviewed-by: André Magalhães <andre.magalhaes@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D6797

The common convention of passing "--" as a separator between
arguments for this command and arguments for the wrapped command
did not work, because it would also pass the "--" through. For
instance, "ade configure --native -- --device=foo" should pass
"--device=foo" to the configure script instead of interpreting it
as ade's own --device option.

Signed-off-by: Simon McVittie <smcv@collabora.com>
Reviewed-by: André Magalhães <andre.magalhaes@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D6796

Subcommands with no REMAINING arguments, such as "ade uninstall",
would previously accept and ignore unknown arguments. This could be
very bad if the command is something like "ade uninstall --dry-run".

Signed-off-by: Simon McVittie <smcv@collabora.com>
Reviewed-by: André Magalhães <andre.magalhaes@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D6795

Because we set args to a new value, any arguments passed to
"ade configure" were ignored.

Apertis: https://phabricator.apertis.org/T3978



Signed-off-by: Simon McVittie <smcv@collabora.com>
Reviewed-by: Justin Kim <justin.kim@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6774

Python arguments are passed by reference, so if we append to these
arrays, the caller's copy of the list is modified. This seems unlikely
to be intended.

Signed-off-by: Simon McVittie <smcv@collabora.com>
Reviewed-by: Justin Kim <justin.kim@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6773

The handler is already present since the info and sysroot subcommand
are already having a --path parameter.

Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6412

The sysroot tag parser was not tested and a regression occured
To catch future regressions make use for ade info and check
that the expected sysroot info can be read during tests.
Add support for --path to the ade info subcommand parser work
in temporary test directory.

Apertis: https://phabricator.apertis.org/T3777



Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6390

That error was not defined and triggered an Exception in the Exception.
It was not raised either. Use SysrootManagerError instead

Apertis: https://phabricator.apertis.org/T3777



Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6402

Recent parsing changes introduced this regression.

Apertis: https://phabricator.apertis.org/T3777



Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D6383

The current parsing expressions assume release is in the form of
YY.QQ e.g. 16.09. While this is true for Apertis, it's not mandatory for
derivatives so drop that assumption.

While fixing that, move to using match names to make the regexps a bit
more foolproof.

Also split up the function parsing the string from a version file
(/etc/image_version) and from the sysroot version tag as they are
fundamentally different, the current regexp handled both in extremely
subtle and error-prone ways.

Apertis: https://phabricator.apertis.org/T3555



Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Reviewed-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6264

The SysrootVersion class was trying to define a date and build index
from the build identifier. While this is true by convention for the
Apertis builds, this may break in the future.

Instead treat the build identifier as an opaque string consisting of just
digits and dots.

While there, also make all SysrootVersion arguments optional to improve
readability and rename author to variant (which is its intended
meaning).

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Reviewed-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6263

sudo is used to run ribchesterctl to install a bundle on the target,
but it is not used to run ribchesterctl to install a bundle on the
simulator. Let's change that.

Change a call to ribchesterctl instead of ribchester on the fly.

Apertis: https://phabricator.apertis.org/T3556



Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D6200

We suppose the intent from the user when configuring an existing
directory is to rebuild all the binaries for the new configuration. For
example, the user could have changed the sysroot.

Apertis: https://phabricator.apertis.org/T3557



Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6141

In order to configure different target and build in a different
directory, the build-dir parameter will run configure from a
subdirectory of the root. This allows separating build artifacts
from source code. It allows to build for different targets
at the same time.
Also add support for this in export, run, debug, install, and
uninstall commands.
Do not add it in info command yet.

Apertis: https://phabricator.apertis.org/T3557



Signed-off-by: Frédéric Dalleau <frederic.dalleau@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D6129

`subprocess.PIPE` returns a bytes object which has no attribute
`read`.

Signed-off-by: Justin Kim <justin.kim@collabora.com>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D5358

Louis-Francis Ratté-Boulianne authored 8 years ago and Sjoerd Simons committed 8 years ago

When we only want the GDB server running

Signed-off-by: Louis-Francis Ratté-Boulianne <lfrb@collabora.com>
Reviewed-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D5328

Louis-Francis Ratté-Boulianne authored 8 years ago and Sjoerd Simons committed 8 years ago

We were trying to connect the GDB client even before we got the
listening event. The debugger server thread is now looping forever.

Signed-off-by: Louis-Francis Ratté-Boulianne <lfrb@collabora.com>
Reviewed-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D5327

Louis-Francis Ratté-Boulianne authored 8 years ago and Sjoerd Simons committed 8 years ago

If the --app is not given, we check the installed desktop files
and use the main one if that makes sense.

Signed-off-by: Louis-Francis Ratté-Boulianne <lfrb@collabora.com>
Reviewed-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D5326

There is no reason to not allow e.g. an ssh agent or to prevent ade
looking for keys, so drop the restrictions. Also fixes launching the
install job from eclipse as that doesn't seems to pass a user and relies
on ssh key authentication.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Reviewed-by: Emanuele Aina <emanuele.aina@collabora.com>
Differential Revision: https://phabricator.apertis.org/D5344

If ade execv's gdb then its process will be replaced, potentially
causing the ssh connection to the target device to fail which will kill
the remote gdb server. So instead run gdb in a subprocess, keeping ade
(and thus the ssh connection) alive during the debugging session.

Also instead of using gdb, use gdb-multiarch which is a variant capable
of interacting with gdbservers of multiple architectures (and
understand/parsing the debug symbols of multiple architectures). This
means it can be used to debug both ARM and Intel targets, rather then
just native targets.

Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Reviewed-by: Guillaume Desmottes <guillaume.desmottes@collabora.co.uk>
Differential Revision: https://phabricator.apertis.org/D5305