Try to speed up the single-threaded PDF generation by starting a process
for each CPU core.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

GitLab Pages only really handles a single branch for each project, so
forking is the best way to have the proposed pages be rendered and
published by Pages.

By setting `baseURL` from CI using `$CI_PAGES_URL` we can ensure that
links are set properly instead of always pointing at
https://www.apertis.org/ as that's quite confusing during reviews.

However, for the main project we don't want to use the default GitLab
pages address, but we want to have links using the official
https://www.apertis.org/ address: to the main project now sets
`$BASEURL` as a CI/CD variable to point at https://www.apertis.org/


while forked projects will use the default based on `$CI_PAGES_URL`.

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Martyn Welch authored 4 years ago and Frederic Danis committed 4 years ago

The code of conduct contains for Ekaterina, who is no longer at Collabora.
Update the page to contain the new "conduct@apertis.org" email alias which
has been setup for this purpose.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Peter Senna Tschudin authored 4 years ago and Martyn Welch committed 4 years ago

This patch extends the Contribution process document and the
contribution checklist for adding new designs to Apertis.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Denis Pynkin authored 4 years ago and Denis Pynkin committed 4 years ago

Add the subsection described how to setup the VirtualBox for
USB-OTG boot mode to be used in Apertis SDK VM.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 4 years ago and Denis Pynkin committed 4 years ago

Use 'internal' boot mode for the default boot.
Update photos of BOOT switch used during U-Boot flush to
be more human-friendly on resized page.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Denis Pynkin authored 4 years ago and Denis Pynkin committed 4 years ago

Mention the additional terminal for `minicom` for USB OTG boot
method. Need to mention this explicitly to avoid the problem
then novice user tries to run all steps in the single terminal.
This is preventing the data loss from the U-Boot output via serial
console and allows to see the flash status.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The design documents imported from designs.apertis.org need to be
classified and moved into the structure implemented for documents on the
new website. The content of the documents also need minor edits to ensure
a good look and feel as well as the addition of notes to explain the
context of some documents (such as noting that certain concept documents
have been implemented since they were written and when that occurred).

This commit contains conversion of an initial batch of design documents.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Got out of step whilst adding creation dates to designs and failed to test
properly. A stray space is stopping the dates from being parsed. Remove
the space.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>

Add aliases to the main website page for the index page URLs as redirected
from designs.apertis.org. We are redirecting to the main page as the
contents of designs.apertis.org is in the process of being amalgamated
into the website structure and thus not all the designs content will end
up under one topic.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The old designs.a.o site holds versioned copies of the designs. We are not
going to present these any more, so where appropriate, link to the latest
copy of the document rather than returning 404.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The metadata in each file on designs.apertis.org didn't contain a creation
date, however this is the primary means that we are ordering the documents
by on the new Apertis website so that the newer documents are listed
first.

Whilst the documents don't contain a creation date, they are stored in a
git repository that can provide us with this information. Add the
retrieved dates into the front matter.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

We are using Hugo's "ref" shortcode for links on the website as this helps
to ensure that our links are valid. A number of links that aren't using
this tool have been identified, fix them and the resulting bad page links
found un upstreaming.md.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

A number of documents had previously been copied over from designs.a.o.
Tweak the contribution process document to correct broken titling
(discovered during comparison) and remove duplicates.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Since the `usb_imx` utility is not installed into PATH we have to use
relative or absolute path for calling it.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Whilst the documents copied over from designs.a.o are searchable, they are
not directly accessible from the main page. Add a link to them for now.
This link will be removed once the documents are sorted into the new
category structure.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Peter Senna Tschudin authored 4 years ago and Emanuele Aina committed 4 years ago

This patch fixes a few typos and minor issues and add a concrete example
of how to reproduce the build.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Whilst not important in CI as they will get cleaned up, having
temporary files and directories left after building locally is a pain.
Tweak to clean these files up.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The codeblock tables are being badly formatted, with the column for the
line numbers the same width as that for the content. Tweak the html for
tables with the `lntable` class so that the line numbers are only 3% of
the width.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A mistake whilst scripting replacements resulted in lines with 3
characters being replaced and later removed. Fix these errors.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The PDF generator can't find the images as the paths are relate to
where Hugo will place them on the website. Tweak the path to to point
to the images. We need to convert the SVG files into something the next
step can handle. Don't leave them in the static directory in case this
results in them being exported to the website.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Some of the image links were badly tweaked, resulting in the images not
being found. Correct the links.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

This is required when the cleaning up and recovering space.
Usually, the beta, development and preview releases are the common
candidates for removal

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A number of broken links didn't match the patterns used for automated
replacement. Fix up those that can be manually.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We now have the designs here, don't point to designs.a.o.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

After moving the documents from designs.a.o to www.a.o, there were
quite a few links from the designs.a.o docs pointing to wiki.a.o and
quite a few links from www.a.o pointing to designs.a.o. Replace these
links with Hugo ref shortcodes to avoid hammering the redirects.

This was achieved with the following script:

```

import os
import re
import sys
import toml

from urllib.parse import urlparse

def get_aliases(filename):
    #print("%s: " % filename)
    with open(filename, 'r') as file:
        contents = file.read()
        if not contents[0:3] == "+++":
            return

        if "_index.md" in filename:
            filename = filename.strip(".")
        else:
            filename = filename.split("/")[-1]

        data = toml.loads(contents.split("+++")[1])

        if not "aliases" in data.keys():
            return

        for alias in data["aliases"]:
            if "/old-wiki/" in alias:
                wiki_aliases[alias.replace("/old-wiki/", "/")] = filename

            if "/old-designs/" in alias:
                # We're dropping the per-release links from designs.a.o, so match on shorter path
                design_aliases[alias.replace("/old-designs/latest/", "/")] = filename

def fix_link(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "](%s)" % link

def fix_ref(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "]: %s" % link

def correct_links(filename):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        if not contents[0:3] == "+++":
            return

        fm = contents.split("+++")[1]
        doc = contents.split("+++",2)[2]

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("\]\((?P<link>.*?)\)")
        doc = pattern.sub(fix_link, doc)

        pattern = re.compile("\]: (?P<link>.*)")
        doc = pattern.sub(fix_ref, doc)

        file.seek(0)
        file.truncate()

        file.write("+++")
        file.write(fm)
        file.write("+++")
        file.write(doc)

wiki_aliases = {}
design_aliases = {}

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            get_aliases("%s/%s" %(root, file))

print("wiki_aliases:")
print(wiki_aliases)
print("design_aliases:")
print(design_aliases)

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            correct_links("%s/%s" %(root, file))
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A number of broken links remained after scripted conversion and a
number of spurious code blocks were found in the docs. Clean these up
so that the documents render correctly.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

By default the Goldmark renderer used by Hugo won't render HTML
elements that are present in the markdown, even though this is allowed
in the CommonMark Spec that it claims to implement:

    https://spec.commonmark.org/0.29/#html-blocks



We can get this to work by turning on "unsafe" mode, which means:

    By default, Goldmark does not render raw HTMLs and potentially
    dangerous links. If you have lots of inline HTML and/or JavaScript,
    you may need to turn this on.

As we have such elements and we can guarantee what's being rendered,
turn this on.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A lot of the links in the imported designs are broken:

- HotDoc allowed a `[][link]` format that Hugo doesn't understand
- HotDoc allowed relative links without text, which Hugo doesn't handle
- The location of the images and other media has changed
- The format used for internal page references has is different

Additionally, fix up touched links to other Apertis pages to use the
Hugo "ref" shortcode, which reduces the risk of leaving hanging links.

Conversion completed with the following python script:

```

import os
import re
import sys

def url_munge(match):
    url = match.group('url').strip('.')
    if url[0] != "/":
        url = "/%s" % url
    url = url.replace("media", "images")
    link = ("![](%s)" % url)

    return link

def link_munge(match):
    link = match.group('link')
    new = link.replace("(", "")
    new = new.replace(")", "")
    new = new.replace("`", "")
    new = new.replace("?", "")
    new = new.replace(":", "")
    new = new.replace(",", "")
    new = new.replace("–", "")
    new = new.replace("“", "")
    new = new.replace("”", "")
    new = new.replace(".", "")
    new = new.replace(" ", "-")
    new = new.lower()
    new = (" [%s]( {{< ref \"#%s\" >}} )" % (link, new))

    return new

def link_munge_2(match):
    link = match.group('link')
    new = link.replace("#", "")
    new = new.replace("-", " ")
    new = new.replace(".md", "")
    if not "http" in link:
        link = " {{< ref \"%s\" >}} " % link

    new = ("[%s](%s)" % (new, link))

    return new

for filename in os.listdir(sys.argv[1]):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        # Hotdoc uses `[][Internal Title]` for internal links whilst Hugo uses `[](#internal-title)`
        pattern = re.compile("\[\]\[(?P<link>.*?)\]")
        contents = pattern.sub(link_munge, contents)

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("[^!]\[\]\((?P<link>.*?)\)")
        contents = pattern.sub(link_munge_2, contents)

        # Media links not working
        pattern = re.compile("!\[\]\((?P<url>.*?)\)")
        contents = pattern.sub(url_munge, contents)

        file.seek(0)
        file.truncate()

        file.write(contents)

```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Hotdoc allows "---" to be used for a code block, this is not supported
by Hugo, it needs to use "```". Convert uses of the former with:

  sed 's/^---$/```/' -i *

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Convert the existing metadata stored at the front of each file to
frontmatter formatted for Hugo:

 - Remove authors as we don't have an equivalent for that in Hugo
 - Add in fields that the existing metadata doesn't have
 - Include aliases to allow us to point to the new pages from a
   redirect on designs.apertis.org
 - Ensure we have a title (based on filename if one not present)

All frontmatter created in toml format to stay consistent with existing
documents.

Converted using the following python script:
```

import os
import re
import sys
import toml
import yaml

for filename in os.listdir(sys.argv[1]):
    print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()
        if contents[0:3] == "+++":
            # We have toml
            data = toml.loads(contents.split("+++")[1])
            doc = "+++".join(contents.split("+++")[2:])
        elif contents[0:3] == "---":
            # We have yaml
            data = yaml.load(contents.split("---")[1])
            doc = "---".join(contents.split("---")[2:])
        else:
            # No frontmatter
            data = {}
            doc = contents

        if not "title" in data.keys():
            data["title"] = filename.split(".")[0].replace('_', ' ').capitalize()

        data["weight"] = 100
        data["aliases"] = ["/old-designs/latest/%s" % filename.replace(".md", ".html")]

        data["outputs"] = ["html", "pdf-in"]

        if "authors" in data.keys():
            del data["authors"]

        file.seek(0)
        file.truncate()

        file.write("+++\n")
        file.write(toml.dumps(data))
        file.write("+++")
        file.write(doc)
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Some of the pages lack titles, others are just menu items we don't
need. Tweak the pages with initial pass to give them all titles and
remove stuff we don't need (we will be sorting the designs anyway).

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Copy all the media files from the designs.apertis.org website under
"images" where we currently have all the website media.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Add the designs from designs.a.o with the minimum changes to get the
site to build. This essentially required to trailing marker for the
metadata to be changed from "..." to "---".

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

This file is used throughout our Apertis packaging repositories by all
packages. It specifies which repository component a package is part of

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Document the process for updating packages where we pull from an
upstream (non-Debian) source, such as we do for the LTS v.5.4 kernel.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We have a stub in guidelines for the contribution process. Move the
content over from designs.a.o and move to policies.

Add in alias for `old-designs` to record the location used on
designs.a.o that we can be used once we've dropped all the contents
of designs.a.o as a redirect to the website.

Flag file to be generated as PDF since it's a document from
designs.a.o.

Remove `guidelines` from the main page as it's now empty.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>