Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

After moving the documents from designs.a.o to www.a.o, there were
quite a few links from the designs.a.o docs pointing to wiki.a.o and
quite a few links from www.a.o pointing to designs.a.o. Replace these
links with Hugo ref shortcodes to avoid hammering the redirects.

This was achieved with the following script:

```

import os
import re
import sys
import toml

from urllib.parse import urlparse

def get_aliases(filename):
    #print("%s: " % filename)
    with open(filename, 'r') as file:
        contents = file.read()
        if not contents[0:3] == "+++":
            return

        if "_index.md" in filename:
            filename = filename.strip(".")
        else:
            filename = filename.split("/")[-1]

        data = toml.loads(contents.split("+++")[1])

        if not "aliases" in data.keys():
            return

        for alias in data["aliases"]:
            if "/old-wiki/" in alias:
                wiki_aliases[alias.replace("/old-wiki/", "/")] = filename

            if "/old-designs/" in alias:
                # We're dropping the per-release links from designs.a.o, so match on shorter path
                design_aliases[alias.replace("/old-designs/latest/", "/")] = filename

def fix_link(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "](%s)" % link

def fix_ref(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "]: %s" % link

def correct_links(filename):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        if not contents[0:3] == "+++":
            return

        fm = contents.split("+++")[1]
        doc = contents.split("+++",2)[2]

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("\]\((?P<link>.*?)\)")
        doc = pattern.sub(fix_link, doc)

        pattern = re.compile("\]: (?P<link>.*)")
        doc = pattern.sub(fix_ref, doc)

        file.seek(0)
        file.truncate()

        file.write("+++")
        file.write(fm)
        file.write("+++")
        file.write(doc)

wiki_aliases = {}
design_aliases = {}

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            get_aliases("%s/%s" %(root, file))

print("wiki_aliases:")
print(wiki_aliases)
print("design_aliases:")
print(design_aliases)

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            correct_links("%s/%s" %(root, file))
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A number of broken links remained after scripted conversion and a
number of spurious code blocks were found in the docs. Clean these up
so that the documents render correctly.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

By default the Goldmark renderer used by Hugo won't render HTML
elements that are present in the markdown, even though this is allowed
in the CommonMark Spec that it claims to implement:

    https://spec.commonmark.org/0.29/#html-blocks



We can get this to work by turning on "unsafe" mode, which means:

    By default, Goldmark does not render raw HTMLs and potentially
    dangerous links. If you have lots of inline HTML and/or JavaScript,
    you may need to turn this on.

As we have such elements and we can guarantee what's being rendered,
turn this on.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A lot of the links in the imported designs are broken:

- HotDoc allowed a `[][link]` format that Hugo doesn't understand
- HotDoc allowed relative links without text, which Hugo doesn't handle
- The location of the images and other media has changed
- The format used for internal page references has is different

Additionally, fix up touched links to other Apertis pages to use the
Hugo "ref" shortcode, which reduces the risk of leaving hanging links.

Conversion completed with the following python script:

```

import os
import re
import sys

def url_munge(match):
    url = match.group('url').strip('.')
    if url[0] != "/":
        url = "/%s" % url
    url = url.replace("media", "images")
    link = ("![](%s)" % url)

    return link

def link_munge(match):
    link = match.group('link')
    new = link.replace("(", "")
    new = new.replace(")", "")
    new = new.replace("`", "")
    new = new.replace("?", "")
    new = new.replace(":", "")
    new = new.replace(",", "")
    new = new.replace("–", "")
    new = new.replace("“", "")
    new = new.replace("”", "")
    new = new.replace(".", "")
    new = new.replace(" ", "-")
    new = new.lower()
    new = (" [%s]( {{< ref \"#%s\" >}} )" % (link, new))

    return new

def link_munge_2(match):
    link = match.group('link')
    new = link.replace("#", "")
    new = new.replace("-", " ")
    new = new.replace(".md", "")
    if not "http" in link:
        link = " {{< ref \"%s\" >}} " % link

    new = ("[%s](%s)" % (new, link))

    return new

for filename in os.listdir(sys.argv[1]):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        # Hotdoc uses `[][Internal Title]` for internal links whilst Hugo uses `[](#internal-title)`
        pattern = re.compile("\[\]\[(?P<link>.*?)\]")
        contents = pattern.sub(link_munge, contents)

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("[^!]\[\]\((?P<link>.*?)\)")
        contents = pattern.sub(link_munge_2, contents)

        # Media links not working
        pattern = re.compile("!\[\]\((?P<url>.*?)\)")
        contents = pattern.sub(url_munge, contents)

        file.seek(0)
        file.truncate()

        file.write(contents)

```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Hotdoc allows "---" to be used for a code block, this is not supported
by Hugo, it needs to use "```". Convert uses of the former with:

  sed 's/^---$/```/' -i *

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Convert the existing metadata stored at the front of each file to
frontmatter formatted for Hugo:

 - Remove authors as we don't have an equivalent for that in Hugo
 - Add in fields that the existing metadata doesn't have
 - Include aliases to allow us to point to the new pages from a
   redirect on designs.apertis.org
 - Ensure we have a title (based on filename if one not present)

All frontmatter created in toml format to stay consistent with existing
documents.

Converted using the following python script:
```

import os
import re
import sys
import toml
import yaml

for filename in os.listdir(sys.argv[1]):
    print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()
        if contents[0:3] == "+++":
            # We have toml
            data = toml.loads(contents.split("+++")[1])
            doc = "+++".join(contents.split("+++")[2:])
        elif contents[0:3] == "---":
            # We have yaml
            data = yaml.load(contents.split("---")[1])
            doc = "---".join(contents.split("---")[2:])
        else:
            # No frontmatter
            data = {}
            doc = contents

        if not "title" in data.keys():
            data["title"] = filename.split(".")[0].replace('_', ' ').capitalize()

        data["weight"] = 100
        data["aliases"] = ["/old-designs/latest/%s" % filename.replace(".md", ".html")]

        data["outputs"] = ["html", "pdf-in"]

        if "authors" in data.keys():
            del data["authors"]

        file.seek(0)
        file.truncate()

        file.write("+++\n")
        file.write(toml.dumps(data))
        file.write("+++")
        file.write(doc)
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Some of the pages lack titles, others are just menu items we don't
need. Tweak the pages with initial pass to give them all titles and
remove stuff we don't need (we will be sorting the designs anyway).

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Copy all the media files from the designs.apertis.org website under
"images" where we currently have all the website media.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Add the designs from designs.a.o with the minimum changes to get the
site to build. This essentially required to trailing marker for the
metadata to be changed from "..." to "---".

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

This file is used throughout our Apertis packaging repositories by all
packages. It specifies which repository component a package is part of

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Document the process for updating packages where we pull from an
upstream (non-Debian) source, such as we do for the LTS v.5.4 kernel.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We have a stub in guidelines for the contribution process. Move the
content over from designs.a.o and move to policies.

Add in alias for `old-designs` to record the location used on
designs.a.o that we can be used once we've dropped all the contents
of designs.a.o as a redirect to the website.

Flag file to be generated as PDF since it's a document from
designs.a.o.

Remove `guidelines` from the main page as it's now empty.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We have a stub in guidelines for the contribution checklist. Move the
content over from designs.a.o and move to policies.

Add in alias for `old-designs` to record the location used on
designs.a.o that we can be used once we've dropped all the contents
of designs.a.o as a redirect to the website.

Flag file to be generated as PDF since it's a document from
designs.a.o.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We are moving the contents of apertis-designs over to the website. This
workflow will therefore be obsolete. Delete it now.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We have very little documentation regarding the handling of Debian
revisions but we need to update these more frequently than raw version
numbers. Pull some documentation from a training session guide written
a while back to fill this hole.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The versioning information doesn't sit very well in the releasing
packages guide. Add this to the information on binary package
versioning to make a more holistic page on versioning.

Move the information on dch tags to the version_control guide where
it is mentioned.

The rest of this guide is very out of date. We have a number of other
guides on releasing packages, drop this one.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

These documents are higher level than a guide, move to architecture.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Hugo automatically generates IDs for each title when the HTML pages are
generated. These IDs can be used as anchors in the document, so that we
can provide URLs that jump to the specific title when the link is
followed. Browsers will place the anchor at the top of the window,
however we have a floating navigation bar there which results in the
anchor being obscured, which can be a bit disorientating.

Tweak the CSS to pad the top of the heading, but in a way that doesn't
result in the heading being visibily spaced further from the previous
text. Add override to main page heading to stop it being padded down
the page.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

More of the guides still seem relevant and good information. Tweak the
formatting and move these to guides.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

* Reorder each example under each chunk
* Indent the examples so that they are rendered proper in hugo

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

* Cleanup paragraph
* Add date and weight field
* Use reference shortcodes
* Strip shell prompt

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Moved from ci-package-builder

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

That every rebuild repository should have the following meta attribute
set because otherwise, packages would run into a cyclic build loop.

 rebuild="local" block="never"

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Instead of ``` triple quote ticks, which treats the text as code example

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Moved from repository apertis-infrastructure

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We are now using the gitlab-based package workflow, the guidance on the
git-based package workflow page is out of date:

- We don't use the master branch
- We don't need the `MAINTAINED_IN_GIT.txt` file
- We no longer automatically close tasks based on tags
- We're not using Jenkins for package CI
- We don't submit packages to OBS manually

Delete the page.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Move the doc on module setup, broadly unchanged (minor formatting
improvements).

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The introduction to the linters guide is rather thin. Improve it and
move to `guides/`.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

These documents can be moved to guides more or less as they are.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The build and integration infrastructure documentation page mainly
covers a description of how the binary package versioning is formed.
Rework the page to present this information in a slightly more rounded
way and move it to the archtecture section.

We can forgo a list of the services, links to the services should be
provided where they are needed. We need to remove the link to
mattermost history as that is not publically accessible.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We are rapidly moving from Jenkins to GitLab CI. Remove the
documentation regarding Jenkins as this is of verly little to no use
now.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The version control guide looks useful, but is a little out of date.
Update the guide to use current workflows and move to `guides/`.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The following guides can be moved to `guides/` with minimal changes:

- buildingpatchingandmaintainingtheapertiskernel.md
- gitlab-based_packaging_workflow.md
- howtobuildyourownkernel.md
- howtodevelopondevices.md
- logging.md

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The mainly lists the pages found under `guidelines/`, as the hugo site
now automatically lists these. The rest of the information on this page
is also either broadly out of date or of negligable value. Remove the
index completely as we will be removing the guidelines section anyway.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Signed-off-by: Emanuele Aina <emanuele.aina@collabora.com>