Add aliases to the main website page for the index page URLs as redirected
from designs.apertis.org. We are redirecting to the main page as the
contents of designs.apertis.org is in the process of being amalgamated
into the website structure and thus not all the designs content will end
up under one topic.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The old designs.a.o site holds versioned copies of the designs. We are not
going to present these any more, so where appropriate, link to the latest
copy of the document rather than returning 404.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The metadata in each file on designs.apertis.org didn't contain a creation
date, however this is the primary means that we are ordering the documents
by on the new Apertis website so that the newer documents are listed
first.

Whilst the documents don't contain a creation date, they are stored in a
git repository that can provide us with this information. Add the
retrieved dates into the front matter.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

We are using Hugo's "ref" shortcode for links on the website as this helps
to ensure that our links are valid. A number of links that aren't using
this tool have been identified, fix them and the resulting bad page links
found un upstreaming.md.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

A number of documents had previously been copied over from designs.a.o.
Tweak the contribution process document to correct broken titling
(discovered during comparison) and remove duplicates.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Since the `usb_imx` utility is not installed into PATH we have to use
relative or absolute path for calling it.

Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>

Whilst the documents copied over from designs.a.o are searchable, they are
not directly accessible from the main page. Add a link to them for now.
This link will be removed once the documents are sorted into the new
category structure.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Peter Senna Tschudin authored 4 years ago and Emanuele Aina committed 4 years ago

This patch fixes a few typos and minor issues and add a concrete example
of how to reproduce the build.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Whilst not important in CI as they will get cleaned up, having
temporary files and directories left after building locally is a pain.
Tweak to clean these files up.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The codeblock tables are being badly formatted, with the column for the
line numbers the same width as that for the content. Tweak the html for
tables with the `lntable` class so that the line numbers are only 3% of
the width.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A mistake whilst scripting replacements resulted in lines with 3
characters being replaced and later removed. Fix these errors.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The PDF generator can't find the images as the paths are relate to
where Hugo will place them on the website. Tweak the path to to point
to the images. We need to convert the SVG files into something the next
step can handle. Don't leave them in the static directory in case this
results in them being exported to the website.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Some of the image links were badly tweaked, resulting in the images not
being found. Correct the links.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

This is required when the cleaning up and recovering space.
Usually, the beta, development and preview releases are the common
candidates for removal

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A number of broken links didn't match the patterns used for automated
replacement. Fix up those that can be manually.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We now have the designs here, don't point to designs.a.o.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

After moving the documents from designs.a.o to www.a.o, there were
quite a few links from the designs.a.o docs pointing to wiki.a.o and
quite a few links from www.a.o pointing to designs.a.o. Replace these
links with Hugo ref shortcodes to avoid hammering the redirects.

This was achieved with the following script:

```

import os
import re
import sys
import toml

from urllib.parse import urlparse

def get_aliases(filename):
    #print("%s: " % filename)
    with open(filename, 'r') as file:
        contents = file.read()
        if not contents[0:3] == "+++":
            return

        if "_index.md" in filename:
            filename = filename.strip(".")
        else:
            filename = filename.split("/")[-1]

        data = toml.loads(contents.split("+++")[1])

        if not "aliases" in data.keys():
            return

        for alias in data["aliases"]:
            if "/old-wiki/" in alias:
                wiki_aliases[alias.replace("/old-wiki/", "/")] = filename

            if "/old-designs/" in alias:
                # We're dropping the per-release links from designs.a.o, so match on shorter path
                design_aliases[alias.replace("/old-designs/latest/", "/")] = filename

def fix_link(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "](%s)" % link

def fix_ref(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "]: %s" % link

def correct_links(filename):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        if not contents[0:3] == "+++":
            return

        fm = contents.split("+++")[1]
        doc = contents.split("+++",2)[2]

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("\]\((?P<link>.*?)\)")
        doc = pattern.sub(fix_link, doc)

        pattern = re.compile("\]: (?P<link>.*)")
        doc = pattern.sub(fix_ref, doc)

        file.seek(0)
        file.truncate()

        file.write("+++")
        file.write(fm)
        file.write("+++")
        file.write(doc)

wiki_aliases = {}
design_aliases = {}

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            get_aliases("%s/%s" %(root, file))

print("wiki_aliases:")
print(wiki_aliases)
print("design_aliases:")
print(design_aliases)

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            correct_links("%s/%s" %(root, file))
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A number of broken links remained after scripted conversion and a
number of spurious code blocks were found in the docs. Clean these up
so that the documents render correctly.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

By default the Goldmark renderer used by Hugo won't render HTML
elements that are present in the markdown, even though this is allowed
in the CommonMark Spec that it claims to implement:

    https://spec.commonmark.org/0.29/#html-blocks



We can get this to work by turning on "unsafe" mode, which means:

    By default, Goldmark does not render raw HTMLs and potentially
    dangerous links. If you have lots of inline HTML and/or JavaScript,
    you may need to turn this on.

As we have such elements and we can guarantee what's being rendered,
turn this on.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A lot of the links in the imported designs are broken:

- HotDoc allowed a `[][link]` format that Hugo doesn't understand
- HotDoc allowed relative links without text, which Hugo doesn't handle
- The location of the images and other media has changed
- The format used for internal page references has is different

Additionally, fix up touched links to other Apertis pages to use the
Hugo "ref" shortcode, which reduces the risk of leaving hanging links.

Conversion completed with the following python script:

```

import os
import re
import sys

def url_munge(match):
    url = match.group('url').strip('.')
    if url[0] != "/":
        url = "/%s" % url
    url = url.replace("media", "images")
    link = ("![](%s)" % url)

    return link

def link_munge(match):
    link = match.group('link')
    new = link.replace("(", "")
    new = new.replace(")", "")
    new = new.replace("`", "")
    new = new.replace("?", "")
    new = new.replace(":", "")
    new = new.replace(",", "")
    new = new.replace("–", "")
    new = new.replace("“", "")
    new = new.replace("”", "")
    new = new.replace(".", "")
    new = new.replace(" ", "-")
    new = new.lower()
    new = (" [%s]( {{< ref \"#%s\" >}} )" % (link, new))

    return new

def link_munge_2(match):
    link = match.group('link')
    new = link.replace("#", "")
    new = new.replace("-", " ")
    new = new.replace(".md", "")
    if not "http" in link:
        link = " {{< ref \"%s\" >}} " % link

    new = ("[%s](%s)" % (new, link))

    return new

for filename in os.listdir(sys.argv[1]):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        # Hotdoc uses `[][Internal Title]` for internal links whilst Hugo uses `[](#internal-title)`
        pattern = re.compile("\[\]\[(?P<link>.*?)\]")
        contents = pattern.sub(link_munge, contents)

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("[^!]\[\]\((?P<link>.*?)\)")
        contents = pattern.sub(link_munge_2, contents)

        # Media links not working
        pattern = re.compile("!\[\]\((?P<url>.*?)\)")
        contents = pattern.sub(url_munge, contents)

        file.seek(0)
        file.truncate()

        file.write(contents)

```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Hotdoc allows "---" to be used for a code block, this is not supported
by Hugo, it needs to use "```". Convert uses of the former with:

  sed 's/^---$/```/' -i *

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Convert the existing metadata stored at the front of each file to
frontmatter formatted for Hugo:

 - Remove authors as we don't have an equivalent for that in Hugo
 - Add in fields that the existing metadata doesn't have
 - Include aliases to allow us to point to the new pages from a
   redirect on designs.apertis.org
 - Ensure we have a title (based on filename if one not present)

All frontmatter created in toml format to stay consistent with existing
documents.

Converted using the following python script:
```

import os
import re
import sys
import toml
import yaml

for filename in os.listdir(sys.argv[1]):
    print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()
        if contents[0:3] == "+++":
            # We have toml
            data = toml.loads(contents.split("+++")[1])
            doc = "+++".join(contents.split("+++")[2:])
        elif contents[0:3] == "---":
            # We have yaml
            data = yaml.load(contents.split("---")[1])
            doc = "---".join(contents.split("---")[2:])
        else:
            # No frontmatter
            data = {}
            doc = contents

        if not "title" in data.keys():
            data["title"] = filename.split(".")[0].replace('_', ' ').capitalize()

        data["weight"] = 100
        data["aliases"] = ["/old-designs/latest/%s" % filename.replace(".md", ".html")]

        data["outputs"] = ["html", "pdf-in"]

        if "authors" in data.keys():
            del data["authors"]

        file.seek(0)
        file.truncate()

        file.write("+++\n")
        file.write(toml.dumps(data))
        file.write("+++")
        file.write(doc)
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Some of the pages lack titles, others are just menu items we don't
need. Tweak the pages with initial pass to give them all titles and
remove stuff we don't need (we will be sorting the designs anyway).

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Copy all the media files from the designs.apertis.org website under
"images" where we currently have all the website media.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Add the designs from designs.a.o with the minimum changes to get the
site to build. This essentially required to trailing marker for the
metadata to be changed from "..." to "---".

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

This file is used throughout our Apertis packaging repositories by all
packages. It specifies which repository component a package is part of

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Document the process for updating packages where we pull from an
upstream (non-Debian) source, such as we do for the LTS v.5.4 kernel.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We have a stub in guidelines for the contribution process. Move the
content over from designs.a.o and move to policies.

Add in alias for `old-designs` to record the location used on
designs.a.o that we can be used once we've dropped all the contents
of designs.a.o as a redirect to the website.

Flag file to be generated as PDF since it's a document from
designs.a.o.

Remove `guidelines` from the main page as it's now empty.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We have a stub in guidelines for the contribution checklist. Move the
content over from designs.a.o and move to policies.

Add in alias for `old-designs` to record the location used on
designs.a.o that we can be used once we've dropped all the contents
of designs.a.o as a redirect to the website.

Flag file to be generated as PDF since it's a document from
designs.a.o.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We are moving the contents of apertis-designs over to the website. This
workflow will therefore be obsolete. Delete it now.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

We have very little documentation regarding the handling of Debian
revisions but we need to update these more frequently than raw version
numbers. Pull some documentation from a training session guide written
a while back to fill this hole.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The versioning information doesn't sit very well in the releasing
packages guide. Add this to the information on binary package
versioning to make a more holistic page on versioning.

Move the information on dch tags to the version_control guide where
it is mentioned.

The rest of this guide is very out of date. We have a number of other
guides on releasing packages, drop this one.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

These documents are higher level than a guide, move to architecture.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Hugo automatically generates IDs for each title when the HTML pages are
generated. These IDs can be used as anchors in the document, so that we
can provide URLs that jump to the specific title when the link is
followed. Browsers will place the anchor at the top of the window,
however we have a floating navigation bar there which results in the
anchor being obscured, which can be a bit disorientating.

Tweak the CSS to pad the top of the heading, but in a way that doesn't
result in the heading being visibily spaced further from the previous
text. Add override to main page heading to stop it being padded down
the page.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

More of the guides still seem relevant and good information. Tweak the
formatting and move these to guides.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

* Reorder each example under each chunk
* Indent the examples so that they are rendered proper in hugo

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

* Cleanup paragraph
* Add date and weight field
* Use reference shortcodes
* Strip shell prompt

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

Moved from ci-package-builder

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

That every rebuild repository should have the following meta attribute
set because otherwise, packages would run into a cyclic build loop.

 rebuild="local" block="never"

Signed-off-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>