The license expectations document is a little out of date. Tweak it a bit
and move it along with the license-exceptions document to policies.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

This concept document describes a possible approach to closing the CI loop
using Jenkins. Apertis is moving away from the use of Jenkins and so this
document is now defunct. Remove it.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

The selection of images as defined for the v2019 release has been used for
a number of releases. This is no longer an open concept, add the
documentation to the images page and remove the concept document.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

The license-validation document covers the same ground as the first
portion of the automated license compliance documentation. Drop it to save
duplication of information.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

The design documents imported from designs.apertis.org need to be
classified and moved into the structure implemented for documents on the
new website. The content of the documents also need minor edits to ensure
a good look and feel as well as the addition of notes to explain the
context of some documents (such as noting that certain concept documents
have been implemented since they were written and when that occurred).

This commit contains conversion of an initial batch of design documents.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Got out of step whilst adding creation dates to designs and failed to test
properly. A stray space is stopping the dates from being parsed. Remove
the space.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The old designs.a.o site holds versioned copies of the designs. We are not
going to present these any more, so where appropriate, link to the latest
copy of the document rather than returning 404.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

The metadata in each file on designs.apertis.org didn't contain a creation
date, however this is the primary means that we are ordering the documents
by on the new Apertis website so that the newer documents are listed
first.

Whilst the documents don't contain a creation date, they are stored in a
git repository that can provide us with this information. Add the
retrieved dates into the front matter.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

We are using Hugo's "ref" shortcode for links on the website as this helps
to ensure that our links are valid. A number of links that aren't using
this tool have been identified, fix them and the resulting bad page links
found un upstreaming.md.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Martyn Welch committed 4 years ago

A number of documents had previously been copied over from designs.a.o.
Tweak the contribution process document to correct broken titling
(discovered during comparison) and remove duplicates.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Peter Senna Tschudin authored 4 years ago and Emanuele Aina committed 4 years ago

This patch fixes a few typos and minor issues and add a concrete example
of how to reproduce the build.

Signed-off-by: Peter Senna Tschudin <peter.senna@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A mistake whilst scripting replacements resulted in lines with 3
characters being replaced and later removed. Fix these errors.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Some of the image links were badly tweaked, resulting in the images not
being found. Correct the links.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A number of broken links didn't match the patterns used for automated
replacement. Fix up those that can be manually.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

After moving the documents from designs.a.o to www.a.o, there were
quite a few links from the designs.a.o docs pointing to wiki.a.o and
quite a few links from www.a.o pointing to designs.a.o. Replace these
links with Hugo ref shortcodes to avoid hammering the redirects.

This was achieved with the following script:

```

import os
import re
import sys
import toml

from urllib.parse import urlparse

def get_aliases(filename):
    #print("%s: " % filename)
    with open(filename, 'r') as file:
        contents = file.read()
        if not contents[0:3] == "+++":
            return

        if "_index.md" in filename:
            filename = filename.strip(".")
        else:
            filename = filename.split("/")[-1]

        data = toml.loads(contents.split("+++")[1])

        if not "aliases" in data.keys():
            return

        for alias in data["aliases"]:
            if "/old-wiki/" in alias:
                wiki_aliases[alias.replace("/old-wiki/", "/")] = filename

            if "/old-designs/" in alias:
                # We're dropping the per-release links from designs.a.o, so match on shorter path
                design_aliases[alias.replace("/old-designs/latest/", "/")] = filename

def fix_link(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "](%s)" % link

def fix_ref(url):
    link = url.group('link')

    url = urlparse(link)

    if url.netloc == "wiki.apertis.org":
        if url.path in wiki_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=wiki_aliases[url.path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    if url.netloc == "designs.apertis.org":
        # We're dropping the per-release links from designs.a.o, so match on shorter path
        path = "/%s" % url.path.split('/', 2)[-1]
        print(path)
        if path in design_aliases.keys():
            url = url._replace(scheme="")
            url = url._replace(netloc="")
            url = url._replace(path=design_aliases[path])
            link = " {{< ref \"%s\" >}} " % url.geturl()

    return "]: %s" % link

def correct_links(filename):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        if not contents[0:3] == "+++":
            return

        fm = contents.split("+++")[1]
        doc = contents.split("+++",2)[2]

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("\]\((?P<link>.*?)\)")
        doc = pattern.sub(fix_link, doc)

        pattern = re.compile("\]: (?P<link>.*)")
        doc = pattern.sub(fix_ref, doc)

        file.seek(0)
        file.truncate()

        file.write("+++")
        file.write(fm)
        file.write("+++")
        file.write(doc)

wiki_aliases = {}
design_aliases = {}

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            get_aliases("%s/%s" %(root, file))

print("wiki_aliases:")
print(wiki_aliases)
print("design_aliases:")
print(design_aliases)

for root, dirs, files in os.walk(sys.argv[1]):
    for file in files:
        if ".md" in file:
            correct_links("%s/%s" %(root, file))
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A number of broken links remained after scripted conversion and a
number of spurious code blocks were found in the docs. Clean these up
so that the documents render correctly.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

A lot of the links in the imported designs are broken:

- HotDoc allowed a `[][link]` format that Hugo doesn't understand
- HotDoc allowed relative links without text, which Hugo doesn't handle
- The location of the images and other media has changed
- The format used for internal page references has is different

Additionally, fix up touched links to other Apertis pages to use the
Hugo "ref" shortcode, which reduces the risk of leaving hanging links.

Conversion completed with the following python script:

```

import os
import re
import sys

def url_munge(match):
    url = match.group('url').strip('.')
    if url[0] != "/":
        url = "/%s" % url
    url = url.replace("media", "images")
    link = ("![](%s)" % url)

    return link

def link_munge(match):
    link = match.group('link')
    new = link.replace("(", "")
    new = new.replace(")", "")
    new = new.replace("`", "")
    new = new.replace("?", "")
    new = new.replace(":", "")
    new = new.replace(",", "")
    new = new.replace("–", "")
    new = new.replace("“", "")
    new = new.replace("”", "")
    new = new.replace(".", "")
    new = new.replace(" ", "-")
    new = new.lower()
    new = (" [%s]( {{< ref \"#%s\" >}} )" % (link, new))

    return new

def link_munge_2(match):
    link = match.group('link')
    new = link.replace("#", "")
    new = new.replace("-", " ")
    new = new.replace(".md", "")
    if not "http" in link:
        link = " {{< ref \"%s\" >}} " % link

    new = ("[%s](%s)" % (new, link))

    return new

for filename in os.listdir(sys.argv[1]):
    #print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()

        # Hotdoc uses `[][Internal Title]` for internal links whilst Hugo uses `[](#internal-title)`
        pattern = re.compile("\[\]\[(?P<link>.*?)\]")
        contents = pattern.sub(link_munge, contents)

        # Hotdoc allows empty links like `[](url)`
        pattern = re.compile("[^!]\[\]\((?P<link>.*?)\)")
        contents = pattern.sub(link_munge_2, contents)

        # Media links not working
        pattern = re.compile("!\[\]\((?P<url>.*?)\)")
        contents = pattern.sub(url_munge, contents)

        file.seek(0)
        file.truncate()

        file.write(contents)

```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Hotdoc allows "---" to be used for a code block, this is not supported
by Hugo, it needs to use "```". Convert uses of the former with:

  sed 's/^---$/```/' -i *

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Convert the existing metadata stored at the front of each file to
frontmatter formatted for Hugo:

 - Remove authors as we don't have an equivalent for that in Hugo
 - Add in fields that the existing metadata doesn't have
 - Include aliases to allow us to point to the new pages from a
   redirect on designs.apertis.org
 - Ensure we have a title (based on filename if one not present)

All frontmatter created in toml format to stay consistent with existing
documents.

Converted using the following python script:
```

import os
import re
import sys
import toml
import yaml

for filename in os.listdir(sys.argv[1]):
    print("%s: " % filename)
    with open(filename, 'r+') as file:
        contents = file.read()
        if contents[0:3] == "+++":
            # We have toml
            data = toml.loads(contents.split("+++")[1])
            doc = "+++".join(contents.split("+++")[2:])
        elif contents[0:3] == "---":
            # We have yaml
            data = yaml.load(contents.split("---")[1])
            doc = "---".join(contents.split("---")[2:])
        else:
            # No frontmatter
            data = {}
            doc = contents

        if not "title" in data.keys():
            data["title"] = filename.split(".")[0].replace('_', ' ').capitalize()

        data["weight"] = 100
        data["aliases"] = ["/old-designs/latest/%s" % filename.replace(".md", ".html")]

        data["outputs"] = ["html", "pdf-in"]

        if "authors" in data.keys():
            del data["authors"]

        file.seek(0)
        file.truncate()

        file.write("+++\n")
        file.write(toml.dumps(data))
        file.write("+++")
        file.write(doc)
```

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Some of the pages lack titles, others are just menu items we don't
need. Tweak the pages with initial pass to give them all titles and
remove stuff we don't need (we will be sorting the designs anyway).

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>

Martyn Welch authored 4 years ago and Emanuele Aina committed 4 years ago

Add the designs from designs.a.o with the minimum changes to get the
site to build. This essentially required to trailing marker for the
metadata to be changed from "..." to "---".

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>