aptly: add documentation

Documentation of tasks for maintaining aptly.
Some tasks are recurring and were not documented.

Signed-off-by: Pablo Vigo <pvigo@collabora.com>
Co-authored-by: Ritesh Raj Sarraf <ritesh.sarraf@collabora.com>

docs/aptly.md

@@ -31,3 +31,116 @@ $ gpg --fingerprint
 $ gpg --export-secret-key --armor
 ```
 
+## Increase PVC volume
+
+Aptly generates a large number of files, and it is necessary to periodically 
+increase the size of the PVC where these data are stored.
+
+**IMPORTANT**: Restarting is NOT necessary to increase the PVC volume.
+
+This document describe the process for increasing this volume correctly.
+
+### Find the volume of the aptly instance
+
+It's possible to find the volume ID running the following command:
+
+```get pvc -n aptly-XXXX | awk '{print $3}'```
+
+### Take a snapshot of the volume in DO
+
+From the [control panel](https://cloud.digitalocean.com/), click **Volumes**.
+
+Open the **More** menu for the volume you want to snapshot, then click **Take Snapshot**.
+
+### Increase the Volume
+
+After you create a backup of the volume, you can increase the volume’s size in 
+the control panel. 
+
+From the volume’s **More** menu, choose **Increase size**.
+
+After few seconds, you can see that size of the volume has been increased, but 
+it won't reflect in the machine. We need to do some more steps to make it work.
+
+### Update ansible-playbook value
+
+Increase the volume size updating the **volume_size** value present in the inventory 
+files of the ansible-playbook.
+
+Update to the new **volume_size** for the ```aptly``` instance you want to increase. It should be in-sync with the increased volume size on DO
+
+### Apply the new volume value
+
+Run the ansible-playbook in check mode, to assess the changes to be applied to the cluster
+
+```ansible-playbook -i inventories/production -l aptly-XXXX digital-ocean-k8s/aptly.yaml --diff --check```
+
+The result of the command should only reflect the change in volume size. Once the checked difference is validated, run the ansible-playbook in action/real mode.
+
+```ansible-playbook -i inventories/production -l aptly-XXXX digital-ocean-k8s/aptly.yaml --diff```
+
+This should effectively apply the changes to the cluster. Validate that the changes are active on the cluster.
+
+## Perfom a DB Cleanup
+
+Database cleanup removes information about unreferenced packages and deletes 
+files in the package pool that aren’t used by packages anymore. 
+
+It is a good idea to run this command after massive deletion of mirrors, 
+snapshots or local repos.
+
+It is important to perform the cleanup without any disk operations ongoing. 
+For this reason, it is advisable to temporarily delete the 'ingress' object to 
+ensure that there is no communication with the service while the cleanup is underway.
+
+### Take a snapshot of the volume in DO
+
+From the [control panel](https://cloud.digitalocean.com/), click **Volumes**.
+
+Open the **More** menu for the volume you want to snapshot, then click **Take Snapshot**.
+
+### Backup ingress
+
+Run this command to generate a backup of the ingress
+
+```kubectl get ingress oathkeeper-proxy -n aptly-XXXX -o yaml > ~/ingress-oathkeeper-proxy.yaml```
+
+### Delete ingress
+
+Run this command to delete the ingress and avoid connections
+
+```kubectl delete ingress oathkeeper-proxy -n aptly-XXXX```
+
+### Prepare environment
+
+Port forwarding to connect to the service. 
+The above command will create a port-forward tunnel, mapping port 8080 from the remote pod to port 8080 on the localhost, where kubectl is invoked from.
+
+```kubectl -n aptly-infra-repositories port-forward statefulset/aptly 8080:8080```
+
+Declare necessary variables
+
+```export APTLY_API_URL=http://127.0.0.1:8080```
+
+```export APTLY_API_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXX```
+
+Check connection
+
+```curl -k -H "Authorization: Bearer $APTLY_API_TOKEN" $APTLY_API_URL/api/repos```
+
+### Run DB Cleanup
+
+This command will run the DB cleanup in the aptly instance.
+The execution time will depend on the amount of data available in the instance.
+
+```curl -k -X POST -H "Authorization: Bearer $APTLY_API_TOKEN" $APTLY_API_URL/api/db/cleanup```
+
+### Restore service
+
+Run this command to restore the backed up ingress configuration
+
+```kubectl apply -f ingress-oathkeeper-proxy.yaml```
+
+Validate that the ingress is active and the aptly service is accessible
+
+```curl -k -H "Authorization: Bearer $APTLY_API_TOKEN" https://URL-aptly/path/_aptly/api/repos```