Clean up temporary files and directories

Whilst not important in CI as they will get cleaned up, having
temporary files and directories left after building locally is a pain.
Tweak to clean these files up.

Signed-off-by: Martyn Welch <martyn.welch@collabora.com>