Fleet Management

This controls all of the fleet management software to be able to:

• Bring up new hosts of a certain type
• Perform management operations on a certain host/type

Installing Ansible

Per the instructions here http://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#latest-releases-via-apt-ubuntu

sudo apt-get update
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible

Using ansible

Use s/ansible-playbook to run ansible-playbook, you'll first need to provide the fleet management encryption password, and after it will ask you to delete the locally decrypted files. Unless there was an error this should be perfectly safe.

Using ansible version ansible-playbook 2.9.12

Configuration file

• Avoid bullshit

Adding new hosts

Note: opensmtpd is not installed from ubuntu. The version supplied by Ubuntu (6.8.0p2) has a fairly serious bug where mail cannot be received from all servers. I've manually install opensmtpd-7.4.0p1 from source (./configure; make; sudo make install).

The following manual steps may be needed:

• symlink: /usr/lib/ssl/cert.pem -> /etc/ssl/certs/ca-certificates.crt

1. Add to inventory.yml to the ubuntu_needs_bootstrap role for bootstrapping
2. Run ansible/playbooks/bootstrap-new-hosts.yml
3. Manually set a password for the sufian user (for sudo)
4. Change inventory.yml to add roles
5. Run ansible/playbooks/update-hosts.yml
6. Run various deploy scripts

Note:

• For new hosts, you need to sudo certbot run -d ${DOMAIN} for each domain
• For new hosts, you need to generate a new tarsnap keypair: sudo tarsnap-keygen --keyfile /root/tarsnap.key --user sufian@abstract.properties --machine host-NNNN.abstract.properties and place (encrypted) as ansible/roles/common/files/host-NNNN.abstract.properties.tarsnap.key.aes
• For new hosts, you need to install nodejs apt settings:
  1. curl -sL https://deb.nodesource.com/setup_16.x | sudo bash - (yes seriously, fuck this, what the fuck)
  2. sudo apt-get install -y nodejs
  3. Verify node 16 is installed: node --version
  4. Enable corepack: sudo corepack enable
  5. Update yarn: yarn corepack prepare yarn@stable --activate

Quarterly maintenance

1. Generate new ssh authorized keys for: ansible/roles/common/files/{ansible_config_authorized_keys,sufian_authorized_keys}

• via: ssh-keygen -t ed25519 -a 100 -f out.ed25519.key
• Don't forget to update github / other places that use this public key

2. Update TLS certs for host-0004.abstract.properties
3. On host: sudo certbot --nginx * Note: the certificates may already have been renewed. Check via:
   • sudo cat /etc/letsencrypt/live/abstract.properties/cert.pem | openssl x509 -inform pem -noout -enddate
4. Copy files from host-0004.abstract.properties:"/etc/letsencrypt/live/${DOMAIN}/*" to the appropriate encrypted destination in ansible/roles/static_web/files/certs/${DOMAIN}/*: * rsync -avh --rsync-path="sudo rsync" --copy-links 'ansible-client@host-0004.abstract.properties:/etc/letsencrypt/live/' ansible/roles/static_web/files/certs
5. Encrypt the new files: find ansible/roles/static_web -type f -name '*.pem' | xargs s/encrypt
6. Update TLS certs for host-0005.abstract.properties
7. On host: sudo certbot --nginx * Note: the certificates may already have been renewed. Check via:
   • sudo cat /etc/letsencrypt/live/abstract.properties/cert.pem | openssl x509 -inform pem -noout -enddate
8. Copy files from host-0005.abstract.properties:"/etc/letsencrypt/live/${DOMAIN}/*" to the appropriate encrypted destination in ansible/roles/app_shshararee/files/certs/${DOMAIN}/*: * rsync -avh --rsync-path="sudo rsync" --copy-links 'ansible-client@host-0005.abstract.properties:/etc/letsencrypt/live/' ansible/roles/app_shshararee/files/certs
9. Encrypt the new files: find ansible/roles/app_shshararee -type f -name '*.pem' | xargs s/encrypt
10. Update hosts: s/ansible-playbook ansible/playbooks/update-hosts.yml
11. Verify cert expiry: /etc/cron.daily/check_cert_expiry --no-send
12. Check if kernel needs reboot, reboot if needed

Backups

tarsnap is configured and installed for backups.

Currently, all git repos in /srv/git are shallowly backed up on a weekly basis via cron.weekly.

• By default, only HEAD is backed up.
• Add a periodic-backup-refs file if you want multiple refs backed up

To restore, use the tarsnap key to check the <hostname>:git:<datestring> archive.

Yearly, the tarsnap debian signing key expires, this will cause the tarsnap key to fail:

• Check the instructions here: https://www.tarsnap.com/pkg-deb.html
• The signing key is stored here: ansible/roles/common/files/tarsnap-deb-packaging-key.asc

Upgrading pleroma

Easy instructions (new)

1. Go to the pleroma release directory: cd /opt/pleroma/release
2. Safety -- turn it off: sudo service pleroma stop
3. Download the new version: sudo -u pleroma ./bin/pleroma_ctl update
4. Migrate the db: sudo -u pleroma ./bin/pleroma_ctl migrate
5. (optional) Vacuum the db: sudo -u pleroma psql pleroma -- run VACUUM ANALYZE
6. Safety -- turn it back on: sudo service pleroma start

Hard instructions (old)

1. Get a new version of pleroma: sudo -u pleroma curl 'https://git.pleroma.social/api/v4/projects/2/jobs/artifacts/stable/download?job=amd64' -o /opt/pleroma/pleroma-2023-02-11.zip
2. Stop the service and ensure it's down: sudo sysctl stop pleroma
3. Move prior release in case things go south sudo mv /opt/ple;roma/release /opt/pleroma/release-prev
4. Unzip new version (within /opt/pleroma) sudo -u pleroma unzip pleroma-2023-02-11.zip
5. Run database migrations (if any) cd /opt/pleroma/release sudo -u pleroma bin/pleroma_ctl migrate
6. Start the service again sudo sysctl start pleroma
7. Put the zip in the right place, fix ansible files

Upgrading pleroma frontend (pleroma-fe)

The frontend is installed as static files in /var/lib/pleroma/static

The "dist" directory of pleroma-fe is synced to that folder.

Updates involve building pleroma-fe and syncing the dist directory over.

Monitoring services / healthchecks

Write some cron jobs that run every minute / hour; and if fail post via Slack app hooks.

---

goaccess --log-format='%dT%t\t%h\t%^\t%^\t%^\t%u\t%m\t%v\t%U\t%s\t%b\t' --date-format='%Y-%m-%d' --time-format='%H:%M:%S%z'