Unattended Upgrade Arch Linux – are you serious (part 2) !?!

This is a follow-up of:
Unattended Upgrade Arch Linux – are you serious (part 1) !?!

 

As promised here comes the guide and all relevant information about:

 

uAu – unattended Arch upgrade

 

The following topics are tagged with either [info] or [upgrade] or both. The idea is: when you just want to auto upgrade follow the tags [upgrade] only – when you do not want to auto upgrade follow only the tags with [info] and for the full experience just go through all ;)

 

[info][upgrade] AUR helpers

 

Tbh I wanted to use pacaur because like the pacaur descriptions says: “An AUR helper that minimizes user interaction“. Exactly what we need right?
The problem: the current version of pacaur (v4.x) has a bug (issue #433 and #681) which will be fixed in v5 (hopefully) and we cannot use a workaround because pacaur is missing the feature to choose a custom config file (PR #682). Unfortunately the dev decided to never include the –config file option available in pacman for strange reasons but that’s the way it goes..

While doing all this I found out (much later) that the idea of using pacaur fails for me. The main reason is that it does not refresh the databases in all cases and more important it does not work with the ignores I need (e.g. when just showing the upgrades available). So I switched to yaourt instead and later on to trizen (as yaourt is outdated and not longer maintained).

pacman -S git trizen

[info][upgrade] Prepare & Install

After going through all the stuff I decided to create a package instead of writing all the necessary steps to go.
This means now just doing the following (instead of doing hundreds of steps):

trizen -S aur-comment-fetcher-git python3-memoizedb

cd /tmp
git clone https://github.com/steadfasterX/arch_uau-pkg.git
cd arch_uau-pkg
makepkg -si

[info][upgrade] Prepare to notify about the Arch news feed

The next thing is I want to get informed about the Arch news by mail.
For this I installed the minimalist ssmtp which provides the sendmail like cmd mail. You can skip this step if you already have a tool providing the sendmail like mail command.

sudo pacman -S ssmtp

You need to configure the very simple config in /etc/ssmtp/ssmtp.conf:

# /etc/ssmtp.conf -- a config file for sSMTP sendmail.
#
# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=postmaster

# The place where the mail goes. The actual machine name is required
# no MX records are consulted. Commonly mailhosts are named mail.domain.com
# The example will fit if you are in domain.com and you mailhub is so named.
mailhub=mail.[YOUR-DOMAIN]:[PORT]

# Where will the mail seem to come from?
#rewriteDomain=y
# The full hostname
hostname=[YOUR HOSTNAME]

# use SSL/TLS <-- depends on your provider.
UseTLS=Yes
UseSTARTTLS=Yes

# Username/Password
AuthUser=[YOUR-MAIL-LOGIN]
AuthPass=[YOUR-MAIL-PW]
AuthMethod=LOGIN

The above configuration file contains sensitive data and has to be readable by the user archupdater.
… and in my case ONLY by him so I executed:

chown archupdater /etc/ssmtp/ssmtp.conf
chmod 400 /etc/ssmtp/ssmtp.conf

[info][upgrade] general configuration

/etc/unattended-arch-upgrade.conf:

A self explanatory file which describes inline all available options

/etc/unattended-arch-upgrade.ignore:

A self explanatory file which describes inline all available options

[info] Activate the info cron job

(note: this is handled by the uAu package now!)

Now i want to execute the above info mail by cron with /etc/cron.d/archnews :

# get Arch news plus information about your local packages which need to be upgraded
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
11 11 * * * root sudo -u archupdater /home/archupdater/cron_archnews

Why using user root and then sudo? Well the account archupdater is expired so a normal usage (e.g. su – archupdater or SSH) is not possible but using it with sudo is still working fine (tbh is this a bug?).

[upgrade][info] upgrade exceptions (unsupported by Arch)

pacman let you ignore packages or hold a package as you may/should know. The unattended arch upgrade process will respect those ignored packages and will not update them. You may want to have a different set of ignored packages when running a manual arch update (pacman -Syu or better trizen -Syu) as when doing an unattended one though.

A reason for this could be upgrading critical applications / packages can be excluded from an unattended upgrade and so minimizes the risk for you.
Keep in mind that applications may depend on libraries etc and if you upgrade a library but not the application which uses it this may break your application nevertheless.
So use ignores with care. Always. Not in the case of an unattended upgrade only..
and again: keep in mind that partial upgrades are explicitly not supported: read the Arch statement

If you still want to go on just create the file /etc/unattended-arch-upgrade.ignore:

# PARTIAL UPGRADES ARE NOT SUPPORTED! USE THIS WITH CARE!
# Check this first:
# https://wiki.archlinux.org/index.php/System_maintenance#Partial_upgrades_are_unsupported

# The option IgnorePkg EXTENDS the pacman ignores. This way you can have different ignores
# when doing a regular upgrade and an unattended.
# Format: 1 package in for every ignore! Do not use space delimitated syntax here!
#IgnorePkg=linux-lts
#IgnorePkg=linux

Outcomment and/or adjust the IgnorePkg option to your needs.

Note:
Your pacman ignore definitions will be extended by the above. Means BOTH ignore definition get read and respected for the unattended upgrade.

 

[upgrade] Test & Enable the unattended upgrade

(note: this is handled by the uAu package now!)

Make a single shot to see if all is working first! Modify /etc/systemd/system/unattended-arch-upgrade.timer to a time setting in the next minutes. Then execute:

systemctl start unattended-arch-upgrade.timer

Check if the timer gets activated: systemctl list-timers --all
Wait until the time passed by and check the output of the run:

journalctl -xef -u unattended-arch-upgrade.service

and

less /var/log/unattended-upgrade.log

If all went fine enable the TIMER on next boot (again – not the service. it gets activated by the timer):

systemctl enable unattended-arch-upgrade.timer

 

[upgrade] Recover from a broken auto update

Follow the steps from the TechRapid post

It is not necessary but I would recommend to do all troubleshooting tasks with the user archupdater like this:

sudo -u archupdater [your-command]

The reason is that this way e.g. the cache of that user can be used.

 

[info][upgrade] Notes

IgnorePkg will be respected by the above method so if you adjust /etc/pacman.conf to ignore one or more package(s) it will be excluded from being updated.
The cache of downloaded packages can be found here: /home/archupdater/.cache/

[info][upgrade] sudo permissions

(note: this is handled by the uAu package now!)

We need to adjust sudo to not ask for a password while upgrading with this specific user (note: this is handled by the uAu package now!)
Add a new sudoers file named /etc/sudoers.d/unattended-arch-upgrade:

# auto upgrade (yea the permissions are very high that's why I disable the account when creating the user)
archupdater ALL=NOPASSWD: /usr/bin/pacman *

Do not forget to make it read and writable to root only (note: this is handled by the uAu package now!):

chown root /etc/sudoers.d/unattended-arch-upgrade
chmod 600 /etc/sudoers.d/unattended-arch-upgrade

 

[upgrade] Create systemd services

(note: this is handled by the uAu package now!)

To be able to run some tests after the upgrade etc I created the file /usr/local/bin/unattended-upgrade:

#!/bin/bash
############################################################################
#
# Unattended Upgrade of Arch
#
############################################################################

LOG=/var/log/unattended-upgrade.log
UPDBIN=/usr/bin/yaourt

# TODO: log rotate (I do not want to overwrite by each run..)

echo "$0 has started $(date)"
echo "$0 has started $(date)" >> $LOG

$UPDBIN -Syu --noconfirm --needed --noprogressbar --noedit >> $LOG

systemctl --failed --plain >> $LOG

echo "$0 has finished"

Hint:
Do not use yaourt -q | –quiet as this will stop the installation of some or all AUR packages.

Then I created 2 systemd services. The service itself plus a timer which is well like an extended cron job for the service.

/etc/systemd/system/unattended-arch-upgrade.service

[Unit]
Description=Unattended Arch Upgrade
Requires=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/unattended-upgrade
TimeoutStopSec=900
KillMode=process
KillSignal=SIGINT
User=archupdater

Hints:
The above ensures the network-online is met before doing anything else. It stops the upgrade process after 900 seconds (15 minutes) but in a way pacman / yaourt is able to handle without leaving you with a half installed package. The whole command gets executed as the privileged user archupdater created above.
No [install] section is required as this service gets controlled by the following timer instead!

/etc/systemd/system/unattended-arch-upgrade.timer

[Unit]
Description=Arch Update Timer. Starting the unattended upgrade at a given time.

[Timer]
OnCalendar=Tue,Sat *-*-* 17:16:02
Persistent=false

[Install]
WantedBy=multi-user.target

Hints:
OnCalendar –> systemd.time.7.html
Persistent=true means if missed the last time it starts once when booted immediately.
I personally do not want to do this but you may. I want to ensure only the given days are active because I feel those are the best days for updating (I can be wrong though).
I recommend to start the update at the end of your working day and not in the morning or while actually working.

Skip the following.

(I just leave it here for reference..)

The following will checkout my own package build config pointing to my pacaur fork. The choosen version number 9.9.9 ensures no pacaur release will overwrite my changes. Atm we don’t know HOW the both issues above get fixed so it may requires a change in the systemd service call and so should be done not automatically when upgrading.

git clone https://github.com/steadfasterX/arch_pacaur-pkg.git
cd arch_pacaur-pkg
PATH=$PATH:/usr/bin/core_perl
makepkg -si

When you now make a pacaur -v you will see the version is v4.x-9.9.x . This version string is used internally only and should give you a understanding what my version is based on.
When you do a pacman -Q pacaur you will see version 9.9.x instead. This ensures it gets not upgraded by accident.

… and the result: uAu – [u]nattended [A]rch [u]pgrade !

so what are you waiting for? install uAu!

Check out the README and details here: the uAu package