LEAPP: upgrade fails when using custom certs

2024-07-11Thomas FischerBlog

If you ever tried to upgrade an old RedHat(alike) system to a newer version and got stuck with:

Curl error (60): Peer certificate cannot be authenticated with given CA certificates for….

well then you might have not found a way to solve it. until now :)

This happened to me recently when upgrading from a CentOS 7.9 release to AlmaLinux 8. The process uses the standard leapp tool but the AlmaLinux repos are signed by a CA (atm Amazon) which are unknown to the OS.

Easy! as you might think.. simply:

put all certs which are unknown to /etc/pki/ca-trust/source/anchors/
execute: update-ca-trust
done

well yep. if it would be that easy you would not have reached this post ;) but do the above! This is required in any case. So do it now if you haven’t.

Note: the next is only required if you still have this issue after the above execution! RHEL had finally merged a fix for this issue but that requires leapp 0.19 or later.
So if you still encounter the issue after the above or you have an older leapp version – read on.

The reason this is not the (full) solution is that during the leapp upgrade process a chroot will be created (e.g. /var/lib/leapp/el8userspace).

Unfortunately this will result in a standard chain of CA trust (i.e. without your custom CA’s) and there is no elegant way of adding files (i.e. certificates) to this chroot during the process.
This is fixed in newer leapp versions as stated above but e.g. AlmaLinux provides an older leapp version (as of today 0.15) which does not contain that fix.

Someone suggested to copy the certs manually to the chroot etc, but: the chroot will be re-created (if not in its whole at least the pki part for sure) during the process (in that same link you find a way to disable SSL cert completely with an ugly hack – but I really do NOT recommend doing so!).

So finally how to fix?

edit /etc/leapp/repos.d/system_upgrade/common/libraries/mounting.py
find near the top the
ALWAYS_BIND = []
line and adjust it like this:
ALWAYS_BIND = ['/etc/pki/ca-trust']
again this will work only if your custom CA’s are setup as mentioned above!
now restart the leapp upgrade process and it should pass through!

an Ansible command which achieves the above would be:

ansible <yourHost> \
    --ask-vault-pass \
    -i hosts \
    -m lineinfile \
    -u <ansible-ssh-user>
    -b -a \
    "path=/etc/leapp/repos.d/system_upgrade/common/libraries/mounting.py regexp='ALWAYS_BIND =.*' line='ALWAYS_BIND = [\'/etc/pki/ca-trust\']'" -vv

credits:
– RHEL-11347
– image by rawpixel.com on Freepik

Blog