Better OpenPGP keys handling for signing the repository packages

[primeos-work]

I also ran into #3818 but on Fedora 40. The additional issue here is that updating gpgkey in /etc/yum.repos.d/kubernetes.repo isn't enough here as the Package manager (dnf) has already downloaded the key and, at least in my case, will not try to re-fetch it (even if I disable the repo, run dnf clean all, and enable it again). I didn't have such issues with other repos and the problem here seems to be that only the expiration date of the key was extended instead of rotating the entire key. I'm not a dnf expert but I think most repos are switching to new keys instead of extending the experiation date. AFAIK it should also be possible to fetch/import multiple keys from a single URL so that should help with supporting older and newer packages (in this case only relevant when using an older baseurl with a newer gpgkey URL).

What happened:

I updated the repository:

[root@groot etc]# cat /etc/yum.repos.d/kubernetes.repo
# https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
# https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/change-package-repository/
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key

And I can fetch the current version of the key from the gpgkey URL:

[root@groot etc]# curl -sL https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key | gpg --show-key
pub   rsa2048 2022-08-25 [SC] [expires: 2026-12-29]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid                      isv:kubernetes OBS Project <isv:[email protected]>

But updating still fails:

[root@groot etc]# dnf up
Kubernetes                                                                                                                                                             21 kB/s | 8.9 kB     00:00
Dependencies resolved.
====================================================================================================================================================================================================== Package                                     Architecture                               Version                                                  Repository                                      Size
======================================================================================================================================================================================================Upgrading:
 kubectl                                     x86_64                                     1.31.3-150500.1.1                                        kubernetes                                      11 M

Transaction Summary
======================================================================================================================================================================================================Upgrade  1 Package

Total download size: 11 M
Is this ok [y/N]: y
Downloading Packages:
kubectl-1.31.3-150500.1.1.x86_64.rpm                                                                                                                                  6.7 MB/s |  11 MB     00:01
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Total                                                                                                                                                                 6.7 MB/s |  11 MB     00:01     error: Verifying a signature using certificate DE15B14486CD377B9E876E1A234654DA9A296436 (isv:kubernetes OBS Project <isv:[email protected]>):
  1. Certificate 234654DA9A296436 invalid: certificate is not alive
      because: The primary key is not live
      because: Expired on 2024-11-02T16:21:11Z
  2. Key 234654DA9A296436 invalid: key is not alive
      because: The primary key is not live
      because: Expired on 2024-11-02T16:21:11Z
error: Verifying a signature using certificate DE15B14486CD377B9E876E1A234654DA9A296436 (isv:kubernetes OBS Project <isv:[email protected]>):
  1. Certificate 234654DA9A296436 invalid: certificate is not alive
      because: The primary key is not live
      because: Expired on 2024-11-02T16:21:11Z
  2. Key 234654DA9A296436 invalid: key is not alive
      because: The primary key is not live
      because: Expired on 2024-11-02T16:21:11Z
Kubernetes                                                                                                                                                            8.9 kB/s | 1.7 kB     00:00
GPG key at https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key (0x9A296436) is already installed
The GPG keys listed for the "Kubernetes" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: kubectl-1.31.3-150500.1.1.x86_64
 GPG Keys are configured as: https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

Workarounds:

I can use the following manual workaround to remove the key so that dnf will have to import it again:

[root@groot etc]# rpm --erase gpg-pubkey-9a296436-6307a177
[root@groot etc]# dnf up
Last metadata expiration check: 0:08:07 ago on Thu 21 Nov 2024 02:16:54 PM CET.
Dependencies resolved.
====================================================================================================================================================================================================== Package                                     Architecture                               Version                                                  Repository                                      Size
======================================================================================================================================================================================================Upgrading:
 kubectl                                     x86_64                                     1.31.3-150500.1.1                                        kubernetes                                      11 M

Transaction Summary
======================================================================================================================================================================================================Upgrade  1 Package

Total size: 11 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] kubectl-1.31.3-150500.1.1.x86_64.rpm: Already downloaded
Kubernetes                                                                                                                                                            7.6 kB/s | 1.7 kB     00:00
Importing GPG key 0x9A296436:
 Userid     : "isv:kubernetes OBS Project <isv:[email protected]>"
 Fingerprint: DE15 B144 86CD 377B 9E87 6E1A 2346 54DA 9A29 6436
 From       : https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                              1/1   Upgrading        : kubectl-1.31.3-150500.1.1.x86_64                                                                                                                                             1/2   Cleanup          : kubectl-1.29.10-150500.1.1.x86_64                                                                                                                                            2/2

Upgraded:
  kubectl-1.31.3-150500.1.1.x86_64

Complete!

There should be other workarounds available but this is the easiest one I came up with as there doesn't seem to be a good dnf command for it yet.

What you expected to happen:

dnf update will prompt to (re)import/update the key. This doesn't seem possible so the repo key should be rotated entirely instead of only extending the expiration date.

How to reproduce it (as minimally and precisely as possible):

Install from an older repository with the expired key and then try to update again.

Version 1.27 seems to be the most recent version that offers the old key:

[michael@groot ~]$ curl -sL https://pkgs.k8s.io/core:/stable:/v1.27/rpm/repodata/repomd.xml.key | gpg --show-key
pub   rsa2048 2022-08-25 [SC] [expired: 2024-11-02]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid                      isv:kubernetes OBS Project <isv:[email protected]>

I can provide the exact commands if necessary but the basic steps are the following:

• Uninstall the OpenPGP key: rpm -qa | grep 9a296436 | xargs rpm --erase
• Switch /etc/yum.repos.d/kubernetes.repo back to version 1.27 (or an older version)
• Install a package (e.g., kubectl) from the old repo. You might have to disable gpgcheck. Interestingly the expired key erros seem to be ignored when downgrading via: dnf downgrade kubectl
• Switch /etc/yum.repos.d/kubernetes.repo back to the most recent version (currently 1.31)
• Try to upgrade the package again, e.g.: dnf upgrade kubectl -> it will fail due to the expired key

Anything else we need to know?:

Environment:

Fedora 40 but it should apply to any yum/dnf based system (it might work better with the future dnf5 though - I didn't look at that).

[primeos-work]

Update: I just discovered rpm-software-management/dnf5#1192 so it looks like there's hope on the horizon.

I guess we could consider closing this issue then, given that https://github.com/fedora-copr/copr apparently also doesn't rotate the keys and that rpm-software-management/rpm#3083 already got merged (apparently that PR alone isn't sufficient though: rpm-software-management/dnf5#1192 (comment)).

That said, all (perhaps with some exceptions for rolling releases, etc.?) current RPM based distributions are still affected by this and will likely remain affected for their lifetime (so LTS users remain affected until their distro hits EOL and they upgrade to a newer LTS release).

The updated key expires 2026-12-29:

$ curl -sL https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key | gpg --show-key
pub   rsa2048 2022-08-25 [SC] [expires: 2026-12-29]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid                      isv:kubernetes OBS Project <isv:[email protected]>

[jan-kolarik]

Update: I just discovered rpm-software-management/dnf5#1192 so it looks like there's hope on the horizon.

I guess we could consider closing this issue then, given that https://github.com/fedora-copr/copr apparently also doesn't rotate the keys and that rpm-software-management/rpm#3083 already got merged (apparently that PR alone isn't sufficient though: rpm-software-management/dnf5#1192 (comment)).

That said, all (perhaps with some exceptions for rolling releases, etc.?) current RPM based distributions are still affected by this and will likely remain affected for their lifetime (so LTS users remain affected until their distro hits EOL and they upgrade to a newer LTS release).

The updated key expires 2026-12-29:

$ curl -sL https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key | gpg --show-key
pub   rsa2048 2022-08-25 [SC] [expires: 2026-12-29]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid                      isv:kubernetes OBS Project <isv:[email protected]>

Hi there, I noticed the reference to DNF5 🙂 We plan to deliver this functionality in Q1 2025, following acceptance of the behavior change in our primary Fedora environment.

What you expected to happen:

If that's all that's required, it will be addressed once the ticket is implemented. If you have additional use cases or requirements, please add them to the DNF5 ticket. Thanks!