Avoiding Ansible apt_key
on Debian
The apt_key
Ansible module has been deprecated and there isn’t a drop-in
replacement. There is a workaround available which achieves the same result
but requires more steps. This post describes the issue and presents my riff
on the recommended solution.
This is a retropost: it describes a problem I had at my previous job while upgrading servers from Debian buster to bullseye. Although the information presented here is a bit outdated, I hope it helps someone who finds themselves in a similar situation.
Ansible is great in the way that it encapsulates
the details of common sysadmin tasks in well-maintained modules. A change
to how Debian handles GPG keys for third-party APT repositories means that
we can no longer use the apt_key
module; we now have to do more manual
work to achieve the same result. This is my variation on the recommended
workaround.
If you’ve upgraded from Debian buster to bullseye, and you use third-party APT repositories for some software packages, you’ll have probably noticed a warning like this when upgrading them:
W: https://apt.releases.hashicorp.com/dists/buster/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
What this means is that one can’t use the /etc/apt/trusted.gpg
file as a
global keyring for third-party packages anymore. Also, the apt-key
command is now
deprecated with
bullseye being the last Debian version to use it.
We can get more information about the deprecation by looking at the
DEPRECATION
section of the apt-key(8)
man
page:
If your existing use of apt-key add looks like this:
wget -qO- https://myrepo.example/myrepo.asc | sudo apt-key add -
Then you can directly replace this with (though note the recommendation below):
wget -qO- https://myrepo.example/myrepo.asc | sudo tee /etc/apt/trusted.gpg.d/myrepo.asc
Make sure to use the “asc” extension for ASCII armored keys and the “gpg” extension for the binary OpenPGP format (also known as “GPG key public ring”). The binary OpenPGP format works for all apt versions, while the ASCII armored format works for apt version >= 1.4.
This is all well and good, but as Ansible users we have to manage the
keyring files ourselves now and we can’t use the apt_key
module anymore.
And it did its job so well! Cue “sysadmin-sad-face”.
Oh well, let’s rock on and find a solution.
Cobbling together a workable solution
From the notes section of the Ansible apt_key
module
we have:
The apt-key command has been deprecated and suggests to ‘manage keyring files in trusted.gpg.d instead’.
So each third-party repository has to have its own keyring file in
/etc/apt/trusted.gpg.d
. Good, that’s one piece of the puzzle. How should
we do that as an Ansible task now? Fortunately, the Ansible apt_key
module
documentation
has a suggestion:
- name: One way to avoid apt_key once it is removed from your distro, armored keys should use .asc extension, binary should use .gpg
block:
- name: somerepo |no apt key
ansible.builtin.get_url:
url: https://download.example.com/linux/ubuntu/gpg
dest: /etc/apt/trusted.gpg.d/somerepo.asc
- name: somerepo | apt source
ansible.builtin.apt_repository:
repo: "deb [arch=amd64 signed-by=/etc/apt/trusted.gpg.d/myrepo.asc] https://download.example.com/linux/ubuntu stable"
state: present
That’s cool! So, the new solution still verifies the keys when downloading
and installing them like apt_key
did, right? Right?
Unfortunately, that’s not the case and how to do this is left as an exercise for the reader. Can we glean any ideas from other locations to come up with a more complete solution?
Looking around the web, it seems that most advice from projects or businesses providing third-party APT repositories is to perform steps like these:
- download the project’s APT repository key
- verify its fingerprint by checking against a known value (not all sites mention this though)
- convert the key into the binary GPG format via
gpg --dearmor
- save the file to
/etc/apt/trusted.gpg.d
(or/usr/share/keyrings
) - set the
signed-by
attribute in the APT repository sources configuration to point to the binary keyring file (note that this was implicitly mentioned by the Ansibleapt_key
workaround recommendation).
I think what worries me the most here is the amount of work to do by hand to
import a third-party package’s key. In particular, I’d like to verify the
key’s fingerprint automatically as part of an Ansible task (this is
something that the apt_key
module did). But, as far as I could tell,
there aren’t any Ansible modules doing this as an optional, single step.
I did find ansible-gpg-key
,
but to verify the key with its fingerprint, one has to import the key, and
this is something I’d like to avoid. ansible-gpg-key
seems to want to
install the keys into a central keyring, which I also want to avoid. In
other words, I had to come up with my own workaround for key verification.
After staring at this a while I realised that I could download the key to my
development machine and check the fingerprint there. Then I could create a
checksum for this file which I could use in get_url
to check that the file
that Ansible downloads is correct. At least then I can be very sure that an
Ansible step using get_url
downloads a copy of the file that I’ve verified
by hand.
Also, because the manual fingerprint verification step is only necessary every few years, this is an acceptable amount of effort. It’s just something to keep in mind when handling new keys in the future.
Plan of attack
The plan of attack for the workaround goes like this:
Initial manual preparation on a development machine
- download the key file
- verify the file’s fingerprint using
gpg --show-keys
- if the fingerprint matches, use
sha256sum
to create a checksum for theget_url
module to use
Ansible steps as part of a playbook task
- download the key file with the
get_url
module and use the checksum generated above - convert this to the binary GPG key format (an optional, but common, step);
this will require the
shell
module andgpg --dearmor
- put the key file into the right location; we can do this as part of the
previous step (use the
-o
option togpg --dearmor
) - configure the APT source file appropriately; we can use
apt_repository
for this
Let’s see this in action.
Action stations!
At my previous job,1 we installed HashiCorp Vault by using the HashiCorp Linux Repository as a third-party APT repository. The HashiCorp Official Packaging Guide documents how to set this up.
Let’s set up the HashiCorp Linux Repository to illustrate the process I recommend.
Initial preparation
First, we need to get the key. Download it from the upstream
source into the /tmp
directory on
our local system, calling it hashicorp-key.asc
:2
$ wget -O /tmp/hashicorp-key.asc https://apt.releases.hashicorp.com/gpg
After downloading the GPG key, we need to check that the fingerprint
matches that mentioned on the HashiCorp security
page. To do this, we need to use
the --show-keys
option to the gpg
command:
$ gpg --show-keys /tmp/hashicorp-key.asc
pub rsa4096 2023-01-10 [SC] [expires: 2028-01-09]
798AEC654E5C15428C8E42EEAA16FCBCA621E701
uid HashiCorp Security (HashiCorp Package Signing)
<security+packaging@hashicorp.com>
sub rsa4096 2023-01-10 [S] [expires: 2028-01-09]
We then compare the string
798AEC654E5C15428C8E42EEAA16FCBCA621E701
(which appears under the pub
part of the --show-keys
output) with the fingerprint mentioned on the
HashiCorp security page.3
If the fingerprint matches (and in the current example, it does) we generate
the SHA256 checksum of the file via the sha256sum
program:
$ sha256sum /tmp/hashicorp-key.asc | cut -d' ' -f1
cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029
I’ve used the cut
shell command here to extract the checksum value from
the output. Otherwise, we’d have the filename appear in the output, which
we don’t need.
To use this checksum in an Ansible step, we need to prepend the string
sha256:
to it so that Ansible knows what kind of checksum to generate.
Thus Ansible will be able to correctly compare the downloaded file’s
checksum with the expected checksum. In other words, we specify the
following string in the relevant Ansible step:
sha256:cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029
With this information verified and available, we can define the Ansible
steps to download the key and configure the third-party APT repository.
Then it will be possible to install HashiCorp Vault via apt
.
Ansible steps
The Ansible steps are simply generalisations of the commands one would enter at the command line. There are three steps:
- download the ASC file and check that its contents are what we expect
- “dearmor” the ASC file (turning it into a binary GPG file)
- configure the HashiCorp Linux APT repository
You’ll notice that the HashiCorp Official Packaging
Guide recommends
downloading the ASC file and “dearmoring” it in one step. It’s not possible
to join these steps when implementing this process in Ansible. This is
because the get_url
module needs to download the ASC keyring file and
verify it before continuing. Thus, we can’t bundle downloading and
“dearmoring” into one step. For one-off situations, this is probably fine.
But we want to do this automatically and potentially across several hosts,
hence it’s best to split downloading and “dearmoring” into their component
steps.
Download
To download the upstream GPG file, we use the get_url
module.
- name: download the package signing key for the HashiCorp Debian repository
get_url:
url: https://apt.releases.hashicorp.com/gpg
dest: /tmp/hashicorp-archive-keyring.asc
checksum: "sha256:cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029"
The step specifies the URL of the file we want to download, the checksum to
check it with, and that its download destination is in the /tmp/
directory. I’ve put the file in /tmp
because we’ll be creating a binary
version of it in the next section and hence we don’t need to keep the ASC
file.
Dearmor
An optional, yet rather common, step is to “dearmor” the ASC file,
converting it into a binary GPG file. One does this via the --dearmor
option to the gpg
command.4 For instance:
$ gpg --dearmor -o /etc/apt/trusted.gpg.d/hashicorp-key.gpg /tmp/hashicorp-key.asc
This command will create the file hashicorp-key.gpg
in the
/etc/apt/trusted.gpg.d/
directory.
To convert this shell command into an Ansible step, we use the shell
module.
- name: dearmor the downloaded signing key file
shell: gpg --batch --dearmor -o /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg /tmp/hashicorp-keyring.asc
args:
creates: /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg
Here we’ve told Ansible the full path of the file that this step creates. Thus, Ansible can skip this step if the file already exists the next time it runs the step.
Note that we’ve also used the --batch
option to gpg
which tells gpg
to
never ask for any confirmations and to disallow interactive commands. This
way the gpg
command won’t hang and indefinitely wait for input if
something goes wrong when we run the step.
The -o
option places the binary GPG output file in
/etc/apt/trusted.gpg.d/
. Note that this is not the directory suggested
by the HashiCorp documentation! I’ve chosen instead to use the Debian
standard location for such files. As
mentioned in the Debian secure apt docs:
… the keyrings are stored in specific files all located in the /etc/apt/trusted.gpg.d directory
I think that if one is running Debian servers it makes sense to use the standard Debian locations rather than the locations recommended by third party software. This way, any sysadmin familiar with secure apt on Debian will know where to look for the keyring files.
APT repository configuration
To configure the third-party APT repository, we use the apt_repository
module:
- name: add HashiCorp Linux repository
apt_repository:
repo: deb [arch=amd64, signed-by=/etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com main
state: present
filename: hashicorp-linux
This is much the same as what one would normally enter into
/etc/apt/sources.list
. The main differences to the standard case are:
- the keyring file to verify the upstream repository is specified explicitly
via the
signed-by
attribute, and - the configuration is placed in its own file via the
filename
parameter.
There are some subtleties here with the filename
parameter: Ansible will
implicitly place the file in the /etc/apt/sources.list.d
directory and
will append the extension .list
to the filename. Thus, although we used
hashicorp-linux
for the filename
parameter, the file will actually be
called hashicorp-linux.list
on the filesystem. In other words, don’t use
hashicorp-linux.list
for the filename
parameter because then the file
will be called hashicorp-linux.list.list
. That would be
silly.
All together
Putting all the steps together, we arrive at this final Ansible configuration:
- name: download the package signing key for the HashiCorp Debian repository
get_url:
url: https://apt.releases.hashicorp.com/gpg
dest: /tmp/hashicorp-archive-keyring.asc
checksum: "sha256:cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029"
- name: dearmor the downloaded signing key file
shell: gpg --batch --dearmor -o /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg /tmp/hashicorp-keyring.asc
args:
creates: /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg
- name: add HashiCorp Linux repository
apt_repository:
repo: deb [arch=amd64, signed-by=/etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com main
state: present
filename: hashicorp-linux
And that’s it! All set up Now it’s just a matter of running these steps in an Ansible playbook to roll out the configuration to our servers.
Wrapping up
I would have liked to verify the GPG key automatically by matching its fingerprint with a known value. However, there doesn’t seem to be an Ansible module able to do this. I did consider writing one but decided not to dive down that particular rabbit hole.
Since such keys change only every few years, the effort to download a key to
a development system, verify its fingerprint, and then create a checksum of
the file (so that get_url
can verify the file’s contents) is sufficient
for our purposes here.
I would have liked things to be more automated, but you can’t always get what you want. Maybe in the future, there’ll be an Ansible module which automatically downloads and verifies a GPG key according to a given fingerprint value. Who knows? Time will tell.
Addendum: handling Grafana
If you want to use Grafana and also want to install it via apt
, then
you’ll want to use the get_url
and apt_repository
pattern discussed
here.
Unfortunately, it’s hard to find the fingerprint for the Grafana repository’s GPG key. For instance, the Grafana Debian installation docs don’t mention the keys’ fingerprint. In investigating this issue myself, I managed to stumble across an entry in the Grafana community forum mentioning the Grafana package repository. The package repository web page mentions the fingerprint, so they do publish the fingerprint, just not anywhere obvious.
Note that Grafana refers to its public GPG file with the extension .key
.
Thus, if you want to be consistent with the Debian documentation, you
should use .asc
because this is the extension for an ASCII armored key.
Dearmoring the key, adding it to the system, and configuring the third-party APT repository is then the same as in with the HashiCorp Linux APT repository discussed in this post.
-
I’m now a freelance software developer doing Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and let’s discuss how I can help solve your business’ hairiest problems. ↩
-
Otherwise, its name would be
gpg
. This name isn’t helpful because it’s not clear where it came from and why. ↩ -
At the time of writing, this was under the “Linux package checksum verification” section. ↩
-
As a native speaker of a descendant variant of British English, I find it nice that the GPG developers also implemented this option as
--dearmour
. ↩
Support
If you liked this post and want to see more like this, please buy me a coffee!