I keep getting asked how my setup works, how I use tmux and nvim over ssh… all that good stuff. …
SSH certificates allow system administrators to SSH into machines without having to manage authorized keys in the servers.
In summary, you create a key pair to be used as a Certificate Authority (CA), and add the public key of that key pair to the server:
Then, usually, a system administrator or an automated system creates certificates for the users that need to access the servers.
Those certificates are created with the CA’s private key, the user’s public key, a list of principals and a validity period.
That means that to SSH into a machine, the user needs both their private SSH key and the certificate. The certificate should also match their user (principal) and be used within the time frame in which it is valid.
All those characteristics can help system administrators manage user keys by, well, not managing them at all. They can create short-lived (short being used loosely here) certificates automatically and send to the users, or use some system that creates a new certificate for each access (and those are in fact short lived, a couple of minutes only).
This solves a couple of problems:
- no need to manage authorized keys
- users can share a certificate + private key… but the certificate expire almost instantly, so it’ll be a pain for them to do that, hopefully enough to discourage that behavior
There is, though, a minor inconvenience: you’ll need to pass both your private key and the certificate when SSHing:
ssh -i /tmp/cert -i ~/.ssh/id_ed25519 host.foo.bar
okta, and I’m sure others, work around that by having a
SSH wrapper, so you do:
wrapper ssh host.foo.bar
And the wrapper:
- gets a new certificate for you from their API
sshpassing it as a parameter
This avoids the hassle of having to manually calling the API (or downloading a
cert from a webpage), giving it the right perms (
0600), and then finally
SSHing into the target server.
It’s quite simple: you can have as many CA’s you want. Down to the server, if you want to.
Permissions like “which users can sudo” should probably still be managed by some configuration management tool.
The main thing here is managing, which machines a given user can SSH into.
Let’s try this out, shall we?
We can do it inside Docker, so we don’t risk locking anyone out while playing with it:
mkdir -p /tmp/ca cd /tmp/ca docker run -v $PWD:/tmp/ca -p 2222:22 -it --rm ubuntu
And let’s install OpenSSH Server in it, and create the CA’s key pair:
# within the container: apt update apt install -y --no-install-recommends openssh-server cd /tmp/ca ssh-keygen -f ca -t ed25519
Then, we need to set it up and restart the SSH server:
# within the container: # set up the ca key and authorized principals echo "TrustedUserCAKeys /tmp/ca/ca.pub PasswordAuthentication no" > /etc/ssh/sshd_config.d/ca.conf useradd carlos --create-home # restart ssh service ssh restart
Let’s first copy our public key to the server, in my case, I have it on my SSH agent, so it should look like this:
ssh-add -L > /tmp/ca/carlos.pub chmod 0644 /tmp/ca/carlos.pub # Or, you could copy the public key directly, like so: cp ~/.ssh/id_ed25519.pub /tmp/ca/carlos.pub
Now, we can create a certificate using our public key, and the CA’s private key:
# within the container: ssh-keygen \ -u \ -s /tmp/ca/ca \ -n carlos \ -I id-1 \ -V +1w \ /tmp/ca/carlos.pub
We then copy
/tmp/carlos-cert.pub to our host machine, and use both the
certificate, and the user’s private key to SSH:
ssh \ -i /tmp/ca/carlos-cert.pub \ -i ~/.ssh/id_ed25519 \ # this line might not be needed -F /dev/null \ -o UserKnownHostsFile=/dev/null \ -p 2222 \ carlos@localhost id
You might need to pass the path to your private key as well, in my case, it’s getting it from the SSH Agent.
Issuing a Host Certificate
We can avoid that TOFU warning by issuing a host certificate, instructing the server to advertise it and our client to trust anything from it automatically.
First, let’s create a host certificate:
# within the container: ssh-keygen -s /tmp/ca/ca \ -I "id-1" \ -h \ -z 1 \ /etc/ssh/ssh_host_ed25519_key.pub
-h instead of the
-o. You can also set a validity, principals
(e.g. host name) and more.
Then, we set up the server to advertise it and restart the SSH server, like so:
# within the container: echo "HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub" >> /etc/ssh/sshd_config.d/ca.conf service ssh restart
Finally, on our client, we can create a new known hosts file with the certificate public key:
echo "@cert-authority * CONTENTS_OF_/tmp/ca/ca.pub" > known_hosts
And finally, SSH passing it as a parameter:
ssh \ -i /tmp/ca/carlos-cert.pub \ -F /dev/null \ -o UserKnownHostsFile=./known_hosts \ -p2222 \ carlos@localhost
And, sure enough, it works:
Anatomy of a Certificate
Just wanted to show you what said certificates look like:
# ssh-keygen -L -f /tmp/ca/carlos-cert.pub /tmp/ca/carlos-cert.pub: Type: firstname.lastname@example.org user certificate Public key: ECDSA-CERT SHA256:1mEjon99blahciC4T1Mqj6I06FeFWtl/NGwXXBzwSfk Signing CA: ED25519 SHA256:dmfXhQCffjhtvyIwiFr1Elx/L5EO7/EvbpgCknL1Xg0 (using ssh-ed25519) Key ID: "id-1" Serial: 1 Valid: from 2022-03-04T01:00:00 to 2022-03-11T01:01:40 Principals: carlos Critical Options: force-command /usr/bin/id Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc
Things to notice:
- The key type mentions it is an “user certificate”, might also be a “host certificate”
- It has the fingerprint of my key
- It has the fingerprint of the signing key, in our case, the CA key
- It has an issuer-given ID
- It has a validity timeframe (valid from … to …), might also be forever
- It might have a principals list
- It might have critical options
- This example key was generated using the
-O force-command="/usr/bin/id"option, thus it has a critical option. In the code example above that is not the case.
- If you use
force-commandin your certificates, it is also a good idea to disable
user-rcis permitted, the users might get around the
force-commandrestrictions. You can disable it by passing
- This example key was generated using the
- It might have extensions — in this case, those are the default ones
That’s it. No sorcery required 🙂
- Admins issue a key pair and add its public key as
TrustedUserCAKeyson the servers
- Admins might issue a host certificate and set it up with the
- Admins or automated systems issue certificates for the users, using the CA’s private key, and the user’s public key
- Users SSH into machines using both their private key, and the issued certificate
- Users might set up a
known_hostsfile to avoid TOFU warnings and man in the middle attacks
- User certificates are usually issued to be short-lived and discourage key sharing