Marshaling SSH Private Keys - Why there's always a different block?

Not long ago, when I was building melt, I learned something interesting: if you restore a private key from its seed, and marshal it back to the OpenSSH Private Key format, you’ll always get a different block in the middle.

Why?

That lead to an investigation of how the private key format works. I didn’t find many good references out there, except OpenSSH’s source code.

Let’s start from there, shall we?

We can see in the function sshkey_private_to_blob2 in sshkey.c, there’s this interesting piece of code:

/* Random check bytes */
check = arc4random();
if ((r = sshbuf_put_u32(encrypted, check)) != 0 ||
    (r = sshbuf_put_u32(encrypted, check)) != 0)
	goto out;

We see there that it creates what seems to be a random uint32, and then calls sshbuf_put_u32 two times, adding it to encrypted and expecting it all to succeed.

Interesting… why?

The best clue after that lies in the PROTOCOL.key file:

	uint32	checkint
	uint32	checkint
	byte[]	privatekey1
	string	comment1

checkint… same name… the answer must be here, right? Going further:

Before the key is encrypted, a random integer is assigned to both checkint fields so successful decryption can be quickly checked by verifying that both checkint fields hold the same value.

Aha! So that’s why it’s always different! The checkints are used to check that decryption succeeded. When decrypting the key, we don’t know their value, just that they should be equal.

Cool!

What about Go?

When we started this story, I mentioned I was working on melt, which is written in Go. So far, we’ve looked in to C code, but what about Go?

Turns out there’s an unresolved merge request adding SSH key marshaling into Go’s crypto/ssh package.

We can find the same checks there (called Check1 and Check2) but the code might be a bit easier to read:

// Random check bytes.
var check uint32
if err := binary.Read(rand.Reader, binary.BigEndian, &check); err != nil {
	return nil, err
}
pk1.Check1 = check
pk1.Check2 = check

P.S. If you want to use this in your Go program, I’m keeping a repository with these changes.

Playground

We learned that the Private Key file, in the OpenSSH format, will always be a bit different, even if it’s generated with the same parameters. But… is it still the same key? What happens now?

We can verify that using melt!

Let’s create a new key to play with. You can do so with:

ssh-keygen -t ed25519 -f post

And then run melt to get a mnemonic:

melt ./post >seed

For what its worth, here is the mnemonic for the key I created:

obey axis lecture satoshi deal comic first unfold bomb control attitude lawsuit
this brown often fault myself rabbit assume miss modify riot around punch

Now, let’s restore it a couple of times:

melt restore ./post1 <seed
melt restore ./post2 <seed

Now, let’s check a couple of things, starting with the check sum of the private keys:

sha256sum post post1 post2

a9a08d6ae71412e0397e1c76d9300002d0cb69e484823dd684d217ee07f32081  post
ec7f45126a4bf96a913b66079c1e1773ca809c6b4a653885a1c49e08d2b4d978  post1
cab1849c9560b6705a335192bcb3991ae2ba8ac479d51659e2325aeeb3ab2476  post2

All different… which is expected, due to the checkint we just learned about.

Let’s now check the public keys:

sha256sum post*.pub

562de9510ca7278f3284f9f0114e8dc757b557c92f0b1744514c42eb9c1b0d81  post.pub
8dd282d6ad5a0fa6da2a2054b70ac96c257a58ef4c23715f02b9885329094a27  post1.pub
8dd282d6ad5a0fa6da2a2054b70ac96c257a58ef4c23715f02b9885329094a27  post2.pub

Except for the first one, they are all equal. So what’s the difference between them?

diff -u post.pub post1.pub

--- post.pub    2022-12-07 13:28:36.009649296 -0300
+++ post1.pub   2022-12-07 13:31:34.426816707 -0300
@@ -1 +1 @@
-ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPA8QtyAE1DLpUIY3otmLILZv9XdRlXv37hHEWTGib7p carlos@darkstar
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPA8QtyAE1DLpUIY3otmLILZv9XdRlXv37hHEWTGib7p

Ah, our original key had a memo, and the restored ones don’t. Not a big deal!

What about the private keys’ fingerprints?

ssh-keygen -l -f post > post.finger
ssh-keygen -l -f post1 > post1.finger
ssh-keygen -l -f post2 > post2.finger
cat post*.finger

256 SHA256:KfCFwx1/vfcV/XQqdneOMOdgpbVu4Nxz32buks4MLpI carlos@darkstar (ED25519)
256 SHA256:KfCFwx1/vfcV/XQqdneOMOdgpbVu4Nxz32buks4MLpI post1.pub (ED25519)
256 SHA256:KfCFwx1/vfcV/XQqdneOMOdgpbVu4Nxz32buks4MLpI post2.pub (ED25519)

Again, the same key. The only difference is the memo.

Now let’s check how different the private keys really are:

diff -u post1 post2

--- post1       2022-12-07 13:31:34.426816707 -0300
+++ post2       2022-12-07 13:31:37.870839247 -0300
@@ -1,7 +1,7 @@
 -----BEGIN OPENSSH PRIVATE KEY-----
 b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtz
 c2gtZWQyNTUxOQAAACDwPELcgBNQy6VCGN6LZiyC2b/V3UZV79+4RxFkxom+6QAA
-AIjRy3cc0ct3HAAAAAtzc2gtZWQyNTUxOQAAACDwPELcgBNQy6VCGN6LZiyC2b/V
+AIhYJw2fWCcNnwAAAAtzc2gtZWQyNTUxOQAAACDwPELcgBNQy6VCGN6LZiyC2b/V
 3UZV79+4RxFkxom+6QAAAECX4h38X7OCXJXfaRkl7Dq/Hgw6JmqfklYEN8bo63RD
 DfA8QtyAE1DLpUIY3otmLILZv9XdRlXv37hHEWTGib7pAAAAAAECAwQF
 -----END OPENSSH PRIVATE KEY-----

Just one line–about 12 chars–are different: jRy3cc0ct3HA vs hYJw2fWCcNnw.

All that said, for all intents and purposes it’s the same key, which shouldn’t be a surprise, but it’s good to know anyways!

Hope you enjoyed this trip into how OpenSSH private keys are marshaled, and I’ll see you in the next one!

Cross-posted to the Charm Blog.