Friday, April 16, 2010

Byte Order

We have some software that interacts with third party software. Recently we've been working on using Private/Public key encryption scheme to allow them to send us a small amount of data (encrypting with a public key) and us decrypting with a private key only we know. Pretty standard stuff. We're using RSA, 2048-bit keys. RSA has been around for, well, ever in computer terms. It's a pretty big standard.

We use OpenSSL to do all of our decryption (using rsautl, the -decrypt command and a secret private key). To save on data being passed around, we only push out the modulus to them (these are devices that are operating on bandwith == expensive networks) and use the standard 65537 exponent to make life easy for both of us.

So far so good, right?

We started doing some testing with them. We couldn't decrypt anything they sent us. We gave them a test private key and modulus that we could both use for testing. They couldn't decrypt what they were encrypting either. Wait, what? They created their own private key and modulus, and used it. Everything worked fine for them. They sent it to us. I try to encrypt with the modulus and decrypt with the given private key. No go. Now I'm really confused.

So I look at the private key using RSA and extract the modulus from it. I'm comparing the two and they don't match. Wait, they don't match? Then I look a little closer... they are reversed (well, almost... since they are hex strings I'm looking at, the sets of two are reversed).

It appears that on Windows, CryptoAPI works in Least Significant Byte Order, while OpenSSL works in Most Significant Byte Order (as you may be able to guess from the title). But, Reversing the key order alone doesn't work. I still can't decrypt their data. Then we discover that not only do we have to reverse the modulus, we also have to reverse the output encrypted data.

At least it all works now, but byte ordering sucks.