Trying to identify crypto algorithm

[SiNTAX]

I'm working on a target with crypto and am trying to identify the algorithm.. the target uses RC4 to encrypt communication. At the start it takes the random RC4 encryption key does some XORs with SHA256 hashes on it and then encrypts the entire shebang.

Most logical choice would be for this key exchange packet to be encrypted with public key encryption, although tracing through the code I have my doubts it's RSA.

For one.. it only takes a single 512 bit input key... pubkey encryption would need a private and public key.. so unless I missed some code somewhere, it seems doubtful it actually is public key encryption.

One of the first steps it does is take the unencrypted data block and shifts it into another memory block byte by byte, but the destination memory block only uses 28 bits per 32 bits (0x0FFFFFFF), so for example with input bytes:

01 02 03 04 05

the resulting destination block is (word display):

00030405 00000102

Lateron there's a step where 2 of these block get matrix multiplied.


I've looked at a couple of encryption algorithms but haven't really found one that seems to operate like this, then again.. I'm no crypto jock

Any clues?

[Git]

Have you run Kanal on it, or the crypto analyser plugin for IDA ?

Git

[SiNTAX]

I would, had I known about them A bit rusty on the available rce tools.. just checked with Kanal and while it doesn't detect RSA, it does show the presence of prime numbers and MPI mp_mul.

This is helpful.. thx

[Loki]

Kanal wont detect RSA as its just a mathematical operation using big numbers and doesn't have any recognisable sigs as such (beyond known libraries).

Best idea is probably to determine any mathematical operations, figure the algo used and search on that.

Sounds like a combination of things but if you find the presence of big numbers then check out RSA, elgamal, DSA, ECC and ECDSA.

[SiNTAX]

Found out what the target is using, apparently a version of libtomcrypt. That shift operation I described, that's apparently mp_read_unsigned_bin() called from rsa_exptmod().

Anyway it's always fun finding out what something is and then checking all the labels you slapped onto the reversed code in IDA. Looks like I didn't fail too horribly in identifying stuff