Exetools  

Go Back   Exetools > General > General Discussion

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 03-09-2024, 01:20
eychei eychei is offline
Friend
 
Join Date: Mar 2018
Posts: 57
Rept. Given: 0
Rept. Rcvd 0 Times in 0 Posts
Thanks Given: 34
Thanks Rcvd at 10 Times in 10 Posts
eychei Reputation: 0
Question DES-X Encryption slow / Help needed

Hi everyone,

I did write a small vb.net program to encrypt and decrypt DESX.
Its actually nearly the same as DES but with an additional input and output XOR with two extra keys.

wo8Bl.png

So when I decrypt files it easy really fast taking around 10sec for 100mb.
This is because I can decrypt the complete file first without needing to go from one block to the next.

But when I encrypt files I need to chain 8 byte sized chunks in serial.


My vb.net code looks like this:

Code:
 'DES-X Start
            Dim xor_in_array As Byte() = XORarrayValues(input_whitener_array, ByteToEncrypt)
            Dim ciphertext As Byte() = New Byte(ByteToEncrypt.Length - 1) {}
            Dim ms As New MemoryStream
            Dim cs As New CryptoStream(ms, des.CreateEncryptor(), CryptoStreamMode.Write)
            Dim cut_bytes() As Byte = New Byte(8 - 1) {}
            For i = 0 To (xor_in_array.Length / 8) - 1

                Array.Copy(xor_in_array, CInt(i) * 8, cut_bytes, 0, 8)
                If i = 0 Then
                    Dim xor_iv = XORarrayValues(cut_bytes, Var.iv)
                    ms.Position = 0
                    cs.Write(xor_iv, 0, xor_iv.Length)
                    Dim des_xor_iv = ms.ToArray()
                    Dim xor_out = XORarrayValues(Var.output_whitener, des_xor_iv)
                    Array.Copy(xor_out, 0, ciphertext, CInt(i) * 8, 8)

                Else
                    Dim ciphertext_used() As Byte = New Byte(8 - 1) {}
                    Array.Copy(ciphertext, CInt(i - 1) * 8, ciphertext_used, 0, 8)
                    Dim xor_ciphertext_used = XORarrayValues(cut_bytes, ciphertext_used)
                    ms.Position = 0
                    cs.Write(xor_ciphertext_used, 0, xor_ciphertext_used.Length)
                    Dim des_xor_out = ms.ToArray()
                    Dim xor_out = XORarrayValues(Var.output_whitener, des_xor_out)
                    For j = 0 To xor_out.Length - 1
                        ciphertext(i * 8 + j) = xor_out(j)

                    Next
                    'Array.Copy(xor_out, 0, ciphertext, CInt(i) * 8, 8)
                    'ciphertext = ciphertext.Concat(xor_out).ToArray()



                End If
So I tried different approaches in appending the ciphertext at the end to a bte array so i can write it to file afterwards.
It does not change in speed if I use array.copy / concat or for loop copy.
I also tried with creating 4-8 parallel threads. While this reduces the time needed for encryption, it is still slow for very big files.


How can I increase speed for the encryption algorithm? Hope someone has a solution.

Thx.

-e
Reply With Quote
  #2  
Old 03-09-2024, 06:45
chants chants is offline
VIP
 
Join Date: Jul 2016
Posts: 739
Rept. Given: 37
Rept. Rcvd 48 Times in 30 Posts
Thanks Given: 671
Thanks Rcvd at 1,064 Times in 482 Posts
chants Reputation: 48
XORarrayValues being immutable is certainly hurting perf. You should consider making it mutable where it stores the result in one of the inouy buffers or reuse an output buffer.

Memory allocation is expensive and you overdo it. Also the ToArray calls should be avoided if there is any other way to reference the bytes needed as a view or what have you.
Reply With Quote
The Following 2 Users Say Thank You to chants For This Useful Post:
eychei (03-11-2024), Kurapica (03-10-2024)
  #3  
Old 03-09-2024, 09:15
Gregory Morse Gregory Morse is offline
Friend
 
Join Date: Sep 2023
Location: Cleveland, Ohio
Posts: 65
Rept. Given: 3
Rept. Rcvd 2 Times in 2 Posts
Thanks Given: 33
Thanks Rcvd at 18 Times in 13 Posts
Gregory Morse Reputation: 2
Quote:
Originally Posted by eychei View Post
Hi everyone,

I did write a small vb.net program to encrypt and decrypt DESX.
Its actually nearly the same as DES but with an additional input and output XOR with two extra keys.

Attachment 10479

So when I decrypt files it easy really fast taking around 10sec for 100mb.
This is because I can decrypt the complete file first without needing to go from one block to the next.

But when I encrypt files I need to chain 8 byte sized chunks in serial.



So I tried different approaches in appending the ciphertext at the end to a bte array so i can write it to file afterwards.
It does not change in speed if I use array.copy / concat or for loop copy.
I also tried with creating 4-8 parallel threads. While this reduces the time needed for encryption, it is still slow for very big files.


How can I increase speed for the encryption algorithm? Hope someone has a solution.

Thx.

-e
You can write native code instead of the .Net code. Will be much faster. At least, write the time consuming parts of the code in native. You can use the pre-written libraries available on internet. No need to create your own. Much faster.
Reply With Quote
The Following User Says Thank You to Gregory Morse For This Useful Post:
dyers eve (03-10-2024)
  #4  
Old 03-09-2024, 17:46
chants chants is offline
VIP
 
Join Date: Jul 2016
Posts: 739
Rept. Given: 37
Rept. Rcvd 48 Times in 30 Posts
Thanks Given: 671
Thanks Rcvd at 1,064 Times in 482 Posts
chants Reputation: 48
Quote:
Originally Posted by Gregory Morse View Post
You can write native code instead of the .Net code. Will be much faster. At least, write the time consuming parts of the code in native. You can use the pre-written libraries available on internet. No need to create your own. Much faster.
In the future if you are going to troll with annoying advice, consider not posting. He clearly stated that decryption functioned in a reasonable time. To throw out generic canned nonsense advice like using native code was clearly not what was asked. Although you've been doing this for a while now so it's not surprising.
Reply With Quote
  #5  
Old 03-10-2024, 07:14
dyers eve dyers eve is offline
Friend
 
Join Date: Nov 2023
Posts: 10
Rept. Given: 1
Rept. Rcvd 0 Times in 0 Posts
Thanks Given: 17
Thanks Rcvd at 12 Times in 5 Posts
dyers eve Reputation: 0
Quote:
Originally Posted by chants View Post
In the future if you are going to troll with annoying advice, consider not posting. He clearly stated that decryption functioned in a reasonable time. To throw out generic canned nonsense advice like using native code was clearly not what was asked. Although you've been doing this for a while now so it's not surprising.
You are not the only one who knows everything, @chants.
Whether they are right or wrong, is not for you to decide.
While you may have a better answer, if someone else also wants to post their opinions, let them do so!

@eychie It is better to avoid managed frameworks like .Net when you want to optimise for performance. It is also preferable to code the most compute-intensive parts of the encryption in C++ or better yet, in a memory-safe language like Rust.
Is there a good reason why you chose VB.Net instead of C?
I cannot comment further since I cannot access your attachment due to my low rank. I can help with the code if you provide more information, if you need.
Reply With Quote
The Following User Says Thank You to dyers eve For This Useful Post:
eychei (03-11-2024)
  #6  
Old 03-10-2024, 18:35
chants chants is offline
VIP
 
Join Date: Jul 2016
Posts: 739
Rept. Given: 37
Rept. Rcvd 48 Times in 30 Posts
Thanks Given: 671
Thanks Rcvd at 1,064 Times in 482 Posts
chants Reputation: 48
The picture is just the generic CBC encryption mode. So I'm unclear how you can parallelize it as CBC has dependencies requiring it processed in a line order. But if you are happy with the performance in one direction, then VB.NET can deliver adequate enough performance in the opposite direction if you performance profile and refactoring to make good reuse of memory and avoid unnecessary copies.

Don't worry about the guy with ~15 accounts telling you to abandon ship. Though such advice is fine in general, most of us can read and developed abstract thinking during childhood so we are capable of inferring contextual clues. Unlike this smooth brained individual, yielding too many accounts.
Reply With Quote
  #7  
Old 03-11-2024, 01:49
chants chants is offline
VIP
 
Join Date: Jul 2016
Posts: 739
Rept. Given: 37
Rept. Rcvd 48 Times in 30 Posts
Thanks Given: 671
Thanks Rcvd at 1,064 Times in 482 Posts
chants Reputation: 48
By the way, if you want to parallelize this, you should consider a different cipher mode of operation, preferably Galois Counter Mode (GCM) as the ECB mode has various attack vectors and security risks, which is probably why you are using CBC, but it doesn't scale well for very large data sizes.
Reply With Quote
  #8  
Old 03-11-2024, 22:49
eychei eychei is offline
Friend
 
Join Date: Mar 2018
Posts: 57
Rept. Given: 0
Rept. Rcvd 0 Times in 0 Posts
Thanks Given: 34
Thanks Rcvd at 10 Times in 10 Posts
eychei Reputation: 0
Thanks chants.

Quote:
Originally Posted by chants View Post
By the way, if you want to parallelize this, you should consider a different cipher mode of operation, preferably Galois Counter Mode (GCM) as the ECB mode has various attack vectors and security risks, which is probably why you are using CBC, but it doesn't scale well for very large data sizes.
I am using CBC because the encryption scheme is not done by me it is the software I am trying to recreate.
Reply With Quote
The Following User Says Thank You to eychei For This Useful Post:
chants (03-12-2024)
  #9  
Old 03-11-2024, 22:53
eychei eychei is offline
Friend
 
Join Date: Mar 2018
Posts: 57
Rept. Given: 0
Rept. Rcvd 0 Times in 0 Posts
Thanks Given: 34
Thanks Rcvd at 10 Times in 10 Posts
eychei Reputation: 0
I am using vb.net because I do not know much C.
I also wrote the program first in python but it is also very slow.
Would this be much faster in C?

So to sum it up, there is not much to optimize in the vb.net code speed vise?

How do I make the array mutable?

I will also try to remove the ToArray() copies. Maybe that helps in speed.


-e

Last edited by eychei; 03-11-2024 at 22:59.
Reply With Quote
  #10  
Old 03-12-2024, 07:19
chants chants is offline
VIP
 
Join Date: Jul 2016
Posts: 739
Rept. Given: 37
Rept. Rcvd 48 Times in 30 Posts
Thanks Given: 671
Thanks Rcvd at 1,064 Times in 482 Posts
chants Reputation: 48
.NET JIT is quite good, doing things unsafe is closer to C but there will only be some relatively small increase if you make the VB.NET code highly efficient. XORarrayValues should write directly into one if the two input arguments (and ^= mutable variant used, not x=x^y) or pass a third output buffer and reuse it. Also consider running the VS profiler to see line by line where most time is spent. If you still aren't happy after these basic steps, you can consider whether to spend the time rewriting in C.

Are you sure this isn't just CBC DES and directly decryptable extremely fast with Windows builtin libraries via .NETs System.Security.Cryptography? Ah nevermind, seems there is no way to inject the XORs between blocks. ECB mode and doing manually seems fine.

Last edited by chants; 03-12-2024 at 07:30.
Reply With Quote
Reply

Tags
des, desx

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



All times are GMT +8. The time now is 17:16.


Always Your Best Friend: Aaron, JMI, ahmadmansoor, ZeNiX, chessgod101
( 1998 - 2024 )