SquareD: you should practice doing such basic math in your head As you described the algorithm in your first post, all bytes are handled individually. My advice would be putting the calculator aside and doing the math on a piece of paper. Single bytes are easy to convert to binary notation, just write them down, then manipulate the bits according to the steps the algorithm takes, and see how it turns out. Calculator apps frequently run into overflow/underflow issues, it's just a natural side effect of how they work.
It's just like with every tool - you have to know how to handle it properly to make full use of them.
And by the way, it's very convenient if you are trained enough to read hex and binary just or almost like decimal values