Use correct type sizes to avoid overwriting unrelated data

sig_size is declared as uint32_t, but later typecast to size_t, which is significantly wider on 64-bit architectures.

To avoid potential issues in other places, promote sizes and lengths to size_t/ssize_t where this makes sense.

Check the hash algorithm before (possibly failing to) malloc.

malloc only accepts unsigned sizes, so failing to catch a negative size as an errorcode will result in a malloc call with an enormous value, leading to an allocation failure and a nonsensical error message.