Brute Force Decryption
Brute forcing the decryption key to avoid hardcoding into the binary.
Jump to Section (7)
Learning Objectives
0/1 doneIntroduction
In the beginner modules, payload encryption and decryption were demonstrated and a warning was mentioned about saving the encryption key within the binary. Recall that if the encryption key is saved in plaintext within the binary it can be trivially retrieved. One solution is to encrypt the key with another key and decrypt it at runtime. To avoid hardcoding the key inside the binary, the key is brute-forced.
This module will demonstrate an XOR decryption algorithm where the program has to guess the key through brute forcing.
Key Encryption Process
To perform a key brute force, the encryption and decryption functions require a hint byte. Knowing one byte's value before and after the encryption process makes the decryption process possible. In this case, the first byte has been selected as the hint byte.
For example, if the hint byte is BA and when encrypted it becomes 71, then the decryption process will brute force that value until it is reverted to BA, indicating the correct key was used.

Key Encryption Function
The GenerateProtectedKey function takes a hint byte and
prepends it as the first byte of the plaintext key. It then uses an XOR
encryption algorithm to encrypt the key using a randomly generated key
at runtime.
Note that the PrintHex is a function that prints the input buffer as a hex array, and it is being used to print the plaintext generated key.
/*
- HintByte: is the hint byte that will be saved as the key's first byte
- sKey: the size of the key to generate
- ppProtectedKey: pointer to a PBYTE buffer that will recieve the encrypted key
*/
VOID GenerateProtectedKey(IN BYTE HintByte, IN SIZE_T sKey, OUT PBYTE* ppProtectedKey) {
// Genereting a seed
srand(time(NULL));
// 'b' is used as the key of the key encryption algorithm
BYTE b = rand() % 0xFF;
// 'pKey' is where the original key will be generated to
PBYTE pKey = (PBYTE)malloc(sKey);
// 'pProtectedKey' is the encrypted version of 'pKey' using 'b'
PBYTE pProtectedKey = (PBYTE)malloc(sKey);
if (!pKey || !pProtectedKey)
return;
// Genereting another seed
srand(time(NULL) * 2);
// The key starts with the hint byte
pKey[0] = HintByte;
// generating the rest of the key
for (int i = 1; i < sKey; i++){
pKey[i] = (BYTE)rand() % 0xFF;
}
printf("[+] Generated Key Byte : 0x%0.2X \n\n", b);
printf("[+] Original Key : ");
PrintHex(pKey, sKey);
// Encrypting the key using a xor encryption algorithm
// Using 'b' as the key
for (int i = 0; i < sKey; i++){
pProtectedKey[i] = (BYTE)((pKey[i] + i) ^ b);
}
// Saving the encrypted key by pointer
*ppProtectedKey = pProtectedKey;
// Freeing the raw key buffer
free(pKey);
}
Key Decryption Process
Since the encryption key used to encrypt the key was not stored
anywhere, the decryption function must be able to guess the value of b shown in the GenerateProtectedKey
function. To do so, the decryption function will XOR the first byte of
the key, which is the hint byte, with different keys until the resulting
byte is the original key's hint byte. When that happens, the function
will know that the correct b value was selected. The code snippet below shows this logic.
if (((EncryptedKey[0] ^ b) - 0) == HintByte)
// Then b's value is the xor encryption key
else
// Then b's value is not the xor encryption key, try with a different b value
Continuing from the previous example, when 71 becomes BA then the correct b value has been guessed.
Key Decryption Function
The BruteForceDecryption function needs the same hint byte that was passed to the encryption function.
/*
- HintByte : is the same hint byte that was used in the key generating function
- pProtectedKey : the encrypted key
- sKey : the key size
- ppRealKey : pointer to a PBYTE buffer that will recieve the decrypted key
*/
BYTE BruteForceDecryption(IN BYTE HintByte, IN PBYTE pProtectedKey, IN SIZE_T sKey, OUT PBYTE* ppRealKey) {
BYTE b = 0;
PBYTE pRealKey = (PBYTE)malloc(sKey);
if (!pRealKey)
return NULL;
while (1){
// Using the hint byte, if this is equal, then we found the 'b' value needed to decrypt the key
if (((pProtectedKey[0] ^ b) - 0) == HintByte)
break;
// else, increment 'b' and try again
else
b++;
}
// The reverse algorithm of the xor encryption, since 'b' now is known
for (int i = 0; i < sKey; i++){
pRealKey[i] = (BYTE)((pProtectedKey[i] ^ b) - i);
}
// Saving the decrypted key by pointer
*ppRealKey = pRealKey;
return b;
}
Demo
The image below shows the generation of the XOR-encrypted key. The arrows point to the code that generates the respective console output.

The image below shows the successful brute force and decryption.

Conclusion
Although this brute-forcing approach is simple, it can be used to prevent malware analysts and researchers from dumping the key from the binary file. This forces them to debug the binary to understand how the key is generated which is where the anti-analysis techniques come in handy.
Companion Project & Source Code
4 readable source/config files extracted directly from the archive
// @NUL0x4C | @mrd0x : MalDevAcademy
#include <Windows.h>
#include <stdio.h>
#include <time.h>
#define KEYSIZE 16
// used to print buffer as a array of hex
VOID PrintHex(IN PBYTE pBuf, IN SIZE_T sSize) {
for (int i = 0; i < sSize; i++){
printf("0x%0.2X ", pBuf[i]);
}
printf("\n\n");
}
/*
- HintByte : is the same hint byte that was used in the key generating function
- pProtectedKey : the encrypted key
- sKey : the key size
- ppRealKey : pointer to a PBYTE buffer that will recieve the decrypted key
*/
BYTE BruteForceDecryption(IN BYTE HintByte, IN PBYTE pProtectedKey, IN SIZE_T sKey, OUT PBYTE* ppRealKey) {
BYTE b = 0;
PBYTE pRealKey = (PBYTE)malloc(sKey);
if (!pRealKey)
return NULL;
while (1){
// using the hint byte, if this is equal, then we found the 'b' value needed to decrypt the key
if (((pProtectedKey[0] ^ b) - 0) == HintByte)
break;
// else, increment 'b' and try again
else
b++;
}
printf("[+] FOUND\n[+] Calculated Key Byte : 0x%0.2X \n", b);
for (int i = 0; i < sKey; i++){
pRealKey[i] = (BYTE)((pProtectedKey[i] ^ b) - i);
}
*ppRealKey = pRealKey;
return b;
}
/*
- HintByte: is the hint byte that will be saved as the key's first byte
- sKey: the size of the key to generate
- ppProtectedKey: pointer to a PBYTE buffer that will recieve the encrypted key
*/
VOID GenerateProtectedKey(IN BYTE HintByte, IN SIZE_T sKey, OUT PBYTE* ppProtectedKey) {
// genereting a seed
srand(time(NULL));
// 'b' is used as the key of the key encryption algorithm
BYTE b = rand() % 0xFF;
// 'pKey' is where the original key will be generated to
PBYTE pKey = (PBYTE)malloc(sKey);
// 'pProtectedKey' is the encrypted version of 'pKey' using 'b'
PBYTE pProtectedKey = (PBYTE)malloc(sKey);
if (!pKey || !pProtectedKey)
return;
// genereting another seed
srand(time(NULL) * 2);
// the key starts with the hint byte
pKey[0] = HintByte;
// generating the rest of the key
for (int i = 1; i < sKey; i++){
pKey[i] = (BYTE)rand() % 0xFF;
}
printf("[+] Generated Key Byte : 0x%0.2X \n\n", b);
printf("[+] Original Key : ");
PrintHex(pKey, sKey);
// encrypting the key using a xor encryption algorithm
// using 'b' as the key
for (int i = 0; i < sKey; i++){
pProtectedKey[i] = (BYTE)((pKey[i] + i) ^ b);
}
// saving the encrypted key by pointer
*ppProtectedKey = pProtectedKey;
// freeing the raw key buffer
free(pKey);
}
int main() {
PBYTE pProtectedKey = NULL;
PBYTE pRealKey = NULL;
// 0xBA is the hint byte
GenerateProtectedKey(0xBA, KEYSIZE, &pProtectedKey);
// printing the encrypted key
printf("[+] Protected Key : ");
PrintHex(pProtectedKey, KEYSIZE);
printf("\n\n\t\t\t-------------------------------------------------\n\n");
// attempting to brute force it, using the same hint byte
printf("[i] Brute Forcing The Seed ... ");
if (!BruteForceDecryption(0xBA, pProtectedKey, KEYSIZE, &pRealKey)) {
printf("[!] FAILED \n");
return -1;
}
// printing the decrypted key - should be the same as the one printed by the 'GenerateProtectedKey' function
printf("[+] Original Key : ");
PrintHex(pRealKey, KEYSIZE);
// freeing the remaining allocated buffer
free(pProtectedKey);
free(pRealKey);
printf("[#] Press <Enter> To Quit ... ");
getchar();
return 0;
}
Full Archive Manifest (5 files inside .zip)
- BruteForce/BruteForce.sln
- BruteForce/BruteForce/BruteForce.c
- BruteForce/BruteForce/BruteForce.vcxproj
- BruteForce/BruteForce/BruteForce.vcxproj.filters
- BruteForce/BruteForce/BruteForce.vcxproj.user