Jazz Jackrabbit 2 Online

zlib is a popular free open-source compression system. Jazz Jackrabbit 2 uses it to compress all the data files, to reduce the space used. This article gives a quick overview of how to use zlib, primarily with the Jazz Jackrabbit 2 files.

zlib basics

Details will be given for C and for Visual Basic 6, though zlib can be used with many other languages. To use zlib with C, you will need zlib.lib and zlib.h, while for Visual Basic you merely need zlib.dll. These files are available from the zlib website, which is given at the end of this article.

Only the compress2 and uncompress functions are required. The crc32 is used for verifying the file, but is not needed.

Declaring zlib functions in C

#include <zlib.h>

Declaring zlib functions in Visual Basic

Don’t forget to add “Private“s in front of them if you’re going to use them in your forms.

Declare Function compress2 Lib "zlib.dll" (ByRef dest As Any, ByRef destLen As Any, ByRef src As Any, ByVal srcLen As Long, ByVal Level As Integer) As Long

Declare Function uncompress Lib "zlib.dll" (ByRef dest As Any, ByRef destLen As Any, ByRef src As Any, ByVal srcLen As Long) As Long

Declare Function crc32 Lib "zlib.dll" (ByVal crc As Long, ByRef buffer As Any, ByVal bufferLen As Long) As Long</pre>

How compression is used in JJ2 files

As a bare minimum, all zlib-compressed files in JJ2 files should at least provide the compressed and uncompressed sizes in its header. For example, consider an example zlib section of a file which goes:

44 00 00 00 4F 00 00 00-78 DA E3 60 80 00 17 73 69 30 ED F0 53 98 41 E0-97 30 03 23 90 DD 92 53 C6 78 E7 87 30 C3 7B 31-47 06 35 89 03 FC 20 F9 04 0A DC DB 3C 07 B0 5A-01 55 39 06 95 04 25 86 2B 40 1A 0C 9E 30 00 00-2B 02 0E A6

In this example, the first four bytes (which read 68 in hex) represents the size of the compressed stream (the input data). The next four bytes (which read 79 in hex) represents the size of the uncompressed stream (the output data). Armed with that data, we now know that we will be decompressing a 68-byte buffer into a 79-byte one. Note that zlib streams starts with a “78 DA” when compressed at the maximum level, and this can be very useful for skimming through large files to find out where a zlib stream begins or how many zlib streams there are.

To decompress a zlib file, the uncompress function is used.

Using uncompress in C

For this short snippet we will assume that you have the above “example” file stored as “tutorial.dat”. This assumes you are writing a console program and have already #included anything necessary.

char *in;
char *out;
int CompressedSize;
int UncompressedSize;

int main() {
    FILE *fi = fopen(“tutorial.dat”, “rb”); //Opens tutorial.dat for binary reading
    fread(&CompressedSize, 4, 1, fi);       //Reads the first four bytes and stores it
    fread(&UncompressedSize, 4, 1, fi);     //Reads the next four bytes and stores it

    in = malloc(CompressedSize); //Allocates a size of CompressedSize for input stream
    fread(in, 1, CompressedSize, fi); //Reads CompressedSize bytes from the file
    fclose(fi); //We finished reading the file, no need to use it anymore

    out = malloc(UncompressedSize); //Allocates a size of UncompressedSize for output
    uncompress(out, &UncompressedSize, in, CompressedSize); //Decompresses, YAY :)

    //Here “out” contains the uncompressed data, so do whatever you want with it.
    //For simplicity, the above example is actually a string, so print it to screen:
    printf(out);
}

Once you can understand the concepts of malloc and uncompress, you can decompress just about every file format in JJ2. Don’t forget to free what you malloc though. Note that we pass a pointer to the second argument of uncompress, which will return with the uncompressed size of the buffer. Assuming you already had the right size in the first place, this value will not change. Also, the return value of uncompress determines whether the operation was a success or not. However, to keep things simple, we assume that our zlib stream is valid, and will not error-handle it.

Using uncompress in VB

Here we use the same assumptions: a file tutorial.dat containing the binary data.

Dim inBuffer() As Byte
Dim outBuffer() As Byte
Dim CompressedSize As Long
Dim UncompressedSize As Long

Sub Main()
    Open “tutorial.dat” For Binary Access Read As #1 ‘Opens tutorial.dat for reading
    Get #1, , CompressedSize   ‘Reads the first four bytes, stores in CompressedSize
    Get #1, , UncompressedSize ‘Reads the next four bytes, stores in UncompressedSize

    ReDim inBuffer(CompressedSize – 1) ‘Create buffer of CompressedSize bytes
    Get #1, , inBuffer ‘Reads bytes into inBuffer
    Close #1 ‘Finish reading from file, no need to use it anymore

    ReDim outBuffer(UncompressedSize – 1) ‘Create buffer of UncompressedSize bytes
    Uncompress outBuffer(0), UncompressedSize, inBuffer(0), CompressedSize ‘YAY :)

    ‘Here “outBuffer” contains the uncompressed data, so do whatever you want with it.
End Sub

One thing to note is that ReDim uses “x-1” bytes when you want to allocate x bytes, because the array goes from buffer(0) to buffer(x-1). This is slightly different from how things are in C.

Compressing files

Overall, uncompressing shouldn’t be too difficult, since you already know both the exact compressed AND decompressed size. Trying to compress data however, is more difficult, because you don’t know how big the new zlib stream will be. However, the zlib manual states that the compressed size of a buffer will be at most (1.01 * size + 12) bytes big, regardless of compression level. Here we will use compress2, using a compression level of 9 (maximum).

Using compress2 in C

For this short snippet we will assume that the in buffer already contains a string called “Let us…”. We will compress this into zlib and turn it into a file that is similar to JJ2’s file formats.

char in[] = “Let us attempt to compress this string”;
char *out;
int CompressedSize;
int UncompressedSize;

int main() {
    UncompressedSize = sizeof(in);
    CompressedSize = (int)((float)(UncompressedSize * 1.01) + 12); //Allocate space

    out = malloc(CompressedSize); //Allocates a size of UncompressedSize for output
    compress2(out, &CompressedSize, in, UncompressedSize, 9); //Compresses, YAY :)
    //Note that CompressedSize now holds the real uncompressed size

    //Here “out” contains the compressed data, so do whatever you want with it.
    //For our example, we will write this to testcompress.dat
    FILE *fo = fopen(“testcompress.dat”, “wb”); //Opens testcompress.dat for writing
    fwrite(out, 4, 1, &CompressedSize);   //Writes the compressed size
    fwrite(out, 4, 1, &UncompressedSize);   //Writes the uncompressed size
    fwrite(out, 1, CompressedSize, fo); //Writes compressed stream to file
    fclose(out); //We finished writing to the file, no need to use it anymore
}

Again, if you are implementing this, remember to #include, and don’t forget to free what you malloc. If you are fussy and want your out buffer to be the right size, use realloc to adjust it.

Using compress2 in VB

Same thing as above, string to zlib, write to testcompress.dat.

Dim inBuffer As String
Dim outBuffer() As Byte
Dim CompressedSize As Long
Dim UncompressedSize As Long

Sub Main()
    inBuffer = “Let us attempt to compress this string”
    UncompressedSize = Len(inBuffer) + 1 ‘Add a null-char to the end of the string
    CompressedSize = UncompressedSize * 1.01 + 12 ‘Sets maximum output size

    ReDim outBuffer(CompressedSize – 1) ‘Allocates output space
    compress2 outBuffer(0), CompressedSize, inBuffer, UncompressedSize, 9 ‘YAY :)
    ReDim Preserve outBuffer(CompressedSize – 1) ‘Adjusts output space

    ‘Here “outBuffer” contains the uncompressed data, so do whatever you want with it.
    ‘For out example, we’ll just do something simple – dump it into testcompress.dat
    Open “testcompress.dat” For Binary Access Write As #1 ‘Opens file for writing
    Put #1, , CompressedSize   ‘Writes CompressedSize to the first four bytes
    Put #1, , UncompressedSize ‘Writes UncompressedSize to the next four bytes
    Put #1, , outBuffer        ‘Writes the output buffer to the file
    Close #1 ‘Finish writing to file, no need to use it anymore
End Sub

Again, the Visual Basic function looks slightly harder than C++. Only thing to note is the use of ReDim Preserve. Remember how the UncompressedSize value changes after going through compress2? We adjust the buffer so that it fits exactly the compressed stream, and prevent any problems with Put #1, , outBuffer.

CRC Checking

Finally we get to the crc32 stage. Most of the JJ2 files contain an internal (and some even external) CRC check to verify that the file is not corrupted in any way. Seeing how most programmers are lazy, they will probably skip the CRC mechanism and just assume that file is valid, but it will be included here just for completeness anyway.

The crc32 function takes 3 arguments. The first one is a previous crc value, in case we do the CRC check in chunks. For simplicity, we will always use CRC on the whole buffer at once, which means we leave CRC as 0. buf/buffer is the buffer, and len/bufferLen is the length of the buffer. If you’ve fully understood the uncompress and compress2 functions then there is no need for an example here, since the buffer and length arguments are used here in exactly the same way as before.

Notes

None of the above code has been tested. They were jotted down directly on notepad and have never gone through any stage of compilation.

Related content outside the wiki

• zlib
• Original forum post by Neobeo