About Arctic Engine
Arctic Engine is an open-source, free game
engine released under the MIT license.
Arctic Engine is implemented in C++ and focuses on simplicity. Being a C++
programmer and making games should not be joyless, disillusioning, and
discouraging. In the '80s and '90s, a programmer could make games alone, and
it was fun. Arctic Engine aims at making game development in C++ fun again.
Testing can be fun
Testing the game engine is very important since games are usually no more
robust and performant than the underlying middleware or game engine. Writing
tests by hand is time-consuming and disillusioning, and it may drain the fun
from the development process. So, to my shame, I avoided writing tests in every
way I could. For instance, I used static analyzers to detect bugs. The problem
with static analyzers was the lack of motivation to fix potential issues. You
may be unsure whether a bug is really there, and it can sometimes be hard to
find a way to trigger it.
The other possibility was fuzz testing. I heard about fuzzing but didn't try it
earlier because I thought it was hard to integrate with the project. I could
not be more wrong. It's amazing how little effort it takes to get fuzz testing
up and running with GitLab.
Fuzz testing and what it exposed
Thanks to Sam Kerr for proving me wrong about
fuzzing by actually fuzzing
the sound loader code. Arctic Engine allows loading a sound from a WAV file in
memory. To fuzz the loader's code, you create a small CPP file with a single
function like this:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t * data, size_t size) {
std::shared_ptr<arctic::SoundInstance> result = arctic::LoadWav(data, size);
return 0;
}
Then you add -fsanitize=fuzzer flag to the CMakeLists.txt file and a few
lines to the .gitlab-ci.yml file, and the fuzzing begins! You may want to
drop in a few WAV files to the corpus folder to help the fuzzer and speed up
the process, but that's optional. Ok, it was a little harder than that with the
Arctic Engine because it would output a message and quit upon processing
unsupported file formats. Still, handling file loading errors this way was a
bad idea, and I finally had a reason to fix it.
The fuzzer started crashing Arctic Engine: first, it triggered a signed integer
overflow, a division by zero, and a buffer overrun. And then, the wave loader
got out-of-memory while trying to resample a tiny WAV file with a sampling rate
of 1 sample per second to 44100 samples per second. Wow.
What I liked about fuzzing is that fuzzer actually crashes your program and
provides you the input so you can reproduce the crash. And once you've set up
the test harness, the entire testing process is fully automated, saving you
time and effort. It's like having a personal QA team, you commit your code, and
in a few minutes, you already have it tests-covered.
Then I fuzzed the CSV and the TGA file parsers and expected to find some bugs
in the CSV and none in the TGA. What can I say? You may not find bugs where you
expect them to be and find bugs where you thought there were none. The TGA
loader crashed immediately with a buffer overrun. It did not account for files
containing only a valid header but no actual image data after it.
Plans
I will add a simple HTTP web server and some multiplayer network interaction
code to the Arctic Engine. I was putting it off for quite a while now because I
thought testing would be a pain. Now that I know how easy it is to apply
GitLab's fuzz testing to any data processing code, I'm very optimistic and
somewhat challenged. Like "Can I make it withstand the fuzzer from the first try?".
It makes writing code fun for me once again.
Further reading
About the guest author
Huldra is a senior videogame programmer by day maintainer of the Arctic Engine by night. She started it because she wanted a game engine that kept simple things simple and made complex things possible.
