# Fuzz Testing ### Fuzz Testing Fuzz testing means testing a program with many generated inputs. A normal unit test checks examples that you choose by hand: ```zig try std.testing.expectEqual(@as(u8, 5), parseDigit('5')); ``` A fuzz test checks many inputs automatically. Instead of asking only “does this work for `'5'`?”, it asks a broader question: Can this code survive many possible inputs without crashing, corrupting memory, or violating its rules? #### Why Fuzz Testing Exists Hand-written tests are limited by your imagination. You may test: ```text "123" "0" "999" "abc" "" ``` But real input can be stranger: ```text "\x00" "\xff\xfe" "999999999999999999999999" "12\n34" " 42" "+-1" ``` Fuzz testing helps discover cases you did not think to write. This is especially useful for code that reads external input: parsers decoders file format readers network protocol handlers compression code lexers interpreters command-line argument parsers Anything that accepts bytes from outside the program is a candidate for fuzz testing. #### A Simple Function to Test Suppose we have a small parser: ```zig const std = @import("std"); fn parseSmallNumber(text: []const u8) !u8 { return try std.fmt.parseInt(u8, text, 10); } ``` This function parses a string as an unsigned 8-bit integer. Valid inputs include: ```text "0" "42" "255" ``` Invalid inputs include: ```text "" "abc" "999" "-1" ``` A normal unit test might look like this: ```zig test "parseSmallNumber parses valid values" { try std.testing.expectEqual(@as(u8, 42), try parseSmallNumber("42")); try std.testing.expectEqual(@as(u8, 255), try parseSmallNumber("255")); } ``` That is useful, but narrow. A fuzz-style test asks a more general question: For any byte input, does the parser either return a valid `u8` or return an error safely? #### A Basic Fuzz-Style Test Zig test code can loop through many values. Here is a simple fuzz-style test over all byte values: ```zig const std = @import("std"); fn parseSmallNumber(text: []const u8) !u8 { return try std.fmt.parseInt(u8, text, 10); } test "parseSmallNumber handles every single byte input" { var byte: u16 = 0; while (byte <= 255) : (byte += 1) { const input = [_]u8{@intCast(byte)}; _ = parseSmallNumber(input[0..]) catch { continue; }; } } ``` This test does not require every byte to be valid input. It only requires the function to handle every input safely. If parsing fails, that is acceptable. The test continues. #### Fuzz Testing Checks Properties A fuzz test usually checks a property, not one exact answer. A property is a rule that should always be true. For example, suppose we write a function that reverses a slice in place: ```zig fn reverse(buf: []u8) void { var left: usize = 0; var right: usize = buf.len; while (left < right) { right -= 1; const tmp = buf[left]; buf[left] = buf[right]; buf[right] = tmp; left += 1; } } ``` A good property is: If you reverse a buffer twice, you get the original buffer back. That property can be tested for many inputs: ```zig const std = @import("std"); fn reverse(buf: []u8) void { var left: usize = 0; var right: usize = buf.len; while (left < right) { right -= 1; const tmp = buf[left]; buf[left] = buf[right]; buf[right] = tmp; left += 1; } } test "reverse twice returns the original bytes" { const cases = [_][]const u8{ "", "a", "ab", "abc", "hello", "\x00\x01\x02", "\xff\x00\xff", }; for (cases) |input| { var buf: [32]u8 = undefined; try std.testing.expect(input.len <= buf.len); @memcpy(buf[0..input.len], input); reverse(buf[0..input.len]); reverse(buf[0..input.len]); try std.testing.expectEqualSlices(u8, input, buf[0..input.len]); } } ``` This is not full random fuzzing yet, but it has the same shape: many inputs, one property. #### Generating Inputs with a PRNG You can also generate test data with a pseudo-random number generator. A pseudo-random generator is deterministic when you give it a fixed seed. That is useful for tests because the same test should produce the same result every time. ```zig const std = @import("std"); fn reverse(buf: []u8) void { var left: usize = 0; var right: usize = buf.len; while (left < right) { right -= 1; const tmp = buf[left]; buf[left] = buf[right]; buf[right] = tmp; left += 1; } } test "reverse twice returns the original bytes for generated inputs" { var prng = std.Random.DefaultPrng.init(12345); const random = prng.random(); var round: usize = 0; while (round < 1000) : (round += 1) { var original: [64]u8 = undefined; var buffer: [64]u8 = undefined; const len = random.intRangeAtMost(usize, 0, original.len); for (original[0..len]) |*byte| { byte.* = random.int(u8); } @memcpy(buffer[0..len], original[0..len]); reverse(buffer[0..len]); reverse(buffer[0..len]); try std.testing.expectEqualSlices(u8, original[0..len], buffer[0..len]); } } ``` This test generates 1000 byte arrays of different lengths. For each generated input: copy the original data reverse the copy reverse it again check that it matches the original This catches many mistakes in `reverse`. #### Deterministic Randomness Use a fixed seed in tests: ```zig var prng = std.Random.DefaultPrng.init(12345); ``` Do not use changing seeds by default. A changing seed may find more bugs over time, but it can also make failures hard to reproduce. If a test fails only sometimes, debugging becomes harder. A fixed seed means: same generated inputs same failure same debugging path Once you find a bug, you can turn the failing random input into a normal unit test. #### Turning a Fuzz Failure into a Unit Test Suppose a generated test finds that this input breaks your function: ```zig const failing = [_]u8{ 0xff, 0x00, 0x41 }; ``` Add a direct test: ```zig test "reverse handles bytes found by fuzz testing" { var buf = [_]u8{ 0xff, 0x00, 0x41 }; const original = buf; reverse(buf[0..]); reverse(buf[0..]); try std.testing.expectEqualSlices(u8, original[0..], buf[0..]); } ``` This is called a regression test. It protects you from reintroducing the same bug later. #### Fuzzing Parsers Parsers are excellent fuzz targets because they accept arbitrary bytes. Here is a small parser that accepts only ASCII digits: ```zig const ParseError = error{ Empty, InvalidDigit, }; fn parseDigitsOnly(text: []const u8) !void { if (text.len == 0) return ParseError.Empty; for (text) |ch| { if (ch < '0' or ch > '9') { return ParseError.InvalidDigit; } } } ``` A fuzz-style test can generate many byte strings and require the parser to return safely: ```zig const std = @import("std"); const ParseError = error{ Empty, InvalidDigit, }; fn parseDigitsOnly(text: []const u8) !void { if (text.len == 0) return ParseError.Empty; for (text) |ch| { if (ch < '0' or ch > '9') { return ParseError.InvalidDigit; } } } test "parseDigitsOnly handles generated byte strings safely" { var prng = std.Random.DefaultPrng.init(9876); const random = prng.random(); var round: usize = 0; while (round < 1000) : (round += 1) { var input: [32]u8 = undefined; const len = random.intRangeAtMost(usize, 0, input.len); for (input[0..len]) |*byte| { byte.* = random.int(u8); } parseDigitsOnly(input[0..len]) catch { continue; }; } } ``` This test accepts both success and error. The rule is simple: no crash, no memory bug, no impossible state. #### Testing Stronger Parser Properties A stronger test can check a real property. For `parseDigitsOnly`, if the function succeeds, every byte should be an ASCII digit: ```zig test "parseDigitsOnly succeeds only when all bytes are digits" { var prng = std.Random.DefaultPrng.init(9876); const random = prng.random(); var round: usize = 0; while (round < 1000) : (round += 1) { var input: [32]u8 = undefined; const len = random.intRangeAtMost(usize, 0, input.len); for (input[0..len]) |*byte| { byte.* = random.int(u8); } if (parseDigitsOnly(input[0..len])) |_| { try std.testing.expect(input[0..len].len > 0); for (input[0..len]) |ch| { try std.testing.expect(ch >= '0' and ch <= '9'); } } else |_| { // Errors are allowed for invalid input. } } } ``` Now the test says more than “does not crash.” It checks the meaning of success. #### Fuzz Testing and Memory Safety Fuzz testing is most valuable when combined with Zig’s safety checks. Run tests in a safe mode while developing: ```bash zig test main.zig ``` Debug and safe release modes can catch many problems, including bounds errors and integer overflow. For low-level code, this matters. A fuzz input may reach a branch you did not expect. Safety checks can turn hidden memory bugs into visible test failures. #### What Fuzz Testing Cannot Prove Fuzz testing does not prove your program is correct. It checks many examples, but not all possible examples. A test that runs 1000 inputs still misses many cases. So fuzz testing should complement, not replace: unit tests table-driven tests edge case tests manual reasoning clear API design Use fuzz testing to find surprising inputs. Use normal tests to document known behavior. #### Keep Fuzz Tests Bounded A test suite should run quickly. Do not write beginner fuzz tests that generate millions of inputs every time. Good starting numbers: ```text 100 cases 1000 cases 10,000 cases for small pure functions ``` Choose a count that gives useful coverage without making normal development slow. For expensive code, use fewer cases. #### A Practical Pattern A good beginner fuzz-style test follows this pattern: ```zig test "function obeys property for generated inputs" { var prng = std.Random.DefaultPrng.init(12345); const random = prng.random(); var round: usize = 0; while (round < 1000) : (round += 1) { // 1. generate input // 2. call function // 3. check property } } ``` The important part is not randomness itself. The important part is the property. Ask: What must always be true? Then generate many inputs and check that rule. #### A Complete Example Save this as `main.zig`: ```zig const std = @import("std"); fn reverse(buf: []u8) void { var left: usize = 0; var right: usize = buf.len; while (left < right) { right -= 1; const tmp = buf[left]; buf[left] = buf[right]; buf[right] = tmp; left += 1; } } test "reverse twice returns the original bytes for generated inputs" { var prng = std.Random.DefaultPrng.init(12345); const random = prng.random(); var round: usize = 0; while (round < 1000) : (round += 1) { var original: [64]u8 = undefined; var buffer: [64]u8 = undefined; const len = random.intRangeAtMost(usize, 0, original.len); for (original[0..len]) |*byte| { byte.* = random.int(u8); } @memcpy(buffer[0..len], original[0..len]); reverse(buffer[0..len]); reverse(buffer[0..len]); try std.testing.expectEqualSlices(u8, original[0..len], buffer[0..len]); } } ``` Run it: ```bash zig test main.zig ``` This test does not care what the random bytes are. It only cares about the rule: reversing twice must return the original data. That is the heart of fuzz testing. Pick a rule that should always hold, then attack the code with many inputs.