Automate Bug Finding: Fuzzing C# Code on Windows

In the world of software engineering, bugs can lead to major problems both during development and after deployment. The worst of which can end in very costly security issues. With today’s complex software, automated testing becomes unavoidable. While some procedures such as unit tests and regression tests became widely established, they still require much input from developers in the creation of their test cases.

Therefore in recent years, a testing technique called fuzzing began to gain popularity. Fuzzing involves feeding a program with invalid, unexpected, or random data to see how it responds. By doing this, fuzzing can help identify vulnerabilities and bugs in a program that may not be found through more traditional testing methods. At first glance, it sounds inefficient but modern fuzzing engines use code coverage feedback to guide new input mutations and therefore massively speed up the process.

This blog entry explains the basics of fuzzing and shows how easy it is to find real bugs in a popular C# library.

Basics

Coverage-guided fuzzing works by repeating several steps in a loop:

• Take inputs from a Corpus
• Mutate these inputs and feed them to the Fuzz Target
• The Fuzz Target calls the code that we want to test with the inputs
• Found crashes are saved and the coverage feedback is used to add new inputs to the Corpus

Fuzzing C# Code

Fuzzing is particularly common for programming languages that are not memory safe such as C and C++, with several established Fuzzing Engines such as Libfuzzer and AFL++ having wide community support and popularity.

They are therefore often used as a core engine to provide fuzzing support for other programming languages, one such project is SharpFuzz which enables state-of-the-art fuzzing for C# language code.

Let’s quickly go through the full process, from setup to finding a bug on a real-world C# example.

Setup

For this example, we will use Libfuzzer on Windows. For Linux and AFL support refer to the SharpFuzz Github .

There are only four things we need: the libfuzzer-dotnet-windows.exe , the helpful start script , the SharpFuzz Commandline tool and a Visual Studio project with our fuzzing target.

Download the first two and make sure your Visual Studio installation includes Clang >= 14.0.0 and .NET SDK >= 6.0. Also install the SharpFuzz Commandline tool:

dotnet tool install --global SharpFuzz.CommandLine

If you later get errors with SharpFuzz not being recognized as a command, check if ‘%USERPROFILE%\.dotnet\tools’ is on the Path environment variable.

It’s a good idea to create a simple Visual Studio C# console app. Examples can be found here . Just remember to add the SharpFuzz NuGet Package to the project.

Writing a Fuzz Target

The fuzzing target is probably the most important part of the whole fuzzing process. While it has to be fast and efficient because the bug-finding speed depends on the executions per second, it allows room for creativity. Easy targets are parsers and functions that process complicated structures.

We will keep it simple here and try to fuzz the AngleSharp HTML parser. For this, we just have to write a small fuzzing target:

using SharpFuzz;
using System.Text;

namespace FuzzTest
{
    public class Program
    {
        public static void Main(string[] args)
        {
            Fuzzer.LibFuzzer.Run(data =>
            {
                try
                {
                    var html = Encoding.UTF8.GetString(data);
                    var parser = new AngleSharp.Html.Parser.HtmlParser();
                    parser.ParseDocument(html.ToCharArray(), 0);
                }
                catch { }
            });
        }
    }
}

The Fuzzer.LibFuzzer.Run function is the entry point of the fuzzer which feeds the input data to our code. Keep in mind that if the function you are trying to fuzz is allowed to throw exceptions, you have to catch them to prevent false positives!

Fuzzing

To improve the speed at which the fuzzing engine finds interesting code paths it is recommended to use a dictionary. We will use an HTML dictionary . Now that we have everything ready, let’s start the fuzzing.

We call the start script fuzz-libfuzzer.ps1 downloaded before from the Windows Powershell.

Set the locations correctly and use an empty folder for the corpus. It is needed to save inputs that have new code coverage. This therefore saves the fuzzing progress and allows stopping and continuing at a later time.

.\fuzz-libfuzzer.ps1 -libFuzzer ".\libfuzzer-dotnet-windows.exe" -project ".\FuzzingTargets\FuzzingTargets\AngleSharpFuzzer.csproj" -corpus ".\FuzzingTargets\AngleSharpCorpus" -dict ".\dicts\html.dict"

If everything works correctly the fuzzing starts and runs until a crash is reported. The crashing input is saved to a file and you can use it to reproduce the crash. We found an IndexOutOfRangeException, InvalidOperationException, and a wrong tree construction that leads to a timeout. Overall it took several hours of fuzzing AngleSharps version 1.1.0. If you want to reproduce our findings downgrade the AngleSharp NuGet package version.

We reported the found issues on GitHub and they are fixed in newer versions. Please be careful while reporting found security issues and follow the recommended steps of the respective project.

Some useful Tipps and Tricks

Useful Libfuzzer Flags

While the fuzz-libfuzzer.ps1 script is useful to quickly start fuzzing it does not expose all Libfuzzer functionality, some of which can be important. You can find an overview here . Either extend the helper script or call Libfuzzer directly.

To parallelize the fuzzing process and use all your cores -fork=(Number of parallel jobs) can be used. It is especially useful in combination with -ignore_crashes for continuous fuzzing without stopping after something is found.

Extended script:

param (
    [Parameter(Mandatory = $true)]
    [string]$libFuzzer,
    [Parameter(Mandatory = $true)]
    [string]$project,
    [Parameter(Mandatory = $true)]
    [string]$corpus,
    [string]$dict = $null,
    [int]$timeout = 10,
    [int]$fork = 0,
    [int]$ignore_crashes = 0,
    [string]$command = "sharpfuzz"
)

...

if ($dict) {
    & $libFuzzer -timeout="$timeout" -ignore_crashes="$ignore_crashes" -fork="$fork" -dict="$dict" --target_path=dotnet --target_arg=$project $corpus
}
else {
    & $libFuzzer -timeout="$timeout" -ignore_crashes="$ignore_crashes" -fork="$fork" --target_path=dotnet --target_arg=$project $corpus
}

An identified crashing input can also be long enough to complicate the debugging. This is where -minimize_crash is useful to iteratively reduce the input size while still keeping it crashing.

Direct Libfuzzer call:

./libfuzzer-dotnet-windows.exe --target_path=bin/AngleSharpFuzzer.exe ./crash-8c5a9286d99cc583bb86a75549446d9ab34422a6 -timeout=10 -minimize_crash=1

Fuzzing Blockers

Some code can be hard to fuzz because the code exploration has a difficult time progressing. Typical problems are checksums or exception-catching. If it’s your own or open-source code it is a good idea to remove them from the code during fuzzing.

Better Fuzzing Targets

While we used a really simple fuzzing target for AngleSharp the possibilities are endless.

You can build objects and set parameters by using the input bytes.

You can assert that different libraries or functions return the same results to look for inconsistencies (Differential Fuzzing ).

You can test sanitizers by feeding them input and asserting security vulnerabilities afterward.

There are also many other fuzzing engines available for other use cases such as network and GUI testing.

Conclusion

Overall fuzzing is a great technique to automate bug finding in your code and improve your code quality. We gave a short introduction to the process by showing how easy setting up SharpFuzz to find real-world C# bugs is. Keep in mind that this was only a glimpse of what is possible with fuzzing and always be responsible with reporting security vulnerabilities.