Before the Raygun API limited release last year, we’d been consistently receiving requests for a public API for a long time, to provide a way for our customers to access their Raygun data programmatically. We’re now proud to say we’re providing a public API with a range of endpoints, but it took us a lot of planning and development to get here!

In this post, I’d like to take you back to the beginning of development on our big API project. Specifically, I want to walk through the pivotal decisions we made around testing when we started development on the project, and how (and why) these have paid off.

A question of testing

We had planned the project, laid the groundwork, and were starting to get into development. Early on, we had to ask ourselves: how do we ensure that our various software components work seamlessly together to deliver the desired functionality? And how could we create an environment where we could quickly and confidently introduce new changes? We already had some unit tests, but it seemed like they might not be enough to satisfy our software quality goals. This led us to consider integration tests.

Before I get into the details, let’s clarify the distinction between integration tests and unit tests.

Unit tests vs integration tests

Unit testing focuses on verifying that individual units or components of software are working as expected. A “unit” is typically the smallest part of any software that can be tested, such as a method or a function. By testing each little piece on its own, we can be more confident that the software works well overall. These tests are isolated, meaning external dependencies (like databases or external services) are generally mocked or stubbed out.

For example, we might have an IUserService class which normally makes an API or database request to fetch information about a user. We don’t want our unit tests to rely on a real API or database for a number of reasons (speed, isolation, consistency, control over test scenarios, etc), so we can do something like this:

public void GetUserName_ReturnsCorrectName()
{
    // Arrange
    var mockUserService = new Mock<IUserService>();
    var testUserId = 1;
    var testUserName = "Alice";

    // Setup the mock to return a fake user for a specific ID
    mockUserService.Setup(s => s.GetUserById(testUserId))
      .Returns(new User { Id = testUserId, Name = testUserName });

    var userProcessor = new UserProcessor(mockUserService.Object);

    // Act
    var userName = userProcessor.GetUserName(testUserId);

    // Assert
    Assert.Equal(testUserName, userName);

    // Verify that GetUserById was called with the correct ID
    mockUserService.Verify(s => s.GetUserById(testUserId), Times.Once());
}

Integration testing, on the other hand, is about ensuring that the units or components of software work well when integrated together. It aims to expose the problems that might occur when different units interact with each other. Integration tests often deal with real instances of external dependencies rather than their mocked or stubbed versions–this helps to make the tests realistic, to better catch issues that could happen in production. To illustrate this more clearly, we’ll look at an example of integration testing shortly!

Are integration tests worth the cost?

It’s well known that integration tests are usually not trivial to set up, with a number of potential costs creating a barrier to entry. When deciding whether it was worth it for us to implement integration tests alongside our existing unit tests, we had to consider the following:

• Integration tests often require configuring testing infrastructure alongside your tests (e.g., on a build server), which can take time and lead to brittle tests if the infrastructure ever has issues. This can also make it difficult to run the integration tests in a local development environment.
• Integration tests usually require more effort to write vs. a unit test, because of the need to ensure each test scenario is set up properly. Compounding this across hundreds of tests can really add up on development time.
• Integration tests almost always take longer to execute than unit tests, leading to long CI/CD cycles, wasting developers’ time.

In the end we decided that in our case, the benefits outweighed the costs. We knew that for an API, testing the HTTP and database edges of the application was critical, and unit tests would not be enough to provide confidence in that regard. We also had the advantage of a relatively simple interface–testing did not need to simulate user interactions on a GUI, we could simply send HTTP requests to trigger our tests. And by automating the tests, we would save significant time by reducing the need for manual testing, and adding a “safety net” for any issues we might not find during manual testing. Over the long run, that time saved should more than make up for the time spent implementing the tests.

We also had a secret weapon that eliminated many of the costs, reducing both the infrastructure setup overhead and the test execution time! With this powerful asset, the decision to implement integration tests became obvious.

What might it be? Let me introduce you to…

The WebApplicationFactory

The WebApplicationFactory is a class provided by .NET’s Microsoft.AspNetCore.Mvc.Testing package, described in their documentation as a “Factory for bootstrapping an application in memory for functional end to end tests.” While the description mentions end-to-end tests, the value of running an application in-memory applies to integration tests as well.

As I indicated above, the benefits of this in-memory execution are twofold:

• We can lower the infrastructure overhead and speed up the tests by replacing external services with in-memory equivalents.
  • As you’ll see below, we do this for all external services except for the database because we believed it was important to test using a real database.
• We can reduce the test duration further by relying on the WebApplicationFactory to simulate network requests in-memory, making them both faster and more reliable.

Show me some examples!

Ok, here’s a simplified overview of how we did it, starting with our custom “WebApplicationFactory” class:

public class CustomWebApplicationFactory<TProgram> : WebApplicationFactory<TProgram> where TProgram : class
{
    private const string TestingDatabaseConnectionString = "testing database connection string";

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureTestServices(services =>
        {
            services.AddAuthentication("test")
                    .AddScheme<AuthenticationSchemeOptions, TestAuthenticationHandler>("test", _ => { });

            services.Replace(ServiceDescriptor.Singleton<DbContext>(_ => new DbContext
            {
                ConnectionString = TestingDatabaseConnectionString,
                // other config settings
            }));

            services.Replace(ServiceDescriptor.Singleton<IDistributedCache, MemoryDistributedCache>());
            services.AddInMemoryRateLimiting();

            services.Replace(ServiceDescriptor.Singleton<IPublisher, FakePublisher>()
        });

        builder.UseEnvironment("Development");
    }
}

As you can see, it’s not very complicated. The ConfigureWebHost method is where we set up all the things that we want the integration tests to do differently to a production run of the application.

First we configure a test authentication-handler that allows us to control the authentication status in the tests. Most of the time, we use it to return an authenticated user, but it can also return an unauthenticated user in order to test the behavior of unauthenticated requests. Our authentication system has its own tests so replacing it with a test handler here streamlines these tests significantly.

Then we replace the code that interfaces with our database (called “DbContext” here) with code that is identical except for the database connection string. This allows our tests to interact with a database in the same way they would in production, without modifying any of our production data. Further down, we’ve employed in-memory substitutes or “fakes”, but we believed it paramount for our integration tests to interact with an actual database. This was our primary external dependency, echoing the real-world scenario our API operates under. It ensured our tests validated interactions with the database correctly, mimicking the actual production environment.

To ensure our tests run quickly and to avoid the need to set up any external dependencies other than the test database, we opted to utilize in-memory versions of certain services. For those that didn’t offer such options, we built fakes. You can see these configured at the end of the ConfigureWebHost method. The FakePublisher simply stashes messages in an internal dictionary instead of publishing to an external queue. This makes it easier to verify whether or not messages have been published in our tests - we can simply inspect the FakePublisher’s dictionary.

Usage

Our integration-test classes all make use of a base class that sets up an HttpClient using the CustomWebApplicationFactory. This is a real HttpClient, not a mock, and we use it just like we might use one to make requests in a real application. That’s important because the value of the integration tests lies in them being as close as possible to production. The more the tests deviate from the real-world scenario, the less likely they are to be able to catch a potential real-world issue!

Inside the WebApplicationFactory, the HttpClient is configured to use a TestServer to execute the requests in-memory. So while it is a real HttpClient, it doesn’t have the overhead of performing actual network requests, resulting in the best of both worlds for integration testing.

protected CustomWebApplicationFactory<Program> Factory;
protected HttpClient Client;
protected DbContext DbContext;

[OneTimeSetUp]
public void OneTimeSetUp()
{
    Factory = new CustomWebApplicationFactory<Program>();
}

[SetUp]
public void SetUpBase()
{
    Client = Factory.CreateClient();
    var dbContextFactory = Factory.Services.GetRequiredService<IDbContextFactory>();
    DbContext = dbContextFactoryFactory.CreateDbContext();
}

These methods initialize the fields that we’ll need for all of our tests: the HttpClient to make requests, the DbContext to update or read the database, and the Factory to retrieve any services we might need to inspect during the tests.

An integration test that makes use of all of these fields might look something like this:

[Test]
public async Task ResolveErrorGroup_ErrorGroupIsActive_PublishesEvent()
{
    SetUpErrorGroup(DbContext);

    await Client.PostAsync($"applications/{1}/error-groups/{2}/resolve", null);

    var publisher = (Factory.Services.GetRequiredService<IPublisher>() as FakePublisher)!;
    var messages = publisher.PublishedMessages;

    Assert.That(messages[typeof(ResolvedMessage)], Has.Count.EqualTo(1));
}

In this test:

• For clarity in the example I’ve condensed it into a “SetUpErrorGroup” method, but the first step is to configure the database to the state we need it for the test, using the DbContext from the base class.
• Then we make a request using the Client that was created in the SetUpBase method earlier. This is where the in-memory request goes to our API, which is also running in-memory!
• Lastly, we use the Factory to get our fake publisher and assert that the message was published as expected. In other tests we might save the response from the request and assert based on that instead.

Lessons Learned

Now that it’s been a few months since development started, we’re able to look back and identify a few points that made the integration tests really shine for us:

1. We ran all of our tests (both unit and integration) on every build of our CI server, and they can be run in under 2 minutes on a development machine. This means that the tests are constantly being run, consistently reinforcing our confidence as the code evolves. This helped us release new features more quickly, as we could be sure that new features were not causing any regressions.

2. For some unit tests, we found that once all the external services were mocked, the remainder of the code had very little logic to actually unit test. This highlighted the importance of the integration tests; since much of the API’s functionality hinges on interactions with external services, we found that testing against actual databases and HTTP clients rather than relying on mocks was essential.

3. While the integration tests have been useful for testing external services, one benefit that I didn’t anticipate was that the integration tests have also been great for testing complex behaviors within the application. For example, middleware operations or message/handler systems that are loosely coupled from the rest of the application might be difficult to unit test when isolated. However, because we’re simulating an actual request, the test will traverse through every layer within the application just as it would in a live environment.

Summing up

In this post, we’ve covered some background on the development of Raygun’s public API, along with our decisions around testing and the differences between unit and integration tests. I’ve also let you in on the powerful secret weapon that is the WebApplicationFactory, which you’ll hopefully make good use of!

One of the most important factors in our success with this approach was that we didn’t treat integration tests as an afterthought. We started early and included tests with every feature we added, so we’ve accumulated a good number of them now. Having a large test suite has been really important to build confidence, because if developers feel the tests can’t reliably detect issues introduced by their changes, the tests lose their value.

I hope this account of our integration testing lessons has been insightful. You can follow along with the progress of the Raygun API here!

Read more about Raygun’s monitoring tools for developers here.