Unit Tests Should Go Beyond Interfaces

”Test your interfaces” is incomplete.

There’s a software engineering truism that you should test your interfaces. Broken into its component pieces, this rule states you should:

While solid access control is truly important, the “Test your interfaces” truism is incomplete to the point of doing more harm than good. It forces us to write integration tests, not unit tests. To understand why, we will look at the compromise it tries to broker, see the pitfalls of its approach, and evaluate what other tradeoffs are possible.

We must find ways to safely expose your internal functionality as helpers with interfaces and unit test these.

TL;DR: Use non-private, pure, static functions liberally.


Programming Language Differences

The problems described here are endemic to languages with strict access control semantics. Languages that don’t have that or that allow runtime reflection to change the access control semantics of fields allow you to approach unit testing in different ways.

Nevertheless, the fundamental concepts described here are language-agnostic. (Even in less strict environments, it’s a very bad idea to test your internal implementations instead of their behavior.)

We’ll walk through the concepts here using a piece of Swift code that acts as a component in a user registration system. The most important code parts are inlined, but the full implementation could be interesting.

The code has the following behavior:

  1. Launch sub-units of code that implement the collection of username, phone number, email, and password.
  2. Validate that the password passes some complexity rules, (including not containing username, email, or phone number).
  3. Re-request password until the validation rules are satisfied.
  4. Call up to some owning code using a listener interface to report success.

The Problem: Integration Tests and Combinatorial Explosion

Combinatorial explosion occurs in integration testing as your API grows in size. It refers to the potential complexity of the internal state in a system increasing aggressively as its API is expanded.

This increase in state complexity leads to an increase in setup work that is required in order to test in a desired state.

The main piece of logic in our code is our validatePassword function. It’s what we’d most like to unit test. However, since it’s an internal helper that touches state, we’ve prudently made it private.

private func validatePassword() {
    self.passwordIsValid = false
    self.passwordIssue = nil

    guard let username = self.username,
          let email = self.email,
          let phoneNumber = self.phoneNumber,
          let password = self.password
    else {
        return
    }

    if password.count < 10 {
        self.passwordIssue = .tooShort
    }
    if password.contains(username) {
        self.passwordIssue = .containsUsername
    }
    if password.contains(email) {
        self.passwordIssue = .containsEmail
    }
    if password.contains(phoneNumber) {
        self.passwordIssue = .containsPhoneNumber
    }
    if !passwordContainsAtLeastOneNumber() {
        self.passwordIssue = .noNumbers
    }
    if !passwordContainsAtLeastOneSpecialCharacter() {
        self.passwordIssue = .noSpecialCharacters
    }
    self.passwordIsValid = true
}

This means that in order to test the unit of code responsible for determining if there are issues with the user’s password, we have to spin up a whole mock environment with significant test infrastructure1 to host the BadOnboardingManager and inspect its output through its public methods. That gets annoying really quickly.

func test_PasswordCanNotContainUsername() {
    let mockPasswordCollectorLauncher = MockPasswordCollectorLauncher()
    let mockListener = MockOnboardingManagerListener()

    let onboardingManager = BadOnboardingManager(passwordCollectorLauncher: mockPasswordCollectorLauncher)
    onboardingManager.listener = mockListener

    onboardingManager.start()
    // TODO: assert the username collector launches now.
    onboardingManager.usernameCollectorReturned(username: "ValidUsername")
    // TODO: assert the email collector launches now.
    onboardingManager.emailCollectorReturned(email: "valid@example.com")
    // TODO: assert the phone number collector launches now.
    XCTAssertEqual(mockPasswordCollectorLauncher.launchCallCount, 0)
    onboardingManager.phoneNumberCollectorReturned(number: "+11235554321")
    XCTAssertEqual(mockPasswordCollectorLauncher.launchCallCount, 1)
    XCTAssertEqual(mockPasswordCollectorLauncher.existingPasswordIssueValues.last, nil)

    onboardingManager.passwordCollectorReturned(password: "ValidUsername$123")
    XCTAssertEqual(mockPasswordCollectorLauncher.launchCallCount, 2)
    XCTAssertEqual(mockPasswordCollectorLauncher.existingPasswordIssueValues.last, PasswordIssue.containsUsername)
}

The resulting test isn’t just time-consuming to write, it also fails the cardinal rule of unit tests: Only test one thing at a time. This means that when the test fails, a developer will have to work to pin down exactly what part of the test failed. If the failure was due to changes in the right setup process, it’ll also mean they have to modify a ton of tests!

It fails the cardinal rule of unit tests because it’s an integration test.

One could envision fully testing the logic of a whole program by just inspecting the inputs and outputs that the program’s APIs provide access to. But the setup required before any one test could become astronomical—and it would be duplicated for every near-identical variant of the test.

Unit tests are the answer to this combinatorial explosion of setup state dependence. But how can we use them without sacrificing the safety of our private functions and variables?

Keeping your privates private

It’s good software engineering practice to avoid exposing functionality outside of your program. A reasonable rule of thumb can be to slap private access control on whatever you possibly can and allow more liberal access only where the program’s functionality requires you to do so. When another class has to trigger your class’s function, it can’t be private.

The “Test your interfaces” approach asserts that reasonable unit testing happens by testing only the functionality that need not be private—that the level at which your code must be accessed by external code is a reasonable level to validate it.

This only works if you’re not doing significant amounts of logic within the private portion of your code. If you are, you hit combinatorial explosion and get unreadable, fragile, tests like test_PasswordCanNotContainUsername above.

Taking a deeper look at the rules of thumb that have gotten us this far provides a potential way out of our quandary: The assertion that you should constrict access to your helper methods is founded on the assumption that those helper methods manipulate state.

It’s not public functions that are bad. It’s any public means to change private state which is dangerous!

In the example code, the password’s verification state is stored in a var in the BadOnboardingManager that the validatePassword function belongs to. Simplified, it does roughly this:

private func validatePassword() {
    if /* `self.password` fails validation */ {
        self.passwordIsValid = false
    } else {
        self.passwordIsValid = true
    }
}

That’s a pretty shoddy way to pass state around an app.

If instead, your helper method was stateless, it would be safe to expose publicly. With a small change, the fact that external code can call the helper method doesn’t make the code more fragile since it doesn’t let the external code manipulate the class’s internal state.

public func isValidPassword(_ password: String) -> Bool {
    if /* local `password` fails validation */ {
         return false
    } else {
         return true
    }
}

However, this is not a real solution in production code. Why not?

Writing software for longevity at developer scale

While there’s no immediate issue with that code, it leaves the codebase in a more vulnerable state. Once there’s an exposed helper function whose safety is dependent on its lack of side effects, the codebase is one simple mistake away from a state modifying exposed helper.

public func isValidPassword(_ password: String) -> Bool {
    self.potentialPassword = password
    if /* local `password` fails validation */ {
         return false
    } else {
         return true
    }
}

After all, how would another author or code reviewer know that your intentions for your public function were for it to be public, but only as long as it doesn’t change to modify state? Sure, you could sprinkle the function with comments to appeal to your co-committers (or future self) to take heed, but humans mess up all the time.

You know what doesn’t mess up? A compiler doesn’t mess up. Trust your compiler.

To make this code safe for other humans to change, we want to find a way to let the compiler know—and enforce—that the function is outside of the scope of our private state. There are a few different ways to do this and they have their own tradeoffs:

  1. We could make the helper function part of its own class (perhaps PasswordValidator). The class would then implement an interface like PasswordValidating that the OnboardingManager class would require a conforming instance of. This is the OOP-iest answer.
  2. We could make the helper function a free-floating global function (perhaps in some helper package). This is safe because we know it’s pure (side-effect-less) function. We could call this function in the OnboardingManager, or accept it as a parameter with or without a typealias (aliased name for the function signature/type). This is the most functional answer.
  3. Something in the middle.

(1.) and (2.) are the best answers (for their respective language paradigms) in a situation where you can factor the code out to be something you’ll plausibly reuse. However, if the functionality is naturally a member of the class (i.e., OnboardingManager) that’s unlikely to be reused elsewhere, breaking the code out into a helper construct could just end up meaning pulling it out of the context in which it is most understandable.

If the code is not likely to be used elsewhere and is most understandable in the context of the class, you should leave it there.

To do so safely, lean on your compiler by making your helper function static to prevent it from touching class scope state.

public static func isValidPassword(_ password: String) -> Bool {
   /* Any attempt to touch self here fails compilation */
   if /* local `password` fails validation */ {
         return false
    } else {
         return true
    }
}

In this manner, one can use the compiler to prevent a helper function from being able to access a state that’s private to its class, while preserving our ability to test its behavior concisely and directly.

func testPasswordCanNotContainUsername() {
    let (didPass, issue) = 
        OnboardingManager.
            validatePassword(
                password:"ValidUsername$123",
                username: "ValidUsername",
                email: "valid@example.com",
                phoneNumber: "+11235554321"
            )

    XCTAssertFalse(didPass)
    XCTAssertEqual(issue, PasswordIssue.containsUsername)
}

As long as your codebase doesn’t already abuse static state by storing important global state there already, static functions are inherently out of the flow of state, and an excellent way of ensuring that helper functions are explicit about the state they can change (only what you pass in) and so can’t accidentally hurt the flow of your program logic.

Code review should then consider any un-static-ing of methods as a potentially risky act—which should, of course, already be true since it gives the method potentially destructive access to all of the instance state!

You can, of course, continue to use your existing integration tests, but now they can be refined to their intended purpose—testing the connections between units.

  1. The test infrastructure required to run stateful integration tests of this sort involves mocked versions of every one of the objects whose visible methods are called with values computed here. The values computed here would then be inspected in the those mocks.

    See the test infrastructure examples gist.