Code Audit

Essays: Code Audit

Code Review is a fundamentally incremental process, where you start with working code, and analyze proposed changes with the goal of continuing to have working code. What if instead you've been handed an entirely unfamiliar chunk of code - from a subcontractor, a vendor, a founder, or "the Internet" - and it has grown and evolved without ever having been subjected your review processes?

In the extreme case, you only have the code, but perhaps the only testsuite the code has is that "it's working in production already", and it's now your problem to keep it that way.

Testing

Obviously1 your first task is going to be to start coming up with tests that represent "it works in production" in a more fine-grained way - even if that's just "stand up a staging system, run these commands against it" (in a repeatable, and preferably automated, way.) This leans into the idea that "what the system does now is Correct because someone is paying for it", which isn't the most sophisticated expression of intent but it's a fine starting point.

Documentation

If you have documentation you're in better than average shape, as it's usually a higher level expression of what features people cared about enough to write down - it can also capture "what we thought it should do" which can be clearer guidance than "what it happens to have ended up doing". (This is also why documentation is more important than you think it is for highly technical software products - trying to explain what's going on can lead to "huh, that is stupid when you say it out loud" Ah-Hah!-moments that can feed directly back into valuable customer-facing improvements. This is also why many documentation people have an inner product manager that you'd be well served to take advantage of...)

The Code

Once you're ready to look at the code, you'll probably want to go back and forth between identifying the structure and figuring out where things go, and examining the details of how this is accomplished. Ideally you're recording this analysis for your coworkers, and producing internal documentation for future use - while you may get a "good enough/can't use this" result, it's not the most important part, especially if you do end up actually accepting the code. Also, recorded decisions are good draft material for a more formal auditing policy guide, if you end up doing a lot of this work.

Code Structure

Depending on the kind of code you're working with, the structure may be more discoverable "outside-in" rather than "top-down", but in either case you want to start looking for

This structure should provide guidance for what kinds of tests you should be writing and which parts of the system you want to be able to demonstrate still work after future changes.

(In an system where all changes go through code review, you're generally showing that "all tests still pass" - for an initial audit like this, you are instead likely to write tests that seem entirely reasonable and should pass but don't - so you need to have some way of tracking "expected" failures, so you can put pressure on fixing them soon, even if you can't do it immediately.)

Code Details

If you have fully automated tooling around your coding standards (formatting, cyclomatic complexity, comment density, type annotations) it's worth giving those a shot to "see what happens". If they don't fail outright, they can give you an initial view of what parts of the system are likely to need attention soonest; if they do, then at least you have a clear early cleanup task to schedule.

Security is usually a system design issue, but there are things that can stand out directly in code - and that tools can find - and there are often domain-specific checklists like OWASP that can suggest areas to look at where even shallow examination can expose problems. These are often in the form of "simple industry-wide problems that we should have gotten beyond, some time in the last three decades", like "examine all sql queries for injection attacks" or "be suspicious of all explicit memory allocation".


  1. Michael Feathers, "Working with Legacy Code" specifically coins the term "Legacy code" to mean "any code that doesn't have sufficient tests for you to safely make real changes to it."