nelhage

A surprisingly controversial perspective on software engineering and computers in general is the idea that There Are No Mysteries - that computers are entirely "knowable"¹ and it's primarily a matter of how deep you choose to look. (That choice may be limited by resources or access, but not by possibility.) This particularly applies to plumbing around software - npm packages, pypi modules, and Debian packages aren't magic, and actually help you figure out where things are coming from.²

Complexity is still a battle

If you always had to understand every system in its entirety, debugging might actually be intractable - the "spaghetti code" moniker to describe this arose in the 1980s.³ Instead, we generally build code in "layers of abstraction" - which sounds clever except it's really an emergent property of starting with something simple and building one layer on top of it, then doing the same thing again. The main "modern" distinction is that we're more likely to have a formal API defining the layer as part of a remote service, than having it be local function calls; even "local" services often come from a local container rather than the same machine, which allows larger subsystems to be properly isolated - even your develop-and-test "local database" is likely to be an entirely isolated machine that just happens to share a CPU.

The important part of these layers is that each layer only "sees" (communicates with) immediately adjacent ones.⁴ The term for when an upper layer sneaks through and uses a lower layer's interfaces directly is "layering violation".

The Good Stuff is the Next Level Down

Understanding how a system behaves at the surface or user level is sufficient when it is working but often falls apart when you're trying to figure out why it isn't working. A great deal of "debugging" is about peeling back that surface and looking at the mechanisms underneath to get clarity on what is "really" going on.

This isn't an argument about purity - just that the layer immediately below the visible part is what's most directly causing things, properly or improperly.

Just as the layering itself is an emergent property, so is debugging by layer - you've got something wrong at the surface so you go step by step on the surface until you get a wrong result - and then you open up that layer and do the same thing. Much of the time you can repeat that with the each lower layer - sometimes you need to synthesize inputs to keep the operations at that layer "narrow", but that's fine, as long as you can show that it corresponds to the upper level problem.

How do you get to the next level

Tooling varies depending on what you're starting with. On Linux, the "big" layer is "your user space code vs. the kernel", and the tool for looking at that boundary in particular is strace. (Realistically you are probably using libraries and frameworks above that and should look at them first! You'll normally be examining those with debuggers or simply by printing out return values. Still, the kernel/user boundary is where a lot of interesting problems happen.)

To avoid turning this into an strace tutorial, I'll just suggest you practice by running it on something known to work, like ls, and something known to fail, like ls /nonexistant and see what the differences are. (To go sideways a little, a file not existing is often merely a fact and not an error - but if it should be there in normal circumstances, it is common to program in such a way that your code assumes it is present - since you're treating the absence as exceptional, and you're going to have an error handling path anyway that can sometimes be a simpler⁵ structure.)

Higher level environments often have built-in tools - when developing javascript in a web context, you have an entire in-browser console and inspection system that lets you focus on a problematic area of a page (document) and go from there to the related code, since the problems you care about are often more directly visible (at least to end-users) from the page perspective rather than from the flow of logic.

Stories

This isn't just theory, but it helps to have actual cases to show it realistically mattering. Most of these are very situational and don't necessarily lead to modern debugging techniques, so you'll find them under Debugging Tales instead.

Other history: Literature

"The Cuckoo's Egg" and "With Microscope and Tweezers" cover the Morris Worm incident - notably, the part where Bill Sommerfeld noticed that a piece of code was running as the nobody user, and leveraged that into a clean-room duplication of the buffer-overrun attack that was one of the worm's more successful vectors. (The nobody part had mostly been dismissed by other people investigating, but turned out to be significant.)

Other history: 6.004

MIT teaches this directly in 6.004, Computation Structures which (as taught in the late 1980s) built a small computer all the way up from

• The "Digital Abstraction" - the idea that you could use amplifier circuits to slam a continuous voltage to one extreme or the other, and oversimplify⁶ those levels into 1/0 true/false logic⁷
• Simple multi-bit arithmetic units, first as chips and wires, then as small pre-made circuit boards⁸ (containing the same chips and wires.)
• An instruction set (basically a counter and some gates attached to the arithmetic units) and storage
• Toggle switches for entering code

and finally a tool chain on a host system to assemble (and later compile and optimize) higher level code to run on this hand-built system.

As you crawl "up" the stack during the course, you don't just build the lower levels, you build a belief in the lower levels - voltages representing logic levels, representing numbers, representing code, with nothing hidden⁹ about how you got there.

Nelson Elhage wrote Software Engineers should keep lab notebooks discusses the value of keeping detailed notes - more than just a screen recording, including notes on intent (on what you were trying to do or how you were interpreting results.) The article emphasizes it primarily for retrospective analysis (both immediate and longer term) but I've also found that one side effect of keeping such detailed notes (in a searchable manner) is that you can also find that snippet of awk code or those weird lsblk arguments you figured out that time, even if the problem itself is unrelated.

(See also Lab Notebooking for the Software Engineer for basic mechanisms for applying this technique.)

• Advantage over AI: snippets that actually work and you have a clear record that they did, what happened, and when - "Oh $this stopped working after 16.04 and we need to figure out $that instead" is important.