Debugging Makefiles (1)

Newsflash

Since this article disappeared from the web I'll post it here again:

Debugging Makefiles

Practical debugging tips for real-world development

By John Graham-Cumming

John is a cofounder of Electric Cloud, which focuses on reducing software build times. He can be contacted at This email address is being protected from spambots. You need JavaScript enabled to view it..


Donald Knuth once said that, given a choice between two languages, he'd choose the one with the better debugger. For that reason, he'd probably never choose to use Make. Its debugging facilities are virtually nonexistent.

In this article, I address this problem by providing practical debugging tips I've used to debug real-world Makefiles. To illustrate, I use GNU Make because of its wide platform support and rich language. Unfortunately, that rich language doesn't include much in the way of debugging. In fact, GNU Make's -d command-line flag seems to produce output more useful to the author of GNU Make than to someone facing Makefile problems.

The only GNU Make command-line option that's really useful for Makefile debugging is -p, and all it does is dump the contents of GNU Make's built-in rules and macros, and any rules or macros defined in the Makefile. Granted, that's an enormous amount of information that can be useful in debugging, but hardly counts as a debugger.

Broadly, Makefile debugging problems fall into three categories:

  • What's the value of macro X and how did it get it?
  • Why did file foo get built?
  • How does Make log file output relate to the rules in the Makefile?

What's the Value of Macro X?

Frequently, Makefile debugging requires getting the value of a Make macro. Because Make doesn't have a debugger, it's hard to find out the value of a macro without a great deal of searching through Makefiles for macro definitions.

Look at this example Makefile:

 X=$(YS) hate $(ZS)
 Y=dog
 YS=$(Y)$(S)
 Z=cat
 ZS=$(Z)$(S)
 S=s
 .PHONY: all
 all:
    @echo done

If you follow the definitions of X, Y, S, YS, and ZS, you can discover that X has the value dogs hate cats. In a real Makefile, unwinding the definition of a macro is next to impossible. The simplest way to find out the value of a macro in a Makefile is to use the $(warning...) function. When GNU Make encounters $(warning...), it prints the message contained in the warning along with the name of the Makefile and line number. $(warning...) is GNU Make's printf-style of debugging.

Just adding $(warning X is $(X)) as the last line of this Makefile causes GNU Make to print:

Makefile:11: X is dogs hate cats

But the printf-style is just as inflexible in Make as it is in other languages. The Makefile has to be constantly modified and rerun while digging down to discover the cause of a problem. What's needed is a more interactive approach.

You can add a GNU Make pattern rule that enables the name (or names) of the macros to be printed to be specified on the GNU Make command line. This approach is more flexible than modifying the Makefile; all you need is a single pattern rule to print the value of any macro. Adding this line to the previous example Makefile means that it's possible to type make print-X to get the value of X, or even make print-X print-XS print-Y to get the values of a number of macros:

print-%: ; @echo $* is $($*)

This works because GNU Make attempts to build the goals specified on the command line. If you type make print-X, the print-X goal matches the print-% rule, with % matching the name of the variable (in this case, X). The part matched by % is stored in the GNU Make automatic variable $*; $* is used twice in the command for the print-% rule: First it's used to print the name of the variable, then it's used to get that variable's value by writing $($*). When $($*) is seen, GNU Make expands $* (for example, to X), then expands the resulting macro reference (for example, $(X)).

There are two problems with the print-%-style:

  1. If the macro's value changes inside a Makefile (which is an unusual, but not impossible, event), print-% only gives the final value of the macro (the value of the macro once all the Makefiles have been parsed). The only good solution to this is to revert to the $(warning...) style.
  2. Also, GNU Make lets macros have local scope in the form of target-specific macros. The example Makefile can be modified to include a target-specific value for the X macro in the scope of the all rule:
.PHONY: all
all: X = $(ZS) love $(YS)
all:
   @echo done

If all is being built, then X has a different value in that rule than anywhere else in the Makefile. The print-% method can be extended to print the value of target-specific macros:

print-%: ; @$(error $* is $($*))
$(filter-out print-%,
       $(MAKECMDGOALS)): 
   $(filter print-%,
       $(MAKECMDGOALS))

First, the print-% rule has been modified so that when it runs, it prints the same message as before, but stops GNU Make with an error. This is required because to get the target-specific value of the macro you need to tell GNU Make to build those targets, and since you don't actually want GNU Make to build (you just want the target-specific macro scope to be set up), the fatal error created with $(error...) stops GNU Make once it has printed the value of the macro requested.

To get the target-specific macro scope, the improved version specifies that the print-% rule is a prerequisite of the target for which there may be a target-specific macro. This is done using normal GNU Make syntax. For example, if you want to print the value of X as it's seen from within the all rule, type make all print-X. The second line in this improvement translates to:

all: print-X 

The $(filter-out...) first removes everything on the command line (stored in the GNU Make variable MAKECMDGOALS) that doesn't match the pattern print-% (for example, just the all in all print-X); the $(filter...) does the opposite: It just keeps the things that do match print-% (for example, just the print-X in all print-X).

Because target-specific macros are inherited by any prerequisites of a target, writing all: print-X means that the print-X rule (which is handled by the pattern rule print-%) will have any target-specific macros defined for all. That's exactly what you want, and typing make all print-X on the aforementioned example outputs:

Makefile:12: *** X is cats love dogs. Stop.

The target-specific macro-aware version of print-% can still output the global value of a macro by omitting any targets on the command line. For example, typing make print-X outputs the global value of X:

Makefile:12: *** X is dogs hate cats. Stop.

In addition to the value of a macro, it can also be helpful to find out where it was defined (for example, was it defined in a Makefile, from the environment, or perhaps overridden on the Make command line?).

GNU Make provides the $(origin...) function that returns a string describing how a macro was defined. A modification to the print-% rule makes it print this information in addition to the value of the macro:

print-%: ; @$(error $* is $($*) (from $(origin $*)))

And so make print-X now prints:

Makefile:12: *** X is dogs hate cats (from file). Stop.

The (from file) indicates that X was defined in a Makefile. If X had come from the environment, it would read (from environment). If X were overridden on the Make command line—for example, by typing make print-X X=foo—you would see this output:

Makefile:12: *** X is foo  (from command line). Stop.

One final enhancement to print-% is to get it to print the definition as well as the value of a macro. GNU Make's confusingly named $(value...) function does just that:

print-%: ; @$(error $* is $($*) ($(value $*)) 
                        (from $(origin $*)))

Now typing make print-X gives this useful debugging output:

Makefile:12: *** X is dogs hate cats  ($(YS) hate $(ZS))
                        (from file). Stop.

For more about GNU Make macro debugging, see my article "Makefile Debugging: Tracing Macro Values" (www.jgc.org), which shows how to make GNU Make divulge every place a macro is used in a Makefile.

Why Did File foo Get Built?

Here's an example Makefile that builds two object files (foo.o and bar.o) from corresponding C source files (foo.c and bar.c). The two object files are linked together to create an executable called "runme":

 .PHONY: all
 all: runme
 runme: foo.o bar.o
   @$(LINK.o) $^ -o $@
 foo.o: foo.c
 bar.o: bar.c
 %.o: %.c
   @$(COMPILE.C) -o $@ $<

Now suppose that runme, foo.o, and bar.o are all up to date. Altering foo.c clearly rebuilds foo.o and relinks runme. In this example, you can follow the chain of dependencies and see that runme being relinked could have been caused by a change to foo.c. In a larger Makefile, following dependencies is next to impossible.

If you used GNU Make's -d option to follow the chain foo.c caused foo.o to build, which caused runme to relink, you'd be faced with walking through 556 lines of debugging information. Imagine trying the same thing on a real Makefile. However, a small change to the Makefile causes GNU Make to reveal exactly the chain of events. Inserting the following in the Makefile causes GNU Make to output information about each rule that's being executed, and why it was executed:

OLD_SHELL := $(SHELL)
SHELL = $(warning [$@ ($^) 
            ($?)])$(OLD_SHELL)

Here's the output when foo.c had just been modified but everything else was up to date:

Makefile:11: 
    [foo.o (foo.c) (foo.c)] 
Makefile:5: [runme (foo.o bar.o) 
    (foo.o)]

There's a lot of information in those lines. Each line has the name of the Makefile and the line number of the rule that's executing (line 11 is the pattern rule %.o: %.c used to build foo.o from foo.c, and line 5 is the rule to link runme). Within the square brackets there are three pieces of information: the target that's being built (first the Makefile is building foo.o and then it's building runme); then the full list of prerequisites of that target (so foo.o is built from foo.c, and runme is built from both foo.o and bar.o); finally, there are the names of any prerequisites that are newer than the target (foo.c is newer than foo.o, and foo.o is newer than runme).

The second piece of information in parentheses gives the reason a target was being built: It's the list of newer prerequisites that force a rebuild. If the parentheses are empty, it indicates that there are no newer prerequisites and that the target is being rebuilt because it does not exist.

How this works is what I call the "SHELL hack." GNU Make stores the path of the shell it uses to execute commands in the SHELL variable. It's possible to modify SHELL to set your own shell. If you do, GNU Make checks the value of SHELL every time it is about to execute a rule. That checking per rule gives us the opportunity to do something on a per-rule basis. In the aforementioned code the actual shell is stored in OLD_SHELL by grabbing the value of $(SHELL) using an immediate assignment (:=). Then the SHELL is modified by preprending it with a call to $(warning...). The $(warning...) uses three GNU Make automatic variables to print the name of the target being built ($@), the complete list of prerequisites ($^), and the list of prerequisites that are newer than the target ($?).

Since GNU Make expands SHELL every time it wants to run a rule, and it does this after setting the automatic variables for the rule, the $(warning...) gets expanded giving the output shown earlier. This SHELL hack is powerful and can be used to great effect in relating GNU Make log file output to Makefiles.

How Does Make Log File Output Relate?

In the previous section, a complete build (for example, where runme, foo.o, and bar.o were missing before typing make) produces no output. That's because all the commands in the Makefile were prefixed by @, which prevents GNU Make from echoing them.

GNU Make's -n option provides a way to echo all the commands without executing them. This is handy for dry runs of the Makefile. The output from make -n on the example Makefile is:

 g++ -c -o foo.o foo.c
 g++ -c -o bar.o bar.c
 cc foo.o bar.o -o runme

In this example, it's relatively easy to relate these lines to specific rules in the Makefile; however, it can be hard on large, real-world Makefiles. And since -n doesn't actually run these commands, it's no help when trying to figure out why a rule failed to execute correctly.

One solution is to replace every @ with $(AT). Setting AT to @ in the Makefile makes it behave as before. Setting AT to blank means that commands are echoed. For example, the Makefile can be modified to use $(AT) instead of @:

 .PHONY: all
 all: runme
 runme: foo.o bar.o
   $(AT)$(LINK.o) $^ -o $@
 foo.o: foo.c
 bar.o: bar.c
 %.o: %.c
   $(AT)$(COMPILE.C) -o $@ $<
 AT := @

This behaves exactly as before, except that it's possible to override the definition of AT on the command line, causing GNU Make to print the commands it is executing. Running make AT= effectively strips the @ from each command and produces the same output as make -n, except that the commands are actually run.

Of course, modifying every command in a Makefile might not be realistic.

The SHELL hack can be modified to cause GNU Make to output command information. Almost all shells have an -x option that causes them to echo the commands that they are about to execute. By modifying SHELL to include -x (and a $(warning...)) that outputs the location of the rule in the Makefile) and wrapping the modification in an ifdef, you can enable/disable detailed debugging information as needed. Here's the SHELL hack for command dumping:

 ifdef DUMP
 OLD_SHELL := $(SHELL)
 SHELL = 
    $(warning [$@])$(OLD_SHELL) -x
 endif

If DUMP is defined then SHELL is modified to add the -x option and output the name of the target being built. If foo.o, bar.o, and runme are missing, typing make DUMP=1 produces this output using the SHELL hack:

 Makefile:11: [foo.o]
 + g++ -c -o foo.o foo.c
 Makefile:11: [bar.o]
 + g++ -c -o bar.o bar.c
 Makefile:5: [runme]
 + cc foo.o bar.o -o
 runme

Each file being built is in square brackets with the name of the Makefile and the location where the rule to build that file can be found. The shell outputs the exact command to be used to build that file (prepended by +).

In just four lines of GNU Make code, you can turn an empty log file into one with complete file, line number, and command information. This is useful when debugging a failing rule. Just rerun the Makefile with DUMP=1 to find out what was being built and where it was in the Makefile. The only disadvantage to the SHELL hack is that it slows GNU Make. That's because GNU Make tries to avoid using the shell if it can; for example, if GNU Make is doing a simple g++ invocation, then it can avoid the shell and execute the command directly. If SHELL is modified in a Makefile, GNU Make disables this optimization.

Extra Resources

If these ideas whet your appetite for better GNU Make debugging, look at the GNU Make Debugger project (http://gmd.sf.net). GMD is an interactive debugger for GNU Make written entirely using GNU Make functions that provides breakpoints and the ability to examine the value and definition of any macro at runtime.

And if you do something really cool building on the SHELL hack, drop me a line. I'd love to hear about it.