The Conifer Systems Blog

Our GNU make whitepaper mentions that, if you’re using Visual C with GNU make, you can use the /showIncludes compiler option to help generate .d files, much like you can with the -MD option to gcc.  I thought I’d post a Python code snippet to illustrate.

One thing that’s important to remember is that, as the Visual C documentation points out, /showIncludes prints its results to stderr and not stdout.  This means that you cannot simply redirect stdout to a file and then parse it.  You generally don’t want to redirect stderr to a file, because this will hide other error messages.  Also, there’s no sense in creating a bunch of temporary files that you’ll just have to delete later.

Fortunately, the Python subprocess module gives us extensive control over redirection of stdin, stdout, and stderr, so we can simply write a wrapper script around the compiler that takes care of everything.  We’ll create a script named cl.py that wraps cl.exe.  It will automatically add the /showIncludes option and merge the stdout and stderr from cl.exe into a single pipe that we can parse for /showIncludes information.

cl.py‘s usage is very simple.  Normally, you might run the command cl /c foo.c to create foo.obj.  Now you run python cl.py /c foo.c to create both foo.obj and foo.d.

Without further ado, here’s the script. Enjoy!

# cl.exe wrapper to create .d file from /showIncludes
# Python 2.5 or later required
from __future__ import with_statement
import sys, os, subprocess

# Determine path for .obj and .d files
# This is probably not 100% robust; you may need to tweak it depending on how
# you are invoking cl.exe.
cmdline = sys.argv[1:]
source = None
target = './' # default is current directory
for arg in cmdline:
    if arg.startswith('/') or arg.startswith('-'):
        # A compiler flag
        if arg[1:3] == 'Fo':
            target = arg[3:].replace('\\', '/')
    else:
        # This must be the source file name (assume there is only one)
        source = arg
if target.endswith('/'):
    # Default object file name is source file with extension changed to .obj
    target = target + source[0:source.rfind('.')] + '.obj'
if target.startswith('./'):
    target = target[2:] # chop off unnecessary ./ prefix

# Name of .d file is name of object file with extension changed to .d
depend_file = target[0:target.rfind('.')] + '.d'

# Build cl.exe command line with /showIncludes
# Assumption: cl.exe is in the user's PATH
cmdline = ['cl.exe', '/showIncludes'] + cmdline

# Launch the cl.exe process with stdout/stderr merged into a single pipe
p = subprocess.Popen(cmdline, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# Parse the cl.exe output to build up a set of dependencies
deps = set()
for line in p.stdout:
    if line.startswith('Note: including file:'):
        dep = line[21:].strip()
        dep = os.path.normpath(dep)
        dep = os.path.normcase(dep)
        dep = dep.replace('\\', '/') # use forward slashes for path separators
        dep = dep.replace(' ', '\ ') # escape spaces in paths with a backslash
        deps.add(dep)
    else:
        sys.stdout.write(line)

# Wait for cl.exe to exit, then return an error if it failed
ret = p.wait()
if ret != 0:
    sys.exit(1)

# Write the .d file, one dependency per line
with open(depend_file, 'wt') as f:
    for dep in sorted(deps):
        print >>f, '%s: %s' % (target, dep)

One warning: spawning an extra Python process on each source file can hurt performance.  Ideally, something along the lines of the above would be built directly into GNU make, but I digress.