The Conifer Systems Blog

The Voodoo Magic School of Software Engineering

18 comments

Computers are deterministic.

Well, OK, not always. A single-threaded, sequential program that doesn’t interface with the outside world is deterministic, but once user input, hardware devices, and networking get involved, these create asynchronous events that can be nondeterministic in character. The same is true if two programs, each independently deterministic, write to the same files in the file system at the same time. Preemptive multithreading adds more nondeterminism, and computers with multiple CPU cores add still more.

At the same time, we have also built a whole family of software abstractions that take these initially nondeterministic constructs and push them back into the realm of determinism. Yes, user input is asynchronous, but if the window system puts it in a queue instead of giving us a callback, and we only have a single thread pulling messages out of that queue, we’re back to deterministic behavior. The arrival of network packets is nondetermistic, but if we use blocking TCP sockets, the operating system takes care of most of the dirty work for us and all we have to do is send and receive bytes. A 3D graphics processor works in parallel with the CPU, but when you call glReadPixels, the OpenGL driver first waits for the previous rendering to finish so it doesn’t give you random incompletely rendered garbage.

Multithreading remains a hard case, but you aren’t forced to use it, and with careful use of mutexes, semaphores, etc., multithreaded programs can be perfectly deterministic.  We can also hope that future programming environments will be able to automatically extract more parallelism from our software, so that we won’t have to write multithreaded code to get speedups on multi-core CPUs.

Even if your program has nondeterministic behavior, though, one simple fact remains: everything that happens on a computer happens for a reason.  There is a logical explanation for why it happened.  (That explanation may be “hardware failure” or “cosmic ray”, but these are rare, and even so, they are still explanations.)  It may be hard to determine that reason, but with enough digging, it is usually possible.

So, it may be tempting to take a sledgehammer to your computer when it misbehaves, but when we have our “professional software engineer” hats on, we ought to be able to do better.

Let’s take GNU make as an example.  If you’re read our whitepaper, or even if you’re just an occasional user of GNU make, you probably know that occasionally a build fails “for some weird inexplicable reason.”  You type “make”, and instead of a correct binary popping out, you get a strange error message.  Or, worse, it looks like it succeeded, but you get a binary that doesn’t run.  So you type “make clean” and “make”, and then hope it will work this next time around.

So say that second try fails… what now?  Well, you can send an email to other people and see if they’re hitting the same problem.  You can look at the status of your automated build system and see whether it’s hitting the same failure.  But suppose the problem really does seem to be unique to your computer.  It’s possible your source control tree might be corrupted in some strange way.  (This has happened to me a good number of times.)  You might type “p4 sync -f”, but even that only fixes files that exist in the depot — it doesn’t delete files that exist but shouldn’t.  So maybe you “rm -rf” your entire tree and get a brand new one from scratch.  Perhaps that will fix it!  If it doesn’t, well, then you’re really in trouble…

At this point it’s tempting to throw up your hands in despair and admit defeat.  But if we really want to keep going… well, does it happen if I do the build on another computer?  I could reinstall my OS — maybe that will fix it?

This is, in a nutshell, what I call the “voodoo magic school of software engineering.”  We run into a problem — a build fails.  Rather than understanding the problem, determining its root cause, and addressing the root cause so it won’t happen again, we flail about, trying random stuff in the hope that something will magically fix it.  When we get to the point of “let’s reinstall the OS — maybe that will help!”, we might as well just be sticking pins in a voodoo doll.

We may well “fix” the problem this way, but we won’t ever determine the root cause.  If we don’t determine the root cause, how can we know that the problem won’t happen again to someone else or even to ourselves?  How do we know that our “fix” really fixed the problem and didn’t just paper it over, ready to reemerge at any moment?

We don’t need to track down the root cause of everything that goes wrong on a computer.  When I load some random web page and it comes up garbled, I usually just hit “refresh” and often it magically works on the second try.  Or, sometimes I come back to the site later and hope that the site owner has fixed their bug.  Either way, I don’t waste time trying to figure it out or even report the problem, unless the web site is really important to me.

That’s fine and well when we are talking about someone else’s web page.  Someone else’s web page — someone else’s problem.  But when we’re talking about our own software, our own web page, our own build flow — anything that affects our day-to-day workflow, or anything that we ship to customers and must take responsibility for — we owe it to ourselves as professional software engineers to drill down into problems and address their root causes so they don’t happen again.

When your build fails, it didn’t fail for an “inexplicable reason.”  Sometimes your tree really is corrupted and “rm -rf” is the correct answer, but even if so, how did your tree get corrupted in the first place?  For example, did some previous build step delete some files by accident?  Did some runaway script corrupt it?  Sometimes the root cause may just turn out to be user error, but even so, it’s helpful to understand our own mistakes so we don’t repeat them.

But more often, when these things happen, I find that there is a real problem to be fixed.  A common example is a makefile with incomplete dependencies.  If you’re missing a dependency, things may not be rebuilt correctly, and there’s no telling what might go wrong after that.  Tracking down the missing dependency isn’t a waste of time.  It’s an investment in the future.


Written by Matt

September 11th, 2008 at 2:14 pm

18 Responses to 'The Voodoo Magic School of Software Engineering'

Subscribe to comments with RSS or TrackBack to 'The Voodoo Magic School of Software Engineering'.

  1. […] Normally, when debugging, I would recommend starting from first principles and trying to understand the problem’s root cause before doing anything else.  Without a complete understanding of the bug, your chances of implementing a complete and correct fix are poor.  The last thing you want to do is apply a random fix that “seems to work” without understanding why it works; at that point, you’re a member of the Voodoo Magic School of Software Engineering. […]

  2. […] for the root cause of each problem and fix the root cause so the problem cannot happen again.  (I’ve written previously on this topic.)  One simple method Toyota adopted to identify root causes is called the “5 Whys.”  […]

  3. Hi and welcome to my website . I’m Teresa.
    I have always dreamed of being a writer but never dreamed I’d make a career of it. In college, though, I assisted a fellow student who needed help. She could not stop telling me how well I had done. Word got around and someone asked me for writing help just a week later. This time they would compensate me for my work.
    During the summer, I started doing research papers for students at the local college. It helped me have fun that summer and even funded some of my college tuition. Today, I still offer my writing services to students.

    Professional Writer – Teresa Martins – //collegewriterz.com/]Collegewriterz Team

    teresamart

    5 Feb 19 at 9:02 am

  4. My name is Noah Wilkinson. And I am a professional Content writer with many years of experience in writing.

    My main focus is to solve problems related to writing. And I have been doing it for many years. I have been with several groups as a volunteer and have assisted people in many ways.
    My love for writing has no end. It is like the air we breathe, something I cherish with all my being. I am a full-time writer who started at an early age.
    I’m happy that I`ve already sold several copies of my poems in different countries like USA, Russia and others too numerous to mention.
    I also work in a company that provides assistance to many clients from different parts of the world. People always come to me because I work no matter how difficult their projects are. I help them to save energy, because I feel happy when people come to me for writing help.

    Academic Writer – Noah Wilkinson – Myessaypartner Corps

    eesawalt

    14 Feb 19 at 5:14 am

  5. Hey, my name is Cynthia!

    I`m an academic writer and I`m going to change your lifes on?e and for all
    Writing has been my passion since early years and now I cannot imagine my life without it.
    Most of my poems were sold throughout Canada, USA, China and even India. Also I`m working with services that help people to save their time.
    People ask me “Hey, Cynthia, I need your professional help” and I always accept the request, `cause I know, that only I can solve all their problems!

    Academic Writer – Cynthia Nava – Weuniteconference Confederation

    cynthianav

    18 Feb 19 at 9:00 am

  6. Hi everyone , I’m Penny.
    Welcome to my website . I started writing in middle school after a creative writing assignment for my English teacher. I did creative writing for a while before I thought about doing something else.
    I had always loved doing non-fiction writing because I’m passionate about learning. When you combine writing ability with a love of learning, research paper writing only makes sense as a job.
    I’m passionate about assisting the students of the future in their school career. When they don’t like their assignment , I am there to help.

    Penny – Academic Writing Help – //www.mantecateachers.com/]Mantecateachers Corp

    pennyallis

    20 Feb 19 at 12:42 pm

  7. Onetogel

    15 Mar 19 at 1:31 pm

  8. Riatoto

    15 Mar 19 at 1:32 pm

  9. Champion4d

    17 Mar 19 at 4:48 am

  10. ???????,??????????! .

    Augustpsk

    25 Apr 19 at 11:11 am

  11. Welcome to my blog ,
    I’m Greta Brock.
    If you’ve ever been too busy and couldn’t finish a research paper , then you’ve come to the right place. I work with students in all areas of the writing technique. I can also write the paper from start to finish.
    My career as a scholarly writer started during high school. After learning that I was very able in the field of academic writing, I decided to take it up as a profession.

    Skilled Academic Writer- Greta- Thescientistexpo Corps

    gretabrock

    7 May 19 at 5:32 am

  12. I’m an expert writer who loves to bring smiles to people’s face.

    Writing is what I do for a living and I am so passionate about this. I have worked with several companies whose mission is to help people solve writing problems.
    I love traveling and have visited several places in the past few years.
    I’m happy to have written several books that have contributed positively to the lives of many. My books are available in several parts of the world. And I’m currently working with companies that help people save energy. Being a part of this team has open more opportunities for me to excel as a writer. I have worked with different people and met many clients as a writer.
    I can handle any kind of writing and provide nothing but the best. People come to me all the time to ask if I can solve their assignment problems and I accept. I find pleasure in assisting them to solve their problems as a writer.

    Professional Academic Writer – Greta – Thescientistexpo Company

    gretabrock

    13 May 19 at 2:05 am

  13. I’m an expert writer who loves to bring smiles to people’s face.

    Writing is what I do for a living and I am so passionate about this. I have worked with several associations whose mission is to help people solve problems.
    I love traveling and have visited several countries in the past few years.
    I’m happy to have written several books that have contributed positively to the lives of many. My books are available in several parts of the world. And I’m currently working with service providers that help people save time. Being a part of this team has open more opportunities for me to excel as a writer. I have worked with different people and met many clients as a professional.
    I can handle any kind of writing and provide nothing but the best. People come to me all the time to ask if I can solve their assignment problems and I accept. I find pleasure in helping them to solve their problems as a professional.

    Professional Academic Writer – Beverly Pierce – Sergreport Team

    beverlypier

    20 May 19 at 8:50 am

  14. Hi guys, my name is Jean!

    I`m a professional writer and I`m going to change your lifes on?e and for all
    Writing has been my passion for a long time and now I can`t imagine my life without it.
    Most of my poems were sold throughout Canada, USA, Old England and even India. Also I`m working with services that help people to save their time.
    People ask me “Hey, Jean, I need your professional help” and I always accept the request, `cause I know, that only I can solve all their problems!

    Professional Academic Writer – Jean Hartman – Byrumsocialstudies Confederation

    jeanhartm

    28 May 19 at 3:42 am

  15. Hello ,
    I’m Amber Crouch.
    If you’ve ever been too busy and couldn’t finish a research paper , then you’ve come to the right place. I assist students in all areas of the writing steps . I can also write the paper from start to finish.
    My career as an academic writer started during my school years . After learning that I was very capable in the field of academic writing, I decided to take it up as a profession.

    Skilled Academic Writer- Amber- Rhythmnmotionstudio Company

    ambercrou

    3 Jun 19 at 3:09 am

  16. Hi everyone , I’m Amber.
    Welcome to my homepage . I started writing in high school after a creative writing assignment for my English teacher. I did creative writing for almost a year before I thought about doing something else.
    I had always loved doing research assignments because I’m passionate about learning. When you combine writing ability with a love of learning, dissertation writing only makes sense as a job.
    I’m passionate about helping the students of the future in their school career. When they don’t have time for their paper , I am there to help.

    Amber Crouch – Academic Writing Help – Vahealthchartbook Company

    ambercrou

    10 Jun 19 at 3:00 am

  17. Hello and welcome to my webpage. I’m Liana.
    I have always dreamed of being a book writer but never dreamed I’d make a career of it. In college, though, I aided a fellow student who needed help. She could not stop complimenting me . Word got around and someone asked me for to help them just a week later. This time they would compensate me for my work.
    During the summer, I started doing research paper writing for students at the local college. It helped me have fun that summer and even funded some of my college tuition. Today, I still offer my research paper writing to students.

    Professional Writer – Liana Lynn – Literaturematters Confederation

    lianaly

    11 Jun 19 at 3:39 am

  18. Hello All
    Thanks for checking out my writing website . My name is Amiee.
    I have worked a long time in this niche. My aptitude for writing started at a young age. I wrote journaled as a child and eventually went on to work with my school newspaper.
    This early tryst into journalism eventually led me to academic writing. There is plenty of work for skilled writers. I specialize in research papers , but have the skills to do all types of academic writing.
    Contact me for more information about rates and a price quote. I’m looking forward to helping you.

    Academic Writer – Amiee Bernard – Jenniferpalmerstudio Band

    amieebern

    12 Jun 19 at 4:56 am

Leave a Reply