Watch out for compiler bugs related to volatile variables

At my new job, I've been diving back into embedded software development. During my time as an industrial automation application engineer, I stopped working quite as close to the hardware. It's definitely fun to be digging around in registers and assembly code again.

I came across this article called on Embedded.com called "When good compilers go bad, or What you see is not what you execute". It talks about all kinds of ways that compilers can end up generating incorrect (or sometimes technically correct but unexpected) source code. I've got a rule of thumb when debugging that in that vast majority of cases "you didn't discover a bug in the compiler/operating system". It's good to keep in mind that that isn't always the case.

One reference that especially caught my eye was to a paper entitled "Volatiles Are Miscompiled, and What to Do about It". They looked at 13 different C compilers and found out that all thirteen of them had issues with volatile variables.

Volatile variables are very frequently used in the embedded world. Memory-mapped hardware registers (used to read digital and analog inputs for example) are often accessed via volatile pointers in C. The volatile qualifier tells the C compiler "this variable could change at any time, so every time I ask you to read or write it, I really mean it!"

The authors of the paper found that every single compiler they tested failed to correctly implement the volatile qualifier in at least a few of their test cases. However, there are a few things you can do to help protect yourself:

• Don't turn on optimization. They couldn't prove it, but they're pretty sure that all (and definitely most) of these kinds of bugs are caused by the compiler's optimizer.
• Wrap uses of volatile variables in function calls. Because functions are (obviously) used way more than volatile variables, there tend to be very few bugs related to incorrect optimizations of function calls. Hiding volatile accesses in function calls makes it much less likely the access to a volatile variable will get optimized away (assuming the function isn't inlined).
• Don't use volatile variables for inter-thread communication, use semaphores instead. This one deserves a post all of its own.
• When the code just absolutely has to work, verify the assembly or machine code your compiler produces by hand.

One thing I really liked about the paper is that they used valgrind to help automate their testing. When I've run into some really tough to debug memory and pointer issues, valgrind has always been a huge help. I wish it were easier to use it in embedded programming projects.