In contrast to application software, embedded software generally have multi-threaded designs. The main driving factor behind this is a complex environment in which multiple events are occurring in parallel: user(s), inputs from peripherals or network etc. Drivers in particular are also affected in this context, maybe even more so as because they may be servicing multiple applications and HW components simultaneously. In addition, the driver models of OSs are generally not very specific about threading models, therefore the drivers become passive servers of multi-threaded and sometimes buggy clients, which makes it hard to foresee which function of the driver will be called when.

So, this context prepares almost all conditions for several race conditions. In my experience, most race conditions are caused by a few common situations, and in this post I’d like to summarize my knowledge about them.

Foundation

For this discussion, I will use the stream interface driver model of Windows CE, as I have the most practical experience with it and it’s similar to the character device driver model of Linux, and unlike Linux it’s part of a stable API. The most important interface functions of this driver model are:

• Init, Deinit & PreDeinit
• Open, Close & PreClose
• Operational functions: Read, Write, Seek, IOControl etc.

A driver can typically control multiple devices, a ubiquitous example is a serial port driver controlling COM1, COM2 etc. devices. To accommodate this, a driver typically defines a data structure for the driver (let’s call it s_driver) and a data structure for the device (s_device). In a driver there will be an s_driver instance and multiple s_device instances (1:n relationship between the s_driver and the s_device). Init and Deinit initialize/deinitialize the s_driver and Open and Close initialize/deinitialize an s_device.^[1]However, do not take this for granted, as you may also see driver designs which mix up driver- and device-related data.

Another common element of the design is a thread/routine that handles events triggered by the device HW. This is not described in the driver model of the OS, since the OS doesn’t care about how the HW interacts with the driver.

Simultaneous requests from the client(s) and from the HW

Problem: A driver may receive multiple simultaneous requests from the client(s) (i.e. user(s) of the driver) and the HW controlled by the driver. If they are not synchronized properly the driver will exhibit race conditions.

To solve this problem, mutexes should be used to synchronize accesses. Here, the most important point is the relationship of the mutex to the data it is protecting. The simplest solution would be one mutex for the s_driver and one mutex for the s_device. Whenever a data in one of these structures is going to be changed, its mutex should be locked. This rule will keep you safe whether you are in the interrupt routine handling a request from the HW, in a client-called function or in a worker thread you have in the driver.

In more complex drivers, you might have to have further data structures, e.g. for an Ethernet driver the data for packet transmission and reception can be largely independent from each other, so it can make sense to define sub-data structures under the s_device, and protect each of them by a different mutex. This will increase complexity, but probably bring about higher performance e.g. by not necessarily blocking packet transmission during packet reception.

Calling an external component

Problem: The driver is using an external component (maybe another driver). It has to release its mutex before calling this component, otherwise nested locking of mutexes may lead to performance bottlenecks or even deadlocks. But releasing the mutex can lead to a race condition within the driver.^[2]This is another pattern: your design should find the sweet spot between synchronizing too tightly which can lead to performance bottlenecks and possibly deadlocks and synchronizing too loosely … Continue reading

The solution to this problem should build up on the relationship you defined between the mutex and the data protected by it. Once you have the set of data protected by the particular mutex, you can think about when this data is “consistent”, i.e. when would it not be a problem if another routine working on the data interrupted this routine and acted upon and changed this data further. A simple example could be a list and a counter holding the number of elements in this list in the data structure. When you add an element to this list, the counter should be incremented. Between the two changes, the data structure is not consistent, and the mutex shouldn’t be released. But afterwards you may release the mutex safely.

So, if you ensure that before calling the external component, the data structure is brought to a consistent state, you may release the mutex before calling it, too. After the component returns you have to re-lock the mutex and continue with the operation. You should also take into account that the parts of data you might have copied to some local variables before releasing the mutex may not be actual anymore.

A similar situation also exists between the s_driver and the s_device. You don’t want to keep the s_driver’s mutex locked when you are working in the s_device, otherwise one device’s operations will unnecessarily block another device. Again, if you take care that you are leaving the s_driver in a consistent state, you may release its mutex before starting to work with the s_device.

Shutdown

Problem: Driver receives request to close, but there may be ongoing operations. Worse yet, more operation requests might come after the close request.

As a solution, use a two-way locking scheme. First, when you receive the close request, set a flag in the s_device so that new operations are rejected from that point on. There is a small dilemma here: this flag will reside in the s_device, which will be freed at the end of Close just before returning, so you will also lose the flag. This dilemma should ideally be solved by the fact that when a client calls Close on the driver’s handle, it mustn’t call any operation afterwards. However, some clients may be buggy in that they do call other operations after they call a Close, most commonly from their other threads. PreClose helps at this point for separating the freeing of the s_device from the setting of the flag, but ultimately you may have to fix the bug at the client-side.

Then, in Close, start waiting until all operations are finished, and close the driver. To implement this waiting, you need some kind of a counter or a list of ongoing operations. If waiting on close will be a problem in your context, the only other option you have is to reject the close request. You can also vary your design between these options (e.g. wait for a while for ongoing operations to finish before giving up and rejecting the close request). However, do also keep in mind that only few clients check the return value of the Close call.

Needless to say, for checking or setting every variable (e.g. the counter of ongoing operations or the flag for closing) in the s_device you need to lock its mutex.

The same problem also exists for the s_driver. You can use another flag in it that would be set in PreDeinit. Again you need a counter or a list of all s_device instances to ensure that you are not deinitializing the driver while some device instances are still open.

Startup

Problem: The driver might be used before it has finished its initialization.

Here, the driver model of the OS already prepares most of the solution. In contrast to Close, even buggy clients cannot request any operations before Open returns an handle. In the same way, Open cannot be called before Init returns successfully. The driver just has to care that it’s completely ready for an Open before returning from Init, and completely ready for an operation before returning from Open.

Conclusion

I believe these (simultaneous requests, calls to external components, shutdown and startup) are the most common and impactful conditions giving way to race conditions. Try to identify them in your designs, and hopefully these solution ideas will help you avoid the race conditions.

As always, I’d be very much interested in your ideas, as well as other ideas you came across in the literature. I also would like to ask if it would make sense to implement a driver framework that wraps these ideas in a more reusable form.