Follow the Real-Time Operating Model

20 October 2010 (Last Updated October 20th, 2010 18:30)

Selecting an overqualified real time operating system (RTOS) can affect a project's time to market. Comparing design outcomes between various RTOS solutions, Jerry Krasner, Embedded Market Forecasters, offers advice for medical device developers.

Follow the Real-Time Operating Model

With embedded markets becoming increasingly competitive the value of RTOS technology has shifted. Developers are confronted by greater design complexities and limited opportunity.

Trends indicate that developers no longer have the time to individually research available processors, tools and operating systems; write, compile and debug code; integrate software and hardware and test the final solution. Yet new vertical market application products are emerging weekly, placing increasing demands on embedded developers.

Gone are the days when it was considered fashionable to craft one's own operating system and tinker with building in tools. The results were predictably horrendous and characterised by unreliable products, overdue development schedules and costly staffing.

More than a decade later the industry is blessed with powerful development tools and a plethora of operating systems that offer developers so many performance choices that it sometimes leads to confusion. Today, embedded developers continue to be divided as far as the best way to satisfy their application's need for real-time control, scheduling and interrupt management.

Roll your own?

Although commercial RTOS products are plentiful and support dozens of microprocessors it is strange that developers are still encouraged to use 'roll your own' (RYO) operating systems. Over the past five years, EMF data shows that RYO design outcomes fare worse than any commercial RTOS. Yet the debate continues. The same has been shown for embedded Linux compared with non-commercial Linux.

RTOSs are crucial to the effective function of a medical device. Selecting the right one, commercially available or RYO, is a decision that requires consideration. But what questions should device developers be asking, what should they be looking for from their system and how do legal requirements affect these choices?

"EMF data shows that roll your own design outcomes fare worse than any commercial RTOS."

New research from Embedded Market Forecasters (EMF) indicates that there are measurable factors that contribute to the appropriate choice of an operating system (or RTOS) for different medical device developments. These include time-to-market, design outcomes measured by the closeness of final designs to pre-design expectations and the need to provide an accurate CDRH audit trail for product failures that can risk the existence of the company. EMF published a detailed assessment of this issue and a set of recommendations to medical device executives and product developers.

According to the results of an EMF survey of embedded developers, certain operating systems consistently outperform others in terms of helping developers get their projects to market on time or even ahead of schedule. This is also true of the value of measuring how close final design outcomes compared with pre-design expectations – poorer design outcomes usually take a toll on time to market, increased demands for customer support and removal of features. None of these are good for any product, and for a medical device company, they can be crushing.

RTOS selection challenges

Commercial operating systems form a continuum of functionality, performance and price. These operating systems range from those that offer a basic pre-emptive scheduler, contain a few key system services, are usually inexpensive, come with modifiable source code and are royalty free, to those more sophisticated operating systems that include functionality beyond the basic scheduler and can be expensive.

With such a variety of operating systems and features to choose from it can be difficult to decide which is best for a given project. Many developers make their decision based on performance, functionality or compatibility with their choice of compiler, debugger and other development tools. Many use integrated development environments (IDE) that enable them to develop over a wider range of RTOSs and to use Eclipse-based tools.

RTOS selection has been a matter of preference and convenience. Development teams tend to resist change in the absence of perceived benefits of change. Since many RTOSs offer similar technical capabilities and a broad range of software and hardware ports, there are few technically compelling reasons to change. However, if developers reach beyond the traditional criteria and look at time to market and other design outcome measures, then some RTOSs rise above others in keeping developers on time and within budget and getting the product to market faster.

Importance of time to market

Looking at comparative project costs, an OS not usually associated with the RTOS leaders can be shown to be an effective selection for medical device usage (Table 1). However, if one is dealing with mission critical design issues that require DO 178 B Level A or MILS security levels, then a more powerful and robust OS would be appropriate. One can argue that for patient monitoring, there is really no need for real-time processing.

For patient-monitoring applications, where the highest frequency response is 100Hz, there is little need for high levels of security. As long as alarm systems are designed independent of the RTOS and create alarms in any failure mode, it's not like having an aircraft falling out of the sky. Well-designed alarm systems trump the need for extraordinary OS considerations. This doesn't mean that one can grab any OS, but with hundreds of millions of ThreadX, MontaVista Linux and Nucleus applications running worldwide, one can choose at this level for many applications.

Time to market significantly affects the overall cost of a project. EMF's 2010 survey of embedded developers (528 respondents) shows that the average number of software developers per project is 14.7.

"An RTOS that is overqualified for the application may have a negative impact on time-to-market and final design outcomes."

If the average cost per engineer is $150,000 per year, then the average cost for each month of delay is $183,750. Knowing from EMF data that on average 47% of embedded designs are behind schedule by an average of 3.8 months, one can calculate the average expected loss due to design delays per project as ($185,750)*.47*3.8 = $328,177.

EMF data averaged over the past four years shows that ThreadX, Nucleus and Micrium have the highest percentages of on-time or ahead of schedule completion with fewer software developers / projects than better known OS such as VxWorks, LynxOS and Integrity. ThreadX, for example, has consistently been in the 62-70% range for on time project completion during the past four years.

Although complexity cannot be measured by lines of code alone, it was interesting to see that only Green Hills' Integrity OS averaged significantly higher total lines of code per development than the others. Notably, these calculations do not take into account the cost of lost revenues because the product was delivered late. Time to market is important in every industry segment, but in short product lifecycle segments, such as consumer products, it often makes the difference between profit and loss.

Developers need to consider which OSs produce the best design outcomes for the application under development. Table 2 presents the percentage of final designs that are within 30% of pre-design expectations. EMF believes that final designs not within this range constitute poor design outcomes. One has to balance that up-front costs to purchase a superior time-to-market RTOS are very small compared with the expenditures experienced as a consequence of late design completion, and the benefits of getting to market ahead of competitors. Given such dramatic implications, it makes sense that such analysis should become part of every company's best practices.

Spotlight on Linux

In 2007 EMF published an analysis from the extensive survey data that EMF compiles year-over-year from embedded developers. The findings and analysis were based on factual data. In 2007 the EMF report on embedded Linux generated angst among RTOS vendors who thought that Linux was an inferior product. The 2007 data was revisited and updated in 2009 showing that Linux remains a viable OS choice for most embedded developments and a preferred choice for many. The 2007 / 2009 EMF conclusions were:

  • designing with a Linux OS is no longer an expensive and risky undertaking
  • using a supported commercial Linux OS is more cost effective to an in-house RYO Linux development undertaking
  • Linux can be used in a mission-critical environment that requires MILS, EAL certification or POSIX conformance, when used in protected memory under a certified RTOS
  • the use of Linux for embedded designs is no longer restricted to a few applications.

Furthermore, EMF design outcome data demonstrated that developers can consider using a Linux OS without concern that their end product will be inferior to end products produced with a proprietary RTOS.

"For patient monitoring there is really no need for real-time processing."

EMF looked at the RoI between commercial Linux and RYO Linux developments. It was assumed that RYO Linux developments were the most cost-effective option because there are no subscriptions or support costs: simply download the code and begin development. But was this really true? Which Linux developments are really free?

Assuming that the loaded costs of commercial and RYO developers are the same, Table 3 shows that commercial Linux developments enjoy a 16.1% RoI advantage.

One can argue that an RoI based on the average total number of lines of code per project developer would provide a better comparison. EMF data shows that, on average, the total lines of code for developers using commercial Linux offerings are 44.9% larger than for RYO developments.

One can speculate as to why there is this difference in code size. One point of view is that commercial developments can add more functionality and do cooler things because they are spending less time just getting things to work.

One can argue that commercial Linux offerings enjoy a more efficient development process. Of particular interest is that all commercial Linux OS outperformed RYO Linux with MontaVista Linux having the best RoI and design outcomes.

Keep it real

Without question, traditional criteria and analysis still matter. Emotional appeals to non-applicable certification standards do considerable harm and tend to influence developers to focus away from having the RTOS selection based on meeting the specific functional requirements of the application.

An RTOS that is overqualified for the application may have a negative impact on time-to-market and final design outcomes. This is because additional capabilities, beyond what is required by the needs of the application, make these RTOSs more complicated and harder to use.

For patient monitoring applications the expected critical frequency is 100Hz, the use of a costly OS that guarantees a 10µs response is unreasonable, expensive and power hungry. Base the selection of the OS and the processor on the system's requirements. If secure communications are required, existing applications can be run on an MILS-certified OS as a guest OS without the need to rewrite the existing code.

Every embedded system of reasonable complexity will require an OS or RTOS. Managers and developers understand that it is more efficient to spend their time designing the application rather than RYO.