Intro to Real-Time Linux for Embedded Developers

When embedded projects call for for a real-time operating system, Linux developers often turn to PREEMPT-RT, the real-time kernel patch, to get it done.

“The PREEMPT_RT patch (aka the -rt patch or RT patch) makes Linux into a real-time system,” said Steven Rostedt, a Linux kernel developer at Red Hat and maintainer of the stable version of the real-time Linux kernel patch.

The thing is, in most cases real-time requirements on embedded projects can be met without turning to a real-time operating system, he said via email. To developers, a real-time system “does what you expect it to do when you expect it to do it.” That’s all.

“It’s about no surprises,” he said last month in his “Inside the RT patch” presentation at the Embedded Linux Conference (see the slides or video, below).

That means depending on the project’s requirements, any OS can be considered real-time. So when is the RT patch needed?

Here Rostedt discusses when to use a RTOS; compares the RT patch with mainline Linux; defines “hard” vs. “soft” real-time requirements; tells developers how to get started with the RT patch; and estimates when he expects PREEMPT-RT to be fully incorporated into the mainline Linux kernel. 

As Dave Stewart said in an ABS panel discussion, people often assume that embedded means real-time but that’s not always the case. He said, “There’s been a dawning realization that you can achieve the vast majority of what you need to with embedded without a classically-oriented real-time system.” Do you agree?

Steven Rostedt: Yes. In fact, a lot of people I’ve talked to think they need real-time for their work when they actually do not. One of my colleagues a long time ago stated that all operating systems are real-time. That is, they all have some kind of deadline, even Windows. If you hit a key and the computer doesn’t respond in say 5 minutes, you are likely to throw the computer out the window. It failed to meet its deadline. When your deadlines are big enough, pretty much any operating system will do.

Of course there’s caveats to this rule.

When and how does real-time Linux come into embedded development?

It all comes down to what your requirements are. An ATM probably doesn’t require a real time operating system, because it’s not too hard to create code that would let it make its deadlines. It just needs to be clean (non buggy) code, not real-time.

Now, if your embedded device has some “must have” deadlines to respond to then the PREEMPT_RT patch would probably be sufficient. I’ve heard that some of the Garmin navigational devices run with Linux patched with the PREEMPT_RT kernel.

What PREEMPT_RT gives you over the normal kernel is not only faster response times, but more importantly, it removes all unbounded latencies. An unbounded latency is where the amount of delay that can occur is dependent on the situation.

For example, with unfair reader writer locks, where the writer has to wait for there to be no readers before it can take the lock. Because new readers can continually take the lock at any time while the writer is waiting, the writer may never get the lock. This is a prime example of an unbounded latency on the writer.

A bounded latency is the fair reader writer lock, where new readers will block if there’s a writer waiting. This way the writer has to wait for at most a fixed number of readers (at most one per CPU minus the one the writer is on).

There’s also the case of priority inversion, but I covered that in my talk at ELC, (See the video, below).

I heard from Steve Sakoman that embedded Linux doesn’t work when the device has hard real-time requirements (and in some cases, any real-time requirements). How does the real-time patch meet those requirements when embedded Linux falls short?

Honestly, I hate the terms “Hard real-time” and “Soft real-time.” How hard is hard?  I used to work for Lockheed Martin, and there we did have the classic “hard real-time”. Basically that’s for a small component, which is part of a bigger component, but you can test every branch that the software on that component can take. It ends up being a custom piece of software, even when you start out with something like VxWorks. Probably the best real-time OS was DOS, because you can customize it to whatever requirements you need.

But once the software needs to handle a more general situation, it grows out of being mathematically provable. Then you start to lose that classic “Hard” real-time everyone talks about. There’s a lot of systems out there that call themselves “Hard” real-time, where they probably would not meet these requirements.

For those that need hard real-time, it’s usually to control something that will end up killing people if something goes wrong. Everything else pretty much does not need that classic “Hard” real-time OS.

That said, PREEMPT_RT gives you something that’s very close. I will be the first to tell you that I wouldn’t want the PREEMPT_RT kernel to be controlling whether or not the plane I’m flying on crashes. But it’s good enough for robotics, stock exchanges, and for computers that have to interface with the “Hard” real-time software. PREEMPT_RT has been used on computers that have gone into space.

PREEMPT_RT is a “hardening” of Linux. It’s far from mathematically provable, but if there’s an unbounded latency that’s in PREEMPT_RT, we consider that a bug, and work hard to fix it. Our goal is to make Linux as hard as possible. 😉 That’s probably what they meant about Linux not being able to handle real-time, as the mainline Linux kernel is filled with unbounded latencies. With unbounded latencies comes non-deterministic behavior.

Mainline Linux can do things really fast most of the time, but every so often, you get that outlier that could cause a large delay. Ever see your mouse jitter? With a properly setup PREEMPT_RT kernel, your desktop experience can be made much smoother.

I said before that a lot of people want real-time when they don’t really need it, and even fewer people need hard real-time when they think they need it. I gave a talk in Brazil once titled “Who needs Real Time Linux? (Not you!)”, where I basically stated that it’s really a small niche of people that need real-time.

One place that PREEMPT_RT comes in handy is with musicians. They were our earliest testers. I asked one person who reported a bug to test a patch, and he asked me for some advice in applying it. I asked him if he was a computer programmer, and he replied, “No, I’m a guitarist”. He told me that the RT patch was great for getting reliable recordings, as jitter from mainline Linux would cause a scratching sound. I thought it was really cool that non computer focused people had a use for our work.

How is the RT patch used in embedded development?

That’s easy. Install it, and run your program. That’s it!  The big advantage of PREEMPT_RT over other real-time Linux implementations is that PREEMPT_RT makes Linux itself real-time, where as the others usually create a small micro kernel that runs like a hypervisor and Linux runs as a task. That’s not really Linux. Your real-time tasks must be modified to communicate with the micro kernel.

With PREEMPT_RT, if your program runs on the stock Linux kernel, it runs on PREEMPT_RT as well. You can easily make a “real-time bash”. Our kernel supports all the posix commands to modify the priority of the tasks in your system. PREEMPT_RT makes interrupts run as threads, so you have the ability to modify the priorities of interrupts as well, and have user space tasks run at even a higher priority than interrupts. Having interrupts run as threads lets you prioritize the interrupt handlers even when the hardware does not support it.

What’s your advice to embedded developers who want to use the RT patch?

Download and install it 🙂 It’s that easy. If you can work with Linux, than using the RT patch is no different.

When does it make more sense to work with a distro that has already incorporated the RT patch, versus starting from scratch?

As I work for Red Hat, I would obviously say to use a distro 🙂  But as Red Hat focuses on servers, I would recommend to the embedded world to find someone like a Thomas Gleixner and contract them to get you going.

There’s actually a small number of PREEMPT_RT developers and we are already overworked. We try to give advice on the mailing lists when we can, but then there’s those companies that take advantage of that. If you report bugs, we are glad to fix them, but as most users (besides musicians) of the PREEMPT_RT patch are companies, they can afford to hire someone to help them out.

We love to help the community, but our customers come first. If you plan on using the RT patch, it would be prudent to budget for an expert (person or company) to help you out. As with all real-time projects, they are easy to screw up if you do not know what you are doing. Real-time is the gun to shoot yourself in the foot with.

What challenges still remain for RT Linux in embedded development and how are they being addressed?

I don’t know how embedded development is any different than normal development. We test PREEMPT_RT on x86, ARM and PowerPC regularly. Now, we still have things remaining that needs to be addressed.

The biggest change still outside of mainline is the conversion of spin_locks into mutexes. That changes a lot of assumptions about spin_lock side effects. For example, they no longer disable preemption. There’s some code in the kernel that assumes that.

We are slowly solving all the issues, and it seems that today the RT patch is much more acceptable than it was 5 years ago. Back then we had to get parts of the RT patch into the kernel via Trojan horses. We would do a clean up of the kernel code and slip in some RT patch features. Linus knew we were doing this, but as long as we helped clean up the rest of the kernel he was fine.

In fact, the quality of Linux on large scale SMP machines is largely due to the RT patch. PREEMPT_RT can trigger bugs that may take 64 to 1024 CPUs to trigger. Because we found them first, Linux can easily run on 64 and 1024 CPU machines.

Over the years PREEMPT_RT has proven itself as an asset to Linux. Because real-time requires a clean code base, the RT patch developers spent a lot of time cleaning up Linux.

Real-time is also much more sensitive to bugs than the mainline kernel is, thus the PREEMPT_RT patch was able to reliably trigger bugs in mainline that may take months to trigger normally. We would then fix the bug and send a patch to mainline. A bug that the RT patch would trigger in a few hours may take a few months to trigger in normal Linux. If it were triggered there, it may not be easily reproducible, and probably forgotten.

Linux really does owe a lot of gratitude towards the RT patch. 🙂