[PlanetCCRMA] Fwd: [ANNOUNCE] 3.0-rc7-rt0

Fernando Lopez-Lezcano nando at ccrma.Stanford.EDU
Tue Jul 19 18:01:33 PDT 2011

Aha.... finally!
So there's now hope for fc15 + rt kernels + systemd...

-------- Original Message --------
Subject: [ANNOUNCE] 3.0-rc7-rt0
Date: Wed, 20 Jul 2011 02:37:17 +0200 (CEST)
From: Thomas Gleixner <tglx at linutronix.de>
To: LKML <linux-kernel at vger.kernel.org>
CC: linux-rt-users <linux-rt-users at vger.kernel.org>,        Peter 
Zijlstra <peterz at infradead.org>, Ingo Molnar <mingo at elte.hu>, 
Carsten Emde <ce at ceag.ch>, Clark Williams <williams at redhat.com>, 
"Paul E. McKenney" <paulmck at linux.vnet.ibm.com>,        Kumar Gala 
<galak at gate.crashing.org>,        Ralf Baechle <ralf at linux-mips.org>

Dear RT Folks,

I'm pleased to announce the first drop of the 3.0-rc7 based RT

It's been quite a while since 2.6.33-rt, but I went through a very
painful experience while trying to get a 2.6.38-rt stabilized. The
beast insisted on destroying filesystems with reproduction times
measured in days and the total refusal to reveal at least a
minimalistic hint to debug the root cause. Staring into completely
useless traces for months is not a very pleasant pastime.

That's the very first problem in the RT history which I gave up on.

[The truth: Linus avoiding the final 2.6.42 release made all my
  ultimate plans go down the drain ... ]

Though while trying to analyse the problem I had plenty of time to
twist my brain around the existing RT approach and its shortcomings.

The main issue which RT is fighting with is the ever growing per cpu
variable usage and the assumptions which are built around it. The
existing RT approach to work around this with PER_CPU_LOCKED
constructs and hand the CPU number around simply does not work anymore
because the number of sites which need to be patched is way too large
and the resulting mess in the code is neither acceptable nor

After lenghty and fruitful discussions with Peter Zijlstra - thanks a
lot Peter! - we finally agreed on trying a totally different approach
to tackle these issues: disabling migration over spinlock and get_cpu
sections. This had been discussed before, but nobody ever considered
to sit down and make it work.

This keeps the semantics which are expected by the per cpu users,
while keeping the regions preemptible. As a side effect, it allows us
to run softirq handlers directly from irq threads on local_bh_enable
which was a long desired feature to lower the performance impact of

Changing this required a major refactoring of the RT patch queue,
which took some time as I had to go through every single patch, fold
fixes back into the right places and sort them into various categories:

  - Mainline ready (raw lock annotations, infrastructure patches, code

  - Preparatory (_rt()/_nort() variants of preempt_*(), local_irq_*(),
    BUG*(), WARN*() and the annotations in various places)

  - Base patches (Reworking the slab/page_alloc code, bit_spinlock
    replacements, migrate disable infrastructure ...)

  - Full RT patches (sleeping spinlocks and the resulting fixups here
    and there)

In course of that exercise I weeded out a lot of historically grown
hackery and dropped stuff which was not essential for getting it up
and running. Thanks to Carsten for reintegrating the tracer addons
which he's using for the OSADL test farm:


I probably have missed a few bits and pieces, but the overall outcome
is stable and survived testing on various systems. The latency
behaviour with cyclictest is on par with 33-rt at least on x86_64/32.

The overall patch size has shrunk significantly and the readability
(except for the missing changelogs in various patches) is at an
acceptable level.

If you download the quilt tarball, you'll find various sections:

- upstream fixes: Stuff broken upstream which we managed to trip
   over. This section contains real weird stuff from simple fixes, over
   mainline code which claims to contain (complete bogus) RT support up
   to an archaeologic bug in the floppy driver code.

   8 patches (size 8892)
   7 files changed, 59 insertions(+), 51 deletions(-)

- upstream submitted: Stuff which is on LKML already and needs some
   follow up.

   4 patches (size 9741)
   4 files changed, 81 insertions(+), 119 deletions(-)

- upstream ready: Stuff which needs a bit polishing and upstream

   79 patches (size 232566)
   192 files changed, 1204 insertions(+), 1097 deletions(-)

- upstream needs work: Stuff which should go upstream, but needs some
   or lots of care.

   7 patches (size 164120)
   49 files changed, 3292 insertions(+), 253 deletions(-)

- the real rt stuff:

   125 patches (size 280665)
   162 files changed, 4327 insertions(+), 592 deletions(-)

The overall patch is now:
   223 patches (size 680054)
   374 files changed, 8950 insertions(+), 2099 deletions(-)

Compared that to 2.6.33-rt:
   462 patches (size 1396505)
   690 files changed, 15994 insertions(+), 5123 deletions(-)

That's a significant reduction in size and impact. Some of it is due
to the new approach, but we also got quite a lot of the infrastructure
patches upstream in the last few kernel releases. Thanks to all folks
who have helped to get that done, especially to Peter Zijlstra for
getting the preemptible mmu gather problem and lots of the scheduler
issues which we discovered in RT over time sorted out!!!

What's new in 3.0-rt ?

  - No more split soft interrupt threads. We need to analyze whether
    this is a good decision.

  - softirq handling from the end of interrupt threads and on all
    thread sites where a nested local_bh disabled section ends

  - SPARSE interrupts and IOMMU interrupt remapping work now

    and CONFIG_PREEMPT_RT_FULL. RT_BASE covers some of the more complex
    changes (e.g. mm/* where we substitute interrupt disabled sections
    with per cpu locks and the bit_spinlock to spinlock conversion).
    RT_BASE allows us to test and verify these changes independently of
    the big RT_FULL modifications. That's mainly a debugability and
    maintainability issue.

What's the state:

    We've done quite some testing on x86 32/64 bit and basic tests on
    some ARM/MIPS/POWERPC platforms. Thank God, no file system eating so
    far :)

    Given the fact that it is a major rewrite it's amazinlgy stable and
    I consider it to be the best -rt1 release we ever had. That doesn't
    mean that there are no bugs, since it has not had the proper test
    coverage yet.

    Thanks to Carsten, Clark and Peter for all the help to get this far!

Want to help?

    Many people offered help in the past and I had to turn them down so
    far as refactoring that stuff really is not a task which can be
    shared easily. Though now is the point where I can use all the help
    you promised to provide.

    What's needed?

    - Testing, testing, testing ... you know the drill (good bug
      reports are 98% of the solution)

    - Compare and analyze the performance/troughput impact of the new
      approach with 33-rt

    - Help mainlining the "upstream ready section"

      That means reviewing the patches, cleaning them up, fixing the
      changelogs, submitting them through the proper channels ...

      Please do not blindly pick any of these patches and submit them
      to mailing lists w/o doing the above. Also please coordinate on
      the #linux-rt IRC channel on oftc.net so redundant and
      conflicting work can be avoided

    - Help getting the "upstream needs work" section into shape

      All of these patches need a close look and (especially the
      hwlatency detector) major cleanups. Please coordinate with the
      patch authors and lookout for previous discussions of some of
      those on LKML.

    - Tend to the FIXME annotations in the RT stuff section

      I have annotated some places with /* FIXME ... comments. These
      sections are not for the faint hearted and need some serious
      review and thought.

    - Help with the RCU modifications

      That's an easy one. We have a volunteer signed up for this
      involuntarily already. Thanks Paul!

    - Twist your brain around the schedulability impact of the
      migrate_disable() approach.

      A really interesting research topic for our friends from the
      academic universe. Relevant and conclusive (even short notice)
      papers and/or talks on that topic have a reserved slot in the
      Kernel developers track at the Realtime Linux Workshop in Prague
      in October this year.

Enough marketing, here comes the real stuff.

   Patch against 3.0-rc7 can be found here:


   The split quilt queue is available at:


There is no git tree for now.

I'm not yet convinced that moving RT to git was a good idea as quilt
allows me to move stuff around in a way more flexible manner. So for
now no git version until someone comes up with a brilliant idea which
allows me to keep my workflow sane (do not even try to suggest stgit &

That said, have fun and make sure that you have the fire extinguisher
ready when you start using this!


To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

More information about the PlanetCCRMA mailing list