[PlanetCCRMA] Performance degradation with kernel-rt-3.14.17-200.rt9.1.fc20.ccrma.x86_64

[Fernando Lopez-Lezcano]

On 09/05/2014 11:39 PM, Michael J. Wilson wrote:
> Hello all,
>
> I was also noticing performance issues with recent versions of the RT
> kernel (CPU load reported by Jack about 12% higher than expected, xruns
> under light loads).  I can't seem to set the performance scaling governor:
>
> $ su -c 'echo performance >
> /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor'
> bash: line 0: echo: write error: Invalid argument
>
> even thought it appears to be available:
>
> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
> performance powersave

Yes, indeed. This does not work if the new intel pstate driver is loaded 
and controlling the cpus. I don't know why the old style governors are 
still listed (a bug?).

> This is on:
>
> $ uname -r
> 3.14.17-200.rt9.1.fc20.ccrma.x86_64+rt
>
> It works as expected with the standard kernel.

"Standard" means the Fedora kernel? Which version? And "work as 
expected" means you don't get the intel_pstate driver?

> I wanted to try to debug further before posting to the list but didn't
> have time to look into it; it sounds like the same or a related issue
> though.
>
> For reference, running this:
> $ su -c 'echo "100" > /sys/devices/system/cpu/intel_pstate/min_perf_pct'
>
> does seem to give me the CPU load I'd expect when running Jack in
> realtime mode, but I didn't do a very strict comparison.

In my experience with this option the cpu horsepower you get is the same 
or better than with the older "performance" governor. The cpu speed is 
_not_ fixed to a hard number, as apparently the cpu itself decides at 
what exact frequency to run. But the fluctuation is near the top end of 
the frequency range of the cpu. Do this to take a look before and after:

   watch grep MHz /proc/cpuinfo

I think (just a guess) that what happens is that the intel_pstate driver 
is not coupled with the scheduler. So, when running tasks with deadlines 
as is the case of jackd the scheduler can switch a task to a core that 
is running at a low frequency (or was just switched to that state), the 
pstate driver can't react instantly to the new load, and by the time the 
frequency is upped jackd runs out of cpu and you get an xrun.

You can also disable the intel_pstate driver completely on boot by 
adding "intel_pstate=disable" in the kernel boot line (without the 
quotes). Presumably you get back the old governor based speed control.

Recently, when testing a piece that pretty much drives the machine (a 4 
core laptop, Intel(R) Core(TM) i7-4900MQ CPU @ 2.80GHz) to run at 90% 
cpu usage I also found it useful to disable hyperthreading (this is 
something that had never happened to me before but I had read about). 
You can do that from the kernel boot line with the "noht" parameter - 
insert it without the quotes, of course.

Or you can disable the second cpu of the each pair of cores that share 
the same id. In my laptop:

----
# egrep "core id|processor" /proc/cpuinfo
processor	: 0
core id		: 0
processor	: 1
core id		: 0
processor	: 2
core id		: 1
processor	: 3
core id		: 1
processor	: 4
core id		: 2
processor	: 5
core id		: 2
processor	: 6
core id		: 3
processor	: 7
core id		: 3
----

And then, for example:

echo 0 > /sys/devices/system/cpu/cpu1/online

(and the same for cpu5, 5, and 7)

With this I get less (or no) xruns.

I still get periodic errors in /var/log/messages about softirqs not 
being able to be started[*] on those cpus, but with no side effects that 
I can see.

Hope this is helpful...
-- Fernando


[*] For example:

----
Sep  6 12:52:08 localhost /etc/gdm/Xsession: glibtop: 'intr 3332147 76 
11188 0 0 0 0 0 0 1 11132 0 0 486438 0 0 0 33 0 0 0 0 0 0 39 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 259585 326 22002 32 254551 369 198 25 22 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0
Sep  6 12:52:08 localhost /etc/gdm/Xsession: ctxt 11771613
Sep  6 12:52:08 localhost /etc/gdm/Xsession: btime 1410030403
Sep  6 12:52:08 localhost /etc/gdm/Xsession: processes 3905
Sep  6 12:52:08 localhost /etc/gdm/Xsession: procs_running 1
Sep  6 12:52:08 localhost /etc/gdm/Xsession: procs_blocked 0
Sep  6 12:52:08 localhost /etc/gdm/Xsession: softirq 1312278 9 797056 
564 27118 259776 5 8589 218925 236 0
Sep  6 12:52:08 localhost /etc/gdm/Xsession: ' does not start with 
'cpu7': Resource temporarily unavailable
----