[PlanetCCRMA] Performance degradation with kernel-rt-3.14.17-200.rt9.1.fc20.ccrma.x86_64

Sun Sep 7 16:12:40 PDT 2014

On 09/07/2014 03:26 AM, Oleg Samarin wrote:
>
> I recompiled kernel-rt-3.14.17 with the latest intel_pstate
> driver from
> https://github.com/torvalds/linux/blob/16405f98bca8eb39a23b3ce03e241ca19e7af370/drivers/cpufreq/intel_pstate.c
> The behaviour rest the same as with intel_pstate from kernel-rt-3.14.17.

That's not good news :-(
Just out of curiosity, what processor do you have?

> I made more testing.
>
> Under 3.14.17, when I just play GO, jack DSP load becomes 100%, but CPU core speed
> leaves 1600-1800 mhz and the sound is terrible.
>
> Here is a i7z

(Ah, cool, I did not know about this program.)

>output at that time:
>
>          Core [core-id]  :Actual Freq (Mult.)	  C0%   Halt(C1)%  C3 %   C6 %
>          Core 1 [0]:	  2143.82 (21.44x)      31.1    44.8    33.5    2.09
>          Core 2 [1]:	  1994.62 (19.95x)      47.9    60.4    10.5       1
>          Core 3 [2]:	  2077.24 (20.77x)      34.7    33.2    44.7       1
>          Core 4 [3]:	  2284.95 (22.85x)      45.4    29.5    34.3    5.74
>
>
> If I run mprime torture
> test (a tool that loads CPU hard) with 1 single thread, the CPU speed
> becomes 4500-4800 mhz, jack dsp load decreases and I can play GO without
> any sound problems. It could be a temporary workaround for me.
>
> Here is a i7z output with one mprime thread:
>
>          Core [core-id]  :Actual Freq (Mult.)	  C0%   Halt(C1)%  C3 %   C6 %
>          Core 1 [0]:	  4284.18 (42.85x)        31    39.3    20.6       1
>          Core 2 [1]:	  4305.80 (43.07x)      30.6      38    22.2       1
>          Core 3 [2]:	  4102.73 (41.04x)      98.4       0       0       0
>          Core 4 [3]:	  4131.24 (41.32x)      38.2    24.2    28.3       1
>
>
> echo "100" > /sys/devices/system/cpu/intel_pstate/min_perf_pct increases
> CPU does not speed up so much so it cann't be a workaround.

How much faster does it go with min_perf_pct set to 100?

> Under 3.12.12 playing GO activates CPU speed up itself, so running
> mprime is not necessary.

It would seem the thresholds for changing the frequency have changed. Or 
the underlying algorithm. But setting min_perf_pct to 100 should bypass 
that. What do you see in the other variables in 
/sys/devices/system/cpu/intel_pstate?

Did you try to run without the intel_pstate driver? 
(intel_pstate=disable in the kernel boot line).

> The i7z output is here:
>
>          Core [core-id]  :Actual Freq (Mult.)	  C0%   Halt(C1)%  C3 %   C6 %
>          Core 1 [0]:	  4502.27 (45.02x)	28.8      42    18.8       1
>          Core 2 [1]:	  4503.14 (45.03x)	24.7    52.3      14       1
>          Core 3 [2]:	  4524.26 (45.24x)	16.1      20    57.5       1
>          Core 4 [3]:	  4521.34 (45.21x)	69.8       0    11.9       1
>
> May be the source of the problem is not in intel_pstate, but in the scheduler?

I think that is unlikely. I imagine it could be possible - a different 
scheduling of processes could affect how the frequency is switched in 
the cpus.

> Seems under 3.12 the most part of jack load belongs to the Core 4 that enforces CPU to increase its frequency,
> but under 3.14 it is spreaded between all cores and their workload is not sufficient for speeding up.

I could be. I do not know how intel_pstate works but I think the 
decision is made on each cpu, there is no "master" cpu AFAIK. I read 
some comments online that the algorithms on the pstate driver are 
changing from release to release and are not "stable".

> I'd like to port intel_pstate from 3.12.12 to 3.14.17 for checking
> whether intel_pstate is an issue, but I couldn't find the rt-3.12.12
> source rpm. Where can I get it?

Here:
http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetcore/20/SRPMS/repoview/kernel.html

By the way, this only happens in the rt patched kernels? What is the 
behavior in the latest Fedora kernel?

Keep us updated, and good luck!
-- Fernando

> В Сб, 06/09/2014 в 16:28 +0400, Oleg Samarin пишет:
>> В Пт, 05/09/2014 в 15:17 -0700, Fernando Lopez-Lezcano пишет:
>>> On 09/05/2014 12:35 PM, Oleg Samarin wrote:
>>>> Hello!
>>>
>>> Hi!,
>>>
>>>> I use Fedora 20 with PlanetCCRMA realtime kernel.
>>>>
>>>> After upgrading the kernel from kernel-rt-3.12.12-300.rt19.1.fc20.ccrma to
>>>> the last one - kernel-rt-3.14.17-200.rt9.1.fc20.ccrma, the huge sound
>>>> distortion started occurring. When I boot my station with kernel-rt-3.12.12,
>>>> everything seems all right.
>>>
>>> Did you try to confirm by, for example, looking at the output of top in
>>> both cases? Can you confirm that the cpus are stuck to low frequencies?
>>
>> I use Jack audio stack and can see the DSP load with qjackctl.
>> Every time I can hear audio distortion, qjackctl shows 100% DSP.
>>
>> When I just start GrandOrgue, the DSP load is about 15% and top shows
>> that GO utilises 19% of CPU.
>>
>> When I start playing with GO under kernel-rt 3.14.17, qjackctl shows
>> 100% of DSP load and top shows that GO uses 121 % of CPU and the
>> distortion begins. At the same time i7z indicates, that CPU frequency is
>> between 1600-1800 MHz.
>>
>> The same under kernel-rt 3.12.12 brings about 40-50 % in qjackctl and
>> about 60% CPU of GO in top. i7z indicates CPU frequency 4500-4800 MHz.
>>
>>
>>> What does /proc/cpuinfo report in your case?
>>
>> It is difficult to play GO and capture /proc/cpuinfo at the same time.
>>
>>>
>>>> The most probably reason is a bug in intel_pstate driver, that prevents CPU
>>>>    speeding up: https://bugzilla.kernel.org/show_bug.cgi?id=70941,  There is a
>>>> patch that should solve this problem.
>>>
>>> Is your system actually using the pstate_intel driver? If so, what
>>> happens if you try to get all cpus to run as fast as possible:
>>>
>>>     echo "100" > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>>>
>>> Does that make a difference?
>>
>> Yes, it looks better, but the CPU freq (3800-4000 Mhz) is still less
>> than under 3.12.12.
>>
>>>
>>>> Could you include this patch in the next build of kernel-rt?
>>>
>>> Sure, but it would be nice to get more feedback.
>>> -- Fernando
>>>
>>
>
>