Using and Understanding the Real-Time Real-Time Cyclictest Benchmark Cyclictest results are the most frequently cited real-time Linux metric. metric. The core concept of Cyclictest is is very simple. simple. o!ever the test options are are very extensive. The meaning meaning of Cyclictest results appear simple "ut are actually quite complex. This talk talk !ill explore and explain the the complexities complexities of Cyclictest. #t the end of the the talk$ the audience !ill !ill understand ho! Cyclictest results descri"e the potential real-time performance of a system.
%rank Ro!and$ &ony 'o"ile Communications
(cto"er )*$ )+, ,,+)*+)/
0hat Cyclictest 'easures Latency of response to a stimulus. external interrupt triggers 1clock expires2 - possi"le delay until 3R4s ena"led - 3R4 handling - cyclictest is !oken - possi"le delay until preemption ena"led - possi"le delay until cyclictest is highest priority priority - possi"le delay until other process is preempted - scheduler overhead transfer control to cyclictest
0hat Cyclictest 'easures Latency of response to a stimulus. Causes of delay list on previous slide is simplified5 - order !ill vary - may occur multiple times - there are additional causes of delay
'any factors can increase latency - additional external interrupts - &'3 - processor emerging from sleep states - cache migration migration of data used "y !oken process - "lock on sleeping lock - lock o!ner gets priority "oost - lock o!ner schedules - lock o!ner completes scheduled !ork - lock o!ner releases lock$ loses priority "oost
o! Cyclictest 'easures Latency 1Cyclictest 6seudocode2 The source code is nearly +++ lines$ lines$ "ut the algorithm is trivial
Test Loop clock_gettime((&now)) next = now + par->interval while (!shutdown) {
clock_nanosleep((&next))
clock_gettime((&now)) di = calcdi(now next) " update stat-> min max total latenc# c#cles " update the histogram data next += interval
$
The 'agic of &imple This trivial algorithm captures all of the factors that contri"ute to latency. 'ostly. Caveats !ill follo! soon.
Cyclictest 6rogram main() {
or (i = % i ' num_threads i++) { pthread_create((timerthread)) while (!shutdown) { or (i = % i ' num_threads i++) print_stat((statsi) i)) usleep(*%%%%) $ i (histogram) print_hist(parameters num_threads)
$
timerthread12 timerthread(void par) { " thread set up " test loop $
Thread &et Up stat = par->stats pthread_setainit#_np((pthread_sel())) setscheduler(({par->polic# par->priorit#)) sigprocmask((,._/0123))
Test Loop 1as sho!n earlier2 clock_gettime((&now)) next = now + par->interval while (!shutdown) {
clock_nanosleep((&next))
clock_gettime((&now)) di = calcdi(now next) " update stat-> min max avg c#cles " 4pdate the histogram next += interval
$
0hy sho! set up pseudocode7 The timer threads are not in lockstep from time 8ero. 'ultiple threads !ill pro"a"ly not directly impact each other.
&eptem"er )+, update linux-rt-users [rt-tests][PATCH] align thread wakeup times Nicholas Mc Guire !"#-!$-!$ %&$&'( And replies
)This patch pro*ides and additional -A+--align ,lag to cclictest to align thread wakeup times o, all threads as closl de,ined as possi.le/) )/// we need .oth/ same period 0 )random) start time same period 0 snced start time it makes a di,,erence on some .oxes that is signi,icant/)
The 'agic of &imple This trivial algorithm captures all of the factors that contri"ute to latency. 'ostly. Caveats$ as promised.
Caveats 'easured maximum latency is a floor of the possi"le maximum latency - Causes of delay may "e partially completed !hen timer 3R4 occurs - Cyclictest !akeup is on a regular cadence$ may miss delay sources that occur outside the cadence slots
Caveats 9oes not measure the 3R4 handling path of the real RT application - timer 3R4 handling typically fully in 3R4 context - normal interrupt source 3R4 handling5 - irq context$ small handler$ !akes 3R4 thread - 3R4 thread eventually executes$ !akes RT process
Caveats Cyclictest may not exercise latency paths that are triggered "y the RT application$ or even non-RT applications - &'3 to fixup instruction errata - stopmachine12 - module load : unload - hotplug
&olution , 9o not use cyclictest. 5-2 3nstrument the RT application to measure latency
&olution ) Run the normal RT application and non-RT applications as the system load Run cyclictest !ith a higher priority than the RT application to measure latency
&olution ) Typical real time application !ill consist of multiple threads$ !ith differing priorities and latency requirements To capture latencies of each of the threads$ run separate tests$ varying the cyclictest priority
&olution ) ;xample deadline constraint
latenc# constraint
56 app scheduler priorit#
7
critical
8% usec
9%
9*
/
%:*; miss
*%% usec
<
<8
56 app thread
c#clictest priorit#
#side5 Cyclictest output in these slides is edited to fit on the slides (riginal5 c#clictest_%:89 -l*%%%%% -? -p8% -, 6@ % ( 6@ * (
D 7ct@ D 7ct@
Gxample o edit@ c#clictest_%:89 -l*%%%%% -? -p8% -, 6@% @*%%% 6@* @*9%%
Cin@ Cin@
D D
7vg@ 7vg@
<9
Cax@ Cax@
A8 F
<9 Cax@
A8 F
Cyclictest Command Line (ptions 9o 3 really care777 Can 3
9o 3 really care777 c#clictest_%:89 -l*%%%%% -? -p8% 6@%
Cin@ FAF
7vg@ F8*
Cax@
DD
c#clictest_%:89 -l*%%%%% -? -p8% -n 6@%
-l*%%%%% -? -p8% -n
Cin@
D9
7vg@
Cax@
A8
stop ater *%%%%% loops ?uiet priorit# 8% ,2HGI_JJ1 use clock_nanosleep() instead o nanosleep()
3mpact of (ptions 'ore examples Be some!hat skeptical of maximum latencies due to the short test duration. ;xamples are5 ,++$+++ loops ,$+++$+++ loops #r"itrary choice of loop count. =eed large values to properly measure maximum latency>>>
6riority of Real Time kernel threads for next t!o slides BI D A 8 E ** D9D D< D9 DE<
BBI F F F F F F F F F F
, 56B51 20, , * JJ , % JJ , EE JJ , % JJ , EE JJ , * JJ , 9% JJ , 9% JJ , 9% JJ , 9% JJ
2CI ksotir?dK% posixcputmrK% migrationK% posixcputmrK* migrationK* ksotir?dK* ir?K<*-eth;d ir?K
-l*%%%%%
6@%
Cin@ *F8
7vg@ *8E
Cax@ FAEE
6@%
Cin@ *F9
7vg@ *<%
Cax@
<F
-?
no live update
6@%
Cin@ FAF
7vg@ F8*
Cax@
DD
-p8%
,2HGI_JJ1 8%
6@%
Cin@
7vg@
EA
Cax@
F%%
-n
clock_nanosleep
6@% 6@*
Cin@ F
7vg@ DF% 7vg@ D*9
Cax@ Cax@
-? -p8% -a -t
pinned
6@%
Cin@
D9
7vg@
Cax@
A8
-? -p8% -n
,2HGI_JJ1 c_n
6@%
Cin@
D<
7vg@
<<
Cax@
*
-? -p8% -a -n
pinned
6@%
Cin@
D8
7vg@
Cax@
**E
6@% 6@*
Cin@ Cin@
DA D
7vg@ 7vg@
Cax@ Cax@
A9 8
6@% 6@*
Cin@ Cin@
DA D
7vg@ 7vg@
<< <9
Cax@ Cax@
E* ***
-? -p8% -a -t -n pinned
6@% 6@*
Cin@ Cin@
D< D<
7vg@ 7vg@
<<
Cax@ Cax@
E< *%<
-? -p8% -,
88
live update
-? -p8% -a -n -m mem locked -? -p8% -t -n
not pinned
=> -a -t -n
-l*%%%%%%
6@%
Cin@ *FD
7vg@ *8<
Cax@ D8*<
6@%
Cin@ *F9
7vg@ *9%
Cax@
8A%
-?
no live update
6@%
Cin@ F9
7vg@ F8*
Cax@
D*
-? -p8%
,2HGI_JJ1 8%
6@%
Cin@
7vg@
E<
Cax@
D*E
-? -n
clock_nanosleep
6@% 6@*
Cin@ F< Cin@ FF8
7vg@ D*< 7vg@ DF*
Cax@ Cax@
A8F 9%A
-? -p8% -a -t
pinned
6@%
Cin@
D8
7vg@
<<
Cax@
F
-? -p8% -n
,2HGI_JJ1 c_n
6@%
Cin@
DD
7vg@
Cax@
E9
-? -p8% -a -n
pinned
6@%
Cin@
DA
7vg@
Cax@
*<<
6@% 6@*
Cin@ Cin@
DA D
7vg@ 7vg@
<< <9
Cax@ Cax@
8< E<
-? -p8% -t -n
6@% 6@*
Cin@ Cin@
DA DA
7vg@ 7vg@
Cax@ Cax@
8 E*
-? -p8% -a -t -n pinned
6@% 6@*
Cin@ Cin@
DA D<
7vg@ 7vg@
Cax@ Cax@
*<* 88
8<
live update
-? -p8% -a -n -m mem locked
-? -p8% -,
not pinned
=> -a -t -n
&imple 9emo --
&C;9=(R'#L
- single thread - clocknanosleep12$ one thread per cpu$ pinned - clocknanosleep12$ one thread per cpu - clocknanosleep12$ one thread per cpu$ memory locked - clocknanosleep12$ one thread per cpu$ memory locked$ non-interactive
0hat #re =ormal Results7 0hat should 3 expect the data to look like for my system7
;xamples of 'aximum Latency https5::rt.!iki.kernel.org:index.php:C(=%3?6R;;'6TRT6atch @6latformsTestedandinUse!ithC(=%3?6R;;'6TRT
6latforms Tested and in Use !ith C(=%3?6R;;'6TRT Comments sometimes include avg and max latency ta"le is usually stale linux-rt-users email list archives http5::vger.kernel.org:vger-lists.html@linux-rt-users
?raphs of 'aximum Latency ( L.org ?raphs for a !ide variety of machines List of test systems5
https5::!!!.osadl.org:3ndividual-system-data.qa-farm-data.+.html
%ull URL of previous graph https5::!!!.osadl.org:Com"ined-latency-plot-of-all-RT-systems .qa-latencyplot-allrt.+.html7latenciesAsho!noA
Typical command5 cyclictest -l,++++++++ -m -&p -i)++ -hD++ -q ( L Realtime 4# %arm5 https5::!!!.osadl.org:4#-%arm-Realtime.qa-farm-a"out.+.html
( L Latency plots5 https5::!!!.osadl.org:Latency-plots.latency-plots.+.html
#dditional ( L 9ata ( L mem"ers have access to additional data$ such as - the data used to create the graphs - the latency graphs extended in a third dimension$ sho!ing all test runs
&ome Random 3ndividual &ystems 6icked from the ( L spaghetti graph
;ven E"oringF graphs may contain interesting details
Command Line (ptions #n unruly$ out of control$ set of control kno"s
c#clictest --help c#clictest L %:89 4sage@ c#clictest 'options> -a M4C --ainit# -N -/ -c
-2 -d -I
-e -G - -g -h
-H -i - -l -m -C -n -M -o -1 -p -B -? -P -r -5
-s -, -t -t
-6 -u -4 -v -w -R -Q -#
run thread "M on processor "M i possiNle with M4C pin all threads to the processor M4C 4,G2 --Nreaktrace=4,G2 send Nreak trace command when latenc# > 4,G2 --preemptir?s Noth preempt and ir?so tracing (used with -N) 20123 --clock=20123 select clock % = 20123_C1M161M2 (deault) * = 20123_5G706CG --context context switch tracing (used with -N) I,6 --distance=I,6 distance o thread intervals in us deault=9%% --duration=t speci# a length or the test run deault is in seconds Nut OmO OhO or OdO ma#Ne added to modi# value to minutes hours or da#s --latenc#=BC_P1, write BC_P1, to KdevKcpu_dma_latenc# --event event tracing (used with -N) --trace unction trace (when -N is active) C7Q --o_max=C7Q 5eport time in ms (up to C7Q) or histogram overlows --histogram=4, dump a latenc# histogram to stdout ater the run (with same priorit# aNout man# threads) 4, is the max time to Ne Ne tracked in microseconds --histoall=4, same as -h except with an additional summar# column M6L --interval=M6L Nase interval o thread in us deault=*%%% --ir?so r?so tracing (used with -N) 011B, --loops=011B, numNer o loops@ deault=%(endless) --mlockall lock current and uture memor# allocations --reresh_on_max dela# updating the screen until a new max latenc# is hit --nanosleep use clock_nanosleep --nsecs print results in ns instead o us (deault us) 5GI --oscope=5GI oscilloscope mode reduce verNose output N# 5GI 61B6 --traceopt=61B6 trace option B51 --prio=B51 priorit# o highest prio thread --preempto Breempt o tracing (used with -N) --?uiet print onl# a summar# on exit --priospread spread priorit# levels starting at speciied value --relative use relative timer instead o aNsolute --resolution check clock resolution calling clock_gettime() man# times: list o clock_gettime() values will Ne reported with -Q --s#stem use s#s_nanosleep and s#s_setitimer --smp ,tandard ,CB testing@ options -a -t -n and same priorit# o all threads --threads one thread per availaNle processor M4C --threads=M4C numNer o threads@ without M4C threads = max_cpus without -t deault = * 6572G --tracer=6572G5 set tracing unction conigured tracers@ Nlk unction_graph wakeup_rt wakeup unction nop --unNuered orce unNuered output or live processing --numa ,tandard M4C7 testing (similar to ,CB option) thread data structures allocated rom local node --verNose output values on stdout or statistics ormat@ n@c@v n=tasknum c=count v=value in us --wakeup task wakeup tracing (used with -N) --wakeuprt rt task wakeup tracing (used with -N) --dNg_c#clictest print ino useul or deNugging c#clictest B10 --polic#=B10 polic# o realtime thread B10 ma# Ne io(deault) or rr
Thread Behavior (ptions -a M4C --ainit# -c 20123 --clock=20123
-d I,6 -i M6L -m -n -p B51 -P -r -s -,
--distance=I,6 --interval=M6L --mlockall --nanosleep --prio=B51 --priospread --relative --s#stem --smp
-t --threads -t M4C --threads=M4C
-4
--numa
-# B10
--polic#=B10
run thread "M on processor "M i possiNle with M4C pin all threads to the processor M4C select clock % = 20123_C1M161M2 (deault) * = 20123_5G706CG distance o thread intervals in us deault=9%% Nase interval o thread in us deault=*%%% lock current and uture memor# allocations use clock_nanosleep priorit# o highest prio thread spread priorit# levels starting at speciied value use relative timer instead o aNsolute use s#s_nanosleep and s#s_setitimer ,tandard ,CB testing@ options -a -t -n and same priorit# o all threads one thread per availaNle processor numNer o threads@ without M4C threads = max_cpus without -t deault = * ,tandard M4C7 testing (similar to ,CB option) thread data structures allocated rom local node polic# o realtime thread B10 ma# Ne io(deault) or rr ormat@ --polic#=io(deault) or --polic#=rr
side eect sets -d% -h
--histogram=4,
-H
--histoall=4,
dump a latenc# histogram to stdout ater the run (with same priorit# aNout man# threads) 4, is the max time to Ne Ne tracked in microseconds same as -h except with an additional summar# column
Benchmark and &ystem (ptions -I
--duration=t
-l 011B, --loops=011B,
speci# a length or the test run deault is in seconds Nut OmO OhO or OdO ma#Ne added to modi# value to minutes hours or da#s numNer o loops@ deault=%(endless)
-e
write BC_P1, to KdevKcpu_dma_latenc#
--latenc#=BC_P1,
9isplay (ptions -g C7Q -h
--o_max=C7Q --histogram=4,
-H -C -M -o 5GI -? -u -v
--histoall=4, --reresh_on_max --nsecs --oscope=5GI --?uiet --unNuered --verNose
5eport time in ms (up to C7Q) or histogram overlows dump a latenc# histogram to stdout ater the run (with same priorit# aNout man# threads) 4, is the max time to Ne Ne tracked in microseconds same as -h except with an additional summar# column dela# updating the screen until a new max latenc# is hit print results in ns instead o us (deault us) oscilloscope mode reduce verNose output N# 5GI print onl# a summar# on exit orce unNuered output or live processing output values on stdout or statistics ormat@ n@c@v n=tasknum c=count v=value in us
9e"ug (ptions -N 4,G2 -/ -2 -G - - -1 61B6 -B -5
-6 -w -R -Q
--Nreaktrace=4,G2 --preemptir?s --context --event --trace --ir?so --traceopt=61B6 --preempto --resolution
send Nreak trace command when latenc# > 4,G2 Noth preempt and ir?so tracing (used with -N) context switch tracing (used with -N) event tracing (used with -N) unction trace (when -N is active) r?so tracing (used with -N) trace option Breempt o tracing (used with -N) check clock resolution calling clock_gettime() man# times: list o clock_gettime() values will Ne reported with -Q 6572G --tracer=6572G5 set tracing unction conigured tracers@ Nlk unction_graph wakeup_rt wakeup unction nop --wakeup task wakeup tracing (used with -N) --wakeuprt rt task wakeup tracing (used with -N) --dNg_c#clictest print ino useul or deNugging c#clictest
9e"ug (ptions =o time to descri"e in this talk ooks to invoke various tools that can capture the cause of large latencies
(ptions Trivia (ptions parsing is not ro"ust - example , @ affinity !ill "e + c#clictest -t -l*%% -a% c#clictest -t -l*%% -a % c#clictest -t -l*%% -a -a% @ affinity !ill "e G$ !ith no error message c#clictest -t -l*%% -a -a % -a
cpu ainit#
(ptions Trivia (ptions parsing is not ro"ust - example ) c#clictest -ant 6@ % (FAE8) B@ % @*%%% 2@
F%E* Cin@
c#clictest -an -t 6@ % (FAE8%) B@ % @*%%% 2@ 6@ * (FAE8*) B@ % @*9%% 2@
*EF8 Cin@ *F89 Cin@
-a -n -t
cpu ainit# clock_nanosleep() one thread per cpu
(ptions Trivia (ptions parsing is not ro"ust Best 6ractice5 - do not com"ine options - specify each separately !ith a leading H-H
Third 9ata %ormat Report each latency c#clictest -? -n -t* -p <8 -l *%%%% -v
itting the RT sched throttle :proc:sys:kernel:schedrtruntimeus :proc:sys:kernel:schedrtperiodus cyclictest5 &C;9%3%( priorityA/+ "ackground load5 - continuous - &C;9%3%( priorityAD+
itting the RT sched throttle :proc:sys:kernel:schedrtruntimeus :proc:sys:kernel:schedrtperiodus cyclictest5 &C;9=(R'#L "ackground load5 - continuous - &C;9%3%( priorityAD+
itting the RT sched throttle 0hy is this measurement interesting777
?ives a picture of ho! much cpu is =(T used "y the real time tasks
Unusual Uses of Cyclictest Rough measurement of response time of a real time application$ !ithout instrumenting task. cyclictest latency AI task latency J task !ork duration This is not an accurate measurement$ "ut it does provide a rough picture.
Response Time of a Task
Cyclictest5 1,2 &C;9%3%( priorityA/+ "aseline latency 1)2 &C;9%3%( priorityA+ approximation of task response time
Real time application5 - "usy loop 1random num"er of iterations2$ follo!ed "y a short sleep - &C;9%3%( priorityAD+
Response Time of a Task
Cyclictest5 1,2 &C;9%3%( priorityA/+ "aseline latency 1)2 &C;9%3%( priorityA+ approximation of task response time
Real time application5 - recursive scp of a panda root file system - &C;9%3%( priorityAD+ - no guarantee of sleep "et!een scp EtransactionsF - response time may include multiple transactions
9emo - oscilloscope c#clictest_%:89 -t* -n -p8% -i*%% -o*% -v S T oscilloscope >KdevKnull
oscilloscope screen shot
%ourth 9ata %ormat Report time of each histogram overflo! &hould "e in next version of cyclictest 1+./K72 c#clictest -? -h <%% -g *%%%
The same information can "e extracted from the third data format 1-v2$ "ut this method is lo!er overhead.
%inding and Building git clone git5::git.kernel.org:pu":scm:linux:kernel:git:clrk!llms:rt-tests.git
source5 src:cyclictest:cyclictest.c self-hosted5 make self-hosted !ithout =U'#5 make M4C7=% cross-"uild !ithout =U'#5 make M4C7=% 22=U{251,,_21CB0G$gccU
Revie! - &imple methodology captures all sources of latency fairly !ell - (ptions must "e used !ith care - (ptions are po!erful - 9ifferent data formats are each useful - 9e"ug features can capture the cause of large latencies
T; ;=9
Thank you for your attention...
4uestions7
o! to get a copy of the slides
,2 leave a "usiness card !ith me )2 frank.ro!andMsonymo"ile.com
