Benchmarking Perl

brian d foy Stonehenge Consulting Ser vices March 14, 2006 Chicago UniForum

Know where you are

A benchmark is a point of reference for a measurement. The term originates from the chiseled horizontal marks that sur veyors made into which an angle-iron could be placed to bracket (bench) a leveling rod, thus ensuring that the leveling rod can be repositioned in the exact same place in the future.
http:/ /en.wikipedia.org/wiki/Benchmark

Single Points

Multiple Points

Nobody is ferpect
Its only relative It has to be the same answer next time If something changes, the reference is meaningless

Computer Benchmarks
Standard Performance Evaluation Corporation (SPEC) CPU (2000, 95, 92) Graphics Web ser vers
http:/ /www.spec.org/

All Things Being Equal...

There are lies, damned lies, and benchmarks Everyone has an agenda You dont run testbeds as production Skepticism wins the day

Language Shootout
Benchmarking programming languages?
How can we benchmark a programming language? We can't - we benchmark programming language implementations. How can we benchmark language implementations? We can't - we measure particular programs. http:/ /shootout.alioth.debian.org/

Performance
A major factor in determining the overall productivity of a system, performance is primarily tied to availability, throughput and response time.

http:/ /www.comptia.org/sections/ssg/glossary.aspx

Performance (2)
A performance comprises an event in which generally one group of people behave in a particular way for another group of people

http:/ /en.wikipedia.org/wiki/Performance

Performance (3)
Your investment's activity over time. Past performance does not guarantee future results.
My accountant

How many metrics can you name?

Availability Disk Use Concurrent Users CPU Time Completion Time Memory Use Uptime Bandwidth Use Net work Lag Responsiveness Binary Size

Programmer Time

Power Speed Ease of use

Pick Any Two

You havent said Perl

Perl is just a programming language Its a High Level Language Measure Perl the same as other things Measure Perl against itself Compare the results

TIMTOWTMI

Its not just speed

Memory Use
use Devel::Size qw(size total_size); my $size = size("A string"); my @foo = (1, 2, 3, 4, 5); my $other_size = size(\@foo); my $foo = { a => [1, 2, 3], b => {a => [1, 3, 4]} }; my $total_size = total_size($foo);

Memory Use (2)

use Devel::Peek; Dump( $a ); Dump( $a, 5 ); DumpArray( 5, $a, $b, ... ); mstat "Point 5";

Need PERL_DEBUG_MSTATS

Database query time

$ DBI_PROFILE=2 db_script
DBI::Profile: 0.001015s (5 calls) db_script @ YYYY-MM-DD HH:MM:SS '' => 0.000024s / 2 = 0.000012s avg (first 0.000015s, min 0.000009s, max 0.000015s) 'SELECT mode,size,name FROM table' => 0.000991s / 3 = 0.000330s avg (first 0.000678s, min 0.000009s, max 0.000678s)

Benchmark.pm comes with Perl...

...and it sux...
Sux : Sucks :: Kwalitee : Quality

Common misuse
use Benchmark 'cmpthese'; my @long = ('a' .. 'z', ''); my $iter = shift || -1; cmpthese( $iter,{ long_block_ne long_block_len long_bare_ne long_bare_len } );

=> => => =>

q{grep q{grep q{grep q{grep

{$_ ne ''} {length} $_ ne '', length,

@long}, @long}, @long}, @long},

http:/ /www.perlmonks.org/index.pl?node_id=536503

Whats wrong with this picture?

Rate long_bare_ne long_block_len long_block_ne long_bare_len 3635361/s 3869054/s 3872708/s 3963159/s bare_ne -6% 7% 9% block_len -6% -0% 2% block_ne -6% -0% -2% bare_len -8% -2% -2% --

Mac OS X.4.5 15 G4 Powerbook perl5.8.4

Rate long_bare_len 2805822/s long_bare_ne 2805822/s long_block_ne 2840569/s long_block_len 2885232/s bare_len -0% 1% 3% bare_ne -0% -1% 3% block_ne -1% -1% -2% block_len -3% -3% -2% --

This is perl, v5.8.4 built for darwin-2level Summary of my perl5 (revision 5 version 8 subversion 4) configuration: Platform: osname=darwin, osvers=7.3.1, archname=darwin-2level uname='darwin albook.local 7.3.1 darwin kernel version 7.3.1: mon mar 22 21:48:41 pst 2004; root:xnuxnu-517.4.12.obj~2release_ppc power macintosh powerpc ' config_args='' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-pipe -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strictaliasing', optimize='-Os', cppflags='-no-cpp-precomp -pipe -fno-common -DPERL_DARWIN -no-cpp-precomp -fnostrict-aliasing' ccversion='', gccversion='3.3 20030304 (Apple Computer, Inc. build 1640)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='' libpth=/usr/lib libs=-ldbm -ldl -lm -lc perllibs=-ldl -lm -lc libc=/usr/lib/libc.dylib, so=dylib, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dyld.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup'

Millions of times a second?

Do something useful
use Benchmark 'cmpthese'; our @long = ('a' .. 'z', ''); my $iter = shift || -1; cmpthese( $iter,{ long_block_ne long_block_len long_bare_ne long_bare_len } );

=> => => =>

q{my q{my q{my q{my

@array @array @array @array

= = = =

grep grep grep grep

{$_ ne ''} {length} $_ ne '', length,

@long}, @long}, @long}, @long},

These numbers make sense

Rate long_block_ne 31210/s long_block_len 32119/s long_bare_ne 32237/s long_bare_len 32755/s block_ne -3% 3% 5% block_len -3% -0% 2% bare_ne -3% -0% -2% bare_len -5% -2% -2% --

Dont Trust! Verify!

use Benchmark 'cmpthese'; our @long = ('a' .. 'z', 0 .. 10_000, '');

my $iter = shift || -1; cmpthese( $iter,{ long_block_ne long_block_len long_bare_ne long_bare_len } );

=> => => =>

q{my q{my q{my q{my

@array @array @array @array

= = = =

grep grep grep grep

{$_ ne ''} {length} $_ ne '', length,

@long}, @long}, @long}, @long},

It takes longer to do more

Rate long_bare_ne 59.8/s long_block_ne 60.4/s long_block_len 60.9/s long_bare_len 61.9/s bare_ne -1% 2% 4% block_ne -1% -1% 3% block_len -2% -1% -2% bare_len -3% -3% -2% --

Theory of Measurement
Obser vation changes the universe Nothing is objective Tools have inherent uncertainities

Precision is repeatability

Accuracy is getting the right answer

precision != accuracy

you want both

Benchmark.pms benchmark
Uses a null loop as a control sub { } Its just a timer Subtracts the null loop time Introduces an error of about 7%

$ time perl script.pl real user sys 0m0.293s 0m0.130s 0m0.036s

http:/ /ppt.perl.org/

The map is not the terrain

my @selected () my @selected my @selected my @selected = grep { ... } @array; = grep { ... } @array; = @array; = ( ... ); ;

$ perl -MO=Concise -e 'grep { lc } @array' a 1 2 7 6 3 9 8 5 4 <@> leave[1 ref] vKP/REFC ->(end) <0> enter ->2 <;> nextstate(main 2 -e:1) v ->3 <|> grepwhile(other->8)[t4] vK/1 ->a <@> grepstart K*/2 ->7 <0> pushmark s ->4 <1> null lK/1 ->4 <1> null sK/1 ->7 <@> scope sK ->7 <0> ex-nextstate v ->8 <1> lc[t2] sK/1 -><1> ex-rv2sv sK/1 ->9 <$> gvsv(*_) s ->9 <1> rv2av[t3] lKM/1 ->6 <$> gv(*array) s ->5

$ perl -MO=Concise -e '() = grep { lc } @array' d 1 2 c 3 8 7 4 a 9 6 5 b <@> leave[1 ref] vKP/REFC ->(end) <0> enter ->2 <;> nextstate(main 2 -e:1) v ->3 <2> aassign[t5] vKS/COMMON ->d <1> ex-list lK ->b <0> pushmark s ->4 <|> grepwhile(other->9)[t4] lK/1 ->b <@> grepstart lK*/2 ->8 <0> pushmark s ->5 <1> null lK/1 ->5 <1> null sK/1 ->8 <@> scope sK ->8 <0> ex-nextstate v ->9 <1> lc[t2] sK/1 -><1> ex-rv2sv sK/1 ->a <$> gvsv(*_) s ->a <1> rv2av[t3] lKM/1 ->7 <$> gv(*array) s ->6 <1> ex-list lK ->c <0> pushmark s ->c <0> stub lPRM* ->-

$ perl -MO=Concise -e 'my @selected = grep { lc } @array' e 1 2 d 3 8 7 4 a 9 6 5 b c <@> leave[1 ref] vKP/REFC ->(end) <0> enter ->2 <;> nextstate(main 2 -e:1) v ->3 <2> aassign[t6] vKS ->e <1> ex-list lK ->b <0> pushmark s ->4 <|> grepwhile(other->9)[t5] lK/1 ->b <@> grepstart lK*/2 ->8 <0> pushmark s ->5 <1> null lK/1 ->5 <1> null sK/1 ->8 <@> scope sK ->8 <0> ex-nextstate v ->9 <1> lc[t3] sK/1 -><1> ex-rv2sv sK/1 ->a <$> gvsv(*_) s ->a <1> rv2av[t4] lKM/1 ->7 <$> gv(*array) s ->6 <1> ex-list lK ->d <0> pushmark s ->c <0> padav[@selected:2,3] lRM*/LVINTRO ->d

Premature optimization is the root of all evil

Tony Hoare

What do I benchmark?

Find the bad parts

Prole the application rst Find out whos taking all the time/ memory/net work Compare situations Fix that rst

perl -d:SmallProf script

Better algorithms do better

Summary
Decide what is important to you Realize you have bias Report the situation Dont turn off your brain Make predictions that you can verify

Further Reading
Benchmarking, The Perl Journal #11, http:/ /www.pair.com/~comdog/Articles/benchmark.1_4.txt Wasting Time Thinking About Wasted Time, http:/ /www.perlmonks.org/?node_id=393128 Proling in Perl, http:/ /www.ddj.com/documents/s=1498/ddj0104pl/