[MAGEEC] [BEEBS] Plackett-Burman Initial Review

[George Field]

Hi all,

I've just finished doing a bit of analysis on a small subset of GCC passes
for BEEBS. I still need to pull in some of the recent changes to BEEBS,
namely deleting the benchmarks that are no longer part of the suite (the
gdb-* benchmarks seem to be skewing the results somewhat) - but the results
are still interesting.

I've ran 16 tests 3 times, testing the first 12 optional passes. Possibly
the most interesting thing I've produced from the energy measurements is
the following graph:

https://raw.githubusercontent.com/ks07/beebs/plb/plb/main_effects_test.png

I was aiming for a clustered bar chart, but instead settled on stacked
columns, as there were far too many data points and the chart was
cluttered. To explain the chart: I've plotted the percentage change between
the average energy usage of each benchmark with each pass enabled and
disabled. Thus, a negative value shows that the pass reduced the energy
usage of a benchmark. In terms of the chart produced - bands below the x
axis are where the benchmark had reduced energy,  whereas those above used
more.

You'll notice that no GCC pass was universally good or bad wrt the energy
usage of our benchmarks. However, it's clear that the majority of passes
have a tendency to either improve or impair the energy usage, on average.

Another, more detailed look at the main effects shows the 3 best, and 3
worst passes for each benchmark.

https://github.com/ks07/beebs/blob/plb/plb/best_passes.txt

In this file, you'll see the name of the benchmark, followed by the best 3
passes and the percentage change on their energy usage. Following that are
the 3 worst.

One thing I noticed is that 2dfir had disproportionately large magnitudes
in the energy changes. Therefore, I excluded 2dfir from the chart linked
earlier. I will see if this changes after I've pulled in recent beebsv2
changes.

I believe comparing the means of enabled vs disabled is the way to
determine main effects. However, I'm not sure how to determine whether or
not the difference is statistically significant - if you look at the raw
data (https://github.com/ks07/beebs/blob/plb/plb/rudimentary_analysis.txt),
a large portion of the energy changes are very small (for example,
1.953e-14%).

Thanks,
George
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mageec.org/pipermail/mageec/attachments/20140811/1998f42d/attachment.html>