President
October 2013
Topics
The Value of Certified Scores, Peer Pressure, and Sustained Benchmark DevelopmentMost people recognize EEMBC as a reputable benchmark consortium, defining and developing benchmarks for the embedded industry, as well as defining clear and concise rules on how benchmarks must be run. However, ever since the beginning of EEMBC (circa 1997), one of our primary goals has been to supply certified scores to the industry. Certification entails a rather laborious process performed by the EEMBC Technology Center to both recreate the benchmark setup and subsequently verify the accuracy and repeatability of the benchmark results, prior to allowing members to publish their scores. While certification is still required for many of the EEMBC benchmarks, we have relaxed our policy for some of our newer benchmarks (i.e. CoreMark and AndEBench). The motivation behind this policy change for CoreMark was to allow users to freely publish their scores and therefore, rely on peer pressure to monitor erroneous or deceitful scores. Nevertheless, some EEMBC members still request certifications for their CoreMark results, just to demonstrate their commitment to providing dependable and independently verified data to their customers. Certification for CoreMark results also makes sense because EEMBC provides source code and requires the user to build the binary and run it, and although CoreMark contains various self-checking mechanisms, there is still room for error (and it’s been demonstrated). But we’re still relying on all of you CoreMark users to submit your scores and add to our online database – certified or not. The more data we have, the easier it is for people to cross-check and detect suspicious results. For AndEBench, we have also implemented a similar policy change that does not require certification to publish. Unlike CoreMark, AndEBench is downloaded as a binary to your Android device and simply requires the push of a button to run it. Obviously, this ease of use makes it possible for anyone with an Android device to run AndEBench, and as a result, over 7000 users have done so. Even though users only have access to binary, the source code of AndEBench, as well as the entire definition and development of the benchmark is open to all EEMBC members (to ensure fairness to all). The typical user of AndEBench has no desire to ‘game’ the benchmark, although hackers have been known to alter the device clock frequency to produce non-standard results (Incidentally, this wouldn’t be a bad thing if the modified frequency was publicly known). However, the bigger concern lies with those who have the ability to alter the platform. For example, in a recent article by Anand Shimpi and Brian Klug, the authors point out how OEMs are detecting benchmarks and preventing CPU throttling to achieve higher performance. This wouldn’t be such a bad thing if EVERY device followed a similar procedure – at least the essence of apples-to-apples comparison would still be maintained. But again this highlights the value of a credible certification methodology, or at least relying on savvy analysts and users to sniff out the foul-smelling results. It also puts the onus on EEMBC and its community of industry leaders to continually evolve its benchmarks to stay one step ahead of the score manipulators (or at least a ½ step ahead!). New Search and Compare Feature for AndEBenchAndEBench popularity continues to grow and so do all the device scores on our website. As a matter of fact, there’s gotten to be so many device scores that it made nearly impossible to compare. So we have added a new search and compare feature that allows you to compare phones and tables of different brands, devices, CPUs, and GPUs. If you don’t have AndEBench installed yet on our Android phone or tablet, you can access it for free from GooglePlay™ and the Amazon™ Appstore for Android. Also, earlier this year I indicated that we were working on a second-generation of AndEBench 2.0 that will focus on both hardware- and platform-level tests. The new AndEBench will include a wide range of CPU tests, memory bandwidth and latency tests, storage tests, a 3D test, and a host of platform tests that will replicate the performance of Android applications by using common services that many of these applications use; these services include GUI rendering, XML parsing, image operations, and cryptography. This will be the ultimate Android device benchmark – expect to see it before the end of the year. EEMBC encourages all vendors and manufacturers to join the consortium’s working group to contribute to the development of AndEBench 2.0. To join the working group or gain access to the source code for AndEBench, contact Markus Levy for details. EEMBC Exhibit at Linley Processor ConferenceOn October 16th and 17th, 2013, The Linley Group will host the 7th Annual Linley Tech Processor Conference in Santa Clara, CA. Exhibits are open on October 16 from 16:40-18:00, and EEMBC President (Markus Levy) and Director of Technology (Shay Gal-On) will be on hand to answer any questions. The conference is free to attend for qualified attendees, and it focuses on processors for enterprise- and carrier-communications systems. In addition to the 27 technical presentations by experts from the companies leading the industry, the two-day conference program will include a keynote session covering technology and market trends in communications systems. EEMBC in the NewsArticles utilizing AndEBench:
On GoingDo you need a comprehensive, reliable, unbiased benchmark to test the performance of processors with floating-point units? You can now license the new EEMBC FPMark suite. Uniquely, FPMark contains single (32 bit) and double (64 bit) precision workloads, as well as a mixture of small, medium, and large data sets to support everything from microcontrollers to high-end processors. The EEMBC FPMark Suite uses 10 diverse kernels to generate 53 workloads, each of which self-verify to ensure correct execution of the benchmark. Non-members may obtain the entire FPMark suite, including source code and documentation, for only $495 ($195 for universities and research programs). Follow us on Twitter. Sign up at @eembc_org. Visit us on Linkedin. http://www.linkedin.com/company/eembc If you do not wish to receive e-mail from EEMBC, you can un-subscribe. EEMBC sends no more than one e-mail per month to registered users at www.eembc.org. Continuing your subscription ensures you'll be notified when new scores and other important announcements are available. |