FAQ

  1. What is EEMBC's mission?
  2. Why aren't benchmark scores available for all embedded processors?
  3. What steps does EEMBC take to ensure that the published benchmark results are accurate and honest?
  4. What's with the extra E at the beginning of EEMBC?
  5. How do I qualify to be an EEMBC member?
  6. What are the main steps in becoming a member of EEMBC?
  7. Is my whole company included in our EEMBC membership? Can more than one person in a member company access the benchmark source code?
  8. What sort of person should be the representative from our company to the EEMBC Board of Directors?
  9. Do we own the benchmark source code once we have become a member?
  10. How are membership dues spent?
  11. Can I just audit EEMBC activities?
  12. Does EEMBC have benchmarks for DSPs?
  13. EEMBC has such a variety of benchmarks, how do I determine which one to use?
  14. What process does EEMBC use to select specific benchmark algorithms?
  15. Does EEMBC perform board-level or system-level benchmarking?
  16. In out-of-the-box benchmarking, a vendor can use any compiler, but cannot alter the source code in any way. Does the compiler have a significant effect on the benchmark score, even in out-of-the-box scores?
  17. What do the out of box scores reveal about compiler performance and/or processor performance?
  18. Must a compiler vendor be an EEMBC member to have scores posted on the EEMBC web site?
  19. EEMBC benchmarks attempt to resemble real-world applications by allowing members to optimize the source code to take advantage of their architectures. How do you define the optimization guidelines so customers can still make apples to apples comparis
  20. What are the main steps in obtaining certified benchmark scores for my processor?
  21. Are any additional tools needed to run the benchmarks?
  22. What information goes into the disclosure form?
  23. What does the certification process cost?
  24. Why does EEMBC have an FFT filter benchmark in both the Telecommunications and Automotive/Industrial benchmarks?
  25. Why don't the Telecomm benchmarks use floating point?

What is EEMBC's mission?

The Embedded Microprocessor Benchmark Consortium (EEMBC - pronounced embassy) is a non-profit organization that works collaboratively with member companies to develop performance benchmarks that target key applications of embedded systems. EEMBC activities include the EEMBC Technology Center, which delivers technical support to members and licensees, certifies benchmark scores, and oversees the development of benchmark suites.

Why aren't benchmark scores available for all embedded processors?

Publication rules vary between benchmarks, but some members simply choose not to publish results, and not every licensee has the bandwidth to port benchmarks to every platform and publish results. If a processor score is not available on the EEMBC website, you may request that score on line or directly with the corresponding processor vendor. Customer requests for scores on specific processors are a key mechanism by which more scores are published, and EEMBC encourages you to request certified scores for any processor you are considering for a design.

What steps does EEMBC take to ensure that the published benchmark results are accurate and honest?

EEMBC offers a certification stamp for members who wish to guarantee that the benchmarks were run according to the specific run rules of the benchmark. The certification process scrutinizes the modified source code, validates that the verification flows (available on some benchmarks) pass, and double checks external hardware interaces with a logic analyzer (if necessary) to verify the integrity of the certified score.

What's with the extra E at the beginning of EEMBC?

Up until 2012, EEMBC stood for the "EDN Embedded Microprocessor Benchmark Consortium". The name has since been changed to drop the "EDN" (as EDN is no longer in business), but we kept EEMBC because for over 20 years that's what we've been known as: it's our identity. Plus it is easier to Google... And heck, the two E's looks pretty cool.

How do I qualify to be an EEMBC member?

You must be a System Developer, Designer and/or manufacturer of embedded microprocessors, microcontrollers, or DSPs, and other devices and architectures, and could benefit from the use of the EEMBC Benchmarks on your products.

What are the main steps in becoming a member of EEMBC?

Fill out the Request Form

Is my whole company included in our EEMBC membership? Can more than one person in a member company access the benchmark source code?

Once you have become an EEMBC member, as many people in your company as you like may download the EEMBC benchmark suites, but each such individual must be advised of the EEMBC rules and the provisions of the EEMBC Member License Agreement. Moreover, each person accessing the members-only area of the EEMBC Web site to download benchmarks or obtain other information must register individually. You are also responsible for seeing that any employees who leave your company do not take the benchmarks outside of your company.

What sort of person should be the representative from our company to the EEMBC Board of Directors?

EEMBC Board of Directors meetings combine both technical and marketing-oriented discussions, and our member representatives include individuals whose roles in their respective companies are purely technical, purely marketing, and everything in between. In general, the most technical discussions in EEMBC take place on the working group level by teleconference and web-meetings outside of Board meetings.

Do we own the benchmark source code once we have become a member?

No, EEMBC owns the benchmark suites; your membership makes you a licensee.

How are membership dues spent?

Your member dues go towards the development, maintenance, and marketing of the benchmarks as well as funding certifications and technical support from the EEMBC Technology Center.

Can I just audit EEMBC activities?

No, you must be an EEMBC member.

Does EEMBC have benchmarks for DSPs?

Yes. Our Digital Entertainment and Telecom benchmark suites contain a number of kernels and applications that are highly relevant for DSPs. Furthermore, EEMBC

EEMBC has such a variety of benchmarks, how do I determine which one to use?

Focus on those parameters that most closely match your application needs. A designer typically uses the benchmark scores as a criterion to pick the best processor for the application. For example, if the application does many table look-ups, then that parameter (in the Automotive/industrial benchmark suite) is most important and the designer will like give the most weight to that benchmark. On the other hand, if the application is a modem, then the benchmark scores in the Telecomm suite become the most important parameters.

What process does EEMBC use to select specific benchmark algorithms?

Algorithms are selected by discussion, consensus, and ultimately a vote of the member companies. Non-members are also invited to suggest benchmark algorithms that you would like to see the consortium develop.

Does EEMBC perform board-level or system-level benchmarking?

The Version 1.0 benchmarks are processor benchmarks. However, the board and/or system can make a significant difference in processor performance (for example, the memory subsystem is one aspect that can dictate performance). EEMBC

In out-of-the-box benchmarking, a vendor can use any compiler, but cannot alter the source code in any way. Does the compiler have a significant effect on the benchmark score, even in out-of-the-box scores?

Absolutely. Feedback from several processor vendors indicates that they have seen performance differences upwards of 40% simply as a result of using a different compiler. This is why vendors such as Green Hills and IAR are actively involved in the EEMBC process.

What do the out of box scores reveal about compiler performance and/or processor performance?

Over time, as more processor vendors submit scores from a variety of compilers running on the same processor, this will provide very valuable information. Also, if you held the compiler family constant and varied the processor, this would be a great way to compare a processor's C friendliness.

Must a compiler vendor be an EEMBC member to have scores posted on the EEMBC web site?

A compiler vendor's results on the web site are not related to whether or not that vendor is an EEMBC member. The processor vendor member typically makes that decision, and they usually publish scores from the compiler that yields the best scores.

EEMBC benchmarks attempt to resemble real-world applications by allowing members to optimize the source code to take advantage of their architectures. How do you define the optimization guidelines so customers can still make apples to apples comparis

EEMBC-allowed optimizations are decided on a benchmark-by-benchmark basis. There are some fundamental restrictions in place to ensure fair comparison. EEMBC allows optimizations that change the source code to take advantage of libraries or hardware accelerators. These restrictions will allow you to demonstrate an architecture's capability and facilitate practical comparisons. You can also use intrinsics ( i.e. C code modification), or change the algorithm in some cases. For example, if EEMBC has a radix-4 implementation of an FFT, we will allow it to change to radix-2. So it is still an FFT, but on some DSPs that makes a big difference.

What are the main steps in obtaining certified benchmark scores for my processor?

1. Log in to EEMBC member section and 'Make a Certification Reservation'.

2. Build and run the benchmark(s) on your platform.

3. Send your platform and tools to EEMBC Technology Center. Depending on the benchmark, you may be required to submit more than one board for verification.

4. Enter your benchmark data in the online disclosure report (in member section), or run the host software that submits the score to the website for you (depends on the benchmark).

Are any additional tools needed to run the benchmarks?

No. For the processor benchmarks, you need to download the benchmark and port these to your board or platform. For a fee, the EEMBC Technology Center can do this for you.

What information goes into the disclosure form?

The disclosure form contains all the information necessary to recreate the benchmark environment. In addition to all the benchmark scores, it ontains board-level information (memory configuration, etc.), compiler information (settings, libraries, etc.), and processor information (core and bus speeds, cache sizes, on-chip peripherals, etc.) After certification is complete, and the scores are published, the associated disclosure is publicly available alongside the benchmark scores.

What does the certification process cost?

As an EEMBC member, you are entitled to certifications per year (certain restrictions apply).

Why does EEMBC have an FFT filter benchmark in both the Telecommunications and Automotive/Industrial benchmarks?

FFTs are used more often in Telecomm applications, but they are becoming increasingly important in automotive and industrial applications. Sensors used in engine knock detection, vehicle stability control, occupant safety systems and flow control, all require sophisticated filtering of the incoming signal to determine what is really happening among a lot of noise.

Why don't the Telecomm benchmarks use floating point?

All of the kernels (Bit Allocation, AutoCorrelation, FFT/iFFT, Viterbi, Biquad IIR, Convolutional Encoder) in the suite are intended to be run (for comparison purposes) using 16-bit fixed point math. There are FP versions of some of the kernels which were used to generate the golden data file sets. At this time, we have no plans to extend the "Out of the Box" code to require/use FP math. You may re-code the kernels, using FP for an optimized version of the benchmark if you wish. At this time, we do not have access to any floating point data sets. The data sets which accompany the AutoCorr kernel are in 16-bit (1.15) fixed point format (that is: 1 sign bit, 15 bits of data) to reduce the precision of the data, you can remove (or zero) bits from the LSBs of each number.

Copyright © EEMBC