NOTE: eembc.org will be down intermittently for maintenance from August 27th to August 31st.

ULPMARK


An EEMBC Benchmark 

ULPMark™ FAQs

How do I submit my ULPMark results? What are the "Run Rules"?

Scores are submitted using the ULPMark host software's "Submit" function. Only license holders of the software may upload scores (this includes corporate and university licensees as well as EEMBC workgroup and board members). The following run-rules apply:

ULPMark-CP and ULPMark-PP

  1. The firmware must be ported to the device under test. Refer to the porting instructions in the respective user manuals. Note that some parts of the firmware are not allowed to be modified.
  2. The device under test must be powered by the official EEMBC Energy Monitor.
  3. Ambient temperature during execution must be higher than 21C.
  4. The device must use a 32kHz external oscillator (+/-50ppm tol.). If scores are reported with an internal oscillator, they must also include a score uploaded with an external. Certification is only done with an external oscillator.
  5. A wake-up frequency of 1Hz needs to be ensured with a tolerance of max. 1% (considering device variances, and run rule ranges for temperature and voltage).

ULPMark-PP

  1. The host software will perform some basic sanity checks: ensuring each slot ran for one-second, and that there were no errors in the power trace (samples outside the range of the hardware).
  2. The ADC sampling frequency may be 1,000 or 1,024 Hz, whichever is easier to implement, with a +/-1% tolerance
  3. The 1MHz PWM must be >950kHz, and the period must be updated every 10ms (+/-5%)

Once a score is submitted, one of two things happen:

  1. If the score was submitted by someone who is not employed by the device manufacturer (aka "cross-posting") they must also include the source code. If the device manufacturer is a member of EEMBC, then an email will be sent to that member alerting them of the upload. The member then has two weeks to review the source code to verify it is a valid project, if not the score will not be published. Otherwise after the review the score will be published. If the device vendor is not a member, the score will be published within two weeks.
  2. If the score was submitted by the manufacturer than the score will be published within two weeks.

A score submitted by a vendor always supersedes a score submitted by anyone else, even if the vendor score is lower.

Certification

Certification is a process that exposes the benchmark to additional scrutiny: EEMBC certification labs review the source code and verify the scores are reproducable. Companies that request certification must also make the source code available internally to EEMBC workgroup members, this adds an additional layer of transparecy.

As a member, request certification by clicking "Certify this result" when submitting a score. You must then send TWO typical boards to EEMBC along with the source code and the electrical hookup instructions. Board modifications are limited to jumpers and solder bridge removal (for isolating high-power devices not used in the test). EEMBC certificaiton labs will then review the source code port to make sure core benchmark code was not modified, validate the electrical connectivity & GPIO, verify the score by running multiple times, and then mark the score as certified on the website.

Am I allowed to use ULPMark scores in my literature? What are the "Publication Rules"?

The ULPMark licensee publishing rights are determined by the ULPMark workgroup, and may differ from other EEMBC benchmarks' publishing rules.

A licensee may not publish ULPMark scores publicly unless the scores are visible (submitted and published) on the EEMBC ULPMark public website. If the scores are on the public website, and they are not certified, that must be indicated in said literature. 

A licensee may share unpublished scores under NDA, but publication status must be stated, as must certification status.

What does 'typical' really mean and does ULPMark address this?

Typical can refer to the workload. There are many ways a vendor might choose to define typical based on their customer's usage models. The ULPMark benchmark standardizes workload profiles to remove this ambiguity by explicitly defining the operations to measure, allowing comparisons between different vendor's products.

Typical can also refer to a statistic distribution in the CPU die manufacturing varation, and power can be very sensitive to these systemic and stochastic variations. The most robust way to define a typical variation would be to measure ULPMark on tens of thousands of die. This is obviously prohibitive. It is entirely possible that a licensee running ULPMark on their board will see a difference in the measured vs reported power numbers if they did not purchase a "typical" part. While it is impossible to prevent this (as it is the nature of semiconductor manufacturing), EEMBC works to reduce this by requiring that vendors supply "typical" material for testing and certification as measured by what they consider the majority of silicon, or the most likely silicon a consumer would receive when purchasing a device off-the-shelf, as opposed to "cherry picking" the best scoring die.

Why doesn't Windows detect the Energy Monitor?

Be sure to install the STMicroelectronics Virtual Com Port driver first. This is required for Windows (but not Linux or macOS).

Is ULPMark a benchmark for 8-, 16-, or 32-bit architectures?

The benchmark is defined under the premise that it is architecture independent.

How is the ULPMark calculated?

Since "bigger is always better" in benchmarking, the ULPMark benchmarks use a formula that takes the reciprocal of the energy values.

ULPMark-CP = 1000 / median(5 runs of the benchmark for 10 seconds)

ULPMark-PP = 10000 / median(5 runs of the benchmark's 10 slots, e.g. 10 seconds)

From an energy perspective, what's the difference between running with an internal RC oscillator and an external crystal oscillator?

Generally speaking, a crystal oscillator requires a higher minimum drive current in order to oscillate due to the higher load capacitance as well as the external pin capacitance and resistances. An internal RC oscillator can have a very small capacitance and resistance and so needs very little drive current.

Does the ULPMark benchmark also calculate battery life?

No. The user would need to calculate this based on the power reported from the Host UI and the "mAh" rating of the battery under consideration.

When building the code for ULPMark, is it allowed to apply compiler optimizations, such as optimize for code size or speed?

Yes. Any optimizations will affect the energy usage.

What are the workload profiles for the ULPMark benchmarks?

The ULPMark-CP (CoreProfile) performs a variety of functions commonly found in ULP applications; among them are memory and math operations, sorting, GPIO interaction, etc.

The main function of the benchmark, which runs in an infinite loop, consists of:

  1. Toggle a GPIO indication port 20 times quickly ending with active indicator.
  2. Run a math_int16 filter and state_machine 24 times, using the cumulative result of the math functions as the input to the state function.
  3. Run an LCD conversion function using the output of the state function as input, with display size set to 8
  4. Run a string-search function on a string chosen based on the the previous step. Test the output value.
  5. Run a byte copy over the size of the chosen string.
  6. Run a bubblesort function on the result of the previous step, and test the result.
  7. Run a permute function for 16 permutes
  8. Indicate end of workload iteration using the GPIO active indicator
  9. Go to sleep and wake up in 1 second

Details on the functions:

  1. The filter function performs a simple filter using 16b math. A routine to filter away the DC content from an AC mains waveform signal. It does this using a heavily damped integrator to estimate the DC level. The current DC level is then subtracted from the signal.
  2. The LCD function will write values to the buffer that will represent LCD characters. The values written are based on the value of the bin parameter, and actual value to write is picked from the lcd_7segment_ascii_table.
  3. The permute function sets a particular bit. The value to modify in table is based on size_mask lower bits of the index. 4. The bit to modify is based on the 3 lower bits of the index. The bit is set if the value parameter is not 0, unset otherwise.
  4. The sort implements a simple bubble sort for 8b data.
  5. The state machine implements a simple state machine using a switch statement.

The ULPMark-PP activates one or more of the following four peripherals in one of 10 fixed configurations, known as slots.

  1. Real Time Clock (RTC)
  2. Analog-to-Digital conversion (ADC) at 1 Hz and 1 kHz
  3. Pulse-width Modulation (PWM) at 32 kHz and 1 MHz
  4. Serial Peripheral Interface (SPI)
Where can I purchase an Energy Monitor (EMON)?

The Energy Monitor is manufacture by STMicroelectronics, but EEMBC provides the benchmarking software. Refer to this link for ordering information:

http://www.st.com/en/evaluation-tools/x-nucleo-lpm01a.html

What is the voltage range of the Energy Monitor?

1800mV to 3300mV with maxumum 50mA dynamic current and ~50nA minumum.

What happens when the maximum energy monitor current is exceeded?

If the device under test draws more than 50 mA, the energy monitor status LED will turn red and the supply voltage will be disabled. You will need to reset the energy monitor.

Is CoreMark used in ULPMark?

Although we would like to use CoreMark, since it is an industry-accepted standard, the energy demand of CoreMark is far beyond the typical workload of an ultra-low power application. Hence, the ULPMark working group has developed a different set of workloads that are more relevant and real-world.

What is the roadmap for the evolution of the benchmark?

Phase 1 of the benchmark is called ULPMark-CP, for Core Profile, and is primarily focused on the active or run mode (code execution). Other phases currently in development, will cover more complex applications to test other functions, such as smart peripherals and various power modes of the microcontroller.

Phase 2 is called called ULPMark-PP, for Peripheral Profile, and its focus is on four common peripherals.

Future variations of the benchmark will focus on different low-power CPU and peripheral configurations.