eembc.logopeter.torelli
Issue: Q3, 2019 Peter Torelli, President

The EEMBC BenchPress

The Embedded Microprocessor Benchmark Newsletter

Topics

MLMark Version 1.0 Launch a Success

EEMBC’s Heterogeneous Compute subcommittee is delighted to announce the launch of the MLMark machine-learning inference benchmark. The team spent a solid year wrestling with the problem of how to reign-in the wide variety of optimizations and configurations that up until now plagued ML performance comparisons. The result is both a meaningful analysis tool and educational jumping-off point for developers and researchers alike. MLMark Version 1.0 deployed three popular vision models on four target frameworks, and we are now focusing on adding scores to the database for those targets. Simultaneously, the working group is developing two new versions: Version 1.0.x will increase the number of accelerators supported and add updated driver and O/S support for existing targets; Version 2.0 will add new models to cover a wider range of hardware and research trends. Not to be forgotten, energy consumption and end-to-end pipeline scenarios are still on our ever-growing to-do list.

The Machine Learning Working Group is actively looking to enlist new members, especially those from the more esoteric realms of AI acceleration, to be a part of defining version 2.0. Please contact peter.torelli@eembc.org for details on how to join.

Read the full press release.

IoTConnect Wi-Fi Benchmark Team Opens up the Throttle

After months spent researching key aspects of Wi-Fi in the IoT space (and initiating into the team several new members from Silicon Labs, TI, Dialog Semiconductor, and Cypress) the IoTConnect Working Group is entering execution mode. This involves outlining the architectural needs for all of the benchmark’s components, such as: changes to the IoTConnect test harness to support higher-power demands; additions to the Radio Manager Gateway to handle interference; extensive configuration-option discovery in the DUT for various application protocols (MQTT, HTTPS, CoAP); and … well, a whole lot more! Wi-Fi is a different beast compared Bluetooth Low Energy, but building on the group’s experience developing EEMBC’s IoTConnect™-BLE benchmark, we are well-positioned to deal with the challenges.

The primary behavioral model has not been defined yet, so there is still time for silicon vendors, integrators, and other Wi-Fi experts to join the working-group and help define the soon-to-be de facto standard for IoT Wi-Fi benchmarking.

More info will be posted here as development progresses.

It’s Time to Talk About RToS

As cloud-based IoT services sprout their tendrils into the edge-device itself, cloud providers are developing their own embedded frameworks to speed time-to-market. Amazon’s acquisition of FreeRTOS and Microsoft’s ThreadX are just two examples of the big players in this space. Today, developers find themselves faced with multiple Real-Time Operating Systems (RToS), and no clear way to evaluate performance tradeoffs. Since most MCU features required for effective real-time processing aren’t covered by traditional core-focused benchmarks, EEMBC is looking for participants to join a new working group dedicated to characterizing RToS performance this fall.

Please contact peter.torelli@eembc.org for more details.

CoreMark™-PRO Meets GitHub

CoreMark-PRO is now available on GitHub, joining CoreMark® and MLMark™ as part of EEMBC’s growing presence in the open-source community. CoreMark-PRO characterizes real-world applications such as Zip compression and XML parsing, as well math operations like single-precision Livermore loops, LINPACK, and Radix-2 FFTs, to name a few. Making it even more applicable to today’s multi-core environment, CoreMark-PRO enables scaling performance analysis through an intuitive POSIX-based thread API to test worker and context parallelism. Finally, all of this goodness is boiled down to a single figure-of-merit that makes comparisons a snap.

Similar to MLMark, a corporate license is now only needed if using results in marketing or PR materials; academics do not need a corporate license to publish results. (But you still need to follow the Run Rules!)

IEEE IRDS Application Benchmarking Report, 2019

Peter Torelli (President of EEMBC) and Anoop Nair (EEMBC Board of Directors, Senior Director at Flex) have been working with Prof. Tom Conte (Georgia Institute of Tech.), Prof. Vijay Reddi (Harvard, MLPerf) and several other industry experts to draft the IEEE International Roadmap for Devices and Standards Application Benchmarking report on Machine Learning, due later this fall. Everyone in the industry is struggling with the same challenges characterizing machine learning performance since the last report was published in 2017, and today there are exponentially more models, more hardware, and more TOPS claims. Fortunately, we brought benchmarks this time. Stay tuned.

Visit the IEEE IRDS website for more information.

New Member Certifications

Members continue to reinforce the value of EEMBC benchmarks in their datasheets through certifications. In this update, STMicroelectronics has certified two of their STM32 processors, the STM32H745XI on CoreMark and the STM32WB55 on CoreMark and ULPMark-CP.

The STM32H745 includes the high-performance Arm® Cortex®-M7 and Cortex®-M4 32-bit RISC cores and achieves a score of 3223 CoreMarks using both cores. The STM32WB55 multiprotocol wireless and ultra-low-power devices embed a powerful and ultra-low-power radio compliant with the Bluetooth® Low Energy SIG specification v5.0 and with IEEE 802.15.4-2011. They contain a dedicated Arm® Cortex® -M0+ for performing all the real-time low layer operation. The WB55 scored 216 CoreMarks running from Flash with ART and prefetch enabled, and 219 CoreMarks with code in SRAM2 and data in SRAM1. ULPMark-CP scores clocked in at 158 ULPMarks powered by 3.0V and 303 at 1.8V.

Question? Comments? Please use our online contact form to share your thoughts.

Want to unsubscribe? Follow this link to be removed from future emails.