Overview

EEMBC's IoTMark-Wi-Fi benchmark examines the energy efficiency of embedded platforms supporting the 802.11 radio protocol, commonly known as Wi-Fi. It does this by emulating the real-world behavior seen by battery-operated wireless devices. The insight provided by this benchmark is crucial for not only evaluating claims of battery-life, but for understanding and optimizing systems so that they can last longer on a single charge.

» Press release

802.11 Wi-FI has connected billions of desktop computers, and new low-power variants are posed to take over the IoT space as well.

Behavioral Model

A behavioral model describes what the benchmark actually does. The execution of the profile must be tightly controlled and completely repeatable for the benchmark to be meaningful. It must also be clearly defined so that measurements are comparible across different vendor's devices. IoTMark-Wi-Fi's behavioral model is split into three phases, two of which are used for scoring. The third is offered for analysis purposes.

Phase One: Connected Idle

Low-power devices spend a significant portion of their time in a mode called connected idle. This is a mode where the DUT (station) indicates to the Access Point (AP) that it is still connected, but will be entering a low-power state and will not be alert for every 802.11 beacon. Instead, the AP and the DUT agree on a sleep interval ahead of time, indicating precisely when the DUT will check to see if it needs to fully power-on. By using what is known as a Delivery Traffic Indication Map, or DTIM, the AP can tell the DUT to wake up and receive its queued data. The DUT DTIM check occurs every sleep interval of beacons. If there is data waiting, then the DUT powers-up to receive and process this data, re-entering connecterd standby after it has completed.

Phase Two: Application Communication

In order to minimize energy, several low-power communication protocols have been developed to operate over TCP/IP on 802.11. MQTT is one such protocol: it is a publication/subscribe model, where a central server is responsible for displatching incoming messages to subscribers. The benchmark uses an MQTT server installed on the AP to communicate with the DUT in a tightly-controlled manner. During this phase of the benchmark, the host system instructs the the DUT to perform both an Rx-Tx and an Rx-only transaction using MQTTS. MQTTS is a TLS enabled mode of MQTT, which means the data packets transmitted are encrypted with AES.

Phase Three: Connected Active Mode

This portion of the benchmark exposes the energy costsd of high-bandwidth communication, and is not part of benchmark score. However, it does allow the user to investigate the real-world power of their device when sending large amounts of data via TCP or UDP. It utilizes an iperf server running on the AP.

Framework

What makes the benchmark effective is the EEMBC Benchmark Framework. The Framework is a combination of hardware, software, and firmware that controls when the device connects to the Access Point (AP), the connection parameters, and when the device sends and receives data using the MQTT communication protocol.

Hardware

The hardware component of the Framework consists of the following:

• A host PC to run the Framework software
• The device under test (DUT)
• The 802.11 Access Point (AP)
• An energy monitor, which supplies and measures power to the DUT; the following energy monitors are supported:
  • STMicroelectronics LPM01A
  • Jetperch Joulescope JS110
  • Keysight N6705 (with 2x6781 or 1x6781+1x6731)
• An IO Manager that acts as an electrically-isolated proxy between the host PC and the DUT
• Additionally, we highly recommend RF isolation, such as the octoScope BOX-26

The diagram below illustrates how these components are connected. The User Guide for the benchmark contains a BOM list for sourcing parts from DigiKey or Farnell.

Software

The software component is a Host UI Runner application. It runs on the host PC and connects to the devices through the USB interface. From the control panel in the GUI, the user has the choice of running the benchmark using the official settings, or adjusting them to perform experiments and gain insight into their platform.

Figure 2. Benchmark control panel.

Figure 3. Zoomed-in energy chart for an MQTT transfer.

Firmware

Since the goal of the benchmark is run the same behavioral profile on a variety of DUTs, tempate source-code is provided for interfacing with the framework. This template code allows the DUT to speak to the Host PC using a thin API. This API is implemented by a developer using their platform's Software Developer Kit (SDK). For example, the API provides a function called "th_wifi_connect", which takes an SSID and passkey. The developer then implements this function using their SDK. There are roughly a dozen functions in the API for this benchmark.

Scoring

The score of the benchmark is computed as follows (power in Watts):

IoTMark-Wi-Fi = 0.217 / (2/3 * Phase-1 Power + 1/3 * Phase-2 Power)

As the power decreases, the score increases. The scale factor adjusts the inverted power into a value somewhere between 10 and 2000. The scalar value 0.217 is roughly the conversion factor for two ideal AA alkaline battery's total charge. As a result of the multiplication, the score somewhat reflects "days of battery life," however real-world batteries vary considerably with discharge curve, heat, manufacturing, and other considerations. So it is not accurate to claim this as the true days of battery life, instead, it is simply a "ballpark" conversion.

Submitting Scores

Scores may be submitted by any licensee. The Host GUI Runner program that comes with the benchmark will enable score submission if all of the basic validation checks succeed after a given run completes.

Obtaining the Benchmark

In order to run the benchmark and submit scores, you must obtain a license.