IoTMark^®-Wi-Fi

An EEMBC Benchmark

« IoTMark Home · About · Run Rules · View Scores

Run Rules

All EEMBC benchmarks have similar run rules, but each group has to account for the benchmark particulars so there may be additional rules included in some. The following rules apply only to the IoTMark-Wi-Fi benchmark.

Explicit Run Rules: Rules checked by the Host UI or explicitly coded in the firmware.
1. Submission must use the settings defined in the behavioral specification.
2. Only the N6705 (6731+6781 or 2x6781), JS110 may be used as an EMON.
3. The score must be collected and submitted with the Host UI runner using the EEMBC firmware.
4. Minimum 3.0V (submitting above is allowed, e.g., board requires 5V).
Implicit Run Rules: specifications that we mandate, but cannot check without certification.
1. The ee_ functions shall not be modified by the submitter.
2. The submitter may re-run repeatedly and choose the score to submit using the GUI “reload” function.
  1. Certification expects that the EEMBC-measured score will be no more than 5% lower than submitted score.
3. The DUT must be a “typical” device SKU that anyone may purchase; the submitter may not “cherry-pick” rare ultra-low power parts.
4. It is allowed—and recommended—to use an RF Isolation chamber.
5. There is no Wi-Fi channel requirement; default is 6.
6. Minimum ambient temperature is 21C; the submitter may not use a cooling chamber.
7. Only one power-supply shall be connected to the DUT: measured VCC. No other power may be supplied. (This does not include the low-side voltage of the level-shifters, which must match the board VCC but come from an unmeasured source.)
8. Any energy used during the benchmark must be drawn from the energy monitor.
9. Tx dBm shall be set to the DUT maximum transmit power (the behavioral model describes a mostly-idle edge node, Tx power is very small fraction of that power).
10. Use of the on-board Broadcom Wi-Fi is not required; the submitter may use a different Wi-Fi adapter on the Radio Manager subject to the following rules:
  1. The Submitter may not use the adapter in a way that conspires with the DUT to cheat the benchmark.
  2. The adapter must be off-the-shelf that anyone may purchase.
  3. If the on-board Broadcom QoS NULL is affecting the score, the following adapters do not exhibit that problem and are supported via the default RT8814 driver:
    1. Alfa AC1900 WiFi Adapter - 1900 Mbps 802.11ac Long-Range Dual Band USB 3.0 Wi-Fi Network Adapter
    2. Netgear Nighthawk AC1900 (A7000-10000S)
    3. Edimax EW-7833UAC AC1750 Dual-Band Wi-Fi USB 3.0 Adapter
11. The HostAP beacon must be 340 bytes.
  1. If a different Wi-Fi adapter is used, the user must run WireShark (or similar) and change the vendor key parameter padding on the HostAP configuration (the command is hostapd set vendor_key N, which persists between reboots).
Submission Rules
1. Only licensees may submit scores.
2. The DUT hardware must be publicly available.
3. The permitted board modifications include: jumpers, switch settings, solder bridge removal, and trace cuts.
4. Submitter must fill out submission disclosure.
  1. https://www.eembc.org/iotmark/iot-wifi/submit/
Certification Rules: Extra steps for requesting a certification.
1. Upload source code port files (minus SDK files that require a license).
  1. Only licensees may view the source code files, these are not made public.
2. Augment submission instructions to include how to compile.
3. Provide a compiler license to EEMBC if using a custom compiler.
4. Ship two typical-SKU boards to EEMBC Tech Center (address on website).
  1. Include return shipper for international certifications.
  2. Include the Radio Manager Wi-Fi adapter, if a different one was used.
5. Notes:
  1. Certifications are offered free for members, and for a fee to licensees.
  2. Certified scores can and should use the “EEMBC Certified” logo.
Publication Rules
1. Scores cannot be discussed publicly without first uploading and published to EEMBC website (so that they can be cited)
  1. Only under NDA with 3rd party non-submitted scores be discussed, and it must be stated clearly that the scores are not published or certified.
  2. To cite a score, use the benchmark score #, or even a link to the viewer for that score.
2. Once a score is submitted it cannot be modified by the submitter.
  1. Modification requests must be sent to `support@eembc.org` for review.
3. If a vendor of the platform (member or not) posts a score, it immediately supersedes any 3rd party scores, i.e., other scores are archived from the scoreboard (not deleted, just hidden).
4. Cross-posting: if someone submits a score for an EEMBC member’s product, the member has 90 days to review the source, verify it isn’t malicious or create their own port score to submit to take the place of the score.
5. If the score is not cross-posted or awaiting certification, it will be published as soon as possible; or it may be delayed to coincide with a product launch by communicating this to EEMBC via email.

IoTMark®-Wi-Fi

Run Rules

IoTMark^®-Wi-Fi