Testing Solutions

For IoT

Bandwidth (GB/s)

Maximum latency (ms)

Time accuracy (ms)

50 Node Provisioning Time (s)

One IoT, Many Things

Not all IoT devices behave the same. Factors such as form factor, and how the device is powered, will play a role in the type of traffic to expect from devices. Here we explore a few well-known device types, and how they exercise the network.

Low Bandwidth Sensors

Simple sensors relying on mesh networks, such as Zigbee, to communicate a few crucial bytes. These devices are usually placed behind a gateway, which aggregates data and sends it periodically. Some types of gateways  handle backpressure gracefully, while other gateways will drop messages under heavy network traffic.

Testers will typically want to test network blackout or high latency scenarios to determine whether mission critical messages are being delivered under the conditions you expect your network to operate within.

Bluetooth Tethering

A large number of IoT devices rely on Bluetooth tethering to connect to the server, sometimes relying on a mobile application to relay information. Those devices typically still operate on battery, yet are able to send larger messages that are transmitted directly to the internet or some other TCP/IP network.

Testers will rely on generating heavy load from these types of messages to test the capacity of the bluetooth host to handle the expected load.

Industrial appliances

Most appliances, especially those deployed in an industrial setting, self-report to a remote server. These devices often communicate over SCADA protocols such as modbus and profibus.

Industrial devices tend to send many messages to help users keep track of critical processes, and many are built with the expectation that their connection to the rest of the devices on the network is low latency and always available.

Testers will want to check end to end message processing, computing the average, mean, and maximum time it takes to receive and process a message. They will also want to test the effects of network infrastructure failures to determine whether industrial processes will stop safely when an ethernet switch fails.

Edge Computing

Edge computing, one of the most powerful forms of IoT, moves big data and AI computation to the farthest reaches of your network infrastructure, using AI software like Tensorflow to perform complex calculations. Those devices can be as simple as specialized Raspberry Pis, yet in spite of their small footprints they generate very large amounts of rich data, such as pictures and videos.

Testers will want to test concurrent transmission of large data segments under heavy system load to ensure that the entire system scales accordingly.

Event Processing

In the old client server paradigm, clients would request information from the server, typically over web or RESTful services. IoT inverses the direction the data flows, with devices sending events to the server. The server must ingest data without loss, and ensure it is processed in near real time. Here we discuss in more detail the different types of use cases and how they impact the network.

Time Series

Time series data are formed from a sequence of data points sent by a device which are indexed by the time at which they were emitted. They are typically used to emit sensor data, such as temperature, wind speed, or UV exposure. Those data points are usually aggregated into hourly or daily averages using big data analytical software, such as Apache Spark.

Testers will want to check that the data aggregated into the time series is complete, or that where data loss has occurred it is within acceptable tolerances. Testers also typically check that they can render the last known value.

Health Checks

Health checks are periodical messages sent by the device providing health information. They exercise the network to make sure the connection is working. They may be as simple as sending a TCP packet, or they may send complex metadata from on-device diagnostic sensors. These data sources are sometimes transmitted alongside time series reporting.

Testers would check that health checks perform as expected. Severing the network connection should change the health check to a failed or unknown status, and reestablishing it should turn revert it back to a status that reflects current diagnostics data.

Remote Commands

Remote commands are executed by the server on a device. This is what happens, for example, in a home automation context when you ask Alexa to turn on the lights. The Alexa Skill sends a message to the IoT device embedded in the bulb. When the processor in the bulb receives the message, an interrupt is triggered..

Remote commands usually come with critical latency requirements, as users don’t want to wait more than a fraction of a second for their lights or appliances to turn on.

Testers will therefore want to test the speed at which a remote command can propagate from an external system interfacing with the server, how long it takes the server to process it, send it to the device, and for the device to properly execute it.

IoT Lifecycle

In the old client server paradigm, clients would request information from the server, typically over web or RESTful services. IoT inverses the direction the data flows, with devices sending events to the server. The server must ingest data without loss, and ensure it is processed in near real time. Here we discuss in more detail the different types of use cases and how they impact the network.

 

Provisioning

In order to make setup easy on the user, IoT devices coming out of the factory floor are usually configured for very unrestricted access to device administration. If these devices are not provisioned by the user and properly secured, they may be vulnerable to a wide variety of attacks. Remote provisioning tools are often used to make this process more efficient, however some remote provisioning methods have been shown to be insecure (for example, vulnerable to DNS spoofing, or opening an insecure port on the local network), in which case attackers can infiltrate those devices and take over their functionality, or even attack the network itself.

Testers should check that the provisioning system is foolproof. With Whiteblock Genesis, it is possible to create environments that recreate man-in-the-middle attacks, detecting insecure network connections and spoofing DNS requests.

Upgrades

Devices also are hard to upgrade. Some devices require a companion device such as an iPhone to perform the download and apply the firmware update. Some devices also support two concurrent OS images, so they can always return to a pristine state in case the upgrade fails.

Testers will want to check upgrades are small enough that they can fit on the device storage. They also should check upgrading is done over a secure line. They should test the trigger of update, whether it is a periodical check or ordered by the remote server. It’s common for remote update servers to require the device to authenticate, however to prevent this upgrade path from becoming a critical attack vector, testers should always validate that the device also checks the authenticity of the server prior to accepting an update.

Supporting Legacy

IoT services age quickly, as not all devices get a chance to upgrade timely. Further, sometimes devices are updated before server software. As a result, it is often necessary to keep all services both backward and forward compatible. Testers will want to use different versions of a firmware against the server deployment to ensure devices without the proper upgrades can still work.

IoT Metadata

IoT is a gold mine for metadata. The business for example will want to know where the device activated, how often it is used, what patterns of usage appear over time. Collecting and managing metadata leads to insights which can be incorporated into the product or sold on the marketplace. Here, we discuss a few aspects of IoT metadata and how best to approach their testing.

 

Metrics

It’s impossible to collect and retain every data point, and this leads to the establishment of metrics, capturing a particular aspect of the product. The business follows those metrics and sets boundaries indicating performance, making sure to listen for alerts indicating problems. Over time, those alerts tend to multiply and it becomes difficult to pay attention.

For this reason, it is necessary to model and test those alerts. Is IT properly notified when devices cannot report data anymore? Is support getting alerts when devices start rebooting often? With Whiteblock Genesis, it is possible to create test scenarios specified to trigger alerts, and make sure those are meaningful.

Know Your Customer

IoT is a revolution for product-minded folks as they can now follow along the activity of their customers. By following along patterns of usage, it becomes possible to build dashboards exposing insights, showing how the product is used in the wild.

With Whiteblock Genesis, it also becomes possible to reproduce and test those patterns. This allows you to ensure that tests are run under typical usage patterns, and to better place those points of collection in your system so that you get the most actionable information possible.

Understanding Failure Modes

Everything fails, and IoT is no different. By performing root cause analysis, the business can better understand where to focus efforts in enhancing quality, be it by enhancing manufacturing standards or retooling connection handling to work with slow Internet connections.

With Whiteblock Genesis, a complete workbench is at the disposal of developers to reproduce failure modes, triggering them on demand. It then becomes possible to identify them and better assist customers.

Development

Whiteblock Genesis can save precious time to developers to stand up networks that replicate their production environment. Whiteblock Genesis can reuse data between runs, and send messages to simulate load as needed. Whether you’re developing the next revision of your IoT product, or completely greenfield IoT systems, you’ll be able to test your system at full scale under the load specifications for which you’re designing.

 

Using Managed Services

The IoT landscape changed with the introduction of managed IoT services from major cloud providers like AWS and Azure. These services help automate provisioning, data collection and commands. Whiteblock Genesis can replicate your production settings from these services so that you can test potential changes without impacting your live running IoT networks.

Testing An IoT Network

An IoT network comprises of devices and the infrastructure. Whiteblock Genesis effortlessly replicates distributed infrastructure components and can manipulate bandwidth, latency, packet loss and other network tools around those components. In the future, Whiteblock Genesis will also support running virtualized devices so they can be tested as part of the ensemble. Contact us to learn more.

IoT Specific Protocols

Whiteblock Genesis can understand MQTT and AMQP long lived connections. We are looking to add support for Bluetooth in the future. Contact us to learn more.

For More Information Contact Us

or

Sign Up For The Beta Below

Get Free Early Access To The Beta

Get Free Early Access To The Beta

You have Successfully Subscribed!