The word platform is massively overloaded today and there now exists a platform for the most nuanced of websites. ClearBlade lives at the other extreme that allows for a highly scalable backend that supports the broadest of solutions. The following describes the technical usage model for ClearBlade as an IoT platform.
Purpose: This section should be informative and factual guide for implementation of an IoT solution.
Background: IoT networks contain the following possible physical elements
Device network - This is where devices all co-exist together. Those devices may all be together in a private cloud or exist together on a mesh environment in a house or industrial facility.
Cloud network - This is the public internet space where devices, laptops, and servers can communicate with one another.
Enterprise network - This represents the traditional protected networks where most enterprises execute their business processes. It is often protected from the public internet via a firewall, software and other rules.
Physical Devices - Many devices can perform activities in the Internet of Things. These devices can be advanced as laptops, ubiquitous as mobile phones, emerging as raspberry pi’s, appealing as an iBeacon or as simple as a basic temperature sensor.
Device Firmware - Software that runs on a device directly. This software may be very advanced and perform operating system like activities or may be simple binaries running in a limited linux environment
M2M communication - A number of protocols that support the transfer of information between devices. This includes bluetooth, RFID, iBeacon, mqtt-sn, zigbee and tcp-ip.
IP communication - To directly participate in the internet a device must receive an IP address.
- ClearBlade platform - middleware server software that provides connectivity to end point devices
- SDK - A library for a specific programming language that simplifies the communication from a client device to the ClearBlade platform
- Client - An endpoint device that will access the clearblade platform for information
- Cloud Integration - A bi-directional communication (typically REST) that supports sharing data or messages with a cloud based service provider often free or purchase via SaaS
- Enterprise Integration - A bi-directional communication via industry protocols for that supports sharing data or executing transaction with an existing enterprise software package, homegrown solution or legacy middleware
ClearBlade supports the creation of a secure highly scalable internet of things that connects to existing systems of record. Below each one of these requirements is defined along with the recommend solution using the ClearBlade platform.
Connect devices with TCP/IP
Low and high powered devices that have TCP/IP capabilities can directly connect to the ClearBlade platform. Each device must authenticate with the platform - even for anonymous access. Authentication may be done via
- HTTP REST endpoint to /
- MQTT Broker connect to ClearBlade auth broker
Point of consideration - MQTT provides significantly higher performance for much less battery and processing power. For devices with limited capability, consider a solution where devices only use MQTT and the ClearBlade platform handles the heavier workload activities.
Once connected, these devices may choose to encrypt traffic between themselves and the ClearBlade server. This decision is influenced by:
- The processing capability of the end device
- The privacy concerns of the data in transit
- The security of the network in use
The devices may include
- Smartphones like Android, iOS, Windows, Ubuntu, FirefoxOS
- Mini-processors like Raspberry Pi, BeagleBone, Arduino, Intel Galileo
These devices may use raw protocol libraries in any language or leverage ClearBlade SDKs for Objective C, Android, Java, GoLang, Python, or NodeJS.
Connect devices without TCP/IP
Some of the most powerful participants in an IoT infrastructure are lower power, minimal processing sensors that report small payloads. These devices do not include network cards for IP based communication nor do they include the processing capability required for encryption. For communication, these devices may have wireless communication over standards like bluetooth, RFID or 802.15.4 and may use protocols like ZigBee or MQTT-sn. Each of these devices will connect to a gateway that is IP connected. The gateway will have the capability to communicate with the ClearBlade platform using TCP/IP over the HTTP(s) or MQTT protocols. Gateways may have processing capability to rapidly implement encryption. Gateways will execute software that receives non TCP/IP communication and relays it to the ClearBlade platform. (see Gateway design)
Gateways provide a mid-point for gathering data from a primitive set of devices and routing that information to a server for analysis and collection. Gateways have the ability to be as lightweight as a mobile phone or as heavy as a traditional server, depending upon environment and computing needs.
Gateways implement local area networking communication via solutions like RFID, BLE, 802.15.4 that talk directly to the sensors and non TCP/IP enabled devices. From those devices information is packaged into a format desired for traveling over the internet. (see follow-on sections for BLE, zigbee, iBeacon recommendations)
Gateways also connect to the ClearBlade platform securely using a ClearBlade SDK. The ClearBlade SDK will allow for authentication from the device. Data from gateway can be sent to the platform via a message publish or an HTTP REST call to a ClearBlade platform endpoint.
Data can be pushed directly from the ClearBlade platform to the Gateway using a message publish. This message publish may be caused by other clients or executed from a service authored within the ClearBlade platform. (See services for more information)
Point of Consideration - If a gateway has limited bandwidth, processing limitations or battery concerns, we recommend using mqtt as a primary or only point of interaction with the ClearBlade platform
In some cases it is desirable to perform a workload or computation logic within the gateway when data needs raking or when the processing capabilities exist. In this case a ClearBlade platform in “cache” mode may be installed on the gateway. This environment will allow for bridging messages or sending data home via REST. Finally this configuration will allow for gateways to perform significant business logic prior to arriving server side. (See recommend deployment architectures - ClearBlade Cache only)
Recommended deployment architectures
Infrastructure topology for IoT is massively ignored by cloud and software vendors today. There are many variables and many concerns for which to account. Below represent different deployment models available with the ClearBlade platform and their attributes / value.
The simplest architecture for ClearBlade is an environment where all devices are IP address enabled.
Public IP Connected
This architecture is for a public network where all IoT devices, the ClearBlade platform and any integration exist in the public internet domain.
Private IP Connected
This architecture is for a set of IoT devices on a private network to communicate with ClearBlade server and enterprise integrations
Private IoT to Public ClearBlade to Public enterprise
This architecture diagram describes a private set of devices connecting to a publically available ClearBlade instance and publically available enterprise infrastructure
Private IoT to Public ClearBlade to Private Enterprise
This architecture describes a private network of devices communicating with a publically available ClearBlade instance that communicates with a private enterprise network
Private IoT to Private Virtualized
Within many corporate networks there are layers of protection. This means that IoT devices may participate on networks not directly accessible to legacy applications or employees
Advanced PERA (Purdue Enterprise Reference Architecture)
The Purdue model gives us a rich network architecture that offers protection from malicious activities. ClearBlade can participate in this architecture by behaving as a secure gateway between layers. This means that its possible to filter and expose data from high-level end users that can access information that is typically stored
This sub-architecture shows a non-persistent ClearBlade instance supporting logic processing on the device network while communicating back to a master ClearBlade instance that will do persistent storage and
ClearBlade Cache only
Messaging information between clients in under 100ms is critical to achieving perceived real time behavior. Removing physical network constraints ClearBlade messaging achieves this goal while also implementing authority and authentication. Traditional REST endpoints cannot do this
Point of Consideration - traditional OS’s do not achieve industry standard real time operating systems like linuxRT may achieve much faster behavior and can be considered for integration with ClearBlade
To protect rogue devices from participating in your network it is important to enforce authentication of each device. Unlike typical users the preferred model of authenticating devices is to embed a list of valid auth-keys in the device firmware. This list of auth-keys will then be used in a specific order for authenticating with the backend. A backend will evaluate those keys for a valid device auth key to device_id paring, a valid date window of operation of validity. Operators should have the ability to disable a devices access by simply disabling the authority of those keys. Additionally a device should be able to update its valid keyset with an update to the locally running firmware.
ClearBlade supports this auth model with the use of a simple collection that contains the device_key, device_id, start_date, end_date, and finally an isValid override fields. The creation of a key set is easily done during the firmware load with a call to a ClearBlade service named “GenerateDeviceKeys”. This call will return a set of keys for loading into the firmware while also inputting them into the DeviceKeyCollection. Once the keys are in place the device will make a call to a ClearBlade service named “AuthenticateDevice” As parameters this services accepts deviceId, device_key. The service will evaluate the sent key with the DeviceKeyCollection. If the key is found to be valid the service will auth into the platform with a device role and produce a token. This token is then returned to the device.
Point of Consideration - This authorization process is extremely flexible and can be used to support device authentication into a large number of third party user registries.
Finally ClearBlade uniquely allows the execution of this login process via either traditional heavy REST or it can be done in a lightweight manner over MQTT.
When the processing power exist for your devices and security is desired encryption should be enforced. There are multiple places where encryption may be desired.
- Device network - If this network is physically accessible or can be scanned by other devices then the data should be encrypted as it travels across wireless protocols.
- Public cloud internet - If you data has any potential for privacy then this is the most important domain to ensure all traffic is encrypted. ClearBlade enforces this by having all communication between its APIs leverage the latest OpenSSL
- Enterprise network - Often traffic will move safely behind firewalls without encryption where environments are more trusted. ClearBlade will use the standard associated with this environment and the integration available to work with. ClearBlade will encrypt at this layer if desired.
Many millions of devices has the ability to put a significant performance load on any backend infrastructure. Your IoT platform should be capable of horizontal scaling. For Request / Response like HTTP this means the backend should be stateless and allow for many machines to be setup in parallel to handle millions of request per second.
For messaging protocols where sessions are held open more broker intelligence is required. An architecture that supports devices connecting to many machines and then brokers being bridged together is the preferred solution.
ClearBlade provides both the stateless REST interactions that can scale horizontally along with the scaling of the message broker. With each new node of ClearBlade a new message broker is made available for devices to connect to. ClearBlade is defaulted to allow 50,000 device connections per broker, but this number can be changed in a setting for handling higher transaction rates.
Gateway vs Direct model
When designing an Internet of THings solution a gateway can be a critical piece of the solution in order to control cost of devices. A gateway can perform the most basic task of relaying information from an endpoint device to an internet based server or it can do advanced activities of compressing data and relaying data intelligently. A gateway is critical to a solution when in a device ecosystem that contains non ip enabled devices.
ClearBlade supports all types of device ecosystems. In the case of direct connections, devices can connect to an internet based ClearBlade server. In the case of a simple pass through gateway the ClearBlade SDK can be used to connect the M2M protocol to an internet based ClearBlade server. In the most flexible scenario of an intelligent gateway it’s recommended to run a headless ClearBlade instance on the gateway device. This instance will cache data in memory only and will execute business rules as desired. Relaying the m2m protocol data into this instance gives the IoT network the ability to monitor / alert / respond to / and provide intelligence at multiple points in the IoT solution.
Many IoT providers fail to see the true opportunity of IoT participating with existing infrastructures. This means that while devices may be unaware of legacy architecture and systems of record, those applications still have meaningful actions to perform to either optimize business process or enhance the value of the device network. IoT devices may replace field data entry or be configured to gather new data depending on new business offerings.
ClearBlade provides a powerful and extremely flexibly Integration engine. This real time code engine allows for logic that execute to talk directly to sockets, protocols, databases or other middleware and COTS applications.
Many cloud services are available that contain public or social data. This data set can be valuable to IoT to both enrich with device information and to better target the use of IoT data. Imagine a store being able to track employees through a store walk through and then matching that behavior to twitter reactions and sentiment.
Through ClearBlade’s integration libraries apps and systems can quickly gather information from these third party cloud services. ClearBlade systems can positively impact those public solutions while also enhancing the apps opportunity to increase overall user value.
Realtime Data Synchronization
A huge challenge and desire for many working in the internet of things is the ability to get data from and to devices in a real time manner. Realtime information means your users and devices can get the latest information when they request it, but also have the ability to get pushed the latest information when it happens. Many solutions attempt to emulate the push process via a “long pole” This ultimately is a poor solution due to battery life and inconsistencies. Instead we see a subscribe publish model replacing the request response model. Today its possible for to use a number of protocols to accomplish including websockets, amqp, xmp, and mqtt. An IoT solution should most likely leverage one of these technologies to be considered capable of supporting real time considerations.
ClearBlade supports extremely rich real time requirements by allowing by HTTP, websockets and MQTT to be used interchangeably with a single authority model and underlying logic and data structures. This means devices can simply subscribe to information and automatically receives any updates. A customizable trigger infrastructure allows for reacting and writing business logic that handles specific events like - humidity warnings setting off business triggers and alerts.
Point of consideration - It is not enough to simply reuse an existing endpoint available over HTTP for most devices. This will result in performance and battery issues that are acceptable in most IoT environments
As data moves from one device to another there is very often a need to modify the structure and size of the payload to optimize for the device and network constraints. We first saw this pain in the failure of many websites to adequately respond and send data to mobile devices. Mobile devices were often on slower carrier networks and lacked the processing power to render the data into long lists when it arrived. Over-all the failure to optimize payload resulted in many poor mobile experiences.
Today we see this happening again with IoT. Very often the urban carrier networks of today have far more capabilities than the small unstable connectivity environments of IoT. For IoT we need the ability to optimize both sending and receiving payloads.
ClearBlade rapidly supports this requirements by allowing for systems of devices to all work together. Those devices can share endpoints of have specific endpoints for their needs. Those endpoints can work off of shared data sources that means a sensor may have a simply mqtt publish of data for sharing temperature while a rich web client will look at the same data with a heavy call to months of temperature data across multiple sensors.
Point of Consideration - Paging of data has never been of more importance. If your platform doesn’t page conservatively by default then you’re likely to encounter a number of issues when immediately rolling out on the IoT networks
Big Data Integration
Many people look to big data to better understand their customers or enhance their processes to better serve their audiences. This opportunity is real and continuously growing. The first challenge to achieve this understanding is to begin collecting data with consistency and concurrency. All too often business are stymied for starting their big data initiatives with IoT because of the simply overwhelming amount of possible data.
ClearBlade eases this pain by providing a conduit to third party analytics and databases. Whether sending analytics to generic cloud vendors or to domain specific processors this can easily be achieved within the platform. The platform can directly route the request of can subscribe to the general activity in the system and send it along as business rules apply
Alarms are critical to making an IoT based network smarter and more productive. An IoT platform should be able to allow for custom alarms when certain thresholds are realized of certain events occur.
ClearBlade allow for triggers and timers that allow for business process to be injected into normal data flow. These triggers can watch data from sensors and store critical information.
Reports are built from an understanding and summary of data gathered over a period of time. This data should be accessible at all times via an API such that both visual representation and data exports can be gathered for an immediate review or high quality physical reports. The data should include both generic information about usage but also information around the domain in question.
ClearBlade by default produces and analytics API that summarizes, events, transactions, usage, and users to easily be represented graphically. These APIs are available live and up to date at all times. Additionally ClearBlade allows for dynamic data structures and customizable report logic to build the preferred data set into a single aggregated view. Finally Clearblade offers triggers allowing for traditional daily rollups and compression of data to simplifiy large data stores into summary information that can more easily be consumed.
BLE / Bluetooth integration
BLE / Bluetooth is a widely used protocol for having non-ip connected devices communicate with more powerful devices that have an IP address. Very often devices in close proximity will share streaming information via bluetooth. Using open source drivers IoT networks can leverage bluetooth to get data from sensors and other devices to IP connected devices. The IP connected devices can then packaged the data appropriately and send it back to the internet for IoT participation.
ClearBlade provides and SDK readily able to communicate data transferred via BLE or bluetooth to Ip enabled gateways or devices. Additionally beacon assets can be inventoried and managed via collections in the ClearBlade platform. The collections may contain custom information including geolocation, brick and mortar information, and mobile phone pairings all for the purpose of gathering future analytics.
iBeacon represent a specific implementation and protocol on top of bluetooth. To build an IoT solution that includes iBeacon support you will need to have iBeacon devices and then firmware or apps that can communicate of the iBeacon standards. There are opensource and proprietary implementations of that specification depending on the device type. Apps will then need to take the data gather in an M2M environment and convert it to an internet based protocol for interoperating in the internet of things.
ClearBlade provides and SDK readily able to communicate data transferred via iBeacons to Ip enabled gateways or devices. Additionally beacon assets can be inventoried and managed via collections in the ClearBlade platform. The collections may contain custom information including geolocation, brick and mortar information, and mobile phone pairings all for the purpose of gathering future analytics.
Zigbee represents a specific implementation and protocol on top of 802.15.4. To build an IoT solution that includes zigbee support you will need to have zigbee enabled hardware devices and then firmware or apps that can communicate of the standards. There are opensource and proprietary implementations of that specification depending on the device type. Apps will then need to take the data gather in an M2M environment and convert it to an internet based protocol for interoperating in the internet of things.
ClearBlade provides and SDK readily able to communicate data transferred via zigbee to Ip enabled gateways or devices. Additionally beacon assets can be inventoried and managed via collections in the ClearBlade platform. The collections may contain custom information including geolocation, brick and mortar information, and mobile phone pairings all for the purpose of gathering future analytics.
MQTT-sn and MQTT provides a similar application interface to developers but is used significantly different in implementation and use case. for a different use case and leverage different underlying technologies. MQTT-sn does not run over TCP/IP but instead uses wireless network protocols like zigbee. In environment where devices do not have IP capabilities MQTT-sn can be used to subscribe and publish
Bridging MQTT specifications - Messages from MQTT-sn can easily be relayed from MQTT-sn to MQTT. A custom MQTT client can provide that needed capability.
Cycling device keys
Unlike humans devices do not have the ability to reset passwords with traditional means with 2nd and 3rd party factor authentication. This constraint removes many concerns with access associated with humans like spoofing or phishing but it requires a more programmatic way to ensure trust. One such way is to assign a set of keys to a device which have an expiration associated with them and can be disabled by the server to force cycling onto the next key.
Depending on the desired model this solution can be built with the ClearBlade platform leverage the authentication token model. End devices can be provided sets of tokens and then use them in the prescribed fashion. The token may even represent no associate authority but simply act as password to be granted authorities. The additional abstraction gives the IoT administrator the ability to rapidly have all devices immediately move to a non compromised token when a breach has occurred, quickly de-authorize certain behaviors in the network or in a worst case scenario immediately disable the entire device network.
Devices participating in IoT network must be traceable and installed with default firmware or apps that handle IoT connectivity. This application that runs on the devices needs the ability to securely connect to any IoT platform.
ClearBlade supports this default application by providing SDKs and open access to protocols used for internet communication. ClearBlade simplifies and communicate a secure authentication model for those devices and can encourage encryption where network and cpu resources are available.
Update device software (aka firmware updates)
That firmware or app must be updatable to ensure longevity and maturity of any project. This update can be received through a third party like an Apple App Store or may be fetched from a file server that contains updated resources. Installed firmware on the device should be aware of how to get the latest version and dynamic replace the executing app and appropriately recycle the executing process.
ClearBlade supports this activity by providing static file hosting that can be referenced via APIs. Those APIs can send firmware update availability along with appropriate file downlocations. The app or firmware may be downloaded and restarted by leveraging SDK in use.
API Device Provisioning
Because devices must be assembled and provisioned in manufacturing scenarios it is important that no human intervention is required to generate device specific keys or SDKs.
ClearBlade exposes all application and administration activities via REST based endpoints ideal for integrating with manufacturer requirements.
This document address the major concepts associated with implementing IoT based solutions. Ultimately this It includes both generic descriptions of the challenges and how ClearBlade uniquely and powerfully solves those issues.
IoT represents the opportunity to make technology an invisible but indispensable part of people’s lives. Technically it represents a significant challenge across the domains of hardware, low level software, communication protocols, security, internet and cloud based interactions, and ultimately data science. The ClearBlade platform provides the versatility required to achieve those goals