Code

Introduction

The ClearBlade IoT Platform utilizes a microservices model powered by a Code Engine. The code engine allows for JavaScript (es5) and has several built-in integrations to interact with your IoT assets. Adhering to microservice practices: a new execution context is sandboxed and created upon invocation. Once the process is complete, it is destroyed.

There are four types of Code assets:

  • Micro Service: A short-lived service that is expected to be completed within a fixed time (or is killed otherwise); default execution timeout is 60 seconds.

  • Stream Service: A service configured to continuously handle requests with an infinite execution timeout.

  • Configuration: A library that contains a key-value object to be used as constants.

  • Library: A custom javascript library that can be imported into one or more code services

For a tutorial on how to set up code services, click here

Purpose

Microservices are a software development technique that allows for a high degree of modularity, scale, and parallelization - all of which are essential for effectively building and running full-scale IoT Applications.

Libraries

Libraries are reusable sets of code that can be imported by one or more code services.

Native Libraries

Native Libraries are available in every System and have an underlying implementation in Golang. Each library is optimized for computational performance.

These libraries can be imported using Requires

Library Desc Docs
clearblade Interacts with Collections, Messaging, Users, Edges, Devices View docs
log Appends messages to a log View docs
http Makes outbound HTTP/S requests View docs
geo Geospatial Computation Library View docs
crypto Creates cryptographical hashes View docs
file_writer Write files to host filesystem View docs
jsutil JS Utility Library View docs
mailer Send email directly from host machine View docs
net_cb Net library View docs
oauth2_lib OAuth Utility Library View docs

Custom Libraries

A developer can create their custom library with custom logic.

Services

A Code Service, or Service, is a short-lived microservice that is expected to complete in a finite time capped by the execution timeout field in its settings (it is killed after the timeout). The default is 60 seconds.

Lifecycle

Invocation

A Service can be invoked in any of the three ways:

  1. A REST request to the API Endpoint
  2. Trigger Event
  3. Timer Event

Our SDKs can be used as a REST Client to accomplish the first REST option

Context

Each Code Service service has a name, and that name corresponds to an autogenerated function name.

For example, a code service named analyze will have an autogenerated function like so:

function analyze(req, resp) {}

This function will have two parameters, similar to Node’s Express library:

req

req is an object containing the following contextual values:

key desc type example
userEmail Email of user who invoked the service string “user@clearblade.com”
params JSON with parameters, structure is based upon type of Invocation Object {“key”:“value”}
systemKey System Key for system in which code service is running string e095a19d0b94e48dee9bff87e5fd01”
systemSecret System Secret for system in which code service is running string “E095A19D0B9ACD9EFA86AEEDFCC001”
userToken Invoking user’s OAuth token string “U1hNa7o4bEXpiE4aGhb-79MQv4…”
isLogging Whether logging is enabled boolean true
userid internal uid associated with invoking user string “fc8cfcea0a96ebebc7f5d4edd414”

resp

resp is an object used to return values and exit the Code Service. Following methods are available in the resp object:

  • resp.success
  • resp.error
  • resp.set
  • resp.status
  • resp.send

resp.success

resp.success = function (object | string){}

function analyze(req, resp) {
  var message = "Everything went well!";
  resp.success(output); // Service exits successfully with "Everything went well!"
}

resp.error

resp.error = function (object | string ){}

function analyze(req, resp) {
  var message = "It broke!";
  resp.error(error); // Service exits with failure with "It broke!"
}

resp.set This method has multiple signatures:

  • resp.set = function(string, string){}

resp.set(keyString, valueString): adds or updates a key in the header to the HTTP response from a service. Example: resp.set("Content-Type", "text/html")

  • resp.set = function(object){}
  • resp.set({}): will set the response headers to those in the map. It replaces the header object so it removes any headers added with resp.set(k,v). Example:
resp.set({
  "Content-Type": "application/json",
  "ClearBlade-UserToken: req.userToken"
})

resp.status

resp.status = function(int){}

resp.status(int): changes the HTTP response code from a service. Example: resp.status(400)

resp.send

resp.send = function(object|string){} resp.send(value): exit the service and send the value back as the body of the response. value will be stringified if not already a string. value will be sent as-is and not wrapped in our typical json response object. Example: resp.send({valueA: "ABC-123-DEF", valueB: 123})

Exit

The Code Service will terminate its execution with resp.success, resp.error, or resp.send.

Note: By default, if no exit method is called and execution reaches the end, resp.success will be called to exit.

Logs

Logs record and store the history of service responses. Each service can store up to 25 of the latest logs with a max size of 100KB each.

Requires

A Code Service can import a Library using the Requires feature. In the Service settings, you can select one or more libraries to import, and that code is accessible from within the Code Service.


The same library can be used for multiple code services

Debugging

There is no native support for debugging line-by-line. Recommended to use log method extensively, like so:

function ServiceA(req, resp) {
  log("Something happened");
  resp.success("All done");
}

Events

Events are actions or occurrences that happen in the ClearBlade System, response to them through execution of the associated Code Service. ClearBlade supports two types of events:

Timers

Timers provide a mechanism where a ClearBlade code service can be scheduled to execute at certain time intervals. Timers are similar to Unix cron jobs, with a similar scheduling mechanism.

Some example Timer configurations:

  • Run every five minutes starting now forever
  • Run 26 times every two weeks starting Friday, December 25th, 2018
  • Run once at midnight

Triggers

Triggers are a mechanism that ties an action, called a Trigger Source, to the invocation of a code service. The trigger source and payload information can be accessed in the code service through the req.params object. When creating a trigger, add log(req) to the Code Editor to log the payload information.

"params": {
    "topic": "ClearBladeTopic",
    "body": "214.312",
    "userId": "8e89e2af0fc9cbeadfd6afb501",
    "trigger": "Messaging::Publish"
}

Event Source

The platform publishes for every event, regardless of whether the event is created programmatically in a service or created externally in the console.

$trigger and $timer are custom ClearBlade topics. It is advisable to use a stream service instead of micro-service when working with $timer and $trigger topics that fire often to prevent overloading the platform.

For more information on publishing event topics, click here

Asset Category Action Topic
Messaging Publish Publish $trigger/messaging/publish
Messaging Subscriptions Upon Subscribe $trigger/messaging/subscribe
Messaging Subscriptions Upon Unsubscribe $trigger/messaging/unsubscribe
Messaging Connection User Connected $trigger/messaging/user/connected
Messaging Connection User Disconnected $trigger/messaging/user/disconnected
Messaging Connection Device Connected $trigger/messaging/device/connected
Messaging Connection Device Disconnected $trigger/messaging/device/disconnected
Data Table Created $trigger/collection/created
Data Table Updated $trigger/collection/updated
Data Table Deleted $trigger/collection/deleted
Data Item Created $trigger/data/created
Data Item Updated $trigger/data/updated
Data Item Deleted $trigger/data/deleted
Data Item Upserted $trigger/data/upserted
User n/a Created $trigger/user/created
User n/a Updated $trigger/user/updated
User n/a Deleted $trigger/user/deleted
Device n/a Created $trigger/device/created
Device n/a Updated $trigger/device/updated
Device n/a Deleted $trigger/device/deleted
Edge/Platform Platform Platform Started $trigger/startstopdisconnect/platformstarted
Edge/Platform Platform Platform Connected on Edge $trigger/startstopdisconnect/platformconnectedonedge
Edge/Platform Platform Platform Disconnected on Edge $trigger/startstopdisconnect/platformdisconnectedonedge
Edge/Platform Edge Edge Started $trigger/startstopdisconnect/edgestarted
Edge/Platform Edge Edge Connected on Platform $trigger/startstopdisconnect/edgeconnectedonplatform
Edge/Platform Edge Edge Disconnected on Platform $trigger/startstopdisconnect/edgedisconnectedonplatform
Timer n/a Create Timer $timer/{timerName}

Examples of Incoming Object

Ensure the relevant permissions for code service are set before using triggers and timers.

Following is a list of categories grouped by action types, showing structures for incoming object:

Messaging (Publish/Subscribe/Unsubscribe):

When there is a Publish/Subscribe/Unsubscribe made to the ClearBlade’s MQTT broker.

// Template
{
  "params": {
      "topic": "<message_topic>",
      "body": "<message_body>",
      "userId": "<user_id>",
      "trigger": "Messaging::<action_type>"
  }
}

// Example
{
  "params": {
      "topic": "ClearBladeTopic",
      "body": "example message",
      "userId": "8e89e2af0fc9cbeadfd6afb501",
      "trigger": "Messaging::Publish"
  }
}

Messaging (UserConnected/UserDisconnected):

When a user Connects/Disconnects to the ClearBlade’s MQTT broker.

// Template
{
  "params": {
      "email": "<user_email>",
      "trigger": "Messaging::<action_type>"
  }
}

// Example

{
  "params": {
    "email": "example@clearblade.com",
    "trigger": "Messaging::MQTTUserDisconnected"
  }
}

Messaging (DeviceConnected/DeviceDisconnected):

When a device Connects/Disconnects to the ClearBlade’s MQTT broker.

// Template

{
  "params": {
    "deviceName": "<device_name>",
    "trigger": "Messaging::<action_type>"
  }
}

// Example

{
  "params": {
    "deviceName": "exampleDevice",
    "trigger": "Messaging::MQTTDeviceDisconnected"
  }
}

Data-Table (CollectionCreated/CollectionUpdated/CollectionDeleted):

// Template 

{
  "params": {
    "collectionId": "<collection_id>",
    "collectionName": "<collection_name>",
    "trigger": "Data::<action_type>"
  }
}

// Example

{
  "params": {
    "collectionId": "ceafb4cf0bbaf5f4ddfbf1b48730",
    "collectionName": "ExampleCollection",
    "trigger": "Data::CollectionCreated"
  }
}

Data (ItemCreated):

// Template

{
  "params": {
    "items": [
      {
        "item_id": "<item_id>"
      }
    ],
    "collectionId": "<collection_id>",
    "collectionName": "<collection_name>",
    "trigger": "Data::ItemCreated"
  }
}

//Example

{
  "params": {
    "items": [
      {
        "item_id": "083196bf-8059-46c0-8280-38234aa73fc1"
      }
    ],
    "collectionId": "ceafb4cf0bbaf5f4ddfbf1b48730",
    "collectionName": "ExampleCollection",
    "trigger": "Data::ItemCreated"
  }
}

Data (ItemUpdated/ItemDeleted):

// Template

{
  "params": {
    "collectionId": "<collection_id>",
    "collectionName": "<collection_name>",
    "trigger": "Data::<action_type>",
    "items": [
    // First row
    // This includes all of the data from the row
      {
        "<column1_name>": <column1_data>,
        "item_id": "<item_id1>",
        "<column2_name>": <column2_data>
      }
    // A second row
      {
        "<column1_name>": <column1_data>,
        "item_id": "<item_id1>",
        "<column2_name>": <column2_data>
      }
    ]
  }
}

// Example

{
  "params": {
    "collectionId": "ceafb4cf0bbaf5f4ddfbf1b48730",
    "collectionName": "ExampleCollection",
    "trigger": "Data::ItemUpdated",
    "items": [
      {
        "hi": 6,
        "item_id": "14b12adf-4c32-4740-8461-546daa660393",
        "low": 3
      },
      {
        "hi": 7,
        "item_id": "3700fa91-ef49-41e8-9fe7-64e9ee6aa3b2",
        "low": 5
      }
    ]
  }
}

User (UserCreated)

// Template

{
  "params": {
    "user": {
      "creation_date": <date_time>,
      "email": "<user_email>",
      "user_id": "<user_id>"
    },
    "trigger": "User::UserCreated"
  }
}

//Example

{
  "params": {
    "user": {
      "creation_date": 2019-06-20T19:43:10Z,
      "email": "example@clearblade.com",
      "user_id": "9c87dfd00beefbd8a0f3cc82f0c001"
    },
    "trigger": "User::UserCreated"
  }
}

User (UserUpdated/UserDeleted)

// Template

{
  "params": {
    "query": {
      "FILTERS": [
        [
          {
            "EQ": {
              "user_id": "<user_id>"
            }
          }
        ]
      ],
      "PAGENUM": <page_number>,
      "PAGESIZE": <page_size>,
      "SELECTCOLUMNS": <columns_selected>,
      "SORT": []
    },
    // Includes all of the data from the row
    "user": {
      "email": "<user_email>"
      <column1_name>: <row_data>
      <column2_name>: <row_data>
      "user_id": "<user_id>"
    },
    "trigger": "User::<action_type>"
  }
}

// Example

{
  "params": {
    "query": {
      "FILTERS": [
        [
          {
            "EQ": {
              "user_id": "9c87dfd00beefbd8a0f3cc82f0c001"
            }
          }
        ]
      ],
      "PAGENUM": 0,
      "PAGESIZE": 0,
      "SELECTCOLUMNS": null,
      "SORT": []
    },
    "user": {
      "email": "example@clearblade.com"
      "name": "Example"
      "age": 22
      "user_id": "9c87dfd00beefbd8a0f3cc82f0c001"
    },
    "trigger": "User::UserUpdated"
  }
}

Device (DeviceCreated/DeviceDeleted)

// Template

{
  "params": {
    "trigger": "Device::<action_type>",
    "device": {
      "allow_certificate_auth": "<boolean>",
      "allow_key_auth": "<boolean>",
      "certificate": "<certificate>",
      "created_date":  "<date_time>",
      "description": "<device_description>",
      "device_key": "<device_key> :: <device_name>",
      "enabled": "<boolean>",
      "last_active_date":  "<date_time>",
      "<columnX_name>": "<columnX_data>",
      "<columnY_name>": "<columnY_data>",
      "system_key": "<system_key>",
      "type": "<device_type>"
    },
    "deviceName": "<device_name>"
  }
}

//Example 

{
  "params": {
    "trigger": "Device::DeviceCreated",
    "device": {
      "allow_certificate_auth": true,
      "allow_key_auth": true,
      "certificate": "",
      "created_date":  2019-06-20T19:43:10Z,
      "description": "",
      "device_key": "ccafb4cf0bd0dcbcadaccaf9ebba01 :: ExampleDevice",
      "enabled": true,
      "last_active_date":  2019-06-20T19:43:10Z,
      "name": "ExampleDevice",
      "state": "active",
      "system_key": "ccafb4cf0bd0dcbcadaccaf9ebba01",
      "type": "Sensor"
    },
    "deviceName": "ExampleDevice"
  }
}

Device (DeviceUpdated)

  // Template

  {
    "params": {
      "changes": {
        "description": "<device_description>"
      },
      "deviceName": "<device_name>",
      "trigger": "Device::DeviceUpdated"
    }
  }

  // Example
  
{
  "params": {
    "changes": {
      "description": "device description"
    },
    "deviceName": "example_device",
    "trigger": "Device::DeviceUpdated"
  }
}

Edge/Platform

  // Template

  {
    "params": {
      "edgeName": "edge_name",
      "trigger": "StartConnectDisconnect::<action_type>"
    }
  }

  // Example

  {
    "params": {
      "edgeName": "example",
      "trigger": "StartConnectDisconnect::<Edge Started>"
    }
  }

Webhooks

A webhook is a mechanism that allows you to execute a code service by targeting a public endpoint. Any HTTP method, such as GET, POST, PUT, DELETE, can be used and will result in the same action, executing the service. The URL we provide can be used by third parties to push real-time data to ClearBlade’s server. A code service can be invoked by multiple webhooks and are sync-able. See tutorial.

Preloaded Services

A micro-service can be preloaded if we turn on the preloaded toggle for it. Preloading allows it to be significantly efficient. Especially when it is being invoked frequently (through a trigger or a webhook). Similar to a regular stream service, you can set the concurrency and auto-balance to true. This way, the requests to invoke a service will be load-balanced across multiple handlers.

Internally these services are getting loaded into a stream service, which has the ability to ingest http requests and returns responses back to the client. Since the stream service is always running, it solves the problem of continuously creating and destroying execution engines. This makes it faster and memory efficient.

Important note: Since your microservice code is part of the internal stream service, we recommend that you use caution if you have globals in your code. Globals will store their state across multiple invocations of the service, which might give undesirable results.

Authorization

These requests are to be added when executing a webhook.

Auth Method Request
clearblade_auth ClearBlade-Usertoken: <USER_TOKEN> See Example
http_basic_auth Authorization: Basic <USER_TOKEN> See Example
payload_auth (GET) Query String: token=<USER_TOKEN> See Example
payload_auth (POST) Request Body: “token”: <USER_TOKEN> See Example
no_auth No Authorization Required. The “Anonymous” role should have Run permission checked for the service. This allows for the no auth webhook to work. See Example

Authorization Examples

ClearBlade Auth

function ExampleWebhookInvoker(req,resp){
    // These are parameters passed into the code service
    var params = req.params
    var http = Requests();
    var options = {
        uri:"https://platform.clearblade.com/api/v/4/webhook/execute/f6edffd60bdc95a5c80f79d0a/ExampleWebhook",
        headers:{
            "ClearBlade-UserToken":"Id19kSWr2vkB4CUcxa5lm6Akzh1jXr6Fw3sxW-ynaaV67EJRLmrHvkqUD8vnYu52XGErDNRpSNJsD5w=="
        },
        body:{
            "data":"hello"
        }
    }
    http.post(options, function(err, response){
        if(err){
            resp.error("Failed to call the webhook");
        }
        resp.success("Success"+ response);
    })
}

HTTP Basic Auth

function ExampleWebhookInvoker(req,resp){
    // These are parameters passed into the code service
    var params = req.params
    var http = Requests();
    var options = {
        uri:"https://platform.clearblade.com/api/v/4/webhook/execute/84c3e6d10bbae683c7ace4dfd701/Webhook_2_HTTP_Auth",
        headers:{
            "Authorization":"Basic Id19kSWr2vkB4CUc5lm6Akzh1jXr6Fw3sxW-ynaaV67EJRLmrHvkqUD8vnYu52XGErPEjDNRpSNJsD5w=="
        },
        body:{
            "data":"Authenticated using Http Auth"
        }
    }
    http.post(options, function(err, response){
        if(err){
            resp.error("Failed to call the webhook");
        }
        resp.success("Success"+ response);
    });
}

Payload Auth GET

function ExampleWebhookInvoker(req,resp){
    // These are parameters passed into the code service
    var params = req.params
    var http = Requests();
    var options = {
        uri:"https://platform.clearblade.com/api/v/4/webhook/execute/f6edffd60bdc95a5c19ff79d0a/Webhook_3_Payload_Auth",
        qs:{
            "token":"Id19kSWr2vkB4CUcxa5lm6Akzh1jXr6Fw3sxW-ynaaV67ELmrHvkqUD8vnYu52XGErPEjDNRpSNJsD5w==",
            "data":"Authenticated using Get method"
        }
    }
    //Using get method
    http.get(options, function(err, response){
        if(err){
            resp.error("Failed to call the webhook");
        }
        resp.success("Success"+ response);
    });
}

Payload Auth POST

function ExampleWebhookInvoker(req,resp){
    // These are parameters passed into the code service
    var params = req.params
    var http = Requests();
    var options = {
        uri:"https://platform.clearblade.com/api/v/4/webhook/execute/84c3e6d10b8ebae3c7ace4dfd701/Webhook_3_Payload_Auth",
        body:{
            "token":"Id19kSWr2vkB4CUcxa5lm6Akzh1jXr6Fw3sxW-ynaaV67EJRLmrHvkqUDnYu52XGErPEjDNRpSNJsD5w==",
            "data":"Authenticated using Http Payload"
        }
    }
    http.post(options, function(err, response){
        if(err){
            resp.error("Failed to call the webhook");
        }
        resp.success("Success"+ response);
    });
}

No Auth

function ExampleWebhookInvoker(req,resp){
    // These are parameters passed into the code service
    var params = req.params
    var http = Requests();
    var options = {
        uri:"https://platform.clearblade.com/api/v/4/webhook/execute/84c3e6d10b8ebae3c7ace4dfd701/Webhook_4_No_Auth"
    }
    http.post(options, function(err, response){
        if(err){
            resp.error("Failed to call the webhook");
        }
        resp.success("Success"+ response);
    });
}

Advanced

Execution Timeout

Code Services are usually used as microservices and are expected to have a finite lifecycle with a default of 60 seconds. However, this execution timeout can be extended to make the services run longer. When execution timeout is set to infinity(“Never” in the console), it’s called a Stream Service. This means the code service will run indefinitely. It allows for the following high-performance use cases:

  • Exponentially faster rate for any operation which is otherwise performed in an event-triggered micro-service.
  • Users can also design stream services to perform bulk operations on collections.
  • Data generation

Best Practices:

Concurrency

Concurrency allows the developer to cap the maximum number of instances of code services that can run on each of the ClearBlade nodes. Any more than this number will kill the oldest code service instance.

Best Practices:

  • For Stream Services, set Concurrency to Other, and then set the number of instances you need. Recommended is 4.

    • For Micro Service, set Concurrency to its default of Infinite, to optimize performance

Run as

This feature is useful when the user who invokes the service needs different permissions than what is required for the service to run.

If a role has permission to execute a code service, then a user with that role can start the execution of the service. If the run as parameter is set, it runs as the user that is stated in the parameter, regardless of the user that invokes it. If the user set in the run as parameter doesn’t have permissions to execute the service (and the permissions to execute all the operations performed within the code service), then the service will fail. If the user invoking the service does not have permission to execute the code service, then the service will fail as well.

For example An end-user (eg: customer1@customer.com) can request for his account information through an accountsInfo service. The user the service executes as needs permissions to the “accounts” collection. The end-user is not supposed to have permissions to the “accounts” collection. This is where the “Run as” feature allows us to have a separation between the end-user and the user the service is “run as”. For the above scenario, the end-user (here customer1@customer.com) will need to execute permissions for the code service. The run as user (here accountInfo_service_user@clearblade.com) will need to execute permissions to the code service along with other permissions for assets accessed in the code service (in this case, “read” permissions to the “accounts” collection).

Auto Restart

This property is valid when the service is a Stream Service. When the box is checked, it restarts service automatically if it stops for any reason, including successes.

Best Practice: Do not check the box until fully testing the service. There could be undesired deadlocks if there are errors in the code.

Auto Balance

This property is valid when the service is a Stream Service. The platform automatically starts a service on each available node with the number of concurrency instances. It kills and restarts services when saved with a new code.

If auto_balance = true :

  • req.params are not allowed
  • A normal code service must be developed first to confirm that it is functioning properly before developing it as a stream service. Do not check the auto-balance box until the code service works.

Failed Runs

If a Code Service exits with resp.error, or throws an uncaught error, it will be stored in Failed Runs. A developer can then fix any issue and retry that failed run with the same input parameters.

Standard Javascript Functions (es5.1)

Method Usage
setTimeout(),setInterval() Available natively in version 9.3.0 or later
atob(), btoa() buffer-node-js library
promises q-promise-library/Available natively in version 9.3.0 or later