Skip to main content

Connections

Overview​

A Connection in this platform is a data integration pipeline that enables the platform to:

  • Connect to external data sources (HTTP APIs, MQTT brokers)
  • Send and receive data to and from other platforms
  • Process and transform data through configurable workflows

Node Types​

The platform supports various node types for building data processing workflows:

Node TypePurposeKey FeaturesUse Cases
ExtractData extraction from payloadsSource selection, JSON path configurationExtract specific fields from incoming data
LoopArray iteration and processingArray validation, item processingProcess multiple items in arrays
LookupReference data lookupsLookup tables, ID mapping (internal to external)Map external identifiers to internal IDs
MapData point mappingField mapping, data transformationMap object keys with data points
SaveGlobal variable storagePersistent variable storageSave and reuse global variables
SplitString parsing and splittingConfigurable source and destination keys; outputs indexed parts (part1, part2, etc.)Parse MQTT topics to extract structured data

Key Features​

1. Test Connection​

The Test Connection feature enables you to verify that your connection to an external data source is working correctly before using it in production.

How It Works​

  • Configuration Validation: Checks that your connection settings (such as host, port, and authentication details) are correct
  • Connectivity Check: Ensures the platform can reach the external data source over the network
  • Authentication Test: Validates that your credentials and permissions are accepted by the external system
  • Response Validation: Confirms that the external system responds as expected

What Users See​

  • Test Button: Each connection in the connections list includes a "Test" button
  • Loading State: While the test is running, a loading indicator is shown
  • Results Modal: After the test completes, a modal displays detailed results, including:
    • Connection status (success or failure)
    • Response data from the external system
    • Any error messages if the test fails
    • Response headers and timing information

This feature helps you quickly identify and resolve configuration or connectivity issues before integrating the connection into your workflows.

2. Export Policy​

The Export Policy feature allows users to save their entire data processing workflow—including all node configurations and connections—for backup and sharing purposes.

What is Exported?​

  • Node Configurations: All settings for each node type (Extract, Loop, Lookup, Map, Save)
  • Node Connections: The links and relationships between nodes in the workflow
  • Node Positions: The visual layout and coordinates of each node on the workflow canvas
  • Entry Point: The node where processing begins
  • Adjacency List: The complete graph structure representing the workflow

Export Options​

  • Copy as JSON: You can easily copy your workflow as a structured, human-readable JSON object to your clipboard for sharing or pasting elsewhere

This feature makes it simple to back up, restore, or share your workflow with others.

3. Import Policy​

The Import Policy feature allows users to import their entire data processing workflow—including all node configurations. This is useful for restoring a workflow from a backup or sharing with others.

What is Imported?​

  • Node Configurations: All settings for each node type (Extract, Loop, Lookup, Map, Save)
  • Node Connections: The links and relationships between nodes in the workflow
  • Node Positions: The visual layout and coordinates of each node on the workflow canvas
  • Entry Point: The node where processing begins
  • Adjacency List: The complete graph structure representing the workflow

Import Options​

  • Paste from Clipboard: You can easily paste your workflow from your clipboard

4. Run/Test Policy​

The Run/Test Policy feature enables users to simulate and validate their data processing policy to see how it will perform in production.

Purpose​

  • Allows users to test their entire policy with sample payloads and headers

How It Works​

  • Sample Data: The test uses the payload and headers obtained from the connection
  • Workflow Execution: Executes the full sequence of nodes in your workflow
  • Result Preview: On hover over a node, you can see the result of that node

What Users See​

  • Run Button: Located in the workflow, allowing you to start the test
  • Node-by-Node Results: For each node, both the input and output data are displayed

Testing Process​

  1. Data Preparation: Loads the payload and headers
  2. Sequential Processing: Executes each node in order, passing data along the workflow
  3. Data Flow Visualization: Illustrates how data moves and changes between nodes

5. Policy Management​

The Policy Management system provides robust tools for handling the lifecycle of your workflows. The main features include:

Save Policy​

  • Purpose: Permanently save the current workflow configuration
  • Process: Stores the complete node graph, settings, and all related metadata
  • Integration: Associates the saved policy with its corresponding connection
  • Persistence: Ensures the workflow is available for production use and future editing

Update Policy​

  • Purpose: Modify and enhance existing workflows as requirements evolve
  • Process: Updates the stored configuration with new changes

Clear Policy​

  • Purpose: Clear the current workflow configuration and reset it for new changes

These management features help maintain a clean, organized, and reliable workflow environment.

Connection Types and Configuration​

The platform supports multiple connection types, each with specific configuration options:

MQTT Connection Configuration​

In short: Lightweight pub/sub messaging, great for IoT and real-time data.

Basic Information​

  • Connection Name*: A descriptive name for your MQTT connection
  • Connection Type*: Select "MQTT" from the dropdown
  • Is Draft: Checkbox to save as draft (optional)

MQTT Configuration Fields​

FieldDescription
HostMQTT broker hostname or IP address
PortMQTT broker port number
Client IDUnique identifier for this MQTT client
UsernameAuthentication username
PasswordAuthentication password
TopicDefault topic for publishing/subscribing
QoS LevelQuality of Service level (0, 1, or 2)
Payload (Base64)Base64 encoded payload for testing
Clean SessionStart with a clean session
RetainedRetain messages on the broker

MQTT Subscriptions​

  • Topic: MQTT topic to subscribe to
  • QoS: Quality of Service level for the subscription
  • Add Subscription: Button to add multiple topic subscriptions

HTTP Connection Configuration​

In short: Standard request–response API calls over HTTP/HTTPS.

Basic Information​

  • Connection Name*: A descriptive name for your HTTP connection
  • Connection Type*: Select "HTTP" from the dropdown
  • Is Draft: Checkbox to save as draft (optional)

HTTP Configuration Fields​

FieldDescription
HostHTTP server hostname or IP address
PortHTTP server port number
SchemeProtocol scheme (HTTP/HTTPS)
PathAPI endpoint path
MethodHTTP request method
PayloadRequest body data (for POST/PUT requests)

HTTP Headers​

  • Header Name: Custom header name (e.g., Content-Type, Authorization)
  • Header Value: Header value (e.g., application/json, Bearer token)
  • Add Header: Button to add multiple custom headers
  • Default Headers: Accept and Content-Type headers are pre-configured

Advanced Configuration Options​

Advanced Configuration​

  • Timeout: Connection timeout in seconds (default: 30)
  • Priority: Connection priority level (default: Normal)
  • Tick Interval: Health check interval (default: 120s)

Retry Configuration​

  • Multiplier*: Exponential backoff multiplier (default: 2)
  • Max Backoff*: Maximum backoff time (default: 30s)
  • Min Backoff*: Minimum backoff time (default: 1s)
  • Max Attempts*: Maximum retry attempts (default: 3)

Options Configuration​

  • Debug: Enable debug logging (optional)
  • Compression: Data compression method (default: Gzip)
Connection Types
  • MQTT: Best for IoT devices, real-time messaging, and lightweight communication
  • HTTP: Ideal for REST API integration, web services, and standard web communication :::

Creating a New Connection​

Step 1: Create Connection​

Click the Create Connection button.

Create Connection Button

Step 2: Configure Connection​

Select the connection type and fill out the form, then click the Create button.

Choose between:

  • MQTT: For IoT devices and real-time messaging
  • HTTP: For REST API integration and web services

We are using the HTTP connection type for this example. :::

Connection Configuration Form

Step 3: View Connection​

After creating the connection, you can see it in the connections list.

Connections List

Step 4: Test Connection​

Click the Test Connection button to test the connection.

Test Results

If the connection is successful, you will see the connection result with payload and headers data. You can also save this data to create a new policy.

Test Connection Results

Creating or Editing a Policy​

Step 1: Access Policy Editor​

Click the Edit Policy button to edit or create the policy.

Edit Policy Button

Step 2: Policy Editor Interface​

After clicking the Edit Policy button, you will be redirected to the policy editor page.

Policy Editor Interface

Policy Editor Features​

  • Node Management: Create or edit a policy by adding or removing nodes in the editor
  • Data Preview: On the left side, you will see a JSON tree displaying the payload and headers data
  • Node Selection: There is an "Add Node" dropdown menu that lists all available node types
  • Node Addition: To add a node, click the "Add Node" dropdown and select the node you want to insert
  • Node Connections: You can add multiple nodes to the policy and connect them to each other using edges

Step 3: Building Your Policy​

Policy with Nodes
Example Policy

In this example, we have added two nodes, "loop" and "lookup", to the policy and connected them using edges.

  • To map an external identifier ID with an internal identifier ID, click the "Open Lookup Table" button within the lookup node to open the lookup table.

Step 4: Configure Lookup Table​

After clicking on the Open Lookup Table button, you will see the lookup table modal.

Lookup Table Modal

Lookup Table Configuration​

  • Select Identifier Type: Choose either "Device" or "Fleet"
  • Select Entity: After choosing the type, select the specific device or fleet from the list and click the "Save" button to confirm your selection
  • Create Missing Entities: If you do not find the desired device or fleet in the list, you can enable the "Create on missing" option and click the "Create and Save" button to add a new device or fleet to the platform
Create Missing Entity Option

Step 5: Configure Data Point Mapping​

Now, we are adding a new map node to map data points. Click on the Edit Mapping Policy button.

Edit Mapping Policy Button

The mapping policy editor modal will now appear.

Mapping Policy Editor Interface​

  • Sample Payload: On the left side, you will see a sample payload
  • Data Point Options: On the right side, you have two options: create a new data point or edit and use an existing data point
Mapping Policy Editor Interface

Step 6: Create New Data Point​

Creating a New Data Point

Create New Data Point Interface

Write the data point name and proto message name, then click the "Create Data Point" button.

Data Point Creation Form
  • Now, you will see the new data point in the list. Click on the data point to select it
  • You can also map multiple data points to the sample payload
Data Point Selection

Step 7: Complete Data Point Mapping​

  • You can see the data point automatically map to the sample payload
  • If you want to map the data point to a different field, you can edit the mapping policy
  • After editing the mapping policy, you can save the policy by clicking the "Save Mapping" button
Data Point Mapping Process

Step 8: Save Mapping Policy​

  • Once you save the mapping policy, it will appear in the policy list
  • In the modal footer, use the "Save Mapping" button to permanently save the policy
Save Mapping Policy

Step 9: Run/Test Policy​

  • Click the "Run/Test Policy" button to run/test the complete policy
  • You can see the node-by-node results on hover over a node
Run/Test Policy Interface

Step 10: Create Policy​

  • After running/testing the policy, you can create a new policy
  • Click the "Create Policy" button to create a new policy
  • You can see a toast message after creating the policy
Create Policy Interface

Policy Editor Features​

1. Export Policy​

  • Click the "Export Policy" button to export the policy
  • You can see the policy workflow copied to the clipboard
Export Policy Interface

2. Import Policy​

  • Click the "Import Policy" button and paste the policy workflow from the clipboard
Import Policy Interface

3. Clear Policy and Update Policy​

  • Click the "Clear Policy" button to clear the policy workflow
  • Click the "Update Policy" button to update the policy workflow
Clear and Update Policy Interface

Summary​

This comprehensive guide has walked you through the complete process of:

  1. Creating a new connection to external data sources
  2. Testing the connection to ensure it works properly
  3. Building a data processing policy using various node types
  4. Configuring lookup tables for ID mapping
  5. Creating and mapping data points for data transformation
  6. Running and testing policies to validate workflows
  7. Managing policies with export, import, clear, and update features

Key Features Covered​

  • Connection Types: Support for MQTT and HTTP connections with comprehensive configuration options
  • Connection Management: Create, test, and manage external data source connections
  • Advanced Configuration: Timeout, retry, and compression settings for robust connections
  • Policy Editor: Visual workflow builder with drag-and-drop node configuration
  • Data Processing: Extract, loop, lookup, map, save, and split operations
  • Policy Testing: Run/test policies with real-time node-by-node results
  • Policy Management: Export, import, clear, and update policy workflows

The connections feature provides a powerful way to integrate external data sources with your platform, enabling real-time data processing and transformation through configurable workflows. This comprehensive system allows you to build sophisticated data pipelines that can handle complex data transformations and integrations.