Blended Edge Logo

Develop an Integration

An integration represents two products passing data back and forth in service of creating a cross-product user experience to benefit the user in some way, typically by eliminating manual data synchronization or automating a process.

In OIH an “integration” doesn’t exist as a distinct entity. Instead, you simply deploy a set of one or more data flows (technically OIH calls them “flows”). A set of flows between two systems defines how data moves bidirectionally between those two systems. Therefore, to “build an integration” in OIH, you build a set of one or more flows. You deploy those flows to a single tenant, which correlates to a specific user.

When defining a reusable product integration, an extra level of design is required. You must design points of optionality or “fill in the blanks” into the flows to allow different logical functions for the flow.

Examples include:

  • Fields set via a user-defined string
  • API credentials
  • Cross-reference maps, aligning metadata across the two endpoints
  • If-logic determining two or different ways to map data

When using these “templated” integrations, you must fill in those blanks on behalf of the user at the time you create or update the flow within that user’s tenant. This design means that all users of that templated integration (set of flows) share identical configurations except for the configurations they specifically enter.

This gives the users flexibility with guardrails so everyone’s integrations can still be easily managed and monitored.

Flows and Multi-tenancy

Open Integration Hub is a multi-tenant platform, which is one of the primary reasons its capabilities support embedded SaaS integration effectively.

Tenants within an OIH environment effectively represent an “account”. It’s a logical separation of ownership for configuration data. It enables/restricts what different users can access in the OIH instance. If end users were logging into OIH directly, this would also impact what they could see or configure.

The following entities are likely to be owned by a tenant:

  • Users (except for administrators or “super” users)
  • Flows
  • Secrets

A productized integration, built for Open Integration Hub, is a set of Flow JSON files, designed to be templates. Using a manual flow deployment workflow (later in this document) or an automated process, those flow templates are merged with tenant-specific information like secrets and configuration input, and deployed into the OIH environment as a flow owned by that tenant.

You should think of a tenant as 1:1 with an end customer, regardless of how many integrations they may have enabled. Multiple flows across multiple integrated systems for a specific end user should live in the same tenant.

You should not ever mix flows or configuration within a tenant for multiple end user accounts, even if you obfuscate away Open Integration Hub from your end user. This violates security best practices.

Flow JSON

Flows are defined in a JSON object that is deployed to the Flow Repository service, associated with a given user’s tenant. Whether templated or single-use, developing an integration means developing one or more of these JSON documents.

The flow JSON contains the following information:

  • Name, description, and other metadata
  • If the flow is scheduled, a cron expression defining when the flow should execute
  • The flow’s owner(s), typically the user that created the flow and the flow’s tenant
  • The components used in the flow and how they are configured (the flow steps)
  • A graph defining the order in which the various component configurations should execute (the sequence of the flow steps)

More fundamental information about flows can be found in the OIH documentation.

Flows are made up of steps, implemented by configurations to use any of the various components available in a given OIH instance. While Blended Edge-managed OIH environments tend to leverage a standardized set of components, any OIH instance can have any OIH-compatible component available. This includes possibilities for fully custom ones.

Components generally include sets of related functions for manipulating data in some way. Components are deployed as individual Docker containers, so they can be scaled horizontally to add capacity.

Each component shares a common schema for metadata like a component name/description, reference to a Secret Service credential ID, and a handful of other fields. Each component has a unique set of “fields” where configuration that is specific to that component resides.

Flow Development Workflow

The following is a recommended workflow for developing a flow JSON document.

  1. Define a new flow, starting with a basic shell that includes only the flow name, optionally a description, the first component to the flow and its graph entry, the owning tenant, and a cron expression if the flow is timed. These can all be changed later if they aren't initially correct. 2)POST the flow to OIH using the “Create a flow” API call. Save the returned flow ID, because you’ll need it for future updates and to start and stop the flow.
  2. Use the “Start a flow” API call and the flow ID to activate a flow.
  3. Verify that the flow executes by either watching for scheduled execution or using Postman to post a test webhook to the flow’s assigned webhook receiving endpoint.
  4. Use the “Stop a flow” API call to deactivate the flow.
  5. Return to your flow JSON document and add additional components, their configurations, and their graph entries.
  6. Update the flow using the “Edit a flow” API call, which will use a PATCH request to overwrite the existing flow configuration. Note, that a flow must be deactivated in order to update it.
  7. Continue making changes, updating the flow in OIH, starting it, executing the flow to test, and deactivating it, until you’ve completed the logic for the entire end-to-end data flow, including any defined points of configuration or variation are required.

JSONata Data Map

One of the most important parts of a data flow is the data transformation map, usually written in JSONata. You can also write data maps in JavaScript, but this is almost never the best practice for a data map.

JSONata is a querying and transformation language designed for JSON. It’s based on the specification for XSL, a language that accommodates similar functions for XML documents. While JSONata is used in many places in OIH, it’s most notably used in the JSONata Transform component, which should be used within a data flow to implement most of the transformation logic between two integrated endpoints.

While there is full support for including many JSONata transformations or no JSONata transformations within a single flow, you should assume that you will always have one primary JSONata transformation that is surrounded by whatever API call(s) retrieve data to be transformed and the API call(s) that write the data, once transformed.

It’s recommended that, as much as possible, user configurations for templated flows be incorporated into the JSONata transformation. This is not always possible, but most configurations can manifest in the map when designed properly.

At a basic level, the JSONata transformation defines how “this field goes to that field” between comparable entities on the source and destination endpoints for the flow. However, these maps can include complex mapping logic, business logic, user configurations, and other data manipulations. This makes the JSONata transformation a powerful tool when defining a data flow.

JSONata transformations are typically larger documents, especially when formatted for readability. There can be hundreds of lines in a single document. However, to include the JSONata map appropriately as a field in a flow JSON, so that the JSONata Transformation component can use the map, it must be minified and reduced to a single line of text. It also must have XML syntax characters escaped.

Therefore, it’s best practice to define the JSONata document as an external file, formatted for readability. Then to apply an update to a flow, a minified version of the JSONata should be copied into the flow JSON in the appropriate field, before the flow JSON is used to update the flow in the OIH instance.

Writing a JSONata Map

It’s helpful to write the JSONata map document while testing it in real-time, so you don’t have to take the time to make flow updates every time you change the map. Currently, the “JSONata Exerciser”, available at https://try.jsonata.org, is the best place to do this.

The JSONata Exerciser has three panels: one for an input JSON object, another for the JSONata you are writing, and a third where a read-only output is printed in real-time as you update your JSONata document.

Typically, you’ll grab the JSON object that is output from the flow step prior to the JSONata Transformation step to use as the input.

To copy a JSONata expression (or map) into a Flow JSON, you’ll need to ensure it will insert as a string that doesn’t invalidate the JSON. The following are recommended:

  • Remove white space and line breaks with the regular expressions “\s+” and “\n”, respectively.
  • Or, in Visual Studio Code, highlight all of the map and click Ctrl + D.
  • Use single quotes whenever possible in your JSONata, because they are interchangeable with double quotes--like JavaScript
  • In cases where you must use double quotes or other JSON syntax characters, escape them when copying into the flow JSON. Note: this makes the JSONata invalid to directly, but OIH will respect the character escapes properly when executing the JSONata expression.
  • Maintain a formatted version of the JSONata expression in a separate file from the flow, so you can edit, source control, and read more easily.