Integration Operator vs. Integration Platform

Doohickey is built to be run in conjunction with a sufficient integration platform. We call Doohickey Cloud, our cloud-hosted API/service, an "Integration Operator". While not exclusively so, Doohickey is most compatible with Open Integration Hub (OIH), an open source integration framework.

This article describes how Doohickey Cloud combines with the APIs and capabilities of the integration platform to provide a fully functional embedded integration solution for software products.

What is an integration operator?

To support native software product integrations at scale, you need to have the following capabilities:

• Build & Managing Integrations
• Operating (i.e. Provisioning) Integrations
• Running Integrations
• Monitoring Integrations

Any integration platform provides these capabilites in some form. This true of full solutions installed "on premise" or "installed" as well as Integration Platform-as-a-Service (iPaaS) providers.

Doohickey takes an intentionally different approach, using a hybrid "as a service" and "installed" architecture. It delegates these capabilities to the part of the architecture that is most advantageous.

Building and Managing

Doohickey Cloud is the repository for your integration templates. These are your integration functions that, when combined with specific user configurations, are able to output a live integration into the runtime (usually Open Integration Hub).

Having a cloud-based repository provides typical "as a service" advantages, including providing a single source of truth accesible via CLI and API.

Note: Integration templates do not store any sensitive or personally identifiable information. The templates themselves also process no actual end user data.

Operating and Provisioning

Doohickey Cloud stores two important end user specific pieces of data: the end user's configuration (e.g. how they want their version of an integration to work) and reference's to the end user's credentials, which are stored in the installed integration platform.

When requested via API, UI, or CLI, these data objects are merged into the appropriate integration templates to generate an runnable integration flow. That flow is publsihed to the connected integration platform, usually Open Integration Hub.

In this way, the integration operator "tells" the integration platform how to work for "this specific user".

Note: It is possible to design templates that require sensitive data to be stored in the config, including user credentials. This is NOT RECOMMENDED, because it will negate the advantage that the integration operator stores no sensitive data.

Running

Doohickey Cloud does not run integrations. It does not process end user data. This responsibility is delegated to the integration platform, usually Open Integration Hub, which is also usually installed into your cloud environment via Kubernetes.

This allows the data processing to happen in your infrastructure and subject to your compliance procedures.

Note: Blended Edge offers a service to run the integration platform on your behalf "as a service". In this case, we will be a data processor.

Monitoring

Note: Some of these features are under development and coming soon.

To maintain working integrations and support their users, you must have visibility into the data has moved through every execution of every integration flow. This is important for providing customer service, specifically when failures occur. The end user isn't going to understand the nuance of your integration architecture. They will simply ask "what happened to my record(s)" and you'll have to go find out.

There are a few challenges with this:

• The data necessary to fully support an integration is often sensitive and subject to compliance policies
• The data is usually not in an easily searchable place, spread across logs, queues, and other places
• The data is not contextualized to the end user's experience with the integration

To provide this monitoring capability, we install an open source collector into your cloud environment, alongside the installed instance of the integration platform. The collector's job is to collect and process every event for consumption and searchability. The processing can (and usually will) include cleansing sensitive data like PII, per your requirements.

The Doohickey Cloud integration operator communicates with this collector and its data store from the cloud. It provides a user experience optimized for supporting complex integrations, but consumes the processed data stored in your environment. This gives you the "as a Service" experience for navigating flow execution data without forcing Doohickey Cloud to be a data processor.

Which API Does What

The responsibilities for providing embedded integrations are shared between the operational API provided by Doohickey Cloud Services and the integration platform's API (and underlying capabilities). You software application should interact with both APIs, typically in the following ways:

Integration Operator (Operational API)

• Loading integrations and their templates
• Triggering Oauth flows (though end user credentials are not stored in Doohickey Cloud)
• Viewing data integration activity
• Deploying end user instances of integrations, using the end user-provided configuration options
• Activate and Deactivate Integrations

Integration Platform API (OIH)

• Store end user credentials directly (if required)
• Inspect underlying log data
• Provision integration platform tenants/accounts
• Inspect specific flows deployed from a template

Using Both APIs Properly

When embedding integrations into your applications or software products, you may interface with both APIs. Whenever possible, expect to integrate with Doohickey Cloud's APIs over the integration platform's. In the event of duplicate functonality, Doohickey includes additional functionality.

You may also store GUID cross-references between the systems, e.g. storing an Open Integration Hub Secret Service record ID in an Integration Config object. Only refer to entities across systems by their GUID. Do not embed or store sensitive information in the system that isn't intended to store it.