Automate data flows

Build a workflow

Integration Studio: triggers, nodes and actions. Build a recurring flow that runs unattended and tells you when it did not.

A workflow is a flow you build once and then stop thinking about: a trigger, a few steps, and somewhere for the result to go. It is what turns a manual import you run every Monday into something that runs every night without you.

Before you begin. You need a connected system and a mapping that you have already previewed. Build the mapping first; a workflow around a wrong mapping only makes the wrong thing happen on a schedule.

The pieces

Trigger: what starts a run. Three kinds:

TriggerStarts whenTypical use
ScheduleA time you setNightly product feed, hourly price update
EventSomething happens hereAn order is placed, a product is published
ManualSomebody presses runMigrations, one-off corrections, testing

Nodes: the steps in between. Read from a connection, filter, transform, branch on a condition, write, notify.

Actions: what a node does at the end: import records, send a payload to an endpoint, write a file to SFTP, send a notification.

Runs: one execution, with its status, duration, counts and errors, listed under Integrations › Runs. The run history is what you will live with, so name workflows so they read clearly in a list: ERP nightly product feed, not Workflow 4. Runs are durable: a run that starts at 03:00 and takes forty minutes survives a restart, and the run list shows where it stopped.

Build it

  1. Go to Integrations › Workflows.
  2. Select New workflow and name it for what it does.
  3. Choose the trigger. For a schedule, set the time in the tenant's timezone and place it where it does not collide: after the source system's own nightly run has finished, and not at the same minute as other feeds. Time triggers are listed together under Integrations › Schedules.
  4. Add a source node and select the credential and the mapping profile. Reference the credential by name rather than entering connection details in the node. That is what lets you repoint the workflow from test to production without editing it.
  5. Add a filter if the run should not cover everything. A feed restricted to one product group is how you test safely on live data.
  6. Add the target action — for a product feed, the product import.
  7. Set error handling: how many rows may fail before the run is abandoned, and whether a partial run is committed or rolled back. For a large catalog feed, tolerating a handful of bad rows and reporting them is usually right. For prices, it is usually not.
  8. Add a notification on failure, to a group address rather than a person.
  9. Save it as inactive first.

Test before you schedule it

  1. Run it manually with a filter that limits it to a handful of records.
  2. Read the run in Integrations › Runs: rows read, created, updated, skipped, failed.
  3. Open two or three affected records and compare them against the source.
  4. Run the same scope again. A second run should produce zero creates.
  5. Widen the filter, run again, and only then activate the schedule.

What to check afterwards

For the first week, look at the run list every morning:

  • Did it run at all? A missing run is worse than a failed one, because nothing alerted.
  • Is the runtime stable? A nightly job creeping from 20 to 90 minutes will eventually overrun into business hours.
  • Are the counts plausible? A feed that updated 40,000 products yesterday and 3 today did not do less work; something upstream changed.
A workflow that writes outward — transmitting orders, pushing stock, sending a catalog — must be idempotent before you schedule it. A retry after a timeout will re-send, and a receiver that does not deduplicate turns one order into two deliveries. See Integration patterns.

Patterns worth copying

  • Nightly full load, weekday deltas. Deltas keep the runtime short; the weekly full load repairs whatever the deltas missed.
  • Staging step before the real one. Import into a restricted channel or a filtered view, check the counts, then release. Slower, and it has saved more catalogs than any validation rule.
  • Separate feeds per data type. Products, prices and stock in three workflows, not one. When prices fail you re-run prices, not the whole night.
  • A canary check. A tiny workflow that reads one known record every hour and alerts if it cannot. It finds a dead connection long before the nightly run does.

Next