Common integration tasks

Bulk exports

Everything, nightly, for BI — export profiles, choosing a format that will not run out of memory, scheduling, and getting the file to where it is needed.

An export is a job, not a request. You ask for it, it runs, and you collect a file — which is why it can move a million rows without touching your per-second API allowance.

Before you begin. Be clear which question the export answers. "Everything" is not a requirement; it is a way of building a job that takes four hours and that nobody can debug. Ask what the receiving side actually joins on and filters by, and export that.

Two different things called "export"

Bulk exportReport export
What it isA file of records from a datasetA CSV of a report you ran
VolumeUp to a million rowsWhat fits a spreadsheet
Who runs itA schedule, or an integrationA person, on the day
WhereData Exchange › Import & ExportAnalytics › Reports
ForFeeding another systemAnswering a question today

If somebody in Controlling wants last quarter's orders by customer, they want a report export, not an integration. Send them there before you build anything.

Define an export profile

A profile is the reusable definition: what to export, which fields, in which format, filtered how. Create it under Data Exchange › Import & Export.

Four decisions:

1. The dataset. Products, prices, orders, customers, stock — whatever your installed apps publish.

2. The fields. Export the ones the receiver uses, and the keys it joins on. Two fields people forget and then need: the business key (SKU, customer number, order number) and a timestamp, so the receiver can tell what changed.

3. Filters. A filter on the profile beats filtering a huge file afterwards. Only published products, only orders since a date, only one channel.

4. Full or delta. Same trade-off as an import, in reverse:

Full exportDelta export
VolumeEverything, every runOnly what changed
RuntimeLongShort
A missed runRepaired by the next oneLeaves a permanent gap
DeletionsVisible — absent means goneOnly if you export a delete signal
Good forNightly, up to a few hundred thousand rowsVery large sets, or frequent runs

A workable compromise, and the one most projects land on: deltas through the week, one full export at the weekend to repair whatever the deltas missed.

Choose the format deliberately

Four are available, and the choice is not only about what the receiver prefers.

FormatGood forWatch out
CSVThe default for anything largeNo types — dates and decimals need a stated convention
XMLReceivers expecting a schema, BMEcat-shaped feedsVerbose; the file is several times larger
JSONSystems that consume JSON nativelyBuffered in memory before writing — a poor choice for the very largest sets
XLSXA human opening itBuffered in memory, and a spreadsheet is not an interface

The practical rule: CSV and XML stream row by row; JSON and XLSX are built in memory first. For a million-row export, pick CSV or XML. If the receiver insists on JSON at that size, split it into several files.

Two conventions to agree once and write into the profile description, because they cause more support tickets than anything else on this page: the decimal separator and the date format. A German-locale CSV with 19,90 read by a system expecting 19.90 produces prices that are wrong by a factor of a hundred, without any error.

Schedule it

Two ways, depending on who is driving.

From a workflow — the usual choice. Build it in Integration Studio: a schedule trigger, the export step, and a delivery step. The run keeps its log, so you can see what happened and re-run it.

From an integration — when the receiving system decides. Trigger the profile and poll the job:

POST /v1/io/profiles/{id}/run     → starts the job
GET  /v1/io/bulk-jobs/{job_id}    → state, rows read / written / rejected
GET  /v1/io/exports/{job_id}/url  → a short-lived signed download URL

Whichever you use, give heavy exports a window of their own. The rate limit is per tenant, so a large export competing with your storefront is a slow storefront — see Rate limits, quotas and fair use. Stagger jobs rather than starting six of them exactly on the hour.

Deliver the file

TargetWhen
A signed download URLThe receiver pulls, on its own schedule. Simplest, and the URL is deliberately short-lived
SFTP folderThe receiver watches a folder. The most common answer for an ERP or BI job — see SFTP and file exchange
A workflow stepThe file has to be transformed, split or posted somewhere on the way

Whatever the target, keep the naming convention: orders_delta_20270312_0200.csv. Never overwrite, and move processed files to an archive folder rather than deleting them. The re-run that saves your evening depends on the file still existing.

What to check

After the first scheduled run:

  • The row counts on the job — read, written and rejected. Rejected rows above zero mean a mapping problem, and a job that "succeeded" with 4,000 rejects is not a success.
  • The file size is plausible. An export that is suddenly a tenth of last night's size is a filter that changed, not a quiet quarter.
  • The receiver actually loaded it, rather than merely receiving it.
  • The run time leaves headroom. A job that takes 50 minutes in a one-hour window will fail the first time your catalog grows.

When it does not work

  • The job never finishes — usually a JSON or XLSX export at a size that should have been CSV.
  • Numbers or dates arrive wrong — the decimal separator or date format convention. Not a bug; an unstated agreement.
  • The file arrives but is empty — a filter on the profile, or a delta export after a full one already took everything.
  • The download URL has expired — signed URLs are short-lived by design. Fetch it as part of the job, not hours later.
  • Nothing arrived at all, and nothing failed — the run did not start. That is the failure nobody notices; see Monitor your integrations.

Next