Most integrations are built as projects. Someone maps some fields, it passes testing, the team moves on, and the integration then runs for years. When an invoice goes missing between two systems, both teams look at their own logs, see no errors, and conclude the problem belongs to the other side.

We think an integration that matters to the business should be run like a small product, with users, a published interface, measures of health and someone accountable for it. Five things separate that from a script that happens to work.

A contract, not just a schema

Write the interface down in a form machines can check, such as OpenAPI for HTTP or AsyncAPI and a schema registry for events. Then test against it in the pipeline, ideally with consumer-driven contract tests, so a breaking change fails a build rather than a customer's order.

Field types are the easy part. The disagreements that hurt are about meaning, so the contract should answer these as well:

  • What is a "customer" here, and is it the same thing as in the other system?
  • Are amounts in cents or dollars, with or without GST? Are timestamps in UTC?
  • Which fields can be null, and what does null mean?
  • What are the rate limits and expected volumes?
  • How will changes be versioned, and how much notice will consumers get before something is retired?

Idempotency

Networks fail in ambiguous ways. If a request times out, the caller can't tell whether the work happened, so it retries. Most message brokers offer at-least-once delivery, which means duplicates are normal behaviour. Without idempotency, an ordinary timeout can become a double payment.

An operation is idempotent if performing it repeatedly has the same effect as performing it once. The usual approach is:

  • The sender generates a unique key for each business operation and reuses it on every retry.
  • The receiver records the key in the same database transaction as the work itself. Doing these separately leaves a gap where a crash causes duplicates.
  • A repeat of a known key returns the original result without doing the work again.
  • Keys are kept for longer than the longest possible retry period.

Retries that don't make things worse

A naive retry loop turns a struggling service into a dead one, as every client piles on at once.

  • Retry only what might succeed later: timeouts, 429 and most 5xx responses. A 400 won't improve with repetition.
  • Use exponential backoff with jitter, so that clients spread out over time.
  • Honour Retry-After headers.
  • Cap the attempts and the total time, and consider a circuit breaker for extended outages.
  • Send messages that still fail to a dead-letter queue, and make sure that queue has an alert, an owner and a tool for replaying messages. A dead-letter queue nobody watches is a slow way to lose data.

Decide about ordering explicitly as well. If an update can overtake a create during a retry, the receiver needs version numbers or timestamps to cope.

Observability that speaks business

A green dashboard of HTTP status codes tells you little. An integration can return 200 all day while silently dropping every order with a particular tax code. Measure things the business would recognise:

  • Reconciliation: orders that left system A compared with orders that arrived in system B, checked on a schedule. This catches what no error log will.
  • Freshness: the age of the oldest unprocessed message, and the time since the last successful sync.
  • Correlation: an ID carried across every hop, so that one transaction can be followed through the logs of all the systems involved.
  • Failure queues: how many items are stuck, and how long they have been there.

An owner

Every integration needs a named team that answers for it end to end, including the failures that turn out to be on the other side. That team maintains the contract, the runbook, the alerting and a list of consumers, and it has a budget for upkeep, because upstream APIs and credentials will keep changing.

If you can't name the owner, you have found your biggest integration risk. Unowned credentials are a common example: an API key that expires, set up by someone who has since left.

What to do this week

Pick the integration that would hurt most if it stopped quietly, and answer these questions:

  1. Who gets alerted if it stops, and how soon would they know?
  2. What happens if the same message arrives twice?
  3. Where do failed messages go, and how many are there today?
  4. When did someone last compare the record counts on each side?
  5. Who owns the credentials, and when do they expire?

Any question you can't answer is your backlog. Start with a daily reconciliation report, which is usually cheap to build and shows quickly whether you have a problem.