Sync data from an API

Step-by-step guide on how to continuously sync data from an API (using a sync template).

Pre-requisite: creation of an integration and at least one connection (step-by-step guide).

Activate a sync template

Nango uses syncs to read data from APIs continuously. For common use cases, templates are available to let you get started fast.

Select your integration in the Integrations tab, and navigate to the Scripts tab. Available sync templates will appear in the Sync Scripts section. Select the relevant one and enable it with the toggle.

Nango will automatically sync the corresponding records in the background for each relevant connection.

Is there no template for your API? Or none matching your exact use case?

Learn more about how to build a custom integration, extend a template or request custom integrations from Nango experts.

Listen for webhooks from Nango

Nango sends webhook notifications to your backend whenever new data is available for a connection & sync combination.

To set this up, go to the Environment Settings tab and configure a Webhook URL to which Nango will send notifications.

The webhook from Nango is a POST request with the following body (reference):

{
    "connectionId": "<string>",
    "providerConfigKey": "<string>",
    "syncName": "<string>",
    "model": "<string>",
    "responseResults": { "<DataModel>": { "added": 123, "updated": 123, "deleted": 123 } },
    "syncType": "INITIAL" | "INCREMENTAL",
    "modifiedAfter": "<timestamp>"
}

Webhooks with non-2xx responses are retried with exponential backoff.

Before using webhooks in production, verify their origin (step-by-step guide).

Fetch the latest data

After receiving a Nango webhook, fetch the latest records using the backend SDK (reference) or API (reference).

Use the modifiedAfter timestamp from the webhook payload as a parameter in your request to fetch only the modified records.

• cURL (standard endpoint)
• Node SDK

curl -G https://api.nango.dev/records \
  --header 'Authorization: Bearer <ENVIRONMENT-SECRET-KEY>' \
  --header 'Provider-Config-Key: <providerConfigKey-in-webhook-payload>' \
  --header 'Connection-Id: <connectionId-in-webhook-payload>' \
  --data-urlencode 'model=<model-in-webhook-payload>' \
  --data-urlencode 'modified_after=<modifiedAfter-in-webhook-payload>' \
  

This returns an array of records conforming to the specified data model.

Each record contains useful metadata automatically generated by Nango:

{
    records:
        [
            {
                id: 123,
                ..., // Fields as specified in the model you queried
                _nango_metadata: {
                    deleted_at: null,
                    last_action: 'ADDED',
                    first_seen_at: '2023-09-18T15:20:35.941305+00:00',
                    last_modified_at: '2023-09-18T15:20:35.941305+00:00',
                    cursor: 'MjAyNC0wMy0wNFQwNjo1OTo1MS40NzE0NDEtMDU6MDB8fDE1Y2NjODA1LTY0ZDUtNDk0MC1hN2UwLTQ1ZmM3MDQ5OTdhMQ=='
                }
            },
            ...
        ],
    next_cursor: "Y3JlYXRlZF9hdF4yMDIzLTExLTE3VDExOjQ3OjE0LjQ0NyswMjowMHxpZF4xYTE2MTYwMS0yMzk5LTQ4MzYtYWFiMi1mNjk1ZWI2YTZhYzI"
}

Cursor-based synchronization

In practice, webhook notifications can be missed, and relying solely on the webhook payload to fetch modified records can cause you to miss some updates.

A more reliable way of keeping track of how far you’ve synced records (for each connection & sync combination) is to rely on record cursors.

Each record comes with a synchronization cursor in _nango_metadata.cursor. Nango uses cursors internally to keep a chronological list of record modifications.

Each time you fetch records, you should store the cursor of the last record you fetched to remember how far you’ve synced (for each connection & sync combination).

The next time you fetch records, pass in the cursor of the last-fetched record to only receive records modified after that record:

• cURL (standard endpoint)
• Node SDK

curl -G https://api.nango.dev/records \
  --header 'Authorization: Bearer <ENVIRONMENT-SECRET-KEY>' \
  --header 'Provider-Config-Key: <providerConfigKey-in-webhook-payload>' \
  --header 'Connection-Id: <connectionId-in-webhook-payload>' \
  --data-urlencode 'model=<model-in-webhook-payload>' \
  --data-urlencode 'cusor=<cursor-of-last-fetched-record>' \
  

So, the overall logic for cursor-based synchronization should be:

1. Receive a webhook notification from Nango
2. Query your database for the cursor of the last-fetched record
3. Fetch the modified records (passing the cursor)
4. Store the modified records
5. Store the last-fetched record cursor

Write back to APIs (2-way syncing)

Write back to APIs with actions (step-by-step guide) or proxy requests (step-by-step guide).

Troubleshoot errors & monitor

Navigate to the Activity tab to inspect potential errors & monitor sync executions.

Questions, problems, feedback? Please reach out in the Slack community.