Syncs

Learn more about syncs.

Overview

Syncs in Nango let you continuously synchronize data from external APIs such as Hubspot contacts, Slack messages, etc.

Syncs replicate external data into Nango, according to a predefined schedule. You can subsequently access the data through a unified API to replicate it into your database. Attached to integrations, syncs apply universally across all connections within that integration, ensuring consistent data replication.

Sync records & models

Syncs are designed to replicate data in a structured manner, transforming it into a list of records that adhere to a predefined model. This model, detailed within the sync configuration, outlines the fields (and their data types) of the output records.

Scheduling and execution

The execution of syncs is based on a schedule you configure, which can be as frequent as every 5 minutes, depending on your plan. Each execution updates the dataset with the latest data from the external API for every connection under the integration. You can start syncs for individual connections or set them to begin for all connections automatically (reference).

Sync schedules & executions can be actively managed per connection through the Nango UI or API, including starting/pausing schedules & canceling/triggering executions. Incremental syncs let you initiate a full refresh, effectively resetting the synced dataset.

Sync endpoints and data caching

Enabled syncs provide access to an endpoint for fetching all synced records, stored within a Nango cache. This highly-available caching supports essential syncing mechanisms like data deduplication, change detection, and heavy replications.

Sync endpoints have an automatically-generated API reference in the Nango UI.

Sync modes: full refresh vs. incremental

Full refresh syncs

These syncs retrieve the entire dataset during each execution, replacing previously synced data. Suitable for smaller datasets due to performance constraints.

Incremental syncs

These only fetch data that has changed since the last execution, appending it to the existing dataset. This mode is more efficient for larger datasets and relies on the external API’s support for querying modified records.

Nango leverages the timestamp of the last sync as a cursor for incremental syncs, allowing for precise data retrieval based on modifications since the previous sync execution.

In the case of incremental syncing, the first sync execution is heavier because it fetches all of the historical data (how far back it should go depends on your script logic). This is called an initial sync execution in Nango. Subsequent sync executions should be lighter, particularly if your syncing frequency is high.

Sync webooks

Nango sends webhooks upon the completion of sync executions that result in new data. These webhooks allow you to promptly fetch the new records from the sync endpoint.

This streamlines the process of reliably replicating data from external APIs to your database, eliminating the need for cron jobs and enabling immediate updates with relevant data as soon as it becomes available.

Detecting created, updated and deleted records

Nango syncs can identify created, updated, and deleted records, though detecting deletions requires additional configuration or coding, varying by sync mode.

Detecting deletions: incremental vs. full refresh syncs

For full refresh syncs

Deletion detection is straightforward—enable it within the integration configuration. As full refresh syncs replace the dataset entirely with each execution, Nango can automatically identify and flag records that no longer exist in the new dataset.

For incremental syncs

This method requires a different approach, as the data is not entirely replaced. The sync script must be adjusted to specifically request information on deleted data from the external API and mark the relevant records as deleted. This capability depends on the external API’s ability to return information about deletions. Nango’s integration templates include examples of scripts designed to handle this scenario.

Error handling

Failed sync attempts don’t automatically retry but will proceed on the next scheduled execution. You have the option to manually trigger a new execution.

Sync scripts

Sync executions are driven by integration scripts, which encapsulate the logic for interacting with external APIs, allowing your application to rely on the Nango unified API. Nango provides script templates for standard use cases, but also supports custom scripting for tailored integration needs.

Sync scripts have the following structure:

import type { NangoSync, ResponseModel } from './models';

export default async function fetchData(nango: NangoSync): Promise<void> {
    const records: ResponseModel[] = [];

    // Your integration code goes here.

    await nango.batchSave(records, 'ResponseModel');
}

The fetchData function starts on the schedule indicated in your integration configuration. The ResponseModel is configurable via the integration configuration and is auto-imported into your script.

Unlike actions, the syncs default function doesn’t take any input parameters or return anything. Sync inputs are passed via the connection metadata. And instead of returning data directly like actions, syncs persist records to Nango’s cache, calling nango.batchSave(), nango.batchUpdate(), and nango.batchDelete(). Your application later fetches the records from the cache.

API unification

Syncs can have records that conform to a unified model, regardless of the specific external API they originate from. This unification simplifies your integration code, enabling you to handle data from various sources in a consistent manner, thus reducing complexity and the potential for errors.

Getting started with syncs

Check out the syncs step-by-step guide or refer to the reference (API / SDK).

Questions, problems, feedback? Please reach out in the Slack community.