v5.0 - v5.5 Data Sync Documentation
  • Overview
  • Release Notes
    • V4.5
    • V4.6
    • v4.7
    • v4.8
    • v4.9
    • v4.10
    • v4.12
    • v4.13
  • v4 Connections Installation Guide
    • v5 Connections and CLI Installation Guide
  • v4 Worker/Listener Installation Guide
    • v5 Worker/Listener Installation Guide
  • Upgrades & Config Changes
  • Builder Guide
    • Overview
    • Types of Data Syncs
    • Subscribing to Event Streams
      • Supported Stream Sources
        • Amazon Simple Queue Service (SQS)
        • Cinchy Change Data Capture
        • Data Polling
        • Kafka Topic
        • MongoDB
        • Salesforce
          • Push Topic
          • Platform Event
    • Configuring a Data Sync
      • Connections Experience & XML Config Reference
        • Info
          • Parameters
            • String Escape
        • Source Config Reference
          • Schema Columns
            • Calculated Column Examples
          • Auth Requests
          • Request Headers
          • Pagination
          • Source Filter
        • Target Destination Config Reference
          • Column Mappings
          • Target Destination Filter
        • Sync Behaviour
        • Post Sync
        • Permissions
        • Jobs
        • Connections Usage Example
      • Data Validation
      • Transformations
        • StringReplacement
      • Supported Data Sources
        • File Based Source
          • Binary File
            • Binary File Data Source Example
          • Delimited File
            • Delimited File Data Source XML Example
          • Fixed Width File
          • Excel
            • Excel Source XML Example
        • Cinchy Event Broker/CDC
          • Cinchy Event Broker/CDC XML Config Example
        • Cinchy Table
          • Cinchy Table XML Config Example
        • Cinchy Query
          • Cinchy Query Data Source Example
        • Copper
        • DB2
        • Dynamics 2015
        • Dynamics
        • DynamoDB
        • Kafka Topic
          • Apache AVRO Data Format
          • Kafka Topic Source Example
        • LDAP
        • MongoDB Collection
          • MongoDB Collection Source XML Example
        • MongoDB Collection (Cinchy Event Triggered)
        • MS SQL Server Query
        • MS SQL Server Table
        • ODBC Table
        • ODBC Query
        • Oracle Table
        • Oracle Query
        • Parquet
        • Polling Event
          • Polling Event Example
        • REST API (Cinchy Event Triggered)
        • REST API
          • REST API XML Example
        • SAP SuccessFactors
        • Salesforce Object (Bulk API)
        • Salesforce Platform Event
        • Salesforce Push Topic
        • Snowflake
          • Snowflake Source XML Example
        • SOAP 1.2 Web Service
      • Supported Sync Targets
        • Cinchy Table
        • DB2 Table
        • Dynamics
        • Kafka Topic
        • MongoDB Collection (Column Based)
        • MS SQL Server Table
        • Oracle Table
        • REST API
        • Salesforce
        • Salesforce Object
        • Snowflake Table
          • Snowflake Table Target XML Example
        • SOAP 1.2 Web Service
    • Common Design Patterns
    • Testing a Data Sync
    • Promoting a Data Sync
    • Scheduling a Data Sync
    • CLI Command List
    • Connections Functions
    • Monitoring
  • Cinchy Platform Documentation
Powered by GitBook
On this page
  • 1. Overview
  • 2. Full File Sync
  • 2.1 Sync Key
  • 2.2 Sync Record Behaviour
  • Delta Sync

Was this helpful?

  1. Builder Guide
  2. Configuring a Data Sync
  3. Connections Experience & XML Config Reference

Sync Behaviour

PreviousTarget Destination FilterNextPost Sync

Last updated 2 years ago

Was this helpful?

1. Overview

When configuring a data sync you must set your sync behaviour. You have two options for this: Full File or Delta.

Full File syncs intake both the source and the destination data and reconcile the records by matching up the sync key. This determines any differences and allows it to perform updates, inserts, ignores, or deletes at the destination.

Delta syncs skip the reconciliation process. In batch syncs, it simply grabs records from the source and inserts it into the destination. In real-time syncs, it may act differently depending on the event type. For example, when using the Cinchy Event Broker/CDC with an insert event, a delta sync will insert the data into the destination, an update event will update, etc.

Delta syncs also have the option to provide an "Action Type Column" for REST API destinations. This reads the value of the source record from a specified column. If the value is "INSERT", then it inserts the record, "UPDATE", then it updates, "DELETE", then it deletes

2. Full File Sync

When using the Full File synchronization pattern there are two distinct sections that must be configured: the Sync Key and the Sync Record Behaviour (Image 1).

2.1 Sync Key

The Sync Key is used as a unique key reference when syncing the data from the data source into your destination. It is used to match up the data between the source and the target, which allows for updates to occur on changed records.

To set this using a config XML, use the following guide:

  • Attributes: name. The name of a column in the destination that you are syncing data into.

<SyncKeyColumnReference
    name:"string">
</SyncKeyColumnReference>

2.2 Sync Record Behaviour

The Sync Record Behaviour is broken down into three subsections which define what action will be taken on certain records.

Values in the attributes section of the config XML for record behaviour are case sensitive.

2.2.1 New Record Behaviour

New Record Behaviour defines what action is taken when a new record is found in the sync source. This can be either Insert or Ignore.

To set this using a config XML, use the following guide:

Attribute
Description
Values

type

The type defines the action upon the new record.

It can either be "INSERT" or "IGNORE".

  • "INSERT" will insert the new record.

  • "IGNORE" will do nothing to the record.

<NewRecordBehaviour type="INSERT" />

2.2.2 Dropped Record Behaviour

Dropped Record Behaviour defines what action is taken when a record is not found in the sync source, but exists in the target. This can be either Delete, Ignore, or Expire.

To set this using a config XML, use the following guide:

Attribute
Description
Values

type

The type defines the action upon the dropped record.

It can either be "IGNORE", "EXPIRE", or "DELETE".

  • "IGNORE" will do nothing to the record.

  • "EXPIRE" will populate a specified expiration timestamp field as the current time. AN expirationTimestampField must be provided. This is a reference to a date/time column in the target that should be updated with the execution timestamp if the record is dropped.

  • "DELETE" will delete dropped records in the target data set.

expirationTimestampField

This attribute is only applicable if the type is equal to "EXPIRE".

The expirationTimestampField is the name of an existing date field to be filled with the current time.

<DroppedRecordBehaviour
    type="EXPIRE"
    expirationTimestampField="string">
    ...
</DroppedRecordBehaviour>

2.2.3 Changed Record Behaviour

Changed Record Behaviour defines what action is taken when a record with a sync key is found in the source and also exists in the target. This can be either Update or Ignore.

To set this using a config XML, use the following guide:

Attribute
Description
Value

type

The type defines the action upon the new record.

It can either be "UPDATE" or "IGNORE".

  • "IGNORE" will do nothing to the record.

  • "UPDATE" will update the record.

<ChangedRecordBehaviour type="UPDATE" />

Delta Sync

When using the Delta synchronization pattern there is one optional configuration that you can choose to provide when running a sync with a REST API destination (Image 2).

The Action Type Column reads the value of the source record from a specified column. If the value is "INSERT", then it inserts the record, "UPDATE", then it updates, "DELETE", then it deletes.

Elements:

This is used in the element when specifying which columns in the Target Table to be utilized as a unique key for the syncing process.

Contained-In:

<SyncKeyColumnReference>
<SyncKey>
<SyncKey>
Image 1: Full File Syncs
Image 2: Delta Syncs