v5.0 - v5.5 Data Sync Documentation
  • Overview
  • Release Notes
    • V4.5
    • V4.6
    • v4.7
    • v4.8
    • v4.9
    • v4.10
    • v4.12
    • v4.13
  • v4 Connections Installation Guide
    • v5 Connections and CLI Installation Guide
  • v4 Worker/Listener Installation Guide
    • v5 Worker/Listener Installation Guide
  • Upgrades & Config Changes
  • Builder Guide
    • Overview
    • Types of Data Syncs
    • Subscribing to Event Streams
      • Supported Stream Sources
        • Amazon Simple Queue Service (SQS)
        • Cinchy Change Data Capture
        • Data Polling
        • Kafka Topic
        • MongoDB
        • Salesforce
          • Push Topic
          • Platform Event
    • Configuring a Data Sync
      • Connections Experience & XML Config Reference
        • Info
          • Parameters
            • String Escape
        • Source Config Reference
          • Schema Columns
            • Calculated Column Examples
          • Auth Requests
          • Request Headers
          • Pagination
          • Source Filter
        • Target Destination Config Reference
          • Column Mappings
          • Target Destination Filter
        • Sync Behaviour
        • Post Sync
        • Permissions
        • Jobs
        • Connections Usage Example
      • Data Validation
      • Transformations
        • StringReplacement
      • Supported Data Sources
        • File Based Source
          • Binary File
            • Binary File Data Source Example
          • Delimited File
            • Delimited File Data Source XML Example
          • Fixed Width File
          • Excel
            • Excel Source XML Example
        • Cinchy Event Broker/CDC
          • Cinchy Event Broker/CDC XML Config Example
        • Cinchy Table
          • Cinchy Table XML Config Example
        • Cinchy Query
          • Cinchy Query Data Source Example
        • Copper
        • DB2
        • Dynamics 2015
        • Dynamics
        • DynamoDB
        • Kafka Topic
          • Apache AVRO Data Format
          • Kafka Topic Source Example
        • LDAP
        • MongoDB Collection
          • MongoDB Collection Source XML Example
        • MongoDB Collection (Cinchy Event Triggered)
        • MS SQL Server Query
        • MS SQL Server Table
        • ODBC Table
        • ODBC Query
        • Oracle Table
        • Oracle Query
        • Parquet
        • Polling Event
          • Polling Event Example
        • REST API (Cinchy Event Triggered)
        • REST API
          • REST API XML Example
        • SAP SuccessFactors
        • Salesforce Object (Bulk API)
        • Salesforce Platform Event
        • Salesforce Push Topic
        • Snowflake
          • Snowflake Source XML Example
        • SOAP 1.2 Web Service
      • Supported Sync Targets
        • Cinchy Table
        • DB2 Table
        • Dynamics
        • Kafka Topic
        • MongoDB Collection (Column Based)
        • MS SQL Server Table
        • Oracle Table
        • REST API
        • Salesforce
        • Salesforce Object
        • Snowflake Table
          • Snowflake Table Target XML Example
        • SOAP 1.2 Web Service
    • Common Design Patterns
    • Testing a Data Sync
    • Promoting a Data Sync
    • Scheduling a Data Sync
    • CLI Command List
    • Connections Functions
    • Monitoring
  • Cinchy Platform Documentation
Powered by GitBook
On this page

Was this helpful?

  1. Builder Guide
  2. Configuring a Data Sync
  3. Connections Experience & XML Config Reference
  4. Source Config Reference

Pagination

PreviousRequest HeadersNextSource Filter

Last updated 2 years ago

Was this helpful?

1. Pagination

When syncing a Data Source, you may have the option to add in additional configuration sections, such as an Pagination, under the "Add a Section" drop down tab in the Connection Experience (Image 1).

There are two types of pagination available (Image 2):

  • Cursor: The cursor is considered to be a key parameter in this type of pagination. You receive a variable named Cursor along with the response. It is a pointer that points at a particular item that needs to be sent with a request. The server then uses the cursor to seek the other set of items. Cursor-based pagination is more complex and is preferred when dealing with a real-time data set.

  • Offset: Offset-based pagination is for parameters with a specific limit (the number of results) and offset (the number of records that need to be skipped). Offset-based pagination is easy to use and is preferred for static data.

Cursor Pagination

To set up cursor pagination, fill in the following parameters (Image 3):

  • Type: Select Cursor

  • Next Page URL JSON Path: This is the JSON Path within the response to the URL for the next page

  • Cursor Key: This is the key used in the query string to specify the cursor value. This is only required if the cursor returned is not a fully qualified URL.

Offset Pagination

To set up offset pagination, fill in the following parameters (Image 4):

  • Type: Select Offset

  • Limit Key: The key used in the query string to specify the limit

  • Limit: The desired page size

  • Offset By: The type of offset that the API does pagination by. This will be either Record Number or Page Number.

  • Offset Key: The key used in the query string to specify the offset

  • Initial Offset: The starting offset

Note that a pagination block is mandatory even if the API does not return results from multiple pages. You can use the following as the placeholder:

<Pagination type="OFFSET" limitField="" offsetField="" limit="0" initialOffset="0" />

<RestApiDataSource
    method="GET or POST"
    format="JSON"
    rootPath="$.result"
    endpoint="https://data.com/api/v1/records/@recordId/details?apiKey=@apiKey">
	<!-- optional -->
  <RequestHeaders>
		<Header name="" isEncrypted="true or false">value goes here</Header>
	</RequestHeaders>
    <!-- optional -->
	<Body>body goes here, only relevant for POST operations</Body>
	<!-- one of the following pagination blocks is required. If there's no pagination you still need a placeholder -->
	<Pagination type="OFFSET" 
	    limitField="name of the field in the querystring to use when passing the limit" 
	    offsetField="name of the field in the querystring to use when passing the offset"
	    limit="20"
	    initialOffset="0">
	<Pagination type="CURSOR"
	    pathInResponse="JSON path to the field in the response containing the next cursor value"
	    cursorPropertyName="name of the field in the querystring to use when passing the cursor">
  <Schema>
			<Column name="$.id" dataType="Text" />
			<Column name="$.variants.field1" dataType="Text" />
            <!-- you need to use the "item" placeholder if you're looking for fields in an array -->
			<Column name="$.colors.item[1]" dataType="Text" />
      <Column name="$.colors.item[2]" dataType="Text" />
	</Schema>
</RestApiDataSource>
Image 1: Adding a Pagination Block
Image 2: Selecting your type of pagination
Image 3: Cursor Pagination
Image 4: Offset Pagination