Kafka Topic

This page describes the required listener config parameters for setting up a sync with a Kafka stream source.

1. Kafka Topic Listener Config
The following parameters should be included in your listener configuration when setting up a real time sync with a Kafka stream source. This process currently supports JSON string or AVRO serialized format.
To listen in on multiple topics, you will need to configure multiple listener configs.
3.1 Topic Column
Name
Description
"topicName"
Mandatory. This is the Kafka topic name to listen messages on.
"messageFormat"
Optional. Put "AVRO" if your messages are serialized in AVRO​
{
  "topicName": "<(mandatory) kafka topic name to listen messages on>",
  "messageFormat": "<(optional) Put "AVRO" if the messages are serialized in AVRO>"
}
3.2 Connections Attributes Column
"bootstrapServers"
Mandatory. List the Kafka bootstrap servers in a comma-separated list. Should be in the form of host:port
"saslMechanism"
This will be either PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512.

SCRAM-SHA-256 must be formatted as: SCRAMSHA256

SCRAM-SHA-512 must be formatted as: SCRAMSHA512
"saslPassword"
The password for your chosen SASL mechanism
"saslUsername"
The username for your chosen SASL mechanism.
"url"
This is required if your data follows a schema when serialized in AVRO. It is a comma-separated list of URLs for schema registry instances that are used to register or lookup schemas.
"basicAuthCredentialsSource"
Specifies the Kafka configuration property "schema.registry.basic.auth.credentials.source" that provides the basic authentication credentials. This can be "UserInfo" | "SaslInherit"
"basicAuthUserInfo"
Basic Auth credentials specified in the form of username:password
"sslKeystorePassword"
Optional. This is the client keystore (PKCS#12) password.
"securityProtocol"
Kafka supports cluster encryption and authentication, which can encrypt data-in-transit between your applications and Kafka.
​
Use this field to specify which protocol will be used for communication between client and server. Cinchy currently supports the following options: Plaintext, SaslPlaintext, or SaslSsl.

Paintext: Unauthenticated, non-encrypted.
SaslPlaintext: SASL-based authentication, non-encrypted.
SaslSSL: SASL-based authentication, TLS-based encryption.

If no parameter is specified, this will default to Plaintext.
{
  "bootstrapServers": "< (mandatory) kafka bootstrap servers in a comma-separated list in the form of host:port>",
  "saslMechanism": "<PLAIN|SCRAM-SHA-256|SCRAM-SHA-512>",
  "saslPassword": "",
  "saslUsername": "",
  "schemaRegistrySettings": {
    "url": "<(optional) This is required if your data follows a schema when serialized in Avro. A comma-separated list of URLs for schema registry instances that are used to register or lookup schemas. >",
   "basicAuthCredentialsSource": "<(optional) Specifies the Kafka configuration property "schema.registry.basic.auth.credentials.source" that provides the basic authentication credentials, this can be "UserInfo" | "SaslInherit">",
   "basicAuthUserInfo": "<(optional) Basic Auth credentials specified in the form of username:password>",
   "sslKeystorePassword": "<(optional) The client keystore (PKCS#12) password>"
  }
  "securityProtocol": "Plaintext | SaslPlaintext | SaslSsl"
}

Data Polling

MongoDB

Last modified 2mo ago

Name	Description
"topicName"	Mandatory. This is the Kafka topic name to listen messages on.
"messageFormat"	Optional. Put "AVRO" if your messages are serialized in AVRO

"bootstrapServers"	Mandatory. List the Kafka bootstrap servers in a comma-separated list. Should be in the form of host:port
"saslMechanism"	This will be either PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512. SCRAM-SHA-256 must be formatted as: SCRAMSHA256 SCRAM-SHA-512 must be formatted as: SCRAMSHA512
"saslPassword"	The password for your chosen SASL mechanism
"saslUsername"	The username for your chosen SASL mechanism.
"url"	This is required if your data follows a schema when serialized in AVRO. It is a comma-separated list of URLs for schema registry instances that are used to register or lookup schemas.
"basicAuthCredentialsSource"	Specifies the Kafka configuration property "schema.registry.basic.auth.credentials.source" that provides the basic authentication credentials. This can be "UserInfo" \| "SaslInherit"
"basicAuthUserInfo"	Basic Auth credentials specified in the form of username:password
"sslKeystorePassword"	Optional. This is the client keystore (PKCS#12) password.
"securityProtocol"	Kafka supports cluster encryption and authentication, which can encrypt data-in-transit between your applications and Kafka. Use this field to specify which protocol will be used for communication between client and server. Cinchy currently supports the following options: Plaintext, SaslPlaintext, or SaslSsl. Paintext: Unauthenticated, non-encrypted. SaslPlaintext: SASL-based authentication, non-encrypted. SaslSSL: SASL-based authentication, TLS-based encryption. If no parameter is specified, this will default to Plaintext.