Skip to main content

Sync API

Authentication​

All API calls, except schema queries, must be authenticated with an API key by Authorization HTTP header:

Authorization: Bearer <api-key>

API keys can be generated in Jitsu UI, on user settings page

Screenshot
tip

API key has a format of {keyId}:{keySecret}. Jitsu never stores key secret, so make sure to copy it right after key creation.

Setting up Sync with Management API​

To setup Sync via Management API you need to know docker image for the source connector you want to use.

Docker image can be found in Service Connector catalog in Jitsu UI

Connector config schema​

This step can be skipped if you already familiar with source connector configuration schema.

GET https://your.jitsu.domain/api/[workspaceId]/sources/spec?package=[package]&version=[version]

When this endpoint is called for the first time for combination of package and version it triggers asynchronous job of loading config specs and returns:

Pending:

{
  "ok": false,
  "pending": true
}

You need to repeat this call until you get final status:

Success:

{
  "ok": true,
  "specs": {
    //...
    "connectionSpecification": {
      // Example of Firebase source config schema
      "type": "object",
      "title": "Firebase",
      "required": [
        "projectId",
        "serviceAccountKey"
      ],
      "properties": {
        "projectId": {
          "type": "string",
          "description": "Firebase Project ID from the Project Settings page"
        },
        "serviceAccountKey": {
          "type": "string",
          "description": "Auth (Service account key JSON)",
          "airbyte_secret": true
        }
      },
      "description": "Firebase (Firestore and User) Source connector"
    }
  }
}

connectionSpecification object contains JSON schema of source connector configuration

Error:

{
  "ok": false,
  "error": "Failed to load specs"
}
Error retry

Final statuses are cached and will be returned immediately on consequent runs.

To make an attempt to retry in case of error add &force=true parameter to the endpoint URL.

&force=true parameter must be omitted in consequent runs

Adding new Service Connector​

POST https://your.jitsu.domain/api/[workspaceId]/config/service

Please check for the base JSON schema of Service Connector object:

https://use.jitsu.com/api/schema/service

Source connector configuration that follows connectionSpecification JSON schema from previous paragraph must be provided in credentials object.

Example​

Request payload new_firebase.json

{
  "id": "fb1",
  "workspaceId": "[workspaceId]",
  "type": "service",
  "name": "New Firebase Service",
  "package": "jitsucom/source-firebase",
  "version": "0.0.2",
  "credentials": {
    "projectId": "[firebaseProject]",
    "serviceAccountKey": "[serviceAccountJson]"
  }
}

Curl command:

curl -X POST 'https://your.jitsu.domain/api/[workspaceId]/config/service' \
 -H 'Authorization: Bearer <api-key>' \
 -H 'Content-Type: application/json'  \
 -d @new_firebase.json

Result:

{
  "id": "fb1"
}

Modifying Service Connector​

To update Service Connector settings use the following endpoint:

PUT https://your.jitsu.domain/api/[workspaceId]/config/service/[serviceId]

Obtaining Streams Catalog​

Before configuring Sync object you need to know what streams are available in configured service connector.

GET https://your.jitsu.domain/api/[workspaceId]/sources/discover?serviceId=[serviceId]

When this endpoint is called for the first time for Service Connector or after changes in Service Connector configuration it triggers asynchronous job of loading catalog and returns:

Pending:

{
  "ok": false,
  "pending": true
}

You need to repeat this call until you get final status:

Success:

{
  "ok": true,
  "catalog": {
    "streams": [
      {
        "name": "users",
        "namespace": "",
        "json_schema": {
          "properties": null
        },
        "supported_sync_modes": [
          "full_refresh"
        ],
        "source_defined_primary_key": [
          [
            "uid"
          ]
        ]
      }
    ]
  }
}

Error:

{
  "ok": false,
  "error": "Failed to load catalog"
}

Refreshing Streams Catalog​

It may be necessary to refresh Streams Catalog due to errors, Service Connector config changes or changes in source data itself

To refresh add &refresh=true parameter to the endpoint URL.

&refresh=true parameter must be omitted in consequent runs.

Adding Sync object​

POST https://your.jitsu.domain/api/[workspaceId]/config/link

Please refer to JSON schema of Sync object:

https://use.jitsu.com/api/schema/link/sync

Example​

Request payload new_sync.json

{
  "id": "[syncId]", // only for modifying existing sync 
  "fromId": "[serviceId]",
  "toId": "[destinationId]",
  "type": "sync",
  "data": {
    "streams": {
      "users": {
        "sync_mode": "full_refresh",
        "table_name": "fb_users"
      }
    }
  }
}

Curl command:

curl  -X POST 'https://your.jitsu.domain/api/[workspaceId]/config/link' \
 -H 'Authorization: Bearer <api-key>' \
 -H 'Content-Type: application/json'  \
 -d @new_sync.json

Result:

{
  "id": "[syncId]",
  "created": true
}

Modifying Sync object​

To update a Sync object use the same endpoint as to add one:

POST https://your.jitsu.domain/api/[workspaceId]/config/link

Running Sync​

Endpoint​

GET https://your.jitsu.domain/api/[workspaceId]/sources/run?syncId=[syncId]

  • syncId - id of Sync object
  • fullSync - optional, boolean, if true - saved state will be deleted and full sync will be performed replacing all source data in destination

Example​

curl -H "Authorization: Bearer <api-key>" \
"https://your.jitsu.domain/api/[workspaceId]/sources/run?syncId=[syncId]"

Response​

Successful response:​
{
  "ok": true,
  "taskId": "[taskId]",
  "status": "https://your.jitsu.domain/api/[workspaceId]/sources/tasks?taskId=[taskId]&syncId=[syncId]",
  "logs": "https://your.jitsu.domain/api/[workspaceId]/sources/logs?taskId=[taskId]&syncId=[syncId]"
}

* You can use status and logs links to check sync status and logs.

Error response:​
{
  "ok": false,
  "error": "Sync is already running",
  "runningTask": {
    "taskId":"[taskId]",
    "status":"https://your.jitsu.domain/api/[workspaceId]/sources/tasks?taskId=[taskId]&syncId=[syncId]",
    "logs":"https://your.jitsu.domain/api/[workspaceId]/sources/logs?taskId=[taskId]&syncId=[syncId]"
  }
}

Sync Status​

Endpoint​

GET https://your.jitsu.domain/api/[workspaceId]/sources/tasks?taskId=[taskId]&syncId=[syncId]

  • syncId - id of Sync object
  • taskId - id of task returned in response of Trigger sync endpoint

Example​

curl -H "Authorization: Bearer <api-key>" \
"https://your.jitsu.domain/api/[workspaceId]/sources/tasks?taskId=[taskId]&syncId=[syncId]"

Response​

Successful response:​
{
  "ok": true,
  "task": {
    "sync_id": "[syncId]",
    "task_id": "[taskId]",
    "package": "jitsucom/source-firebase",
    "version": "0.0.2",
    "started_at": "2023-07-07T08:20:59.000Z",
    "updated_at": "2023-07-07T08:21:07.706Z",
    "status": "RUNNING",
    "description": "CREATED: "
  },
  "logs": "https://your.jitsu.domain/api/[workspaceId]/sources/logs?taskId=[taskId]&syncId=[syncId]"
}

* You can use logs link to check sync logs.

Error response:​
{
  "ok": false,
  "error": "Task [taskId] not found"
}

Sync Logs​

Endpoint​

GET https://your.jitsu.domain/api/[workspaceId]/sources/logs?taskId=[taskId]&syncId=[syncId]

  • syncId - id of Sync object
  • taskId - id of task returned in response of Trigger sync endpoint

Example​

curl -H "Authorization: Bearer <api-key>" \
"https://your.jitsu.domain/api/[workspaceId]/sources/logs?taskId=[taskId]&syncId=[syncId]"

Response​

Successful response:​
2023-07-07 08:21:05.737 INFO [jitsu] Sidecar. syncId: [syncId], taskId: [taskId], package: jitsucom/source-firebase:0.0.2 startedAt: 2023-07-07T12:20:59+04:00
2023-07-07 08:21:05.832 INFO [jitsu] Catalog loaded. 36 streams selected
2023-07-07 08:21:05.833 INFO [jitsu] State loaded: {}
2023-07-07 08:21:06.573 INFO [jitsucom/source-firebase] Starting syncing...
Error response:​
Error loading logs for task id [taskId]...