# Add a new business tool or SaaS
There's a process in place that allows app, business tool or SaaS owners to add their service to the list of supported ones. This is the same process we use to expand our existing library of pre-built connectors. In this developer-oriented section, we'll explain how it's done in SyncPenguin.
After the service is added and synchronizations implemented, the SaaS customers can take advantage of reliable two-way synchronization by directly connecting in SyncPenguin dashboard. We also offer a possibility of native integration with SyncPenguin to provide white label integrations in your app (more information here).
It takes a simple and lighweight NodeJS module to be able to use a new service in SyncPenguin. This module is called a data source module. More on the data sources below.
# Data sources
Data source represents a particular data entity provided by any service, business tool or SaaS, and accessible through any kind of API. Examples of data sources: Gmail emails, Google Calendar events, Exchange contacts, Exchange tasks, Salesforce products, Freshsales deals, etc.
SyncPenguin provides a wide range of public (pre-built) data sources you can synchronize your data with. Those include popular mailbox services (Gmail, Exchange, Outlook Office 365, etc.), ERP systems (Salesforce, QuickBooks), CRM systems (Freshsales, EspoCRM, Dynamics 365, etc.), eCommerce platforms (BigCommerce, WooCommerce, Shopify, Shopware, etc.) and many others. You are able to set up synchronization between any of the public sources and your own application or connect two suitable public sources.
For SyncPenguin to be able to communicate with your service, you have to implement a data source module (also called data source app).
# Contact us
Feel free to contact us in case of any questions, or if you want us to build your custom sync according to your requirements for free.
# Data source module
Data source module is a small and lightweight NodeJS module that SyncPenguin uses to access data sources. These modules implement a specific interface and provide basic CRUD (Create, Read, Update and Delete) operations that the platforms uses to synchronize data. The interface looks like this:
module.exports = {
async init(account) {}, // initialization method
async get(id) {}, // action methods
async insert(value) {},
async patch(id, value) {},
async delete(id) {},
async getUpdates(lastSync) {}
}
In most cases, this module is a simple wrapper of HTTP-based REST API. You can use any third-party NodeJS modules that allow simple and reliable implementation of API requests.
SyncPenguin provides simple and intuitive CLI and graphic interface that allows you to implement, debug and test you module.
# Create a data source module
The most convenient way of creating a data source module is through SyncPenguin CLI.
To install the CLI run the following command:
npm i syncpeng -g
# Initialize a folder
Create an empty module folder and run the following command:
syncpeng init -d
The folder is now filled with the following files:
dataSourceApp.js # the module implementation
in.json # an input data file used for running and testing
account.json # a file for access credentials and other sync settings
# Module interface
An empty module interface in dataSourceApp.js looks like this:
module.exports = {
async init(account) {}, // initialization method
async get(id) {}, // action methods
async insert(value) {},
async patch(id, value) {},
async delete(id) {},
async getUpdates(lastSync) {}
}
Upon running, each of the action methods (all except init) should performe the corresponding operation in the data source, and return response (if any).
More details on each method later in the documentation.
# Run and test
To run the module, use the following command:
syncpeng <command>
Where command corresponds to module action methods and can be one of the following: get, insert, patch, delete, getupdates.
To provide an input data to the method you run, use the in.json file.
For instance, to run patch command with certain id and value parameters, specify the following json in the file:
{
"patch": {
"id": "f0c97e9a-d135-46d5-933e-818871d7f850",
"value" {
"subject": "Hello world!",
"count": 7
...
}
}
}
Note that the highest level of in.json contains method name (e. g. "patch": { ... }), as for convenience you can store input for multiple methods at the same time.
The module command response (if any) is returned in the console output.
To redirect the output to a file, use -o <filename> option in the CLI command.
Any errors or warnings are logged to the console output as well.
# INIT method
init method is called one time in an execution scope of the module.
This means that there will be one call to the method, followed by one of multiple calls to actions methods.
It's only parametrs is account object which holds some synchronization metadata along with credentails and other user/system/account ralated information based on what data source API consumes.
It can contain an API key, username/password combination, an access token, etc.
This is how it looks in the sample module:
...
async init(account) {
this.httpClient.setHeader('Authorization', 'Bearer ' + account.apiToken)
}
...
It simply sets API token that is then passed as header to API calls.
# Run
Set the account info in in.json file:
{
...
"init": {
}
...
}
And run init command using SyncPenguin CLI:
syncpeng init
This method has no output.
# GET method
get method is the simplest method in the whole data source module interface.
Its sole responsibility is to fetch a source record (e. g. meeting from Google Calendar) by specified ID and return it as JSON.
Here's how the method is implemented in the sample module:
...
async get(id) {
return await this.httpClient.get(`/cats/${id}`)
}
...
It calls our sample REST API /cats endpoint and returns the record data response as JSON.
# Run
Set the record ID in the in.json file:
{
...
"get": {
"id": "10724907-365c-4c71-8f72-0222f5c435f3"
}
...
}
And run get command using SyncPenguin CLI:
syncpeng get
The response is logged to console output, e. g.:
{
"name": "Simba",
"sex": "female",
"age": 2,
"fluffiness": 100,
"toys": [
{
"name": "ball"
}
]
...
}
# INSERT method
insert method is responsible for creating a data source record with the specified data (e. g. creating meeting in Google Calendar).
Apart from that, the method has to return the ID of the created record.
Here's how the method is implemented in the sample module:
...
async insert(value) {
var cat = await this.httpClient.post(`/cats`, value);
return {
id: cat.id
}
}
...
It calls our sample REST API /cats POST endpoint with the cat data and returns the ID of newly created record.
# Run
Set the record-to-create data in the in.json file:
{
...
"insert": {
"value": {
"name": "Oliver",
"age": 4,
"fluffiness": 47
}
}
...
}
And run insert command using SyncPenguin CLI:
syncpeng insert
The response is logged to console output, e. g.:
{
"id": "2501edde-7415-42b0-ac46-e1f713ab4146"
}
# PATCH method
patch method is responsible for patching data source record with provided data.
Patching means that the field that are provided in the input should be updated, while those that are not should remain the same.
Usually API's provide this type of method. If it's not the case in your situation, you can always do two API calls: one to get the full entity record, then patch only necessary feilds, and finally save the full record. But of course, we recommend minimizing the number of requests for performance purposes.
Here's how the method is implemented in the sample module:
...
async patch(id, value) {
await this.httpClient.patch(`/cats/${id}`, value);
}
...
It calls our sample REST API /cats PATCH endpoint with the cat data and returns the ID of newly created record.
# Run
Set the record-to-patch data in the in.json file:
{
...
"patch": {
"id": "5d3f155a-d7d6-417b-9172-eefc1a3c7d04",
"value": {
"name": "Oliver",
"age": 4
}
}
...
}
And run patch command using SyncPenguin CLI:
syncpeng patch
# DELETE method
delete method is quite simple - it's only job is to remove data source record by the given ID.
Here's how the method is implemented in the sample module:
...
async delete(id) {
await this.httpClient.delete(`/cats/${id}`)
}
...
It calls our sample REST API /cats DELETE endpoint and returns the record data response as JSON.
# Run
Set the record ID in the in.json file:
{
...
"delete": {
"id": "10724907-365c-4c71-8f72-0222f5c435f3"
}
...
}
And run delete command using SyncPenguin CLI:
syncpeng delete
# GETUPDATES method
getUpdates method is the one that is responsible for fetching data source changes that happened since the last sync.
It's only parameter is named lastSync, which can be a timestamp or any other kind of sync token.
It's up to you to decide what's the best value or format suites your API.
Usually lastSync is simply the timestamp of the last time sync happened.
In this case getUpdates should perform an API query based on a LastModified time field or similar to get the recent changes.
Some (usually more sophisticated) APIs support incremental sync. This means that they have an endpoint where you can specify a sync token, and the API returns all changes since the time this token was issued. As response they issue a new token that can be used the next time. For example, Google Calendar APIs support these kind of methods.
The getUpdates response format should look like this:
{
"nextSync": "2019-04-22T14:01:54Z",
"entires": [
{
"id": "6e536457-9028-4ae4-91ce-2019892cf5bf",
"type": "upsert",
"value": {
...
}
},
{
"id": "f763de89-9e91-42da-8e93-1ae670efb2e4",
"type": "delete",
"value": {
...
}
}
]
}
Here nextSync indicates the value of lastSync parameter that will be used the next time the sync is performed.
The type value indicates the type of change that happened, which can be upsert, insert, update or delete.
Here's how the method is implemented in the sample module:
...
async getUpdates(lastSync) {
var nextSync = new Date().toIsoString();
var response = await this.httpClient.get(`/cats?q=lastModified ge ${lastSync}`);
return {
nextSync: nextSync,
entires: response.map(v => {
return {
id: v.id,
type: 'upsert',
value: v
}
});
}
}
...
# Run
Set the record ID in the in.json file:
{
...
"getupdates": {
"lastSync": "2019-04-21T14:01:54Z"
}
...
}
And run getupdates command using SyncPenguin CLI:
syncpeng getupdates
