Sync customization

We take a code-first approach when it comes to sync development and customization. We think that it's much more efficient and allows a higher level of customization as opposed to point-click setup. Note that it has nothing to do with the way customers consume synchronizations: there is no need for any knowledge of coding from the consumer side.

We define a basic framework for the sync process that operates independent and lightweight field mapping and data fetching modules. With the high separation of concerns in mind, SyncPenguin is a perfect solution for a synchronization setup that is clear and easy to debug.

It only takes a field mapping definition to set up a data synchronization process between any two existing data sources. We use mapper apps for this, which act as connectors between two business tools.

Mapper is an extremely simple and lightweight NodeJS module with the following interface:

module.exports = {
  async mapAtoB(value) {}        // map from one data source to another
  async mapBtoA(value) {}        // and vice versa
  async equal(valueA, valueB) {} // are two data records considered equal?
}

After uploading a mapper to SyncPenguin, you are ready to start your synchronization!

As we use NodeJS for mapping (and other) modules, you have the full benefit of using any NPM modules that help you with things like string formatting, time conversions, etc. All values provided to mapper are in a well-defined JSON format, regardless of the data source used.

For most pairings of existing data sources, SyncPenguin provides its own mapper implementation, which you can easily customize in a way that suits your business.

More details on mappers below.

Contact us

Feel free to contact us in case of any questions, or if you want us to build your custom sync according to your requirements for free.

Contact us

Mapper module

SyncPenguin allows users to define field mapping as a small and lightweight NodeJS module. This is very convenient as JavaScript (or TypeScript) is a very popular, easy to learn and simple to use programming language. At the same time, you can take advantage of the huge NPM modules library, which has many already implemented utilities that help you define your field mapping.

A mapper module has the following interface:

module.exports = {
  async mapAtoB(value) {}          // map from one data source to another
  async mapBtoA(value) {}          // and vice versa
  async equal(valueA, valueB) {}   // are two data records considered equal?
}

The mapAtoB and mapBtoA methods are responsible for field mapping when the data is synced one way and the other. A and B denote two data sources of the synchronization.

Method equal is responsible for determining whether two records from both data sources are considered the same or not. This is important in cases when only part of source entity data is synchronized, instead of full mapping of each field.

Let's have a look at a sample mapper module for a clearer picture.

Create a mapper module

The most convenient way of creating a mapper module is through SyncPenguin CLI.

To install the CLI run the following command:

npm i syncpeng -g

Initialize a folder

Create an empty module folder and run the following command:

syncpeng init -m

The folder is now filled with the following files:

mapperApp.js       # the module implementation
in.json            # an input data file used for running and testing

Module interface

As we mentioned before, an empty module interface in mapperApp.js looks like this:

module.exports = {
  async mapAtoB(value) {}          // map from one data source to another
  async mapBtoA(value) {}          // and vice versa
  async equal(valueA, valueB) {}   // are two data records considered equal?
}

More details on each method later in the documentation.

Run and test

To run the module, use the following command:

syncpeng <command>

Where command corresponds to module methods and can be one of the following: ab, ba, equal.

To provide an input data to the method you run, use the in.json file. For instance, to run ab command with certain input value specify the following json in the file:

{
  "ab": {
    "value": {
      "subject": "Hello world!",
      "count": 7
      ...
    }
  }
}

Note that the highest level of in.json contains method name (e. g. "ab": { ... }), as for convenience you can store input for multiple methods at the same time.

The module command response is returned in the console output. To redirect the output to a file, use -o <filename> option in the CLI command.

Any errors or warnings are logged to the console output as well.

AB method

mapAtoB method is responsible for converting a data source record to another data source record. A and B correspond to two data sources you have selected for your sync.

Here's how the method is implemented in the sample module:

...
async mapAtoB(value) {
  return {
    subject: value.summary,
    start: this.convertTimeToExchange(value.start),
    end: this.convertTimeToExchange(value.end),
    isAllDay: value.start.dateTime == null,
    location: value.location,
    description: value.description
  }
}
...

Here convertTimeToExchange is a reusable utitlity method for time conversion implemented inside of the module as well.

Run

Set the input value in in.json and run ab command using SyncPenguin CLI:

syncpeng ab

BA method

mapBtoA method is the exact opposite of mapAtoB method.

Note that overall, the way you map values should be consistent and should not (ideally) lead to any lose of data. Also, sometimes invalid mapping can lead to an infinite synchronization loop.

SyncPenguin automatically analizes and validates your mappers and looks for any signs of data lose or any other potential problems. Also, it has automatic loop detection mechanism that notifies you in case when a loop synchronization happens.

Here's how the method is implemented in the sample module:

...
mapBtoA(value) {
  return {
    summary: value.subject,
    start: this.convertTimeToGoogle(value.start, value),
    end: this.convertTimeToGoogle(value.end, value),
    location: value.location,
    description: value.description
  }
}
...

Run

Set the input value in in.json and run ba command using SyncPenguin CLI:

syncpeng ba

EQUAL method

equal method is responsible for determining whether two records from different data sources are considered equal in the scope of the synchronization.

Usually, only some logical 'intersection' of fields available in both data sources are synchronized, while other fields are ignored. It's up to you and the mapper implementation to decide what fields are in this intersection.

In most cases you should check for equality only in those fields that are used in mapper mapAtoB and mapBtoA methods.

In the sample module it looks like this:

...
equal(valueA, valueB) {
  return {
    equal:
      valueA.summary == valueB.subject &&
      valueA.location == valueB.location &&
      moment(this.convertTimeToExchange(valueA.start)).isSame(moment(valueB.start)) &&
      moment(this.convertTimeToExchange(valueA.end)).isSame(moment(valueB.end)) &&
      valueA.description == valueB.description
  }
}
...

Run

Set the input value in in.json and run equal command using SyncPenguin CLI:

syncpeng equal