sync_clients
Introduction
This is a base module to handle API connections using Advanced Queue.
The goal is to provide a robust framework, that allows for generic API request
handling with build-in error handling and pause mechanism in combination with
queues.
How it works
The idea
- A Sync Client plugin that defines and handles the API connection.
- A corresponding Advanced Queue JobType plugin that does the actual work and uses the Sync Client plugin to make the request(s).
- A dedicated queue configured to the needs of the API and expected JobType processing load and time.
- Leverage as much of Advanced Queue functionalities as possible to handle job failure, duplicate job prevention (if desired) etc.
The Sync Client's plugin annotation defines the corresponding JobType and Queue
ID. From there the Sync Client plugin manager ties it all together.
The goals
- A self-regulating, restoring data sync mechanism.
- Notifications on API issues or errors.
- Notifications on processing errors.
- Processing per data entity, so failures do not block the overall import/export process.
- Easier debugging and retry of failed process jobs.
- Extendable, adjustable to specific use-case/needs.
In practice
- While it is recommended to perform the API request in the queue jobs, to make full use of the pause and retry mechanism, you can combine the pieces in any way you want.
- You can create Sync Client derivatives per API, per endpoint, ... whatever fits your needs.
- You can split data fetching and processing into separate job types, still leveraging the notification mechanism for the data processing job.
- Bulk imports functionality can be easily created by wrapping the initiating code in a drush command or cron task.
Using self replicating queue job, you iterate over a listing endpoint until the last page.
Depending on the amount of data, you can choose- to process the data right away
- to store the data in another job type for processing, per data entity or
- combine multiple,
- ...
- Data POSTed to the site can also be put in a dedicated job type using our base, leveraging the notification mechanism in case of errors.
Build-in functionalities
- The Sync Client base currently contains a mail handler to send notice, warning and error mails and has pause mechanism build-in to stop processing of a queue when there is trouble.
Recipients for these emails can be defined globally or per Sync Client plugin. - The pause mechanism is scoped per Sync Client plugin and bubbles from there to
the corresponding queue.
It has a build-in error level mechanism to temporary halt queue processing for
generic issues like connection timeouts.
Based on certain HTTP response codes, pause state will be enforced, for example
on authentication failure.
Any other errors are to be handled by the JobType. It is up to you to decided to
re-queue the item for another try, or mark the error as a hard fail and if/how
inform somebody about it.
(Current) Limitations
- Needs tweak of Advanced Queue table to convert the payload column data type to JSON, see _sync_clients_check_advancedqueueu_table() in sync_clients.install. This is already outdated, forgot to remove it from the README.
Advanced Queue has issues for:which we can try to use (in favour of our own logic) once they land.
- Needs separate queue (ID) per Sync Client and JobType.
- Pause durations are currently hardcoded.
- Only mail notifications.
While the Sync Client base class was designed to use HTTP requests, we have implementations overriding the request function to for to handle SOAP and FTP connections.
Future plans and ideas
- Configurable pause durations with override option per Sync Client plugin.
- Permission generator for Sync Client plugins for more fine-grained admin access, including pause option only.
- Notification handling using plugins and/or integration with
- Multiple Sync Clients and JobTypes per queue (ID), if this turns out to be beneficial.
Requirements
This module requires Advanced Queue 1.1.
Installation
Install as you would normally install a contributed Drupal module. Visit Installing Modules for further information.
Configuration
Besides the default notification recipients per notification status (error, warning, notices) there is no configuration in the base.
Sync Client plugins need to add their own configuration fields.