Data Pipelines
29 sites
Security covered
This module allows you to ingest, validate, transform, and index datasets, supporting CSV and JSON file formats. It provides a user interface for content editors to manage datasets and allows developers to define custom data transformation and validation pipelines.
Overview
Provides the ability to ingest, validate, transform and index (ElasticSearch) arbitrary datasets.
Features
- Give content editors a place to update datasets
- Support for CSV and JSON files
- Support for CSV and JSON URLs
- Developers can declare a pipeline which is a series of transforms and validation rules
- Data is then transformed, validated and output to a destination
- Default output destinations include JSON and CSV files
Configuration/Installation
- Download and install the module like normal
- Follow the readme to define a pipeline in YML
- Visit /admin/content/datasets to add and manage datasets
Requirements
Requires the following modules:
- Entity module
Ecosystem
Destination plugins
https://www.drupal.org/project/data_pipelines_elasticsearch
https://www.drupal.org/project/data_pipelines_opensearch
Source plugins
https://www.drupal.org/project/data_pipelines_sftp
Known problems
Please use the issue queue to report issues.