This module allows you to ingest, validate, transform, and index datasets, supporting CSV and JSON file formats. It provides a user interface for content editors to manage datasets and allows developers to define custom data transformation and validation pipelines.
Overview
Provides the ability to ingest, validate, transform and index (ElasticSearch) arbitrary datasets.
Features
- Give content editors a place to update datasets
- Support for CSV and JSON files
- Support for CSV and JSON URLs
- Developers can declare a pipeline which is a series of transforms and validation rules
- Data is then transformed, validated and output to a destination
- Default output destinations include JSON and CSV files
Configuration/Installation
- Download and install the module like normal
- Follow the readme to define a pipeline in YML
- Visit /admin/content/datasets to add and manage datasets
Requirements
Requires the following modules:
- Entity module
Ecosystem
Destination plugins
https://www.drupal.org/project/data_pipelines_elasticsearch
https://www.drupal.org/project/data_pipelines_opensearch
Source plugins
https://www.drupal.org/project/data_pipelines_sftp
Known problems
Please use the issue queue to report issues.
Ask your assistant about Data Pipelines
Check compatibility with your Drupal and PHP version, maintenance and security coverage, from current release data. How it works
Depends on
Dependencies of the latest stable release
- Entity API
- file Drupal core
- link Drupal core
- options Drupal core
Required by
1 tracked project depends on this one
- Data Pipelines Elasticsearch 9 sites
Activity
Release Timeline
Releases
| Version | Type | Core | PHP | Release date | |
|---|---|---|---|---|---|
| 2.2.0 | Stable | 10–11 | Mar 19, 2025 | ||
| 2.1.0 | Stable | 10–11 | Mar 17, 2025 | ||
| 2.x-dev | Dev | 10–11 | Mar 7, 2025 | ||
| 2.0.2 | Stable | 10–11 | Nov 20, 2024 | ||
| 2.0.1 | Stable | 10–11 | Nov 19, 2024 | ||
| 2.0.0 | Stable | 10–11 | Nov 19, 2024 | ||
| 1.0.0-alpha27 | Pre-release | 10–11 | PHP 8 | Jun 21, 2024 | |
| 1.0.0-alpha26 | Pre-release | 10 | PHP 8 | Apr 17, 2024 | |
| 1.0.0-alpha25 | Pre-release | 10 | PHP 8 | Mar 5, 2024 | |
| 1.0.0-alpha24 | Pre-release | 10 | PHP 8 | Jan 16, 2024 |