jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
Altinity/clickhouse-sink-connector
The Altinity Sink Connector is a robust data replication tool designed to automatically move data from transactional databases such as MySQL, PostgreSQL, and MongoDB into ClickHouse, a high-performance analytical database. This project facilitates seamless data transfer for analytical purposes, enabling users to leverage ClickHouse's powerful capabilities for data analysis. Key features include initial data dump and load for MySQL, change data capture of new transactions using Debezium, and automatic loading into ClickHouse. The connector supports sources like MySQL and PostgreSQL, with experimental support for other databases, and targets ClickHouse's ReplacingMergeTree engine for efficient data storage and querying. The connector is designed to handle failures gracefully, allowing recovery and restart from issues on either the source or target side. It also manages upstream schema changes automatically and performs checksum-based table comparisons to ensure data integrity. Scalability is a core aspect, with the ability to handle thousands of tables. Deployment options include a lightweight single-process model that transfers data directly from source to target, and a Kafka-based model that uses Kafka as a transport layer between separate source and target processes. The project is distributed as a Docker container for easy deployment and includes comprehensive documentation covering architecture, configuration, monitoring, troubleshooting, and development. It also provides quick start guides for different deployment scenarios and databases. The Altinity Sink Connector is actively maintained by Altinity, which offers commercial support and related services for ClickHouse and analytic applications. The project encourages community contributions and provides various resources such as blog articles, comparison with other technologies, and a roadmap for future development. Overall, the Altinity Sink Connector is a powerful and scalable solution for replicating data from transactional databases to ClickHouse, enabling efficient and reliable analytical workflows.
https://github.com/Altinity/clickhouse-sink-connector
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.