jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
zeromicro/cds
The project "zeromicro/cds" is a data synchronization tool designed specifically for ClickHouse, implemented in Go. It facilitates real-time data syncing from MySQL and MongoDB databases to a ClickHouse cluster with a latency of less than two minutes. The tool is based on the go-zero framework and provides a comprehensive data workflow architecture typical of data warehouses. It supports automatic synchronization, enabling users to keep their ClickHouse data up-to-date with source databases efficiently. The project includes a web interface accessible via a local server, where users can manage data synchronization tasks. Users can create tables in ClickHouse that mirror the structure of source tables from MySQL or MongoDB, generate the necessary SQL for table creation, and send it directly to ClickHouse. It supports both one-time full data synchronization and continuous real-time incremental synchronization, allowing for flexible data management strategies. The synchronization process is user-friendly, with clear steps to connect to source databases and the target ClickHouse instance, select tables, and initiate sync tasks. The tool also provides status monitoring for ongoing tasks and verification of data integrity in ClickHouse. Additionally, it offers commands to start, stop, and clean up the Docker containers used for running the service, simplifying deployment and maintenance. The project documentation includes detailed instructions, architecture diagrams, and examples to help users understand and utilize the tool effectively. It also references best practices for asking questions and reporting bugs, indicating a focus on community support and usability. Overall, zeromicro/cds is a robust solution for organizations needing efficient, real-time data synchronization to ClickHouse from popular databases like MySQL and MongoDB.
https://github.com/zeromicro/cds
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.
Altinity/clickhouse-sink-connector
Altinity Sink Connector is a scalable tool for replicating data from MySQL, PostgreSQL, and MongoDB to ClickHouse for efficient analytical processing.