jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
long2ice/synch
Synch is a data synchronization tool designed to sync data from other databases such as PostgreSQL and MySQL to ClickHouse, including support for ClickHouse clusters. It supports both full and incremental ETL (Extract, Transform, Load) processes, enabling efficient and real-time data synchronization. The tool handles both DDL (Data Definition Language) and DML (Data Manipulation Language) synchronization, including operations like adding, dropping, and changing columns, with full support for DML operations. Synch supports multiple source databases syncing to ClickHouse simultaneously and is compatible with various ClickHouse table engines such as MergeTree, CollapsingMergeTree, VersionedCollapsingMergeTree, and ReplacingMergeTree. Synch uses brokers like Kafka and Redis to manage data streaming and caching, with Redis also used for caching MySQL binlog files and positions. It supports Redis clusters and Kafka for high concurrency scenarios. The tool requires Python 3.7 or higher and integrates with clickhouse-jdbc-bridge for certain ETL operations, especially when using PostgreSQL with auto full ETL enabled. The project provides a command-line interface for managing ETL processes, including commands for full data ETL, producing data from MySQL binlogs to brokers, and consuming data from brokers to insert into ClickHouse. It also supports monitoring by creating a dedicated database in ClickHouse to log synchronization activities. Synch offers Docker Compose configurations for easy deployment with Redis or Kafka brokers, catering to different concurrency needs. It emphasizes the importance of having primary or unique keys in source tables for effective synchronization and notes that DDL sync is not supported for PostgreSQL and that PostgreSQL sync is not fully tested for production use. Overall, Synch is a robust and flexible solution for syncing data from relational databases to ClickHouse, supporting real-time data integration, error reporting via email, and multiple broker options for scalable data streaming.
https://github.com/long2ice/synch
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
Altinity/clickhouse-sink-connector
Altinity Sink Connector is a scalable tool for replicating data from MySQL, PostgreSQL, and MongoDB to ClickHouse for efficient analytical processing.