jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and logical replication service developed by Uber. It is designed to handle large-scale data operations by producing snapshots and real-time change data from thousands of MySQL tables across multiple data centers. The service is also used as a backup solution to snapshot hundreds of terabytes of Schemaless data to storage systems like HDFS and S3, with optional features such as asymmetric encryption and compression to ensure data security and efficiency. StorageTapper supports multiple event sources including MySQL and Schemaless, and can output data to various destinations such as Kafka, HDFS, S3, local files, and experimental support for MySQL, Postgres, and Clickhouse. It supports multiple event formats including Avro, JSON, MsgPack, and SQL, providing flexibility in data handling and integration. Key features of StorageTapper include field filtering, snapshot row filtering, the ability to produce only real-time changes by omitting initial snapshots, ordering of snapshot and changelog events, periodic snapshots, and snapshot index hinting. The service maintains its job state in a MySQL database and automatically distributes jobs among a configurable number of workers. It is designed to be aware of node roles, taking snapshots from slave nodes to reduce the load on master nodes, and streams binlogs from master nodes to ensure better service level agreements (SLAs). StorageTapper is dynamically configurable through a RESTful API or a built-in user interface, allowing for flexible management and operation. It supports deployment on various platforms with detailed build and installation instructions provided for Debian, Ubuntu, and other systems. The project is open-source and licensed under the MIT License. Overall, StorageTapper is a robust and versatile tool for real-time data streaming, backup, and replication in MySQL environments, optimized for large-scale production use cases like those at Uber.
https://github.com/uber/storagetapper
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.
Altinity/clickhouse-sink-connector
Altinity Sink Connector is a scalable tool for replicating data from MySQL, PostgreSQL, and MongoDB to ClickHouse for efficient analytical processing.