jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance data migration and replication tool designed to facilitate ultra-fast data transfers between a variety of database systems including MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse. Written in Rust, ape-dts is lightweight, efficient, and standalone, requiring no third-party components or additional storage, making it ideal for disaster recovery (DR) and migration scenarios. The tool supports both homogeneous and heterogeneous data migrations, enabling snapshot and change data capture (CDC) tasks with the ability to resume from breakpoints. It also offers data checking, revising, and reviewing functionalities to ensure data integrity during transfers. Users can filter and route data at the database, table, and column levels, and can enhance data processing by loading user-defined Lua scripts. Ape-dts implements various parallel algorithms tailored to different source and target databases and task types to optimize performance. It can also operate as an HTTP server, allowing data to be pulled from the source using any language's HTTP client, thus providing flexibility in data consumption. The suite supports a wide range of task types across multiple database combinations, including snapshot, CDC, data check/revise/review, and structure migration. It provides comprehensive documentation and tutorials for different migration scenarios, including MySQL to ClickHouse, PostgreSQL to ClickHouse, and many others. Benchmark tests demonstrate ape-dts's superior performance compared to other tools like Debezium, with higher rows per second (RPS) rates and lower resource consumption. The project is modular, consisting of components for connectors, pipelines, parallel algorithms, task management, and common utilities, with integration tests to ensure reliability. Overall, ape-dts is a robust, versatile, and efficient solution for database replication and migration, particularly suited for environments involving ClickHouse and other popular databases.
https://github.com/apecloud/ape-dts
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.
Altinity/clickhouse-sink-connector
Altinity Sink Connector is a scalable tool for replicating data from MySQL, PostgreSQL, and MongoDB to ClickHouse for efficient analytical processing.