Awesome ClickHouseData Transfer and Synchronization

housepower/clickhouse_sinker

⭐ 533 Go repository created 2018-05-31

clickhouse_sinker is a sinker program designed to facilitate the transfer of Kafka messages into ClickHouse, a fast open-source column-oriented database management system. This project aims to simplify the process of loading data from Kafka, a popular distributed event streaming platform, into ClickHouse for efficient storage and analysis. The tool is built to handle high-throughput data ingestion, making it suitable for real-time analytics and big data applications. It supports seamless integration with Kafka, ensuring reliable and efficient data transfer. The project provides comprehensive documentation and guides to help users get started quickly and understand its features and usage. It is actively maintained and tested, as indicated by its build status and code quality badges. Overall, clickhouse_sinker is a valuable utility for developers and data engineers looking to bridge Kafka and ClickHouse for streamlined data workflows and analytics.

https://github.com/housepower/clickhouse_sinker

big-dataclickhouseclickhouse-bulkclickhouse-serverclickhouse_sinkerdata-engineeringdata-ingestiondata-transferdistributed-event-streamingkafkaopen-sourcereal-time-analyticssinker-program

Also in Data Transfer and Synchronization

jitsucom/jitsu

Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.

getdozer/dozer

Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.

zeromicro/cds

Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.

apecloud/ape-dts

Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.

nikepan/clickhouse-bulk

ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.

uber/storagetapper

StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.

long2ice/synch

Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.

Altinity/clickhouse-sink-connector

Altinity Sink Connector is a scalable tool for replicating data from MySQL, PostgreSQL, and MongoDB to ClickHouse for efficient analytical processing.