jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
ClickHouse/clickhouse-kafka-connect
ClickHouse Kafka Connect Sink is the official Kafka Connect sink connector designed specifically for ClickHouse, a high-performance columnar database management system. This connector facilitates the seamless delivery of data from Kafka topics directly into ClickHouse tables, enabling efficient real-time data ingestion and analytics. The project provides comprehensive documentation on the ClickHouse website, detailing its integration and usage. A notable feature of this connector is its design to achieve exactly-once delivery semantics, ensuring data consistency and reliability during the transfer process. This is critical for applications requiring precise data accuracy without duplication or loss. The connector also includes a unique KeyToValue transformation, which allows users to convert Kafka message keys into values. This transformation is particularly useful for storing the Kafka message key in a separate column within ClickHouse tables, enhancing data organization and query capabilities. By default, this key is stored in a column named `_key` with a String data type, but users can customize the column name as needed. The project encourages community support and collaboration through issue filing on GitHub and participation in the ClickHouse public Slack channel. Overall, ClickHouse Kafka Connect Sink is a robust tool for integrating Kafka with ClickHouse, streamlining data workflows, and supporting advanced data processing and analytics use cases.
https://github.com/ClickHouse/clickhouse-kafka-connect
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.