jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
nikepan/clickhouse-bulk
ClickHouse-Bulk is a utility designed to optimize the process of inserting data into Yandex ClickHouse databases by collecting many small insert requests and sending them as larger bulk inserts. This approach improves efficiency and performance by reducing the overhead associated with multiple small insert operations. The project supports grouping multiple insert requests and sending them to any configured ClickHouse server either by reaching a certain number of requests or after a specified time interval. It supports different data formats such as VALUES and TabSeparated and can handle multiple servers for load distribution or failover. The tool offers flexibility in how queries are sent, supporting query parameters both in the URL and in the request body, along with authentication options including basic authentication. It is configurable via a JSON file or environment variables, allowing users to specify parameters like flush count, flush interval, server addresses, connection timeouts, TLS settings, and debugging options. The tool also includes mechanisms for handling failed inserts by dumping unsent data to disk and retrying later. ClickHouse-Bulk is easy to deploy, with pre-built binaries and Docker images available, or it can be built from source using Go. It exposes metrics compatible with Prometheus for monitoring the health and performance of the insert operations, including counts of received, sent, and failed requests, as well as server status metrics. Overall, ClickHouse-Bulk is a practical solution for applications that generate many small insert queries to ClickHouse, enabling more efficient data ingestion by batching these queries into larger inserts, thus improving throughput and reducing load on the database servers.
https://github.com/nikepan/clickhouse-bulk
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.
Altinity/clickhouse-sink-connector
Altinity Sink Connector is a scalable tool for replicating data from MySQL, PostgreSQL, and MongoDB to ClickHouse for efficient analytical processing.