jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
ClickHouse/pg_clickhouse
The project pg_clickhouse is a PostgreSQL extension designed to enable seamless querying of ClickHouse databases directly from PostgreSQL without the need to rewrite SQL queries. It supports PostgreSQL versions 13 and later, as well as ClickHouse version 23 and later. This extension facilitates analytics queries on ClickHouse, leveraging the strengths of both database systems. The project provides a Docker image for easy setup, which includes the standard PostgreSQL Docker image bundled with the pg_clickhouse extension. Users can quickly get started by running the Docker container and creating the extension within PostgreSQL. The project includes comprehensive documentation, including a reference guide and a tutorial to help users import ClickHouse tables and push down queries efficiently. Performance benchmarks using the TPC-H test case demonstrate significant improvements in query execution times when using pg_clickhouse compared to regular PostgreSQL tables, highlighting the efficiency of query pushdown to ClickHouse. The project supports compilation from source on various Unix-like systems, with detailed instructions for dependencies and build steps on Debian/Ubuntu and RedHat/CentOS. It also supports installation via the PGXN client. The extension requires dependencies such as libcurl, libuuid, libSSL, and build tools like GNU make and CMake. The roadmap for the project focuses on enhancing pushdown coverage for analytic workloads, supporting all PostgreSQL aggregate functions and ClickHouse data types, enabling server and session-level ClickHouse settings, and adding features like lightweight DELETEs, UPDATEs, batch insertion, and support for UNION query pushdown. The project is actively maintained by contributors including David E. Wheeler, Ildus Kurbangaliev, and Ibrar Ahmed, and is copyrighted by ClickHouse and other contributors. Overall, pg_clickhouse bridges PostgreSQL and ClickHouse, allowing users to leverage ClickHouse's analytical capabilities within the familiar PostgreSQL environment, improving query performance and simplifying data analytics workflows.
https://github.com/ClickHouse/pg_clickhouse
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.