jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
ildus/clickhouse_fdw
The project "clickhouse_fdw" is an open-source Foreign Data Wrapper (FDW) designed to integrate the ClickHouse column-oriented database with PostgreSQL. It enables PostgreSQL users to access and query data stored in ClickHouse as if it were a native PostgreSQL table, facilitating seamless data interoperability between the two database systems. The FDW supports PostgreSQL versions 11 through 14 and communicates with ClickHouse using its HTTP interface, requiring dependencies such as libcurl and uuid libraries. The project provides detailed installation instructions, including specific notes for CentOS 7 users to ensure compatibility with modern compilers and necessary library versions. Users can set up sample databases and tables in ClickHouse, load data from external sources, and then create corresponding foreign tables in PostgreSQL using the clickhouse_fdw extension. The FDW supports configuration options such as database name, host, port, and driver type (HTTP or binary protocol). It also allows user mapping for authentication and supports importing foreign schemas directly into PostgreSQL. The project includes examples of querying data from ClickHouse through PostgreSQL, demonstrating the ability to perform SELECT and INSERT operations across the systems. It also supports advanced ClickHouse features like AggregatingMergeTree tables, where users can specify aggregation functions for columns to optimize query performance. Overall, clickhouse_fdw bridges the gap between PostgreSQL and ClickHouse, enabling users to leverage ClickHouse's high-performance analytics capabilities within the familiar PostgreSQL environment. This integration is particularly useful for scenarios requiring efficient analytical queries on large datasets stored in ClickHouse while maintaining transactional and relational capabilities in PostgreSQL.
https://github.com/ildus/clickhouse_fdw
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.