jitsucom/jitsu
Jitsu is an open-source, self-hosted data ingestion platform that enables real-time event data collection and streaming to data warehouses, including ClickHouse, as an alternative to Segment.
Awesome ClickHouse › Data Transfer and Synchronization
QuesmaOrg/quesma
Quesma is a programmable database gateway designed to facilitate seamless database migrations and integrations without requiring changes to the application or client stack. It acts as a translation layer that routes queries from applications to modern database platforms, translating queries on the fly to maintain backward compatibility and enable safe, step-by-step migration processes. This approach eliminates the need for rewriting database query code during migrations, significantly reducing the complexity and risk associated with switching database technologies. One of the key features of Quesma is its ability to connect Kibana or OpenSearch Dashboards to Clickhouse, allowing users to leverage Clickhouse's powerful analytics capabilities while continuing to use familiar visualization tools. The project provides an example setup using Docker Compose that demonstrates how Kibana can connect to both Elasticsearch and Clickhouse through Quesma, showcasing its practical application in real-world scenarios. The example setup includes demo datasets such as generic logs generated by a log-generator application and official Kibana sample data sets for e-commerce and flights, complete with dashboards and log explorers. This makes it easy for users to get started and see the benefits of Quesma in action. Quesma is actively developed and maintained, with a focus on providing a robust and flexible solution for database query translation and migration. It supports modern database platforms and aims to simplify the migration process by handling query translation transparently. The project is licensed under the Elastic License 2.0 and includes developer documentation to assist contributors and users in understanding and extending its capabilities. Overall, Quesma addresses a common challenge in database management by offering a programmable gateway that ensures compatibility and ease of migration between different database systems, with a current emphasis on integrating Kibana/OpenSearch Dashboards with Clickhouse.
https://github.com/QuesmaOrg/quesma
Also in Data Transfer and Synchronization
getdozer/dozer
Dozer is a high-performance real-time data movement tool leveraging CDC to transfer and transform data efficiently into multiple sinks including Clickhouse.
zeromicro/cds
Zeromicro/cds is a Go-based tool for real-time data synchronization from MySQL and MongoDB to ClickHouse, supporting both full and incremental sync with a user-friendly web interface.
apecloud/ape-dts
Ape Data Transfer Suite (ape-dts) is a high-performance, lightweight data migration and replication tool supporting ultra-fast transfers between MySQL, PostgreSQL, Redis, MongoDB, Kafka, and ClickHouse, ideal for disaster recovery and migration scenarios.
housepower/clickhouse_sinker
clickhouse_sinker is a sinker program that efficiently transfers Kafka messages into ClickHouse for real-time data analytics and storage.
nikepan/clickhouse-bulk
ClickHouse-Bulk is a tool that collects many small insert requests and sends them as larger bulk inserts to Yandex ClickHouse servers to improve data ingestion efficiency.
uber/storagetapper
StorageTapper is a scalable real-time MySQL change data streaming, logical backup, and replication service supporting multiple data destinations including experimental Clickhouse integration.
long2ice/synch
Synch is a data synchronization tool that enables full and incremental ETL from PostgreSQL and MySQL databases to ClickHouse, supporting real-time data integration and multiple broker options.