Awesome ClickHouse › ETL and Data Processing
The project "flink-clickhouse-sink" is a high-performance sink connector designed for integrating Apache Flink with the ClickHouse database. It enables efficient loading of streaming data from Flink into ClickHouse, a columnar database management system optimized for real-time analytical queries. The sink is powered by the Async Http Client, which facilitates asynchronous HTTP requests to ClickHouse, enhancing throughput and reducing latency. This library supports two main triggers for data loading: by timeout and by buffer size, allowing flexible and optimized batch processing of data streams. It is compatible with multiple versions of Flink, with a clear version mapping provided to ensure compatibility and stability. The project is distributed via Maven Central, making it easy to include in Java-based projects. Configuration is divided into common global parameters and sink-specific parameters. Global parameters include settings such as the number of writers, queue capacity, timeout duration, retry attempts, and handling of failed records. Sink-specific parameters include the target ClickHouse table and the maximum buffer size for batching data. Users must convert their event data into a CSV-like string format compatible with ClickHouse's insert syntax. The project provides an interface for implementing custom converters to transform user-defined event objects into this format. This design allows flexibility in handling various data types and structures. The sink can be seamlessly integrated into Flink's data stream processing pipeline by adding it as a sink operator with the appropriate configuration and converter. The project also outlines a roadmap for future enhancements, including reading failed records from disk and migrating the build system to Gradle. Overall, flink-clickhouse-sink is a specialized tool aimed at developers and data engineers who need to efficiently stream data from Flink applications into ClickHouse for real-time analytics and storage.
https://github.com/ivi-ru/flink-clickhouse-sink
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.