Awesome ClickHouse › ETL and Data Processing
The project "flink-connector-clickhouse" is a Flink SQL connector designed specifically for integration with the ClickHouse database. It enables seamless interaction between Apache Flink, a powerful stream processing framework, and ClickHouse, a fast open-source columnar database management system. This connector supports both source and sink tables, allowing users to read from and write data to ClickHouse efficiently. Additionally, it provides support for Flink Catalog, facilitating metadata management and table discovery within Flink SQL environments. The connector is powered by the ClickHouse JDBC driver, ensuring reliable and performant communication with ClickHouse servers. It supports a variety of data types including primary data types, maps, and arrays, enabling complex data structures to be handled effectively. Users can configure numerous options such as the ClickHouse JDBC URL, authentication credentials, database and table names, and various performance tuning parameters like batch size, flush intervals, and retry attempts. Advanced features include support for distributed table engines with options to read/write local tables, partitioning strategies for sink operations (balanced, hash, shuffle), and sharding strategies consistent with distributed table definitions. The connector also offers configurable update strategies for handling update-after records, options to ignore delete statements, and custom parallelism settings for sink operations. For lookup operations, the connector provides caching strategies including partial caching with expiration policies and maximum row limits, enhancing query performance by reducing database hits. It also supports retry mechanisms for lookup failures. Overall, this project facilitates robust and efficient integration between Flink and ClickHouse, enabling real-time data processing and analytics workflows that leverage the strengths of both systems. It is suitable for users looking to build scalable, high-performance data pipelines and analytics solutions using Flink SQL and ClickHouse.
https://github.com/itinycheng/flink-connector-clickhouse
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.