Awesome ClickHouse › ETL and Data Processing
The Spark ClickHouse Connector is a project designed to integrate Apache Spark with ClickHouse, a fast open-source column-oriented database management system. This connector is built on the Apache Spark DataSourceV2 API, enabling seamless interaction between Spark applications and ClickHouse databases. It supports various versions of Apache Spark (3.3, 3.4, 3.5) and requires Java 8 or 17 and Scala 2.12 or 2.13 for operation. The connector has evolved over time, initially using a raw gRPC client for communication with ClickHouse but later transitioning to the ClickHouse Official Java Client, which supports the HTTP protocol and extends compatibility with different ClickHouse server versions. Since version 0.6.0, HTTP is the default protocol, and gRPC support has been deprecated and eventually removed by version 0.8.0. The project also underwent a package renaming to better reflect its association with ClickHouse. This connector facilitates efficient data exchange between Spark and ClickHouse, leveraging the strengths of both platforms. It is particularly useful for big data processing and analytics workflows where Spark's distributed computing capabilities are combined with ClickHouse's high-performance analytical database features. The project includes comprehensive testing using Testcontainers and Docker Compose to ensure reliability and compatibility. It provides detailed instructions for building and testing the connector, including running tests with different Spark and Scala versions and against custom ClickHouse images. Overall, the Spark ClickHouse Connector is a vital tool for developers and data engineers looking to integrate Spark's processing power with ClickHouse's analytical capabilities, enabling robust and scalable data processing pipelines.
https://github.com/ClickHouse/spark-clickhouse-connector
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.