Awesome ClickHouse › ETL and Data Processing
Apache Gluten acts as a 'glue' layer, enabling JVM-based SQL engines, particularly Spark SQL, to leverage the high performance of native execution engines like ClickHouse and Velox. Spark, while mature and scalable, faces performance challenges at the operator level. Native engines, with their columnar data formats and vectorized processing, offer significant speedups. Gluten addresses this by reusing Spark's control flow while offloading compute-intensive data processing to the native side. The project transforms Spark's physical plan into a Substrait plan, which is then passed to the native backend via JNI. Currently supporting ClickHouse and Velox, Gluten returns results as a ColumnarBatch, utilizing Spark's Columnar API and Apache Arrow for data representation. Key architectural components include query plan conversion, unified memory management, columnar shuffle, and a fallback mechanism for unsupported operators. It also provides a shim layer for Spark version compatibility and integrates metrics into the Spark UI. Gluten is designed as a plugin, requiring no changes to existing Spark DataFrame APIs or SQL queries, making it easily configurable for users seeking to accelerate their Spark SQL workloads.
https://github.com/apache/gluten
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.
VulcanSQL is an Analytical Data API Framework that simplifies and accelerates the creation of secure, scalable RESTful APIs from databases and data warehouses for AI agents and data applications.