Awesome ClickHouse › ETL and Data Processing
The Airflow ClickHouse Plugin is a highly popular Apache Airflow plugin designed to integrate ClickHouse, a fast open-source column-oriented database management system, with Airflow workflows. It is ranked in the top 1% of downloads on PyPI, indicating its widespread adoption and reliability. The plugin is based on the mymarilyn/clickhouse-driver and offers two main families of operators to interact with ClickHouse: the clickhouse-driver family and the Python DB API 2.0 family. The clickhouse-driver family includes operators such as ClickHouseOperator, ClickHouseHook, and ClickHouseSensor. These operators leverage the Client.execute method from the clickhouse-driver, providing full functionality and efficient native ClickHouse TCP protocol communication. Key features include SQL templating, support for multiple SQL queries in a single operator, detailed logging of executed queries, and support for custom connection parameters like timeouts and compression. This family is recommended for users starting fresh with ClickHouse in Airflow. The Python DB API 2.0 family offers a standardized interface compatible with the Python DB API 2.0 specification. It includes various operators like ClickHouseSQLExecuteQueryOperator and ClickHouseSQLCheckOperator, among others. These operators integrate clickhouse_driver.dbapi with the apache-airflow-providers-common-sql package, making it easier to port Airflow pipelines from other SQL providers such as MySQL or Postgres to ClickHouse. Although this family has limited functionality compared to the clickhouse-driver family, it provides a consistent and familiar interface for SQL operations. The plugin supports multiple versions of Python and Airflow, primarily focusing on Airflow 2.0+ and Python 3.8+. It also provides detailed installation instructions and dependency requirements. The plugin is well-tested across different Airflow and Python versions, ensuring stability and compatibility. Overall, the Airflow ClickHouse Plugin simplifies the integration of ClickHouse with Airflow, enabling efficient execution of SQL queries, monitoring, and data pipeline management within Airflow workflows. It is a valuable tool for data engineers and developers working with ClickHouse and Airflow in their data infrastructure.
https://github.com/bryzgaloff/airflow-clickhouse-plugin
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.