Awesome ClickHouse › ETL and Data Processing
The project "dbt-clickhouse" is a plugin that integrates dbt (data build tool) functionality with ClickHouse, a fast open-source columnar database management system. This plugin enables users to leverage dbt's powerful data transformation and modeling capabilities directly on ClickHouse databases. It supports ClickHouse versions 22.1 and newer and does not currently support replicated tables or the ON CLUSTER functionality. The plugin can be installed via Python package managers like pip, with explicit installation instructions depending on the dbt-core version. Key features supported by dbt-clickhouse include various materializations such as table, view, incremental, materialized view (experimental), distributed table, and distributed incremental materializations. It also supports seeds, sources, documentation generation, tests, snapshots, ephemeral materialization, contracts, and most dbt-utils macros. The plugin addresses specific usage considerations for ClickHouse, such as the unreliability of the SET statement for persisting settings across queries in certain environments and recommends configuring settings via the dbt profile's custom_settings property. The plugin adapts dbt's model relation identifier to ClickHouse's database structure by using a simplified schema.table format since ClickHouse does not support schemas. It provides a detailed example profile configuration for connecting to ClickHouse, including options for driver type, host, port, user credentials, cluster settings, security, retries, compression, timeouts, and other advanced settings. Model configuration options allow users to specify table engines, order by columns, partitioning, sharding keys, primary keys, and unique keys, enabling fine-tuned control over how data is stored and queried in ClickHouse. Overall, dbt-clickhouse facilitates seamless integration of dbt workflows with ClickHouse, enhancing data transformation, testing, and documentation capabilities for users working with ClickHouse databases.
https://github.com/ClickHouse/dbt-clickhouse
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.