Awesome ClickHouse › ETL and Data Processing
ClickBench is a benchmark designed specifically for analytical databases, focusing on workloads typical in areas such as clickstream and traffic analysis, web analytics, machine-generated data, structured logs, and event data. It aims to simulate realistic ad-hoc analytics and real-time dashboard queries. The dataset used in ClickBench is derived from actual traffic recordings of one of the world's largest web analytics platforms, anonymized to preserve essential data distributions. The benchmark includes 43 queries that test various database operations including full scans, filtered scans, index lookups, and relational operations, while also stressing different hardware aspects like storage throughput, CPU core count, and memory bandwidth. The primary goals of ClickBench are reproducibility, compatibility, diversity, and realism. It allows users to reproduce tests quickly using documented, semi-automated scripts on inexpensive cloud VMs. The benchmark uses mostly standard SQL to ensure compatibility across a wide range of database management systems (DBMS), including modern and historical OLAP systems, traditional OLTP systems for baseline comparison, managed database-as-a-service offerings, serverless cloud-native databases, and some NoSQL and specialized time-series databases for reference. ClickBench emphasizes realism by using a dataset with accurate production data distributions, enabling meaningful comparisons of compression, indexing, codecs, and custom data structures. However, it acknowledges limitations such as the use of a single flat table rather than normalized schemas, a relatively small dataset size for modern standards, and the focus on single-node setups for most results. The benchmark runs queries sequentially without concurrent workload testing, and it does not measure system capacity or data loading times for stateless engines. The project encourages contributions by allowing users to add new results or systems through a structured directory and script setup. It provides datasets in multiple formats (CSV, TSV, JSONlines, Parquet) for flexible data loading. Overall, ClickBench serves as a comprehensive, reproducible, and realistic benchmarking tool for evaluating the performance of analytical databases, including ClickHouse, under typical real-world workloads.
https://github.com/ClickHouse/ClickBench
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.
VulcanSQL is an Analytical Data API Framework that simplifies and accelerates the creation of secure, scalable RESTful APIs from databases and data warehouses for AI agents and data applications.