Awesome ClickHouse › ETL and Data Processing
JSONBench is a benchmarking project designed to evaluate and compare the native JSON support capabilities of popular analytical databases. It uses a large real-world dataset consisting of 1 billion Bluesky events formatted as newline-delimited JSON (ndjson), which is publicly available on an S3 bucket. The benchmark aims to provide reproducible, realistic, and fair comparisons of how different database systems handle data analytics on semi-structured JSON data. The project includes detailed documentation and scripts to automate the installation, data loading, query execution, and result collection processes for various databases including ClickHouse, MongoDB, Elasticsearch, DuckDB, and PostgreSQL. The benchmark emphasizes reproducibility by providing documented test setups and shell scripts that allow users to replicate the tests on cloud virtual machines. It focuses on realism by using a production dataset that reflects real-world data distributions, enabling accurate assessment of compression, indexing, and hardware utilization. Fairness is maintained by applying appropriate indexing methods without using query result caching or flattening JSON data into non-JSON columns at insertion time, preserving the original document structure. JSONBench is influenced by the earlier ClickBench project and aims to advance the capabilities of data analytics on semi-structured data. It primarily targets data analytics queries rather than search or data mutation operations. The benchmark does not measure data loading times due to the complexity of data preparation in some systems. Users are advised to provision machines with sufficient resources to handle the large dataset, and the project provides scripts to download datasets of varying sizes and run benchmarks accordingly. The project welcomes contributions to add support for new databases and encourages community involvement. It provides detailed instructions for running benchmarks, retrieving results, and understanding the performance characteristics of different systems when working with JSON data. Overall, JSONBench serves as a valuable tool for database developers and users to assess and improve JSON data analytics performance across multiple platforms.
https://github.com/ClickHouse/JSONBench
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.