Awesome ClickHouse › ETL and Data Processing
DungBeetle is a distributed job server designed to queue and asynchronously execute heavy SQL read jobs, particularly for generating reports from large SQL databases such as MySQL, PostgreSQL, and ClickHouse. It separates the reporting layer from application logic to prevent overloading primary databases during high concurrency report generation. The server supports multiple source databases including MySQL, PostgreSQL, and ClickHouse, and uses MySQL or PostgreSQL as ephemeral result or cache databases where query results are stored in dedicated tables for faster retrieval. This architecture allows user-facing applications to defer report generation requests to a backend job queue, freeing up frontend resources and preventing database bottlenecks. DungBeetle operates as a standalone server exposing HTTP APIs to manage jobs and job groups, supporting operations like listing, posting, status checking, and canceling jobs. SQL queries are loaded from .sql files in a specific format and registered as tasks that can be queued for execution. The system supports multi-process, multi-threaded, asynchronous distributed job queueing with brokers like Redis or AMQP. Key concepts include tasks, which are named SQL queries prepared and registered on server startup, and jobs, which are instances of these tasks queued for execution. Each job has an ID for tracking, and only one job with a given ID can run at a time to prevent duplicate report generation. Results from executed jobs are written to a configured results backend, with tables automatically created to store query outputs in appropriate data types. DungBeetle is ideal for applications with large databases and many concurrent users requesting reports, as it offloads query execution to a distributed job system, reducing load on primary databases and improving user experience by providing asynchronous report generation. The project includes detailed configuration and usage instructions, making it straightforward to deploy and integrate into existing systems.
https://github.com/zerodha/dungbeetle
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.
VulcanSQL is an Analytical Data API Framework that simplifies and accelerates the creation of secure, scalable RESTful APIs from databases and data warehouses for AI agents and data applications.