Awesome ClickHouse › ETL and Data Processing
Bulker is a robust service designed for bulk-loading large volumes of semi-structured data into various databases and data warehouses, including Redshift, Snowflake, BigQuery, ClickHouse, Postgres, and MySQL. It leverages Kafka internally to ensure reliable data streaming and batching. The core functionality of Bulker involves receiving JSON objects via an HTTP endpoint, flattening nested JSON structures, and managing database schemas automatically. This means it dynamically creates or updates tables and columns in the destination database based on the incoming data structure, with intelligent type inference or explicit type hints provided in the JSON. Bulker supports both streaming and batching modes for data ingestion. Streaming mode sends data to the database as soon as it arrives, suitable for low event rates, while batching mode accumulates data and sends it in bulk to optimize performance and reduce the load on the database, which is especially important for cloud data warehouses. It also offers deduplication capabilities by primary key to avoid duplicate records. The service is horizontally scalable, stateless, and cloud-native, making it easy to deploy in containerized environments like Kubernetes. It is also available as a Go library for embedding into applications. Bulker supports a wide range of destinations beyond traditional databases, including cloud storage services like S3 and GCS. Key features include JSON flattening, schema management, implicit and explicit typing, batching optimized for each database (e.g., COPY command for Postgres, batch files for BigQuery), and reliability through Kafka queuing. The project is part of the Jitsu open-source data integration platform ecosystem. Overall, Bulker simplifies the complex process of ingesting and managing semi-structured data in modern data warehouses, providing a scalable, reliable, and efficient solution for data engineers and developers.
https://github.com/jitsucom/bulker
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.