Awesome ClickHouse › ETL and Data Processing
Easy SQL is a library designed to simplify the data ETL (Extract, Transform, Load) development process by allowing developers to write ETL workflows in SQL using an imperative style. It extends standard SQL with a few simple syntaxes that enable SQL statements to be executed sequentially, making the ETL process more intuitive and manageable. The library includes a processor that handles these new syntaxes and is SQL engine agnostic, meaning it can work with any SQL engine as a backend. Currently, Easy SQL supports several popular SQL engines such as SparkSQL, PostgreSQL, Clickhouse, FlinkSQL, Aliyun Maxcompute, and Google BigQuery, with plans to add more in the future. Easy SQL can be installed via pip with optional extras for different backends and tools like CLI and linter. It supports multiple backends, including Spark, PostgreSQL, Clickhouse, and Flink, each with specific setup instructions and examples provided in the documentation. For instance, users can run ETL scripts on a Clickhouse instance by installing the clickhouse backend and executing SQL files with the Easy SQL data processor. The library also supports programmatic ETL execution within code, demonstrated with sample Python snippets using Spark as a backend. The project emphasizes ease of use and flexibility, allowing users to develop ETL pipelines in a familiar SQL environment while leveraging the power of various SQL engines. It also provides debugging support, recommending the use of Jupyter notebooks for interactive ETL development and troubleshooting. The documentation is comprehensive, covering installation, backend-specific instructions, and examples to help users get started quickly. Keywords extracted include ETL, SQL, data processing, SparkSQL, PostgreSQL, Clickhouse, FlinkSQL, Aliyun Maxcompute, Google BigQuery, SQL engine, backend, data pipeline, imperative SQL, debugging, Jupyter, CLI, linter, Python, data transformation, data extraction, data loading, open source.
https://github.com/easysql/easy_sql
PeerDB is a high-performance, PostgreSQL-optimized ETL tool that enables fast, reliable, and cost-effective streaming of data from Postgres to data warehouses, queues, and storage engines, with native integration in ClickHouse Cloud.
YTsaurus is a scalable, fault-tolerant open-source big data platform featuring MapReduce, SQL engine, NoSQL store, and integration with ClickHouse for fast analytics.
Trench is an open-source, production-ready analytics infrastructure built on ClickHouse and Kafka for scalable, real-time event tracking and analytics with GDPR compliance.
Gluten is a middle layer that offloads JVM-based SQL engines' execution, such as Spark SQL, to high-performance native engines like ClickHouse and Velox, leveraging vectorized processing for accele...
Addax is a versatile and extensible open-source ETL tool that supports seamless data transfer between over 20 SQL and NoSQL data sources, including ClickHouse, with easy configuration and deployment options.
DungBeetle is a distributed job server for asynchronously queuing and executing heavy SQL read jobs on MySQL, PostgreSQL, and ClickHouse databases, designed to offload report generation and improve application performance.
ClickBench is a comprehensive and reproducible benchmark designed to evaluate the performance of analytical databases, including ClickHouse, using realistic workloads derived from real-world web analytics data.
DataCap is an integrated software platform for data transformation, integration, and visualization, supporting a wide range of data sources including ClickHouse and other major databases.