Awesome ClickHouse › Examples

toddwschneider/nyc-taxi-data

⭐ 2082 R repository created 2015-11-16

This project provides scripts and tools to download, process, and analyze over 3 billion taxi and for-hire vehicle (FHV) trips data originating in New York City since 2009. The data includes trips from traditional yellow and green taxis as well as ride-hailing services like Uber and Lyft. The raw data is sourced from the NYC Taxi & Limousine Commission (TLC) and has been updated to handle the newer Apache Parquet file format introduced in May 2022, replacing the older CSV format. The repository supports two main database backends for storing and querying this data: PostgreSQL and ClickHouse. For PostgreSQL, the project uses R scripts to convert Parquet files to CSV before importing, while for ClickHouse, it directly loads Parquet files. The data schema includes tables for taxi trips, for-hire vehicle trips, FHV base mappings, NYC census tracts, taxi zones, and weather observations, among others. The project also bundles additional geographic and mapping data such as shapefiles for NYC census tracts and taxi zones, and weather data from Central Park. It offers detailed instructions for setting up the environment, including installing PostgreSQL, PostGIS, and R, downloading raw data, initializing the database, and importing various datasets. The repository is useful for large-scale data analysis, benchmarking database performance, and urban mobility research. It also includes scripts for importing TLC summary statistics and code for comparing taxi and Citi Bike travel times. The project is actively maintained and provides contact information for support and issue reporting.

https://github.com/toddwschneider/nyc-taxi-data

apache-parquetbenchmarkingcensus-tractsclickhousedata-analysisdata-importdata-processingfhv-tripsfor-hire-vehicle-datalyftnycnyc-taxi-&-limousine-commissionnyc-taxi-datanyc-taxi-datasetpostgispostgresqlrshapefilestaxi-tripstaxi-zonestlctrip-datauberurban-mobilityweather-data