Description:
Sky-Pipe is a Prefect data pipeline that integrates Google Cloud Platform and dbt. Specifically, the Sky-Pipe fetches daily exchange data from CoinMarketCap.com and loads this data into Google Cloud Platform, and transforms it with dbt. This pipeline also utilizes GitHub Actions for dev ops.
Technologies:
Prefect/Prefect-GCP
dbt
GCP Secrets Manager
GCP BigQuery
GCP Compute Engine
GCP Cloud Storage
GitHub Actions
GCP Artifact Registry
GCP Cloud Run Jobs