(Day 181) Lending club data engineering project - Done
June 30, 2024
Hello :) Today is Day 181!
A quick summary of today:
- completed and documented my data engineering project
(Day 180) From Kaggle to BigQuery dimension tables - an end2end pipeline
June 29, 2024
Hello :) Today is Day 180!
A quick summary of today:
- finished data modelling in dbt
- set up PROD in dbt
- set up automatic dbt job runs in mage
- created an end to end pipeline
(Day 179) Using Docker, Makefile, and starting Data modelling for my Lending club project
June 28, 2024
Hello :) Today is Day 179!
A quick summary of today:
- continued working on my Lending club data engineering project
(Day 178) Starting 'Lending club data engineering project'
June 27, 2024
Hello :) Today is Day 178!
A quick summary of today:
- started my own data engineering project
- GCP as cloud storage solution
- terraform for infrastructure management
- mage for orchestration
- dbt for modelling
- maybe more to come
(Day 177) Spark for batch processing
June 26, 2024
Hello :) Today is Day 177!
A quick summary of today:
- started Module 5: batch processing from the data eng zoomcamp
- Spark
- Spark operations
- Connecting PySpark to GCS
(Day 176) Testing, Documentation, Deployment with dbt and visualisations with Looker
June 25, 2024
Hello :) Today is Day 176!
A quick summary of today:
- finished Module 4: analysis engineering and using dbt from the data engineering zoomcamp