(Day 180) From Kaggle to BigQuery dimension tables - an end2end pipeline

June 29, 2024

Hello :) Today is Day 180!

A quick summary of today:

finished data modelling in dbt
set up PROD in dbt
set up automatic dbt job runs in mage
created an end to end pipeline

(Day 179) Using Docker, Makefile, and starting Data modelling for my Lending club project

June 28, 2024

Hello :) Today is Day 179!

A quick summary of today:

continued working on my Lending club data engineering project

(Day 178) Starting 'Lending club data engineering project'

June 27, 2024

Hello :) Today is Day 178!

A quick summary of today:

started my own data engineering project
- GCP as cloud storage solution
- terraform for infrastructure management
- mage for orchestration
- dbt for modelling
- maybe more to come

(Day 177) Spark for batch processing

June 26, 2024

Hello :) Today is Day 177!

A quick summary of today:

started Module 5: batch processing from the data eng zoomcamp
- Spark
- Spark operations
- Connecting PySpark to GCS

(Day 176) Testing, Documentation, Deployment with dbt and visualisations with Looker

June 25, 2024

Hello :) Today is Day 176!

A quick summary of today:

finished Module 4: analysis engineering and using dbt from the data engineering zoomcamp

(Day 175) Learning about and using dbt cloud

June 24, 2024

Hello :) Today is Day 175!

A quick summary of today:

basic SQL in BigQuery
continued Module 4: analytics engineering and dbt from the data eng camp and learned about dbt

« Prev 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 Next »