Hello :) Today is Day 171!
A quick summary of today:
- started Data engineering zoomcamp by DataTalksClub
- completed IEUK by Bright Network
Firstly, about the data engineering zoomcamp
I know they deliver good content, and ends up with a solo project that will be a great addition to my portfolio.
This is the camp overview:
Tons of new things to learn, some old friends (in mage.ai), kafka, dlt, spark, dbt. It was a no brainer decision. So I guess at the moment on different days of the week (I need to set up a schedule) I am taking their data engineering camp, their MLOps camp, and LLM camp.
As for covered material for today~
Learned about docker, and using postgres on docker.
Created this dockerfile
And then below are some of the ran comands and steps covered.
Start the postgres server on docker
And access it through the terminal
pgcli -h localhost -p 5432 -u root -d ny_taxi
Then, using python, I uploaded data and I could query it from the terminal where I ran pgclie
For better user experience, I started a pgadmin server in docker
And I could access it on port 8080, however, I could not add a connection to the created table. This is because the two docker instances (postgres and pgadmin) were not connected. To connect them, I used:
docker network create pg-network
And then I ran the postgres and pgadmin docker containers in 1 network
And I could make a connection, view and query the data in pgadmin
However, the above two commands required me to run them in 2 separate terminals. To make this a bit more convenient, I used docker-compose.
We first set up a docker-compose.yaml file
And now I understand what is going on inside this file, and what docker-compose does for us - it combines containers. Awesome!
There are a few videos left from module 1, but everything from today, incuding my command docs are in Module 1 of my repo.
Secondly, for IEUK by Bright Network
The event ended with the submission of some kind of a work sample where we act as a product manager.
I can share a bit of mine. It is the product roadmap that I developed. Overall it was a good exercise because those are new things for me ~
That is all for today!
See you tomorrow :)