Hello :) Today is Day 261!
A quick summary of today:
Continuing with week 3
When your system fails
To build resilient data systems, it’s essential to not only meet stakeholder needs but also plan for potential failures. Planning for failure includes defining key system metrics like availability (uptime), reliability (ability to perform a function), and durability (resistance to data loss). For example, Amazon S3 offers high availability ranging from 99.5% to 99.99%, but 100% uptime is impossible due to unexpected events. Recovery Time Objective (RTO) defines the maximum time a system can be down, while Recovery Point Objective (RPO) defines the acceptable data loss during an outage.
Alongside planning for failure, prioritizing security is crucial. The Zero Trust Security model, unlike the traditional hardened perimeter approach, ensures no users or applications are trusted by default. This reduces the risk of breaches.
Additionally, unforeseen costs and missed revenue opportunities highlight the need for FinOps and scalability. For instance, mismanaging cloud resources can result in unexpected expenses or failure to scale can lead to system crashes. Thus, managing cloud costs dynamically while ensuring scalable architecture is essential to success.
By planning for failure, prioritizing security, and embracing FinOps, data engineers can better serve their organizations, both when systems are performing well and when failures occur.
Batch architectures
Streaming architectures
Choosing tech
Location
Nowadays companies are moving more from on-premise to the Cloud, so learning the Cloud is a good option. However, as many companies still use on-premise, its important to know how to transition.
Monolith vs Modular system
Cost
Build vs Buy
The AWS well-architected framework
It helps us evaluate and improve the solutions we build on AWS
Lab exercise
I should not share detailed info about the lab, so here is just an overview:
-
Three-Tier Architecture: I learned about the three-tier architecture of web applications in AWS, including the presentation, logic, and data tiers. Each plays a vital role in ensuring the web app functions efficiently and can scale.
-
Amazon EC2 Auto Scaling: I explored how EC2 Auto Scaling works to manage a group of EC2 instances, allowing the application to scale up or down based on demand. This helps ensure both performance and cost efficiency.
-
Application Load Balancer (ALB): I understood how the Application Load Balancer (ALB) distributes traffic across multiple EC2 instances, serving as the single point of contact for client requests.
-
VPC and Subnets: I gained insights into Virtual Private Cloud (VPC) and subnets, which allow me to isolate and control access to AWS resources. Subnets help manage public and private traffic within the VPC.
-
Monitoring with Amazon CloudWatch: Using CloudWatch, I learned to monitor CPU usage and network activity during stress tests on my web app. This helps track the performance and behavior of AWS resources in real time.
-
Enhancing Security with Security Groups: I applied security best practices by modifying security group rules to limit access to specific ports and IP addresses. This improved the web app’s security, following the AWS Well-Architected Framework.
-
Reliability with Multi-AZ Deployments: By utilizing multiple availability zones (AZs) for EC2 instances, I ensured that my web app remains highly available and fault-tolerant, even in the event of an outage in one AZ.
-
Auto Scaling Policies: I created dynamic auto-scaling policies based on traffic demand. This ensures the web app scales automatically, maintaining performance while optimizing resource usage and cost.
-
Stress Testing with Apache Benchmark: I performed stress tests on my web app using Apache Benchmark to simulate high traffic and observe how the system reacts under load, making sure it scales efficiently.
For some reason one of the tasks keeps failing. The task is creating a Launch Template for EC2 instances which I do and then starting instances using that template. But when I submit for grading, the test that checks this particular tasks fails. I reported it and it seems coursera is already aware of it and working on a fix. In the meantime I will start week 4.
Week 4
Req. gathering
Examples
The iron triangle
Lab overview
In more detail:
End of course 1:
I just need to wait for the backteam to figure the issue with that 1 assignment and I will officially finish course 1. For now, I am moving to course 2.
Edit after a few hours of writing the above line. I re-did it and I got it. Turns out I will be able to get the certificate after my free trial.
Course 2: Source Systems, Data Ingestion, and Pipelines
Week 1
Types of source systems
ACID principles
- intro to sql
- SQL dbs
- noSQL dbs
- object storage
- logs
The 1st graded item is using AWS’ DynamoDB - all done in a jupyter notebook. Doing CRUD operations
The 2nd graded item is Interacting with amazon S3 object storage
Streaming systems
IAM basics
Networking basics on AWS
Tomorrow I continue with the Week 1 final assignment and quiz.
I am covering lots of material. I am glad I am not a beginner thanks to the DE zoomcamp, the side projects I did and also reading some of the popular DE books (i.e. Fundamentals of DE and Streaming DBs). So far, this course is great. It really focuses on the way of thinking as a data engineer, and also gives example conversation b/e a DE and DS/CTO/Marketing manager that gives an idea of what kind of (stakeholder management) meetings a DE might have.
That is all for today!
See you tomorrow :)