Hello :) Today is Day 358!
A quick summary of today:
- registered for Maria Vechtomova and Başak Tuğçe Eskili’s End-to-End MLOps course
- reading about AI Infrastructure Alliance
- read and listened to a podcast about CRISP-ML(Q)
Registered for Maria Vechtomova and Başak Tuğçe Eskili’s End-to-End MLOps course
It took me ~6 hours to register due to bank restrictions for non-Koreans here in Korea… but thankfully at the end I managed to register. Christmas came 1 day early for me 😆
Why We Started the AIIA and What It Means for the Rapid Evolution of the Canonical Stack of Machine Learning
I saved this article from The AI Engineer’s Guide to Surviving the EU AI Act book as it seemed interesting ~
The AI Infrastructure Alliance (AIIA) talk about the need for a standardized, accessible AI/ML infrastructure stack similar to the LAMP stack in the web dev world. AIIA aims to foster collaboration amongst companies developing AI/ML infrastructure software to create this standardised/canonical stack.
One of the goals is to create a blueprint for enterprises to use for their AI/ML workflows. The below is an early version from one of AIIA’s working groups:
There are already a few companies that provide a platform that covers many of these blocks.
What does the architecture of tomorrow look like?
Many sources cite this architecture by Google MLOps:
But as the article authors mention - it is missing Data - the key behind AI
Where are they storing the data? What kind of system are they using to access that data? How do they control access and version it? They don’t include a storage and data versioning layer at all. The diagram picks up at “data extraction” assuming you already have data storage and scaling perfectly handled.
The above have their structure looking like a combination of multiple pipelines. However, models are never ‘completed’ - they (need to) evolve constantly to match their changing surroundings.
There’s a continual learning cycle that happens as new data comes in and the model updates its understanding of the world and the new model gets deployed to production and the old model gets sunsetted.
A better diagram by Larysa Visengeriyeva
It captures the looping nature of the workflow much better than a linear diagram
The Future and Beyond for AIIA
In the long run we’re looking to deliver the Kubernetes of ML, something that abstracts away all the concepts and communications between different layers of any kind of complex AI/Ml stack people can dream up.
Their goal is to enable plug-and-play and open-source functionality for AI/ML systems that are flexible, fast, and agnostic to the underlying technology - making sure we don’t end up locked by vendors.
They don’t seem active on YT, but I subscribed 😆
Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology
I saved this paper from the same EU AI Act book as well. In the book the author referenced this CRISP-ML(Q) framework and I wanted to check the paper ~
Organizations expect to double the number of machine learning (ML) projects within a year. - the increase in models in various industries demands a standardized process to improve project success rates and efficiency
They talk about the current framework - CRISP-DM focuses on data mining and does not cover the application scenario of ML models inferring real-time decisions over a long period of time
Here is a comparison between the two:
CRISP-ML(Q) focuses on a 6-step approach:
- Business and Data understanding (merged): defining business & ML objectives, data collection, and feasibility assessment
- define the scope of the ML application
- success criteria (business, ML, economic)
- feasibility (applicability, legal constraints, requirements on the app)
- data collection (data version control, cost, time)
- DQ verification (data description, data requirements, data verification)
- review of output docs
- Data prep: cleaning, constructing, standardizing, and selecting relevant features
- select data
- clean data (noise reduction, data imputation)
- construct data (feature eng, data augmentation)
- standardize data (file format, normalisation)
- Modeling: selecting appropriate models, incorporating domain knowledge, training, and assuring reproducibility
- literature research on similar problems
- define quality measures of the model
- model selection
- incorporate domain knowledge
- training
- using unlabaled data and pre-trained models
- model compression
- ensemble methods
- reproducability (model, result, experiment documentation)
- Evaluation: validating performance, robustness, explainability, and comparing results to success criteria
- validate performance
- determine rebustness
- increase explainability for ML practitioners and end users
- compare results with defined success criteria
- Deployment: defining inference hardware, testing under production conditions, ensuring user acceptance, and planning deployment strategy
- define inference hardware
- model eval under prod conditions
- assure user acceptance and usability
- min the risks of unforseen errors
- deployment strategy
- Monitoring and maintenance: monitoring model performance, data drift, hardware degradation, and implementing update procedures
- non-stationary data distribution
- degradation of hardware
- need to update any part of the system when needed if we notice some shift, some change happens
CRISP-ML(Q) aknowledges that MLOps is about building systems that can learn and adapt on the fly - data isn’t static.
Quality assurance methodology is introduced in each phase and task of the process model:
Listening to papers in podcast mode
There is a feature in google’s notebook LM that takes your data sources (1 paper pdf in this case) and makes a podcast-like talk about it. There’s also an interactive mode in beta where you can click join and ask Qs as if you are part of the podcast
Absolutely amazing. ‘Reading’ research has never been easier 😆
That is all for today!
Happy Christmas Eve!
See you tomorrow :)