Machine Learning ML-00.03
The ML project lifecycle
Follow an ML project from framing the problem to monitoring and retraining, and see why data work, not the model, takes most of the time.
The big idea
Tutorials make machine learning look like one step: load a dataset, call fit, print the accuracy. Real projects look nothing like that. Before you can train anything you have to decide what exactly to predict and how you will know it worked, then find, clean and label data. After training you have to ship the model, watch it in production, and retrain it when the world changes under it.
The training step is often the shortest part. A useful mental picture is a restaurant kitchen: the cooking you see on TV takes minutes, but the menu planning, the shopping, the washing and chopping, the plating, the cleaning and the feedback from diners take all day. Skip any of them and dinner fails, however good the chef.
How it works
1. Frame the problem
Decide what you are predicting, for whom, and what decision it changes. "Use AI on our support tickets" is not a problem. "Route each new ticket to the right team, so fewer tickets get reassigned" is. Then pick a metricThe number you use to judge whether a model is good enough, such as accuracy, error in minutes, or recall on fraud. that matches the business goal, and a baselineA simple reference result, such as a rule or a very simple model, that a new model must beat to be worth shipping.: the simplest thing that could work, such as a keyword rule. A model that cannot beat the baseline is not worth shipping.
Sometimes the honest answer at this stage is "no ML yet". If you have no data, or a few rules solve it, start there.
2. Collect and clean data
Find where the examples live (logs, databases, exports), join them, and get labels. Then clean: remove duplicates, fix broken values, decide what to do with missing fields, and check that labels are consistent. This stage is slow, unglamorous and decisive. A simple model on good data beats a fancy model on bad data, and data problems found late cost far more than ones found early. The detail is in Data cleaning .
3. Build features
Turn raw records into inputs a model can use: counts, ratios, dates turned into "hour of day", text turned into numbers. Good features often matter more than the choice of algorithm.
4. Train and 5. Evaluate
Train a model, usually starting simple, on the training data. Then measure it on data it has never seen, held out at the start, using the metric from step 1. Compare against the baseline. Expect to loop between features, training and evaluation many times.
6. Deploy
Put the model where it can make predictions: behind an API, inside an app, or in a nightly batch job. The code that prepares features at prediction time must match the code used in training exactly. When it does not, you get training-serving skewA difference between how data is prepared or distributed at training time and at prediction time, which quietly hurts real-world accuracy.: a model that looked great offline and performs badly live.
7. Monitor and 8. Retrain
The world changes. Customers behave differently in December, fraudsters adapt, a new product launches. A model's accuracy decays silently, so you track input data and prediction quality in production and retrain on fresh data when they drift. That retraining sends you back to the data stage, which is why the diagram is a loop.
Try it
Follow one project around the loop: predicting which support tickets are urgent.
Try this:
- Step through and name the stage where a decision could make the whole project unnecessary.
- At the "Data" step, what would happen if you skipped writing one definition of "urgent"?
- At "Monitor", which signal would have warned you before users complained?
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Rules or a heuristic first | You need something live quickly and want a baseline to measure against | The pattern is too fuzzy for rules to keep up |
| Simple model first | Almost always: fast to train, easy to debug, gets the pipeline working end to end | You already have a working simple model and a clear, measured need for more accuracy |
| Complex model first | Rarely: only when a simple model has been tried and clearly falls short | Debugging a broken pipeline and a complex model at the same time |
Common mistakes
- Starting with the algorithm before agreeing on the metric and the decision it should change.
- Evaluating on data the model has seen during training, then being surprised in production.
- Writing feature code twice, once in a notebook and once in the serving app, so the two drift apart.
- Shipping and forgetting: no monitoring means you learn about decay from angry users.
- Treating the first model as the finish line rather than the start of the loop.
In the wild
A guide of 43 rules from a Google engineer, organized by project phase, from deciding whether you need ML at all to optimizing a mature system. It opens with "Don't be afraid to launch a product without machine learning", advises keeping the first model simple and getting the infrastructure right, and says most gains come from great features rather than great algorithms, which is the lifecycle in this lesson in practice. It also has rules for monitoring freshness and measuring training-serving skew.
Check yourself
-
Which stage usually takes the most time in a real ML project?
Show answer
B. Collecting, cleaning and labeling data Finding, joining, cleaning and labeling data is usually the largest piece of work, and it decides how good the model can be.
-
Your model scored 94 percent offline but performs poorly live. The serving app computes 'customer age' differently from the training notebook. What is this called?
Show answer
B. Training-serving skew When features are computed differently at training and prediction time, the model sees inputs it was never trained on. Share one feature implementation.
-
Why should you define a baseline before training any model?
Show answer
B. So you can tell whether the model adds enough value to be worth shipping If a keyword rule catches most cases, a model must clearly beat it to justify its cost and complexity.
-
Six months after launch, a model's accuracy has slowly dropped because customer behaviour changed. Which stages address this?
Show answer
B. Monitoring and retraining Monitoring catches the drift, and retraining on fresh data brings the model back in line with the world.
Recap
- An ML project is a loop: frame, data, features, train, evaluate, deploy, monitor, retrain.
- Agree on the metric and a simple baseline before training anything.
- Data work usually takes the most time and matters the most.
- Use the same feature code in training and serving to avoid training-serving skew.
- Models decay as the world changes, so monitoring and retraining are part of the job.
Next up
- I know nothing yet learning path
- From notebook to production learning path