Real-world Production ML System
The common architecture of the real-world production machine learning system is shown below:
Types of the Training Process
Static Model – Trained Offline
- We train the model exactly once and then use that trained model for a while.
- Easy to build and test - use batch train & test, iterate until good.
- Still requires monitoring of inputs.
- Easy to let this grow stale.
Dynamic Model – Trained Online
- Continue to feed in training data over time, regularly sync out updated version.
- Use progressive validation rather than batch trining & test.
- Needs monitoring, model rollback & data quarantine capabilities.
- Will adapt to changes, staleness issues avoided.
- Data is continually entering the system and we’re incorporating that data into the model through continuous updates.
Summary
- Static models are easier to build and test.
- Dynamic models adapt to changing data. The world is a highly changeable place.
Types of Inference Process
Offline Inference
- Meaning that you make all possible predictions in a batch, using a MapReduce or something similar. You then write the predictions to an SSTable or Bigtable, and then feed these to a cache/lookup table.
- Upside: don’t need to worry much about cost of inference.
- Upside: can likely use batch quota.
- Upside: can do post-verification on predictions on data before pushing.
- Downside: can only predict things we know about – bad for long tail.
- Downside: update latency likely measured in hours or days.
Online Inference
- Meaning that you predict on demand, using a server.
- Upside: can predict any new item as it comes in – great for long tail.
- Downside: compute intensive, latency sensitive – may imit model complexity.
- Downside: monitoring needs are more intensive.
Data Dependencies
- Reliability
- Versioning
- Necessity
- Correlations
- Feedback Loops
Fairness
Types of Bias
- Reporting Bias
- Automation Bias
- Selection Bias
- Coverage bias
- Non-response bias
- Sampling bias
- Group Attribution Bias
- In-group bias
- Out-group bias
- Implicit Bias
Effective Machine Learning Guidelines
- Keep your first model simple.
- Focus on ensuring data pipeline corrections.
- Use a imple, observable metric for training & evaluation.
- Own and monitor your input features.
- Treat your model configuration as code: review it, check it in.
- Write down the results of all experiments, especially “failures”.
Colab: Intro to Fairness
Note: Cover Picture