Select all of the following statements which are TRUE.
Ridge regression model to predict tenure in right censored data, we are likely to underestimate it because the tenure observed in our data is shorter than what it would be in reality.Why spend a whole lecture on this?
What additional information would you need to evaluate the validity of this claim?
predict_proba=0.9 ≠ 90% certain)Pick one scenario: Discuss 2 negative consequences and 1 thing you’d do to prevent them.
Go to this Google doc: https://tinyurl.com/5n8xf5yj
Explanation 1: https://tinyurl.com/msk2cfkb
Explanation 2: https://tinyurl.com/mt2z9ey5
Explanation 1: Machine learning algorithms, like an airplane’s cockpit, typically involve a bunch of knobs and switches that need to be set.
Explanation 2: Grid search is the process of performing hyper parameter tuning in order to determine the optimal values for a given model.
The effectiveness of these different statements depend on your audience.
In the previous explanations, which one represented a bottom-up explanation and which one a top-down explanation?
The hidden structure in the first explanation
Effective explanations often use the same example throughout the text and code. This helps readers follow the line of reasoning.
RandomizedSearchCV).The first explanation shows the output of the code, whereas the second does not. This is easy to do and makes a big difference.
If it doesn’t serve the audience, it’s noise.
Which one is poor and which one is effective? Why?
Communication 1
“I built a model to predict next week’s avocado prices. The ridge model had an RMSE of 0.79, but the random forest performed better with tuned hyperparameters. The cross-validation score improved after adding lag features. We should use the random forest.”
Communication 2
“Our avocado price forecast reduces weekly price uncertainty by 15%. This lets the procurement team lock in contracts earlier and avoid overpaying during high-volatility weeks, saving an estimated $45k per month.
To deploy: we need 2 days to automate data updates and a weekly accuracy review.
Risk: model performance drops during holiday spikes. Here’s our mitigation plan.”
❌ Poor communication:
“I built a model to predict next week’s avocado prices. The ridge model had an RMSE of 0.79, but the random forest performed better with tuned hyperparameters. The cross-validation score improved after adding lag features. We should use the random forest.”
Result: The manager doesn’t know why this matters, how it affects decisions, or what to do next. No adoption.
✅ Effective reframe:
“Our avocado price forecast reduces weekly price uncertainty by 15%. This lets the procurement team lock in contracts earlier and avoid overpaying during high-volatility weeks, saving an estimated $45k per month.
To deploy: we need 2 days to automate data updates and a weekly accuracy review.
Risk: model performance drops during holiday spikes. Here’s our mitigation plan.”
Result: Clear value, operational impact, required effort, and risks. Enables decision-making.
Key difference: Shift from model-centric communication → decision-ready communication.
Complete the SEoI for this course here
Imagine you are tasked with developing a recommender system for YouTube. You possess data on which users clicked on which videos. After spending considerable time building a recommender system using this data, you realize it isn’t producing high-quality recommendations. What could be the reasons for this?
Questions you have to consider:
predict_probaLet’s imagine that the following claim is true:
Vancouver has the highest cost of living of all cities in Canada.
Now let’s consider a few beliefs we could hold:
The part is bold is called a credence. Which belief is better?
But what if it’s actually Toronto that has the highest cost of living in Canada?
Which belief is better now?
We don’t just want to be right. We want to be confident when we’re right and hesitant when we’re wrong.
In our final exam, imagine if, along with your answers, we ask you to also provide a confidence score for each. This would involve rating how sure you are about each answer, perhaps on a percentage scale from 0% (completely unsure) to 100% (completely sure). This method not only assesses your knowledge but also your awareness of your own understanding, potentially impacting the grading process and highlighting areas for improvement. Who supports this idea 😉?
When you call fit for LogisticRegression it has similar preferences:
correct and confident
> correct and hesitant
> incorrect and hesitant
> incorrect and confident
fit it tries to minimize this metric.Consider the following made-up classification example where target (true y) is binary: -1 or 1. The true \(y\) (y_true) and models raw scores (\(w^Tx_i\)) are given to you. You want to figure out how do you want to punish the mistakes made by the current model. How will you punish the model in each case?
| y_true | raw score ($w^Tx_i$) | correct? (yes/no) | confident/hesitant? | punishment | |
|---|---|---|---|---|---|
| 0 | 1 | 10.00 | yes | confident | None |
| 1 | 1 | 0.51 | yes | hesitant | small punishment |
| 2 | 1 | -0.10 | no | hesitant | |
| 3 | 1 | -10.00 | no | confident | |
| 4 | -1 | -12.00 | yes | confident | |
| 5 | -1 | -1.00 | yes | hesistant | |
| 6 | -1 | 0.40 | no | hesitant | |
| 7 | -1 | 18.00 | no | confident |
This chart is attempting to suggest a relationship between childhood MMR vaccination rates and the prevalence of autism spectrum disorders (AD/ASD) across several countries.
Do you see any problems with this visualization?
Visualizing your data and results could be very powerful but at the same time can be misleading if not done properly.
Some examples from Calling BS visualization videos:
Principles of effective communication
Imagine you’ve created a machine learning model and are eager to share it with others. Consider the following scenarios for sharing your model:
https://cpsc330-moment-predictor.onrender.com/
For this demo, each student should click this link to create a new repo in their accounts, then clone that repo locally to follow the steps to deloy an app.
See notes fom class as well as the lecture demo.
Part 1: Supervised learning on tabular data: ML fundamentals, preprocessing and data encoding, a bunch of models, evaluation metrics, feature importances and model transparency, feature selection, hyperparameter optimization
Part 2: Dealing with other non-tabular data types: Clustering, recommender systems, computer vision with pre-trained deep learning models (high level), language data, text preprocessing, embeddings, topic modeling, time series, right-censored data / survival analysis
Part 3: Communication, Ethics, and Deployment
Lots of room for improvement. Here are some things on my mind.
If you want to further develop your machine learning skills: - Practice! - Work on your own projects - Work hard and be consistent.
For each of the scenarios below
| App | Goal |
|---|---|
| QueuePredictor app | Inform callers how long they’ll wait on hold given the current call volume |
| To-doList App | Keep track of the tasks that a user inputs and organize them by date |
| SegmentSphere App | To segment customers to tailor marketing strategies based on purchasing behaviour |
| Video app | Recommend useful videos |
| Dining app | Identify cuisine by a restaurant’s menu |
| Weather app | Calculate precipitation in six hour increments for a geographic region |
| EvoCarShare app | Calculate number of car rentals in four increments at a particular Evo parking spot |
| Pharma app | Understand the effect of a new drug on patient survival time |
That’s all, folks. We made it! Good luck on your final exam! When you get a chance, please let me know what worked for you and what didn’t work for you in this course.
