Select all of the following statements which are TRUE.
Ridge regression model to predict tenure in right censored data, we are likely to underestimate it because the tenure observed in our data is shorter than what it would be in reality.Why spend a whole lecture on this?
What additional information would you need to evaluate the validity of this claim?
predict_proba=0.9 ≠ 90% certain)Pick one scenario: Discuss 2 negative consequences and 1 thing you’d do to prevent them.
Go to this Google doc: https://tinyurl.com/5n8xf5yj
Explanation 1: https://tinyurl.com/msk2cfkb
Explanation 2: https://tinyurl.com/mt2z9ey5
Explanation 1: Machine learning algorithms, like an airplane’s cockpit, typically involve a bunch of knobs and switches that need to be set.
Explanation 2: Grid search is the process of performing hyper parameter tuning in order to determine the optimal values for a given model.
The effectiveness of these different statements depend on your audience.
In the previous explanations, which one represented a bottom-up explanation and which one a top-down explanation?
The hidden structure in the first explanation
Effective explanations often use the same example throughout the text and code. This helps readers follow the line of reasoning.
RandomizedSearchCV).The first explanation shows the output of the code, whereas the second does not. This is easy to do and makes a big difference.
If it doesn’t serve the audience, it’s noise.
Which one is poor and which one is effective? Why?
Communication 1
“I built a model to predict next week’s avocado prices. The ridge model had an RMSE of 0.79, but the random forest performed better with tuned hyperparameters. The cross-validation score improved after adding lag features. We should use the random forest.”
Communication 2
“Our avocado price forecast reduces weekly price uncertainty by 15%. This lets the procurement team lock in contracts earlier and avoid overpaying during high-volatility weeks, saving an estimated $45k per month.
To deploy: we need 2 days to automate data updates and a weekly accuracy review.
Risk: model performance drops during holiday spikes. Here’s our mitigation plan.”
❌ Poor communication:
“I built a model to predict next week’s avocado prices. The ridge model had an RMSE of 0.79, but the random forest performed better with tuned hyperparameters. The cross-validation score improved after adding lag features. We should use the random forest.”
Result: The manager doesn’t know why this matters, how it affects decisions, or what to do next. No adoption.
✅ Effective reframe:
“Our avocado price forecast reduces weekly price uncertainty by 15%. This lets the procurement team lock in contracts earlier and avoid overpaying during high-volatility weeks, saving an estimated $45k per month.
To deploy: we need 2 days to automate data updates and a weekly accuracy review.
Risk: model performance drops during holiday spikes. Here’s our mitigation plan.”
Result: Clear value, operational impact, required effort, and risks. Enables decision-making.
Key difference: Shift from model-centric communication → decision-ready communication.
Let’s take a 5-min break
Imagine you are tasked with developing a recommender system for YouTube. You possess data on which users clicked on which videos. After spending considerable time building a recommender system using this data, you realize it isn’t producing high-quality recommendations. What could be the reasons for this?
Questions you have to consider:
predict_probaLet’s imagine that the following claim is true:
Vancouver has the highest cost of living of all cities in Canada.
Now let’s consider a few beliefs we could hold:
The part is bold is called a credence. Which belief is better?
But what if it’s actually Toronto that has the highest cost of living in Canada?
Which belief is better now?
We don’t just want to be right. We want to be confident when we’re right and hesitant when we’re wrong.
In our final exam, imagine if, along with your answers, we ask you to also provide a confidence score for each. This would involve rating how sure you are about each answer, perhaps on a percentage scale from 0% (completely unsure) to 100% (completely sure). This method not only assesses your knowledge but also your awareness of your own understanding, potentially impacting the grading process and highlighting areas for improvement. Who supports this idea 😉?
When you call fit for LogisticRegression it has similar preferences:
correct and confident
> correct and hesitant
> incorrect and hesitant
> incorrect and confident
fit it tries to minimize this metric.Consider the following made-up classification example where target (true y) is binary: -1 or 1. The true \(y\) (y_true) and models raw scores (\(w^Tx_i\)) are given to you. You want to figure out how do you want to punish the mistakes made by the current model. How will you punish the model in each case?
| y_true | raw score ($w^Tx_i$) | correct? (yes/no) | confident/hesitant? | punishment | |
|---|---|---|---|---|---|
| 0 | 1 | 10.00 | yes | confident | None |
| 1 | 1 | 0.51 | yes | hesitant | small punishment |
| 2 | 1 | -0.10 | no | hesitant | |
| 3 | 1 | -10.00 | no | confident | |
| 4 | -1 | -12.00 | yes | confident | |
| 5 | -1 | -1.00 | yes | hesistant | |
| 6 | -1 | 0.40 | no | hesitant | |
| 7 | -1 | 18.00 | no | confident |
This chart is attempting to suggest a relationship between childhood MMR vaccination rates and the prevalence of autism spectrum disorders (AD/ASD) across several countries.
Do you see any problems with this visualization?
Visualizing your data and results could be very powerful but at the same time can be misleading if not done properly.
Some examples from Calling BS visualization videos:
Principles of effective communication
