All models are NOT created equal: Evaluation of model performance (Adam Stewart)

Earth Observation Summer School 2026 17 Aug 2026 – 21 Aug 2026, Istanbul, Turkey https://opengeohub.org/events/earth-observation-summer-school-2026 In this lecture, we will learn how to quantitatively evaluate the performance of a machine learning model. First, we walk through various cross-validation procedures and train–val–test splitting methods. Then we learn about common regression and classification metrics and how to handle class imbalance. There are many situations in which we would like to evaluate a model: * Given a model, how likely will this model be to make a correct prediction? * Given ten models, which model is best, and when? * When does the model make mistakes, and why? * Will the model work on different kinds of data? Qualitative analysis can be very subjective: * Biased by personal feelings or opinions * Companies/researchers often cherry-pick good results Quantitative analysis is more objective: * Numbers don't lie (when done correctly!) * Ideally, do both qualitative and quantitative analysis