The last two included a deep dive into historic mortality rates<\/a> as well as studying a beautiful regression formula<\/a>.<\/p>\n\n\n\n One of the most used datasets to teach regression is the diamonds<\/em> dataset. It describes 54’000 diamonds by<\/p>\n\n\n\n The dataset is readily available, e.g. in<\/p>\n\n\n\n Question: <\/strong><\/em>How many times did you use diamonds data to compare regression techniques like random forests and gradient boosting? <\/strong><\/em><\/p>\n\n\n\n Answer:<\/em><\/strong> Probably a lot!<\/p>\n\n\n\n We recently stumbled over a curious fact regarding that dataset. 26% of the diamonds are duplicates<\/strong> regarding In order to illustrate the issue, let us add the two auxilary variables<\/p>\n\n\n\n to the dataset and consider a couple of examples. You can view both R and Python code – but the specific output will differ because language specific naming of group ids.<\/p>\n\n\n Of course, having the same id does not necessarily mean that the rows really describe the same diamond. Why would this be problematic? <\/strong><\/p>\n\n\n\n In the presence of grouped data, standard validation techniques tend to reward overfitting.<\/p><\/blockquote>\n\n\n\n This becomes immediately clear having in mind the 2.01 carat diamond from Table 3. Standard cross-validation (CV) uses random or stratified sampling and would scatter the six rows of that diamond across multiple CV folds. Highly flexible algorithms like random forests or nearest-neighbour regression could exploit this by memorizing the price of this diamond in-fold and do very well out-of-fold. As a consequence, the stated CV performance would be too good and the choice of the modeling technique and its hyperparameters suboptimal.<\/p>\n\n\n\n With grouped data, a good approach is often to randomly sample the whole group<\/em> instead of single rows. Using such grouped splitting<\/strong> ensures that all rows in the same group would end up in the same fold, removing the above described tendency to overfit.<\/p>\n\n\n\n Note 1.<\/strong> In our case of duplicates, a simple alternative to grouped splitting would be to remove the duplicates altogether. However, the occurrence of duplicates is just one of many situations where grouped or clustered samples appear in reality.<\/p>\n\n\n\n Note 2. <\/strong>The same considerations not only apply to cross-validation but also to simple train\/validation\/test splits.<\/p>\n\n\n\n What does this mean regarding our diamonds dataset? Using five-fold CV, we will estimate the true root-mean-squared error (RMSE) of a random forest predicting log price by the four “C”. We run this experiment twice: one time, we create the folds by random splitting and the other time by grouped splitting. How heavily will the results from random splitting be biased?<\/p>\n\n\n The RMSE (11%) of grouped CV is 8%-10% higher<\/strong> than of random CV (10%). The standard technique therefore seems to be considerably biased<\/strong>.<\/p>\n\n\n\n The Python notebook and R code can be found at:<\/p>\n\n\n\n “R <-> Python” continued… A Curious Fact on the Diamonds Dataset<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[10,6,5],"class_list":["post-258","post","type-post","status-publish","format-standard","hentry","category-statistics","tag-lost-in-translation","tag-python","tag-r"],"featured_image_src":null,"author_info":{"display_name":"Michael Mayer","author_link":"https:\/\/lorentzen.ch\/index.php\/author\/michael\/"},"_links":{"self":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/258","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/comments?post=258"}],"version-history":[{"count":74,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/258\/revisions"}],"predecessor-version":[{"id":362,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/258\/revisions\/362"}],"wp:attachment":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/media?parent=258"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/categories?post=258"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/tags?post=258"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Diamonds data<\/h2>\n\n\n\n
price<\/code>, <\/li>carat<\/code>, color<\/code>, cut<\/code>, clarity<\/code>),<\/li>table<\/code>, depth<\/code>, x<\/code>, y<\/code>, and z<\/code>.<\/li><\/ul>\n\n\n\nggplot2<\/code>,<\/li>plotnine<\/code>,<\/li>The curious fact<\/h2>\n\n\n\n
price<\/code> and the four “C” variables. Within duplicates, the perspective variables table<\/code>, depth<\/code>, x<\/code>, y<\/code>, and z<\/code> would differ as if a diamond had been measured from different angles.<\/p>\n\n\n\nid<\/code>: group id of diamonds with identical price and four “C”, and<\/li>id_size<\/code>: number of rows in that id<\/li><\/ul>\n\n\n\nlibrary(tidyverse)\n\n# We add group id and its size\ndia <- diamonds %>% \n group_by(carat, cut, clarity, color, price) %>% \n mutate(id = cur_group_id(),\n id_size = n()) %>% \n ungroup() %>% \n arrange(id)\n\n# Proportion of duplicates\n1 - max(dia$id) \/ nrow(dia) # 0.26\n\n# Some examples\ndia %>% \n filter(id_size > 1) %>%\n head(10)\n\n# Most frequent\ndia %>% \n arrange(-id_size) %>% \n head(.$id_size[1])\n\n# A random large diamond appearing multiple times\ndia %>% \n filter(id_size > 3) %>% \n arrange(-carat) %>% \n head(.$id_size[1])<\/pre><\/div>\n\n<\/div>
import numpy as np\nimport pandas as pd\nfrom plotnine.data import diamonds\n\n# Variable groups\ncat_vars = [\"cut\", \"color\", \"clarity\"]\nxvars = cat_vars + [\"carat\"]\nall_vars = xvars + [\"price\"]\n\nprint(\"Shape: \", diamonds.shape)\n\n# Add id and id_size\ndf = diamonds.copy()\ndf[\"id\"] = df.groupby(all_vars).ngroup()\ndf[\"id_size\"] = df.groupby(all_vars)[\"price\"].transform(len)\ndf.sort_values(\"id\", inplace=True)\n\nprint(f'Proportion of dupes: {1 - df[\"id\"].max() \/ df.shape[0]:.0%}')\n\nprint(\"Random examples\")\nprint(df[df.id_size > 1].head(10))\n\nprint(\"Most frequent\")\nprint(df.sort_values([\"id_size\", \"id\"]).tail(13))\n\nprint(\"A random large diamond appearing multiple times\")\ndf[df.id_size > 3].sort_values(\"carat\").tail(6)<\/pre><\/div>\n\n<\/div><\/div>\n\t\t<\/div>\n\n\n![\"\"](\"https:\/\/lorentzen.ch\/wp-content\/uploads\/2021\/04\/image-8.png\")
price<\/code> (Python output).<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/lorentzen.ch\/wp-content\/uploads\/2021\/04\/image-14.png\")
![\"\"](\"https:\/\/lorentzen.ch\/wp-content\/uploads\/2021\/04\/image-11.png\")
price<\/code> and the four “C”s could coincide purely by chance. Nevertheless: there are exactly six diamonds of 2.01 carat and a price of 16,778 USD in the dataset. And they all have the same color, cut and clarity. This cannot be coincidence!<\/p>\n\n\n\nEvaluation<\/h2>\n\n\n\n
library(ranger)\nlibrary(splitTools) # one of our packages on CRAN\n\nset.seed(8325)\n\n# We model log(price)\ndia <- dia %>% \n mutate(y = log(price))\n\n# Helper function: calculate rmse\nrmse <- function(obs, pred) {\n sqrt(mean((obs - pred)^2))\n}\n\n# Helper function: fit model on one fold and evaluate\nfit_on_fold <- function(fold, data) {\n fit <- ranger(y ~ carat + cut + color + clarity, data = data[fold, ])\n rmse(data$y[-fold], predict(fit, data[-fold, ])$pred)\n}\n \n# 5-fold CV for different split types\ncross_validate <- function(type, data) {\n folds <- create_folds(data$id, k = 5, type = type)\n mean(sapply(folds, fit_on_fold, data = dia))\n}\n\n# Apply and plot\n(results <- sapply(c(\"basic\", \"grouped\"), cross_validate, data = dia))\nbarplot(results, col = \"orange\", ylab = \"RMSE by 5-fold CV\")<\/pre><\/div>\n\n<\/div>from sklearn.ensemble import RandomForestRegressor\nfrom sklearn.model_selection import cross_val_score, GroupKFold, KFold\nfrom sklearn.metrics import make_scorer, mean_squared_error\nimport seaborn as sns\n\nrmse = make_scorer(mean_squared_error, squared=False)\n\n# Prepare y, X\ndf = df.sample(frac=1, random_state=6345)\ny = np.log(df.price)\nX = df[xvars].copy()\n\n# Correctly ordered integer encoding\nX[cat_vars] = X[cat_vars].apply(lambda x: x.cat.codes)\n\n# Cross-validation\nresults = {}\nrf = RandomForestRegressor(n_estimators=500, max_features=\"sqrt\", \n min_samples_leaf=5, n_jobs=-1)\nfor nm, strategy in zip((\"basic\", \"grouped\"), (KFold, GroupKFold)):\n results[nm] = cross_val_score(\n rf, X, y, cv=strategy(), scoring=rmse, groups=df.id\n ).mean()\nprint(results)\n\nres = pd.DataFrame(results.items())\nsns.barplot(x=0, y=1, data=res);<\/pre><\/div>\n\n<\/div><\/div>\n\t\t<\/div>\n\n\n![\"\"](\"https:\/\/lorentzen.ch\/wp-content\/uploads\/2021\/04\/image-12-1024x507.png\")
Final remarks<\/h2>\n\n\n\n