pip install model-diagnostics<\/code><\/li>\n\n\n\n- R:<\/strong> https:\/\/github.com\/mayer79\/effectplots<\/a>
install.packages(\"effectplots\")<\/code><\/li>\n<\/ul>\n\n\n\nThe functionality is best described by its output:<\/p>\n\n\n\n![\"\"](\"https:\/\/lorentzen.ch\/wp-content\/uploads\/2025\/03\/image.png\")
Python<\/figcaption><\/figure>\n\n\n\n![\"\"](\"https:\/\/lorentzen.ch\/wp-content\/uploads\/2024\/11\/effectplot_r-1024x1024.png\")
R<\/figcaption><\/figure>\n\n\n\nThe plots show different types of feature effects relevant in modeling:<\/p>\n\n\n\n
\n- Average observed: Descriptive effect (also interesting without model).<\/li>\n\n\n\n
- Average predicted: Combined effect of all features. Also called “M Plot” (Apley 2020).<\/li>\n\n\n\n
- Partial dependence: Effect of one feature, keeping other feature values constant (Friedman 2001).<\/li>\n\n\n\n
- Number of observations or sum of case weights: Feature value distribution.<\/li>\n\n\n\n
- R only: Accumulated local effects, an alternative to partial dependence (Apley 2020).<\/em><\/li>\n<\/ul>\n\n\n\n
Both implementations…<\/strong><\/p>\n\n\n\n\n- are highly efficient<\/strong> thanks to {Polars} in Python and {collapse} in R, and work on datasets with millions of observations,<\/li>\n\n\n\n
- support case weights<\/strong> with all their statistics, ideal in insurance applications,<\/li>\n\n\n\n
- calculate average residuals (not shown in the plots above),<\/li>\n\n\n\n
- provide standard deviations\/errors of average observed and bias,<\/li>\n\n\n\n
- allow to switch to Plotly<\/strong> for interactive plots, and<\/li>\n\n\n\n
- are highly customizable<\/strong> (the R package, e.g., allows to collapse rare levels after calculating statistics via the
update()<\/code> method or to sort the features according to main effect importance).<\/li>\n<\/ul>\n\n\n\nIn the spirit of our “Lost In Translation” series, we provide both high-quality Python and R code. We will use the same data and models as in one of our latest posts<\/a> on how to build strong GLMs via ML + XAI.<\/p>\n\n\n\nExample<\/h2>\n\n\n\n
Let’s build a Poisson LightGBM model to explain the claim frequency given six traditional features in a pricing dataset on motor liability claims. 80% of the 1 Mio rows are used for training, the other 20% for evaluation. Hyper-parameters have been slightly tuned (not shown).<\/p>\n\n\n
\n\t\t\t\n\t\t\t\t\n\t\t\t\tR<\/div>\n\t\t\t<\/div>\n\t\t\t\tPython<\/div>\n\t\t\t<\/div><\/div>\n\t\t\t<\/div>\n\t\t\t\n\nlibrary(OpenML)\nlibrary(lightgbm)\n\ndim(df <- getOMLDataSet(data.id = 45106L)$data) # 1000000 7\nhead(df)\n\n# year town driver_age car_weight car_power car_age claim_nb\n# 0 2018 1 51 1760 173 3 0\n# 1 2019 1 41 1760 248 2 0\n# 2 2018 1 25 1240 111 2 0\n# 3 2019 0 40 1010 83 9 0\n# 4 2018 0 43 2180 169 5 0\n# 5 2018 1 45 1170 149 1 1\n\nyvar <- \"claim_nb\"\nxvars <- setdiff(colnames(df), yvar)\n\nix <- 1:800000\ntrain <- df[ix, ]\ntest <- df[-ix, ]\nX_train <- data.matrix(train[xvars])\nX_test <- data.matrix(test[xvars])\n\n# Training, using slightly optimized parameters found via cross-validation\nparams <- list(\n learning_rate = 0.05,\n objective = \"poisson\",\n num_leaves = 7,\n min_data_in_leaf = 50,\n min_sum_hessian_in_leaf = 0.001,\n colsample_bynode = 0.8,\n bagging_fraction = 0.8,\n lambda_l1 = 3,\n lambda_l2 = 5,\n num_threads = 7\n)\n\nset.seed(1)\n\nfit <- lgb.train(\n params = params,\n data = lgb.Dataset(X_train, label = train$claim_nb),\n nrounds = 300\n)<\/pre><\/div>\n\n<\/div>\n\nimport matplotlib.pyplot as plt\nfrom lightgbm import LGBMRegressor\nfrom sklearn.datasets import fetch_openml\n\ndf = fetch_openml(data_id=45106, parser=\"pandas\").frame\ndf.head()\n\n# year town driver_age car_weight car_power car_age claim_nb\n# 0 2018 1 51 1760 173 3 0\n# 1 2019 1 41 1760 248 2 0\n# 2 2018 1 25 1240 111 2 0\n# 3 2019 0 40 1010 83 9 0\n# 4 2018 0 43 2180 169 5 0\n# 5 2018 1 45 1170 149 1 1\n\n# Train model on 80% of the data\ny = df.pop(\"claim_nb\")\nn_train = 800_000\nX_train, y_train = df.iloc[:n_train], y.iloc[:n_train]\nX_test, y_test = df.iloc[n_train:], y.iloc[n_train:]\n\nparams = {\n \"learning_rate\": 0.05,\n \"objective\": \"poisson\",\n \"num_leaves\": 7,\n \"min_child_samples\": 50,\n \"min_child_weight\": 0.001,\n \"colsample_bynode\": 0.8,\n \"subsample\": 0.8,\n \"reg_alpha\": 3,\n \"reg_lambda\": 5,\n \"verbose\": -1,\n}\n\nmodel = LGBMRegressor(n_estimators=300, **params, random_state=1)\nmodel.fit(X_train, y_train)<\/pre><\/div>\n\n<\/div><\/div>\n\t\t<\/div>\n\n\n<\/h3>\n\n\n\n
Let’s inspect the (main effects) of the model on the test data.<\/p>\n\n\n
\n\t\t\t\n\t\t\t\t\n\t\t\t\tR<\/div>\n\t\t\t<\/div>\n\t\t\t\tPython<\/div>\n\t\t\t<\/div><\/div>\n\t\t\t<\/div>\n\t\t\t\n\nlibrary(effectplots)\n\n# 0.3 s\nfeature_effects(fit, v = xvars, data = X_test, y = test$claim_nb) |>\n plot(share_y = \"all\")<\/pre><\/div>\n\n<\/div>\n\nfrom model_diagnostics.calibration import plot_marginal\n\nfig, axes = plt.subplots(3, 2, figsize=(8, 8), sharey=True, layout=\"tight\")\n\n# 2.3 s\nfor i, (feat, ax) in enumerate(zip(X_test.columns, axes.flatten())):\n plot_marginal(\n y_obs=y_test,\n y_pred=model.predict(X_test),\n X=X_test,\n feature_name=feat,\n predict_function=model.predict,\n ax=ax,\n )\n ax.set_title(feat)\n if i != 1:\n ax.legend().remove()<\/pre><\/div>\n\n<\/div><\/div>\n\t\t<\/div>\n\n\nThe output can be seen at the beginning of this blog post.<\/p>\n\n\n\n
Here some model insights:<\/strong><\/p>\n\n\n\n\n- Average predictions closely match observed frequencies. No clear bias is visible.<\/li>\n\n\n\n
- Partial dependence shows that the year and the car weight almost have no impact (regarding their main effects), while the
driver_age<\/code> and car_power<\/code> effects seem strongest. The shared y axes help to assess these.<\/li>\n\n\n\n- Except for
car_weight<\/code>, the partial dependence curve closely follows the average predictions. This means that the model effect seems to really come from the feature on the x axis, and not of some correlated other feature (as, e.g., with car_weight<\/code> which is actually strongly correlated with car_power<\/code>).<\/li>\n<\/ul>\n\n\n\nFinal words<\/h2>\n\n\n\n\n- Inspecting models has become much relaxed with above functions.<\/li>\n\n\n\n
- The packages used offer much more functionality. Try them out! Or we will show them in later posts ;).<\/li>\n<\/ul>\n\n\n\n
References<\/a><\/h2>\n\n\n\n\n- Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models.<\/em> Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059\u20131086. doi:10.1111\/rssb.12377.<\/li>\n\n\n\n
- Friedman, Jerome H. 2001. Greedy Function Approximation: A Gradient Boosting Machine.<\/em> Annals of Statistics 29 (5): 1189\u20131232. doi:10.1214\/aos\/1013203451.<\/li>\n<\/ol>\n\n\n\n
R script<\/a> , Python notebook<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"This post introduces new Python and R functionality how to get a quick summary of any model.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,17,9],"tags":[10,6,5],"class_list":["post-1774","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-programming","category-statistics","tag-lost-in-translation","tag-python","tag-r"],"featured_image_src":null,"author_info":{"display_name":"Michael Mayer","author_link":"https:\/\/lorentzen.ch\/index.php\/author\/michael\/"},"_links":{"self":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/1774","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/comments?post=1774"}],"version-history":[{"count":10,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/1774\/revisions"}],"predecessor-version":[{"id":1875,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/1774\/revisions\/1875"}],"wp:attachment":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/media?parent=1774"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/categories?post=1774"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/tags?post=1774"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}