In this post, I’d like to tell the story of my journey into the open source world of Python with a focus on scikit-learn. My hope is that it encourages others to start or to keep contributing and have endurance for bigger picture changes.<\/p>\n\n\n
Back in 2015\/2016, I was working as a non-life pricing actuary. The standard vendor desktop applications we used for generalized linear models (GLM) had problems of system discontinuities, manual error prone steps and the lack of modern machine learning capabilities (not even out-of-sample model comparison).<\/p>\n\n\n\n
Python was then on the rise for data science. Numpy, scipy and pandas had laid the foundations, then came deep learning alias neural net frameworks leading to tensorflow and pytorch. XGBoost was also a game changer visible in Kaggle competition leaderboards. All those projects came as open source with thriving communities and possibilities to contribute.<\/p>\n\n\n\n
While the R base package always comes with splendid dataframes (I guess they invented it) and battle proven GLMs out of the box, the Python site for GLMs was not that well developed. So I started with GLMs in statsmodels<\/a> and generalized linear mixed models (a.k.a. hierarchical or multilevel models) in pymc<\/a> (then called pymc3). My first open source contributions in the Python world were small issues in statsmodels and a little later the bug report pymc#2640<\/a> about memory alignment issues which was caused by joblib#563<\/a>.<\/p>\n\n\n\n To my great surprise the famous machine learning library scikit-learn<\/a> did not have GLMs, only penalized linear models and logistic regression, but no Poisson or Gamma GLMs which are essential in non-life insurance pricing. Fortunately, I was not the first one to notice this lack. There was already an open issue scikit-learn#5975<\/a> with many people asking for this feature. Just nobody had contributed a pull request (PR) yet. <\/p>\n\n\n\n That’s when I said to myself: It should not fail just because no one implements it. I really like open source and gained some programming experience during my PhD in particle physics, mainly C++. Eventually, I boldly (because I was still a newbie) opened the PR scikit-learn#9405<\/a> in summer 2017.<\/p>\n\n\n\n This PR turned out to be essential for the development of GLMs and for becoming a scikit-learn core developer. I dare say that I almost got crazy trying to convince the core developers that GLMs are really that useful for supervised machine learning and that GLMs should land in scikit-learn. In retrospective, this was the hardest part and it took me almost 2 years of patience and repeating my arguments, some examples comments are given below:<\/p>\n\n\n comment example 1<\/p>\n\t\t\t “I can only repeat myself: I’d prefer to have this functionality in scikit-learn for several reasons (your review, opinion and ideas, very official\/trustworthy library, more efficient maintainance, effort to release this pr as its own library, \u2026). comment example 2<\/p>\n\t\t\t “I see it a bit different. Scikit-Learn like R glm and glmnet is trusted world-wide and can be used in many companies, whereas it might be difficult to get any of the existing GLM libraries on pypi (h2o excluded) into production (no offense intended). That being said, I’d like to return the question and ask you: What exactly has to be fulfilled in order for a GLM PR to be merged into scikit-learn? Once that is clarified, I’ll think about starting a collaboration for this.”<\/em> link<\/a><\/p>\n\n<\/div>\n\t\t<\/div>\n\n comment example 3<\/p>\n\t\t\t …<\/em><\/p>\n\n\n\n guidance – maintenance<\/strong> miscellaneous<\/em><\/strong><\/p>\n\n\n\n …<\/em><\/p>\n\n\n\n further steps<\/strong> [link<\/a>]<\/p>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n\n As I wanted to demonstrate the full utility of GLMs, this PR had become much too large for review and inclusion: +4000 lines of code with several solvers, penalty matrices, 3 examples, a lot of documentation and good test coverage (and a lot of things I would do differently today).<\/p>\n\n\n\n The conclusion was to carve out a minimal GLM implementation using the L-BFGS solver of scipy. This way, I met Roman Yurchak<\/a> with whom it was a pleasure to work with. It took a little \ud83c\udde8\ud83c\uddedSwiss chocolate\ud83c\udf6b incentive to finally get scikit-learn#14300<\/a> (still +2900 loc) reviewed and merged in spring 2020. Almost 3 years after opening my original PR, it was released in scikit-learn version 0.23<\/a>!<\/p>\n\n\n\n I guess it was mainly this work and perseverance around GLMs that catched the attention of the core developers and that motivated them to vote for me: In summer 2020, I was invited to become a scikit-learn core developer and gladly accepted.<\/p>\n\n\n\n My work on GLMs was easily extensible to other estimators in the form of loss functions. Again, to my surprise, loss functions, a core element for supervised learning, were re-implemented again and again within scikit-learn. So, based on Roman’s idea in #15123<\/a>, I started a project to unify them, and by unifying also extending several tree estimator classes with poisson and gamma losses (and making existing ones more stable and faster).<\/p>\n\n\n\n As loss functions are such important core components, they have basically 2 major requirements: be numerically stable and fast. That’s why I went with Cython (preferred way for fast code in scikit-learn) in scikit-learn#20567<\/a> and guess which loop it closed? Again, I met segfault errors caused by joblib#563<\/a>. This time, it motivated another core developer to quite an investment in fixing it in joblib#1254<\/a>.<\/p>\n\n\n\n Another story branch is the dedicated GLM Python library glum<\/a>. The authors took my original way too long GLM PR as a starting point and developed one of the most feature rich and fastest GLM implementations out there. This is almost like a dream come true.<\/p>\n\n\n\n A summary of my contributions over those 3 intensive years as scikit-learn core developer are best given in several categories.<\/p>\n\n\n\n A summary of my contributions in terms of code may be:<\/p>\n\n\n\n From the release notes<\/a> and the github PRs<\/a> (where one would miss a few) a more details list of important PRs<\/p>\n\t\t\t <\/p>\n\n<\/div>\n\t\t<\/div>\n<\/div>\n\n\n Among the biggest changes in newer scikit-learn history are two scikit-learn enhancement proposals (SLEP)<\/p>\n\n\n\n For both, I did one of the 2 obligatory reviews. Then, maybe the technically most challenging review I can remember was on:<\/p>\n\n\n\n Keep in mind that review(er)s are by far the scarcest resource of scikit-learn.<\/p>\n\n\n\n I also like to mention PR#25753<\/a> which changed to government to be more inclusive, in particular with voting rights.<\/p>\n\n\n\n Just before the end, a few critical words must be allowed.<\/p>\n\n\n\n One of the greatest rewards of this journey was that I learned a lot, about Python, machine learning, rigorous reviews, CI\/CD, open source communities, endurance. But even more so, I had the pleasure to meet and work with some kind, brilliant and gifted people like Roman Yurchak, Alexandre Gramfort, Olivier Grisel, Thomas Fan, Nicolas Hug, Adrin Jalali and many more. I am really grateful to be a part of something bigger than its parts. <\/p>\n","protected":false},"excerpt":{"rendered":" In this post, I’d like to tell the story of my journey into the open source world of Python with a focus on scikit-learn. My hope is that it encourages others to start or to keep contributing and have endurance for bigger picture changes.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[6],"class_list":["post-1346","post","type-post","status-publish","format-standard","hentry","category-stories","tag-python"],"featured_image_src":null,"author_info":{"display_name":"Christian Lorentzen","author_link":"https:\/\/lorentzen.ch\/index.php\/author\/christian\/"},"_links":{"self":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/1346","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/comments?post=1346"}],"version-history":[{"count":38,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/1346\/revisions"}],"predecessor-version":[{"id":1388,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/posts\/1346\/revisions\/1388"}],"wp:attachment":[{"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/media?parent=1346"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/categories?post=1346"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lorentzen.ch\/index.php\/wp-json\/wp\/v2\/tags?post=1346"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Becoming a scikit-learn core developer<\/h2>\n\n\n\n
To be more explicit for the moment: If it takes longer than the end of 2019 (+-), I’ll consider to release it as separate library.”<\/em> link<\/a><\/p>\n\n<\/div>\n\t\t<\/div>\n\n
As a GLM user on a fairly regular basis, I’d be happy to help as good as I can. Feel free to reach out to me. As to maintenance, I think a unified framework would even lower the burden. I can also imagine to give some support for maintenance.<\/em><\/p>\n\n\n\n\n
Besides further commits to this PR, let me know how I can help you best.<\/em><\/p>\n\n\n\nSummary as core developer<\/h2>\n\n\n\n
Further directions<\/h3>\n\n\n\n
Pull requests<\/h3>\n\n\n\n
\n
LinearModelLoss<\/code> and NewtonSolver<\/code> (newton-cholesky) for GLMs like LogisticRegression<\/code> and PoissonRegressor<\/code> as well as further solver improvements<\/li>\n\n\n\nQuantileRegressor<\/code> (linear quantile regression) and quantile\/pinball loss for HistGradientBoostingRegressor<\/code> (HGBT). BTW, linear quantile regression is much harder than GLM solvers!<\/li>\n\n\n\nSplineTransformer<\/code><\/li>\n\n\n\n\n
\n
Reviewing and steering<\/h3>\n\n\n\n
\n
championed by Thomas Fan<\/a>, implemented in PR#23734<\/a> v1.2, further developments like PR#27315<\/a> for polars in future v1.4<\/li>\n\n\n\n
championed by Adrin Jalali<\/a>, base implementation PR#22083<\/a><\/li>\n<\/ul>\n\n\n\n\n
Lessons learned<\/h3>\n\n\n\n
\n
predict<\/code> implicitly using a 50% threshold instead of returning a predicted probability (what predict_proba<\/code> does) a bit poor. Hard to change!!! At least, PR#26120<\/a> might improve that to some extent.<\/li>\n\n\n\nfit<\/code>, transform<\/code> and predict<\/code>. But the current trend of modern model classes like gradient boosted trees (XGBoost, LightGBM, HGBT) don’t need a preprocessing pipeline anymore, e.g., they can natively deal with categorical features and missing values. But it is hard to pass the information which feature to treat as categorical through a pipeline, see scikit-learn#18894<\/a>.<\/li>\n\n\n\nColumnTransformer<\/code> is just very complicated and prohibits a lot of optimizations (mostly for categoricals)\u2014which is one of the reasons glum is faster.<\/li>\n<\/ul>\n\n\n\n