This post shows how filling histograms can be done in very different ways thereby connecting very different areas: from gradient boosted trees to SQL queries to one-hot encoding. Let’s jump into it!<\/p>\n\n\n\n
Modern gradient boosted trees (GBT) like LightGBM<\/a>, XGBoost<\/a> and the HistGradientBoostingRegressor<\/a> of scikit-learn all use two techniques on top of standard gradient boosting:<\/p>\n\n\n\n The filling of the histograms is often the bottleneck when fitting GBTs. While filling a single histogram is very fast, this operation is executed many times: for each boosting round, for each tree split and for each feature. This is the reason why GBT implementations have dedicated routines for it. We look into this operation from different angles.<\/p>\n\n\n\n For the coming (I)Python code snippets to work ( As a starter, we create a small table with two columns: bin index and value of the hessian.<\/p>\n\n\n\n\n
# %%<\/code> indicates a new notebook cell), we need the following imports.<\/p>\n\n\n\nimport duckdb # v0.5.1\nimport matplotlib.pyplot as plt # v.3.6.1\nfrom matplotlib.ticker import MultipleLocator\nimport numpy as np # v1.23.4\nimport pandas as pd # v1.5.0\nimport pyarrow as pa # v9.0.0\nimport tabmat # v3.1.2\n\nfrom sklearn.ensemble._hist_gradient_boosting.histogram import (\n _build_histogram_root,\n) # v1.1.2\nfrom sklearn.ensemble._hist_gradient_boosting.common import (\n HISTOGRAM_DTYPE\n)<\/pre><\/div>\n\n\n\n
Naive Histogram Visualisation<\/h2>\n\n\n\n
def highlight(df):\n if df["bin"] == 0:\n return ["background-color: rgb(255, 128, 128)"] * len(df)\n elif df["bin"] == 1:\n return ["background-color: rgb(128, 255, 128)"] * len(df)\n else:\n return ['background-color: rgb(128, 128, 255)'] * len(df)\n\ndf = pd.DataFrame({"bin": [0, 2, 1, 0, 1], "hessian": [1.5, 1, 2, 2.5, 3]})\ndf.style.apply(highlight, axis=1)<\/pre><\/div>\n\n\n\n