Need help with Imbalance-XGBoost?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

jhwjhw0123
166 Stars 36 Forks MIT License 53 Commits 6 Opened issues

#### Description

XGBoost for label-imbalanced data: XGBoost with weighted and focal loss functions

!
?

# Imbalance-Xgboost

This software includes the codes of Weighted Loss and Focal Loss [1] implementations for Xgboost 2 in binary classification problems. The principal reason for us to use Weighted and Focal Loss functions is to address the problem of label-imbalanced data. The original Xgboost program provides a convinient way to customize the loss function, but one will be needing to compute the first and second order derivatives to implement them. The major contribution of the software is the drivation of the gradients and the implementations of them.

## Software Release

The project has been posted on github for several months, and now a correponding API on Pypi is released. Special thanks to @icegrid and @shaojunchao for help correct errors in the previous versions. The codes are now updated to version 0.7 and it now allows users to specify the weighted parameter \alpha and focal parameter \gamma outside the script. Also it supports higher version of XGBoost now.

From version 0.7.0 on Imbalance-XGBoost starts to support higher versions of XGBoost and removes supports of versions earlier than 0.4a30(XGBoost>=0.4a30). This contradicts with the previous requirement of XGBoost<=0.4a30. Please choose the version fits your system accordingly.

## Installation

Installing with Pypi will be easiest way, you can run:

pip install imbalance-xgboost


If you have multiple versions of Python, make sure you're using Python 3 (run with

pip3 install imbalance-xgboost
). Currently, the program only supports Python 3.5 and 3.6.

The package has hard depedency on numpy, sklearn and xgboost.

## Usage

To use the wrapper, one needs to import imbalance_xgboost from module imxgboost.imbalance_xgb. An example is given as bellow:

from imxgboost.imbalance_xgb import imbalance_xgboost as imb_xgb


The specific loss function could be set through special_objective parameter. Specificly, one could construct a booster with:

Python
xgboster = imb_xgb(special_objective='focal')

for focal loss and
Python
xgboster = imb_xgb(special_objective='weighted')

for weighted loss. The prarameters $\alpha$ and $\gamma$ can be specified by giving a value when constructing the object. In addition, the class is designed to be compatible with scikit-learn package, and you can treat it as a sk-learn classifier object. Thus, it will be easy to use methods in Sklearn such as GridsearchCV to perform grid search for the parameters of focal and weighted loss functions.
Python
from sklearn.model_selection import GridSearchCV
xgboster_focal = imb_xgb(special_objective='focal')
xgboster_weight = imb_xgb(special_objective='weighted')
CV_focal_booster = GridSearchCV(xgboster_focal, {"focal_gamma":[1.0,1.5,2.0,2.5,3.0]})
CV_weight_booster = GridSearchCV(xgboster_weight, {"imbalance_alpha":[1.5,2.0,2.5,3.0,4.0]})

The data fed to the booster should be of numpy type and following the convention of:
x: [nData, nDim]
y: [nData,]
In other words, the xinput should be row-major and labels should be flat.
And finally, one could fit the data with Cross-validation and retreive the optimal model:
Python CV
focalbooster.fit(records, labels) CVweightbooster.fit(records, labels) optfocalbooster = CVfocalbooster.bestestimator_ optweightbooster = CVweightbooster.bestestimator
After getting the optimal booster, one will be able to make predictions. There are following methods to make predictions with imabalnce-xgboost:
Method predict

Python rawoutput = optfocalbooster.predict(datax, y=None)
This will return the value of 'zi' before applying sigmoid.
Method predict_sigmoid

Python sigmoidoutput = optfocalbooster.predictsigmoid(datax, y=None) 
This will return the \hat{y} value, which is p(y=1|x) for 2-lcass classification.
Method
predict
determine


Python
class_output = opt_focal_booster.predict_determine(data_x, y=None)

This will return the predicted logit, which 0 or 1 in the 2-class scenario.
Method
predicttwoclass


Python
prob_output = opt_focal_booster.predict_two_class(data_x, y=None)

This will return the predicted probability of 2 classes, in the form of [nData * 2]. The first column is the probability of classifying the datapoint to 0 and the second column is the prob of classifying as 1.
To assistant the evluation of classification results, the package provides a score function
scoreevalfunc()
with multiple metrics. One can use
makescorer()
method in sk-learn and
functools
to specify the evaluation score. The method will be compatible with sk-learn cross validation and model selection processes.

Python import functools from sklearn.metrics import make
scorer from sklearn.modelselection import LeaveOneOut, crossvalidate

# retrieve the best parameters

xgboostoptparam = CVfocalbooster.bestparams

# instantialize an imbalance-xgboost instance

xgboostopt = imbxgb(specialobjective='focal', **xgboostopt_param)

# initialize the splitter

loo_splitter = LeaveOneOut()

# 'mode' can be [\'accuracy\', \'precision\',\'recall\',\'f1\',\'MCC\']

scoreevalfunc = functools.partial(xgboostopt.scoreeval_func, mode='accuracy')

# Leave-One cross validation

looinfodict = crossvalidate(xgboostopt, X=x, y=y, cv=loosplitter, scoring=makescorer(scoreevalfunc))

In the new version, we can also collect the information of the confusion matrix through the correct_eval_func provided. This enables the users to evluate the metrics like accuracy, precision, and recall for the average/overall test sets in the cross-validation process.

Python

# 'mode' can be ['TP', 'TN', 'FP', 'FN']

TPevalfunc = functools.partial(xgboostopt.scoreevalfunc, mode='TP') TNevalfunc = functools.partial(xgboostopt.scoreevalfunc, mode='FP') FPevalfunc = functools.partial(xgboostopt.scoreevalfunc, mode='TN') FNevalfunc = functools.partial(xgboostopt.scoreevalfunc, mode='FN')

# define the score function dictionary

scoredict = {'TP': makescorer(TPevalfunc), 'FP': makescorer(TNevalfunc), 'TN': makescorer(FPevalfunc), 'FN': makescorer(FNeval_func)}

# Leave-One cross validation

looinfodict = crossvalidate(xgboostopt, X=x, y=y, cv=loosplitter, scoring=scoredict) overalltp = np.sum(looinfodict['testTP']).astype('float')  More soring function may be added in later versions.

## Theories and derivatives

You don't have to understand the equations if you find they are hard to grasp, you can simply use it with the API. However, for the purpose of understanding, the derivatives of the two loss functions are listed.
For both of the loss functions, since the task is 2-class classification, the activation would be sigmoid:
$y_{i}&space;=&space;\frac{1}{1+\text{exp}(-z_{i})}$
And bellow the two types of loss will be discussed respectively.

### 1. Weighted Imbalance (Cross-entropoy) Loss

And combining with $\hat{y}$, which are the true labels, the weighted imbalance loss for 2-class data could be denoted as:
$l_{w}&space;=&space;-\sum_{i=1}^{m}(\alpha\hat{y}_{i}\text{log}(y_{i})+(1-\hat{y}_{i})\text{log}(1-y_{i})))$
Where $\alpha$ is the 'imbalance factor'. And $\alpha$ value greater than 1 means to put extra loss on 'classifying 1 as 0'.
$\frac{\partial&space;L_{w}}{\partial&space;z_{i}}&space;=&space;-\alpha^{\hat{y}_{i}}(\hat{y}_{i}-y_{i})$
And the second order gradient would be:
$\frac{\partial&space;L_{w}^{2}}{\partial^{2}&space;z_{i}}&space;=&space;\alpha^{\hat{y}_{i}}(1-y_{i})(y_{i})$

### 2. Focal Loss

The focal loss is proposed in [1] and the expression of it would be:
$\dpi{150}&space;L_{w}&space;=&space;-\sum_{i=1}^{m}\hat{y}_{i}(1-y_{i})^{\gamma}\text{log}(y_{i})&space;+&space;(1-\hat{y}_{i})y_{i}^{\gamma}\text{log}(1-y_{i})$
The first order gradient would be:
$\dpi{150}&space;\small&space;{\frac{\partial&space;L_{w}}{\partial&space;z_i}&space;=&space;\gamma(y_i+\hat{y}_{i}-1)(\hat{y}_{i}+(-1)^{\hat{y}}y_{i})^{\gamma}\text{log}(1-\hat{y}_{i}-(-1)^{\hat{y}_{i}}y_{i})&space;+&space;(-1)^{\hat{y}_i}(\hat{y}_{i}+(-1)^{\hat{y}_i}y_{i})^{\gamma+1}}$
And the second order gradient would be a little bit complex. To simplify the expression, we firstly denotes the terms in the 1-st order gradient as the following notations:
$\dpi{150}&space;\begin{cases}&space;g_1&space;=&space;y_{i}(1-y_{i})\\&space;g_2=&space;\hat{y}_i&space;+&space;(-1)^{\hat{y}_i}y_{i}\\&space;g_3&space;=&space;y_i&space;+&space;\hat{y}_{i}-1\\&space;g_4&space;=&space;1-\hat{y}_i-(-1)^{\hat{y}_i}y_i\\&space;g_5&space;=&space;\hat{y}_i&space;+&space;(-1)^{\hat{y}_i}&space;y_i&space;\end{cases}$
Using the above notations, the 1-st order drivative will be:
$\dpi{150}&space;\large&space;\frac{\partial&space;L_w}{\partial&space;z_i}&space;=&space;\gamma&space;g_3&space;g_2^{\gamma}&space;\text{log}(g_4)&space;+&space;(-1)^{\hat{y_i}}g_5^{\gamma&space;+&space;1}$
Then the 2-nd order derivative will be:
$\dpi{150}&space;\frac{\partial^{2}&space;L}{\partial&space;z_{i}^{2}}&space;=&space;g_{1}\{\gamma[(g_2^{\gamma}+\gamma&space;(-1)^{\hat{y}_{i}}g_3&space;g_2^{\gamma&space;-&space;1})\text{log}(g_4)-\frac{(-1)^{\hat{y}_i}g_3&space;g_2^{\gamma}}{g_4}]&space;+&space;(\gamma+1)g_5^{\gamma}\}$

## Paper Citation

If you use this package in your research please cite our paper:

@misc{wang2019imbalancexgboost,
title={Imbalance-XGBoost: Leveraging Weighted and Focal Losses for Binary Label-Imbalanced Classification with XGBoost},
author={Chen Wang and Chengyuan Deng and Suzhen Wang},
year={2019},
eprint={1908.01672},
archivePrefix={arXiv},
primaryClass={cs.LG}
}