Data Types

Data Types

In summary, nominal variables are used to “name,” or label a series of values. Ordinal scales provide good information about the order of choices, such as in a customer satisfaction survey. Interval scales give us the order of values + the ability to quantify the difference between each one. Finally, Ratio scales give us the ultimate–order, interval values, plus the ability to calculate ratios since a “true zero” can be defined.

https://www.mymarketresearchmethods.com/types-of-data-nominal-ordinal-interval-ratio/

Gradient Tree Boosting

A More General View of Ensembles

Now that we have know about

A More General View of Ensembles

People realized that the very successful Boosting method was in essence
Boosting = a very general meta-algorithm for optimization of the mapping function from input variables to output target variables.

This algorithm chooses multiple weak functions that are combined together, just as the ensemble of decision trees are for Random Forests.

What is the Gradient Though?

We’ve already used them

Doing Better

This idea can then be generalized so that each new weak learner is explicitely treated as a function that points directly away from the gradient of the current combined function.

Gradient Tree Boosting

Given some tree based ensemble model then, represented as a function

$$T_i(X)\rightarrow Y$$

Gradient Tree Boosting

$$L(Y, T(X)) = \sum_i Y-T_i(X)$$

Gradient Tree Boosting

Gradient Tree Boosting explicitly uses the gradient

$$\nabla L(Y,Ti(X))=[ \nabla{w_i} L(Y,T^{w_i}_i (X))]$$

of the loss function of each tree to fit a new tree

$$h(X)= Ti(X) - \sum_i \nabla{T_i}L(Y,T_i(X))$$

and then add it to the ensemble.

There is also further optimization of weighting functions for each tree and various regularization methods which can be done.

The popular algorithm XGBoost\cite{xgboost} implements approach.