Feature Selection

Filter Based

We specify some metric and based on that filter features.
- chi-square test
- fisher score
- correlation coefficient
- variance threshold

Wrapper-based

Wrapper methods consider the selection of a set of features as a search problem.
- Sequential Feature Selection
- Forward/Stepwise/Backward Selection

Embedded

Embedded methods use algorithms that have built-in feature selection methods.
- Lasso
- Tree based models

The procedure starts with an empty set of features [reduced set]. The best of the original features is determined and added to the reduced set. At each subsequent iteration, the best of the remaining original attributes is added to the set.

The procedure starts with the full set of attributes. At each step, it removes the worst attribute remaining in the set.

Greedy procedure where, at each iteration, we choose the best new feature to add to our selected features based on a cross-validation score.
That is, we start with 0 features and choose the best single feature with the highest score.
The procedure is repeated until we reach the desired number of selected features.

Augment with noisy data
Can apply this approach to Tree based models: XGBoost, DecisionTree, RandomForrest
The idea is that we can inject noisy data features into our input training dataset when training model
Here we can perform cross-validation with n-folds where at each n-fold inject noisy features in each iteration and determine the threshold in which the first noisy feature is selected when computing feature importance.
We select the raw features that are less than this threshold and append to a list.
We do this iteratively for each fold and then take a set of the final appended list.
This approach can help minimize model overfitting.