Computes information gain of single feature and target vector.
calc_ig(feature, target, len_target, pos_target)
feature | feature vector. |
---|---|
target | target. |
len_target | length of the target vector. |
pos_target | number of positive cases in the target vector. |
A numeric
vector of length 1 representing information gain in nats.
The information gain term is used here (improperly) as a synonym of mutual information. It is defined as: $$IG(X; Y) = \sum_{y \in Y} \sum_{x \in X} p(x, y) \log \left(\frac{p(x, y)}{p(x) p(y)} \right)$$
In biogram package information gain is computed using following relationship: \(IG = E(S) - E(S|F)\)
During calculations \(0 \log 0 = 0\). For a justification see References.
The function was designed to be afast subroutine of
calc_criterion
and might be cumbersome if directly called by a user.
Cover TM, Thomas JA Elements of Information Theory, 2nd Edition Wiley, 2006.
tar <- sample(0L:1, 100, replace = TRUE) feat <- sample(0L:1, 100, replace = TRUE) calc_ig(feat, tar, 100, sum(tar))#> [1] 0.0007015338