Package 'SLmetrics'

Description

A generic function for the (normalized) accuracy in classification tasks. Use weighted.accuracy() for the weighted accuracy.

Usage

## S3 method for class 'factor'
accuracy(actual, predicted, ...)

## S3 method for class 'factor'
weighted.accuracy(actual, predicted, w, ...)

## S3 method for class 'cmatrix'
accuracy(x, ...)

## Generic S3 method
accuracy(...)

## Generic S3 method
weighted.accuracy(
...,
w
)

Arguments

Value

A <numeric>-vector of length 1

Definition

Let $\hat{\alpha} \in [0, 1]$ be the proportion of correctly predicted classes. The accuracy of the classifier is calculated as,

$\hat{\alpha} = \frac{\#TP + \#TN}{\#TP + \#TN + \#FP + \#FN}$

Where:

• $\#TP$ is the number of true positives,

• $\#TN$ is the number of true negatives,

• $\#FP$ is the number of false positives, and

• $\#FN$ is the number of false negatives.

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate model
# performance
cat(
  "Accuracy", accuracy(
    actual    = actual,
    predicted = predicted
  ),

  "Accuracy (weigthed)", weighted.accuracy(
    actual    = actual,
    predicted = predicted,
    w         = iris$Petal.Length/mean(iris$Petal.Length)
  ),
  sep = "\n"
)

Description

The auc()-function calculates the area under the curve.

Usage

## S3 method for class 'numeric'
auc(y, x, method = 0L, presorted = TRUE, ...)

## Generic S3 method
auc(
 y,
 x,
 method = 0,
 presorted = TRUE,
 ...
)

Arguments

Value

A <numeric> vector of length 1

Definition

Trapezoidal rule

The trapezoidal rule approximates the integral of a function $f(x)$ between $x = a$ and $x = b$ using trapezoids formed between consecutive points. If we have points $x_0, x_1, \ldots, x_n$ (with $a = x_0 < x_1 < \cdots < x_n = b$) and corresponding function values $f(x_0), f(x_1), \ldots, f(x_n)$, the area under the curve $A_T$ is approximated by:

$A_T \approx \sum_{k=1}^{n} \frac{f(x_{k-1}) + f(x_k)}{2} \bigl[x_k - x_{k-1}\bigr].$

Step-function method

The step-function (rectangular) method uses the value of the function at one endpoint of each subinterval to form rectangles. With the same partition $x_0, x_1, \ldots, x_n$, the rectangular approximation $A_S$ can be written as:

$A_S \approx \sum_{k=1}^{n} f(x_{k-1}) \bigl[x_k - x_{k-1}\bigr].$

See Also

Other Tools: cov.wt.matrix(), preorder(), presort()

Examples

## 1) Ordered x and y pair
x <- seq(0, pi, length.out = 200)
y <- sin(x)

## 1.1) calculate area
ordered_auc <- auc(y = y,  x = x)

## 2) Unordered x and y pair
x <- sample(seq(0, pi, length.out = 200))
y <- sin(x)

## 2.1) calculate area
unordered_auc <- auc(y = y,  x = x)

## 2.2) calculate area with explicit
## ordering
unordered_auc_flag <- auc(
  y = y,
  x = x,
  presorted = FALSE
)

## 3) display result
cat(
  "AUC (ordered x and y pair)", ordered_auc,
  "AUC (unordered x and y pair)", unordered_auc,
  "AUC (unordered x and y pair, with unordered flag)", unordered_auc_flag,
  sep = "\n"
)

Description

A generic function for the (normalized) balanced accuracy. Use weighted.baccuracy() for the weighted balanced accuracy.

Usage

## S3 method for class 'factor'
baccuracy(actual, predicted, adjust = FALSE, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.baccuracy(actual, predicted, w, adjust = FALSE, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
baccuracy(x, adjust = FALSE, na.rm = TRUE, ...)

## Generic S3 method
baccuracy(
  ...,
  adjust = FALSE,
  na.rm  = TRUE
)

## Generic S3 method
weighted.baccuracy(
  ...,
  w,
  adjust = FALSE,
  na.rm  = TRUE
)

Arguments

Value

A numeric-vector of length 1

Definition

Let $\hat{\alpha} \in [0, 1]$ be the proportion of correctly predicted classes. If adjust == false, the balanced accuracy of the classifier is calculated as,

$\hat{\alpha} = \frac{\text{sensitivity} + \text{specificity}}{2}$

otherwise,

$\hat{\alpha} = \frac{\text{sensitivity} + \text{specificity}}{2} \frac{1}{k}$

Where:

• $k$ is the number of classes

• $\text{sensitivity}$ is the overall sensitivity, and

• $\text{specificity}$ is the overall specificity

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate the
# model
cat(
  "Balanced accuracy", baccuracy(
    actual    = actual,
    predicted = predicted
  ),
  
  "Balanced accuracy (weigthed)", weighted.baccuracy(
    actual    = actual,
    predicted = predicted,
    w         = iris$Petal.Length/mean(iris$Petal.Length)
  ),
  sep = "\n"
)

Description

This dataset contains features extracted from the wavelet transform of banknote images, which are used to classify banknotes as authentic or inauthentic. The data originates from the UCI Machine Learning Repository.

Usage

data(banknote)

Format

A list with two components:

features

A data frame with 4 variables: variance, skewness, curtosis, and entropy.

target

A factor with levels "inauthentic" and "authentic" representing the banknote's authenticity.

Details

The data is provided as a list with two components:

features

A data frame containing the following variables:

variance

Variance of the wavelet transformed image.

skewness

Skewness of the wavelet transformed image.

curtosis

Curtosis of the wavelet transformed image.

entropy

Entropy of the image.

target

A factor indicating the authenticity of the banknote. The factor has two levels:

inauthentic

Indicates the banknote is not genuine.

authentic

Indicates the banknote is genuine.

Source

https://archive.ics.uci.edu/dataset/267/banknote+authentication

Description

A generic function for the concordance correlation coefficient. Use weighted.ccc() for the weighted concordance correlation coefficient.

Usage

## S3 method for class 'numeric'
ccc(actual, predicted, correction = FALSE, ...)

## S3 method for class 'numeric'
weighted.ccc(actual, predicted, w, correction = FALSE, ...)

ccc(
 ...,
 correction = FALSE
)

weighted.ccc(
 ...,
 w,
 correction = FALSE
)

Arguments

Value

A <numeric> vector of length 1.

Definition

Let $\rho_c \in [0,1]$ measure the agreement between $y$ and $\upsilon$. The classifier agreement is calculated as,

$\rho_c = \frac{2 \rho \sigma_{\upsilon} \sigma_y}{\sigma_{\upsilon}^2 + \sigma_y^2 + (\mu_{\upsilon} - \mu_y)^2}$

Where:

• $\rho$ is the pearson correlation coefficient

• $\sigma_y$ is the unbiased standard deviation of $y$

• $\sigma_{\upsilon}$ is the unbiased standard deviation of $\upsilon$

• $\mu_y$ is the mean of $y$

• $\mu_{\upsilon}$ is the mean of $\upsilon$

If correction == TRUE each $\sigma_{i \in [y, \upsilon]}$ is adjusted by $\frac{1-n}{n}$

See Also

Other Regression: huberloss.numeric(), mae.numeric(), mape.numeric(), mpe.numeric(), mse.numeric(), pinball.numeric(), rae.numeric(), rmse.numeric(), rmsle.numeric(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) fit a linear
# regression
model <- lm(
  mpg ~ .,
  data = mtcars
)

# 1.1) define actual
# and predicted values
# to measure performance
actual    <- mtcars$mpg
predicted <- fitted(model)

# 2) evaluate in-sample model
# performance
cat(
  "Concordance Correlation Coefficient", ccc(
    actual     = actual,
    predicted  = predicted,
    correction = FALSE
  ),
  "Concordance Correlation Coefficient (corrected)", ccc(
    actual     = actual,
    predicted  = predicted,
    correction = TRUE
  ),
  "Concordance Correlation Coefficient (weigthed)", weighted.ccc(
    actual     = actual,
    predicted  = predicted,
    w          = mtcars$mpg/mean(mtcars$mpg),
    correction = FALSE
  ),
  sep = "\n"
)

Description

A generic function for Cohen's $\kappa$-statistic. Use weighted.ckappa() for the weighted $\kappa$-statistic.

Usage

## S3 method for class 'factor'
ckappa(actual, predicted, beta = 0, ...)

## S3 method for class 'factor'
weighted.ckappa(actual, predicted, w, beta = 0, ...)

## S3 method for class 'cmatrix'
ckappa(x, beta = 0, ...)

ckappa(
 ...,
 beta = 0
)

weighted.ckappa(
 ...,
 w,
 beta = 0
)

Arguments

Value

If micro is NULL (the default), a named <numeric>-vector of length k

If micro is TRUE or FALSE, a <numeric>-vector of length 1

Definition

Let $\kappa \in [0, 1]$ be the inter-rater (intra-rater) reliability. The inter-rater (intra-rater) reliability is calculated as,

$\kappa = \frac{\rho_p - \rho_e}{1-\rho_e}$

Where:

• $\rho_p$ is the empirical probability of agreement between predicted and actual values

• $\rho_e$ is the expected probability of agreement under random chance

If $\beta \neq 0$ the off-diagonals in the confusion matrix is penalized before $\rho$ is calculated. More formally,

$\chi = X \circ Y^{\beta}$

Where:

• $X$ is the confusion matrix

• $Y$ is the penalizing matrix and

• $\beta$ is the penalizing factor

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate model performance with
# Cohens Kappa statistic
cat(
  "Kappa", ckappa(
    actual    = actual,
    predicted = predicted
  ),
  "Kappa (penalized)", ckappa(
    actual    = actual,
    predicted = predicted,
    beta      = 2
  ),
  "Kappa (weigthed)", weighted.ckappa(
    actual    = actual,
    predicted = predicted,
    w         = iris$Petal.Length/mean(iris$Petal.Length)
  ),
  sep = "\n"
)

Description

The cmatrix()-function uses cross-classifying factors to build a confusion matrix of the counts at each combination of the factor levels. Each row of the matrix represents the actual factor levels, while each column represents the predicted factor levels.

Usage

## S3 method for class 'factor'
cmatrix(actual, predicted, ...)

## S3 method for class 'factor'
weighted.cmatrix(actual, predicted, w, ...)

## Generic S3 method
cmatrix(
 actual,
 predicted,
 ...
)

## Generic S3 method
weighted.cmatrix(
 actual,
 predicted,
 w,
 ...
)

Arguments

Value

A named $k$ x $k$ <matrix>

Dimensions

There is no robust defensive measure against mis-specifying the confusion matrix. If the arguments are correctly specified, the resulting confusion matrix is on the form:

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) summarise performance
# in a confusion matrix

# 4.1) unweighted matrix
confusion_matrix <- cmatrix(
  actual    = actual,
  predicted = predicted
)

# 4.1.1) summarise matrix
summary(
  confusion_matrix
)

# 4.1.2) plot confusion
# matrix
plot(
  confusion_matrix
)

# 4.2) weighted matrix
confusion_matrix <- weighted.cmatrix(
  actual    = actual,
  predicted = predicted,
  w         = iris$Petal.Length/mean(iris$Petal.Length)
)

# 4.2.1) summarise matrix
summary(
  confusion_matrix
)

# 4.2.1) plot confusion
# matrix
plot(
  confusion_matrix
)

Description

A generic function for the diagnostic odds ratio in classification tasks. Use weighted.dor() weighted diagnostic odds ratio.

Usage

## S3 method for class 'factor'
dor(actual, predicted, ...)

## S3 method for class 'factor'
weighted.dor(actual, predicted, w, ...)

## S3 method for class 'cmatrix'
dor(x, ...)

## Generic S3 method
dor(...)

## Generic S3 method
weighted.dor(
 ...,
 w
)

Arguments

Value

A <numeric>-vector of length 1

Definition

Let $\hat{\alpha} \in [0, \infty]$ be the effectiveness of the classifier. The diagnostic odds ratio of the classifier is calculated as,

$\hat{\alpha} = \frac{\text{\#TP} \text{\#TN}}{\text{\#FP} \text{\#FN}}$

Where:

• $\text{\#TP}$ is the number of true positives

• $\text{\#TN}$ is the number of true negatives

• $\text{\#FP}$ is the number of false positives

• $\text{\#FN}$ is the number of false negatives

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)


# 4) evaluate model performance
# with Diagnostic Odds Ratio
cat("Diagnostic Odds Ratio", sep = "\n")
dor(
  actual    = actual, 
  predicted = predicted
)

cat("Diagnostic Odds Ratio (weighted)", sep = "\n")
weighted.dor(
  actual    = actual,
  predicted = predicted,
  w         = iris$Petal.Length/mean(iris$Petal.Length)
)

Description

The entropy() function calculates the Entropy of given probability distributions.

Usage

## S3 method for class 'matrix'
entropy(pk, dim = 0L, base = -1, ...)

## S3 method for class 'matrix'
relative.entropy(pk, qk, dim = 0L, base = -1, ...)

## S3 method for class 'matrix'
cross.entropy(pk, qk, dim = 0L, base = -1, ...)

## Generic S3 method
entropy(
 pk,
 dim  = 0,
 base = -1,
 ...
)

## Generic S3 method
relative.entropy(
 pk,
 qk,
 dim  = 0,
 base = -1,
 ...
)

## Generic S3 method
cross.entropy(
 pk,
 qk,
 dim  = 0,
 base = -1,
 ...
)

Arguments

Value

A <numeric> value or vector:

• A single <numeric> value (length 1) if dim == 0.

• A <numeric> vector with length equal to the length of rows if dim == 1.

• A <numeric> vector with length equal to the length of columns if dim == 2.

Definition

Entropy:

$H(pk) = -\sum_{i} pk_i \log(pk_i)$

Cross Entropy:

$H(pk, qk) = -\sum_{i} pk_i \log(qk_i)$

Relative Entropy

$D_{KL}(pk \parallel qk) = \sum_{i} pk_i \log\left(\frac{pk_i}{qk_i}\right)$

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), fbeta.factor(), fdr.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) Define actual
# and observed probabilities

# 1.1) actual probabilies
pk <- matrix(
  cbind(1/2, 1/2),
  ncol = 2
)

# 1.2) observed (estimated) probabilites
qk <- matrix(
  cbind(9/10, 1/10), 
  ncol = 2
)

# 2) calculate
# Entropy
cat(
  "Entropy", entropy(
    pk
  ),
  "Relative Entropy", relative.entropy(
    pk,
    qk
  ),
  "Cross Entropy", cross.entropy(
    pk,
    qk
  ),
  sep = "\n"
)

Description

A generic function for the $F_{\beta}$-score. Use weighted.fbeta() for the weighted $F_{\beta}$-score.

Usage

## S3 method for class 'factor'
fbeta(actual, predicted, beta = 1, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.fbeta(actual, predicted, w, beta = 1, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
fbeta(x, beta = 1, micro = NULL, na.rm = TRUE, ...)

## Generic S3 method
fbeta(
 ...,
 beta  = 1,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.fbeta(
 ...,
 w,
 beta = 1,
 micro = NULL,
 na.rm = TRUE
)

Arguments

Value

If micro is NULL (the default), a named <numeric>-vector of length k

If micro is TRUE or FALSE, a <numeric>-vector of length 1

Definition

Let $\hat{F}_{\beta} \in [0, 1]$ be the $F_{\beta}$ score, which is a weighted harmonic mean of precision and recall. $F_{\beta}$ score of the classifier is calculated as,

$\hat{F}_{\beta} = \left(1 + \beta^2\right) \frac{\text{Precision} \times \text{Recall}} {\beta^2 \times \text{Precision} + \text{Recall}}$

Substituting $\text{Precision} = \frac{\#TP_k}{\#TP_k + \#FP_k}$ and $\text{Recall} = \frac{\#TP_k}{\#TP_k + \#FN_k}$ yields:

$\hat{F}_{\beta} = \left(1 + \beta^2\right) \frac{\frac{\#TP_k}{\#TP_k + \#FP_k} \times \frac{\#TP_k}{\#TP_k + \#FN_k}} {\beta^2 \times \frac{\#TP_k}{\#TP_k + \#FP_k} + \frac{\#TP_k}{\#TP_k + \#FN_k}}$

Where:

• $\#TP_k$ is the number of true positives,

• $\#FP_k$ is the number of false positives,

• $\#FN_k$ is the number of false negatives, and

• $\beta$ is a non-negative real number that determines the relative importance of precision vs. recall in the score.

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fdr.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fdr.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate class-wise performance
# using F1-score

# 4.1) unweighted F1-score
fbeta(
  actual    = actual,
  predicted = predicted,
  beta      = 1
)

# 4.2) weighted F1-score
weighted.fbeta(
  actual    = actual,
  predicted = predicted,
  w         = iris$Petal.Length/mean(iris$Petal.Length),
  beta      = 1
)

# 5) evaluate overall performance
# using micro-averaged F1-score
cat(
  "Micro-averaged F1-score", fbeta(
    actual    = actual,
    predicted = predicted,
    beta      = 1,
    micro     = TRUE
  ),
  "Micro-averaged F1-score (weighted)", weighted.fbeta(
    actual    = actual,
    predicted = predicted,
    w         = iris$Petal.Length/mean(iris$Petal.Length),
    beta      = 1,
    micro     = TRUE
  ),
  sep = "\n"
)

Description

A generic function for the False Discovery Rate. Use weighted.fdr() for the weighted False Discovery Rate.

Usage

## S3 method for class 'factor'
fdr(actual, predicted, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.fdr(actual, predicted, w, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
fdr(x, micro = NULL, na.rm = TRUE, ...)

## Generic S3 method
fdr(
 ...,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.fdr(
 ...,
 w,
 micro = NULL,
 na.rm = TRUE
)

Arguments

Value

If micro is NULL (the default), a named <numeric>-vector of length k

If micro is TRUE or FALSE, a <numeric>-vector of length 1

Definition

Let $\hat{\alpha} \in [0, 1]$ be the proportion of false positives among the preditced positives. The false discovery rate of the classifier is calculated as,

$\hat{\alpha} = \frac{\#FP_k}{\#TP_k+\#FP_k}$

Where:

• $\#TP_k$ is the number of true positives, and

• $\#FP_k$ is the number of false positives

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fer.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fer.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate class-wise performance
# using False Discovery Rate

# 4.1) unweighted False Discovery Rate
fdr(
  actual    = actual,
  predicted = predicted
)

# 4.2) weighted False Discovery Rate
weighted.fdr(
  actual    = actual,
  predicted = predicted,
  w         = iris$Petal.Length/mean(iris$Petal.Length)
)

# 5) evaluate overall performance
# using micro-averaged False Discovery Rate
cat(
  "Micro-averaged False Discovery Rate", fdr(
    actual    = actual,
    predicted = predicted,
    micro     = TRUE
  ),
  "Micro-averaged False Discovery Rate (weighted)", weighted.fdr(
    actual    = actual,
    predicted = predicted,
    w         = iris$Petal.Length/mean(iris$Petal.Length),
    micro     = TRUE
  ),
  sep = "\n"
)

Description

A generic function for the false omission rate. Use weighted.fdr() for the weighted false omission rate.

Usage

## S3 method for class 'factor'
fer(actual, predicted, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.fer(actual, predicted, w, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
fer(x, micro = NULL, na.rm = TRUE, ...)

## Generic S3 method
fer(
 ...,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.fer(
 ...,
 w,
 micro = NULL,
 na.rm = TRUE
)

Arguments

Value

If micro is NULL (the default), a named <numeric>-vector of length k

If micro is TRUE or FALSE, a <numeric>-vector of length 1

Definition

Let $\hat{\beta} \in [0, 1]$ be the proportion of false negatives among the predicted negatives. The false omission rate of the classifier is calculated as,

$\hat{\beta} = \frac{\#FN_k}{\#TN_k + \#FN_k}$

Where:

• $\#TN_k$ is the number of true negatives, and

• $\#FN_k$ is the number of false negatives.

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fmi.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fpr.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate class-wise performance
# using False Omission Rate

# 4.1) unweighted False Omission Rate
fer(
  actual    = actual,
  predicted = predicted
)

# 4.2) weighted False Omission Rate
weighted.fer(
  actual    = actual,
  predicted = predicted,
  w         = iris$Petal.Length/mean(iris$Petal.Length)
)

# 5) evaluate overall performance
# using micro-averaged False Omission Rate
cat(
  "Micro-averaged False Omission Rate", fer(
    actual    = actual,
    predicted = predicted,
    micro     = TRUE
  ),
  "Micro-averaged False Omission Rate (weighted)", weighted.fer(
    actual    = actual,
    predicted = predicted,
    w         = iris$Petal.Length/mean(iris$Petal.Length),
    micro     = TRUE
  ),
  sep = "\n"
)

Description

The fmi()-function computes the Fowlkes-Mallows Index (FMI), a measure of the similarity between two sets of clusterings, between two vectors of predicted and observed factor() values.

Usage

## S3 method for class 'factor'
fmi(actual, predicted, ...)

## S3 method for class 'cmatrix'
fmi(x, ...)

## Generic S3 method
fmi(...)

Arguments

Value

A <numeric>-vector of length 1

Definition

The metric is calculated for each class $k$ as follows,

$\sqrt{\frac{\#TP_k}{\#TP_k + \#FP_k} \times \frac{\#TP_k}{\#TP_k + \#FN_k}}$

Where $\#TP_k$, $\#FP_k$, and $\#FN_k$ represent the number of true positives, false positives, and false negatives for each class $k$, respectively.

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate model performance
# using Fowlkes Mallows Index
cat(
  "Fowlkes Mallows Index", fmi(
  actual    = actual,
  predicted = predicted
  ),
  sep = "\n"
)

Description

A generic function for the False Positive Rate. Use weighted.fpr() for the weighted False Positive Rate.

Other names

Fallout

Usage

## S3 method for class 'factor'
fpr(actual, predicted, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.fpr(actual, predicted, w, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
fpr(x, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
fallout(actual, predicted, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.fallout(actual, predicted, w, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
fallout(x, micro = NULL, na.rm = TRUE, ...)

## Generic S3 method
fpr(
 ...,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
fallout(
 ...,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.fpr(
 ...,
 w,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.fallout(
 ...,
 w,
 micro = NULL,
 na.rm = TRUE
)

Arguments

Value

If micro is NULL (the default), a named <numeric>-vector of length k

If micro is TRUE or FALSE, a <numeric>-vector of length 1

Definition

Let $\hat{\gamma} \in [0, 1]$ be the proportion of false positives among the actual negatives. The false positive rate of the classifier is calculated as,

$\hat{\gamma} = \frac{\#FP_k}{\#TN_k + \#FP_k}$

Where:

• $\#TN_k$ is the number of true negatives, and

• $\#FP_k$ is the number of false positives.

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows:

## set seed
set.seed(1903)

## actual
factor(
  x = sample(x = 1:3, size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] B A B B A C B C C A
#> Levels: A B C

Here, the values 1, 2, and 3 are mapped to A, B, and C, respectively. Now, suppose your model does not predict any B's. The predicted vector of factor() values would be defined as follows:

## set seed
set.seed(1903)

## predicted
factor(
  x = sample(x = c(1, 3), size = 10, replace = TRUE),
  levels = c(1, 2, 3),
  labels = c("A", "B", "C")
)
#>  [1] C A C C C C C C A C
#> Levels: A B C

In both cases, $k = 3$, determined indirectly by the levels argument.

See Also

Other Classification: ROC.factor(), accuracy.factor(), baccuracy.factor(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fmi.factor(), jaccard.factor(), logloss.factor(), mcc.factor(), nlr.factor(), npv.factor(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), recall.factor(), roc.auc.matrix(), specificity.factor(), zerooneloss.factor()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), huberloss.numeric(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) recode Iris
# to binary classification
# problem
iris$species_num <- as.numeric(
  iris$Species == "virginica"
)

# 2) fit the logistic
# regression
model <- glm(
  formula = species_num ~ Sepal.Length + Sepal.Width,
  data    = iris,
  family  = binomial(
    link = "logit"
  )
)

# 3) generate predicted
# classes
predicted <- factor(
  as.numeric(
    predict(model, type = "response") > 0.5
  ),
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 3.1) generate actual
# classes
actual <- factor(
  x = iris$species_num,
  levels = c(1,0),
  labels = c("Virginica", "Others")
)

# 4) evaluate class-wise performance
# using False Positive Rate

# 4.1) unweighted False Positive Rate
fpr(
  actual    = actual,
  predicted = predicted
)

# 4.2) weighted False Positive Rate
weighted.fpr(
  actual    = actual,
  predicted = predicted,
  w         = iris$Petal.Length/mean(iris$Petal.Length)
)

# 5) evaluate overall performance
# using micro-averaged False Positive Rate
cat(
  "Micro-averaged False Positive Rate", fpr(
    actual    = actual,
    predicted = predicted,
    micro     = TRUE
  ),
  "Micro-averaged False Positive Rate (weighted)", weighted.fpr(
    actual    = actual,
    predicted = predicted,
    w         = iris$Petal.Length/mean(iris$Petal.Length),
    micro     = TRUE
  ),
  sep = "\n"
)

Description

The huberloss()-function computes the simple and weighted huber loss between the predicted and observed <numeric> vectors. The weighted.huberloss() function computes the weighted Huber Loss.

Usage

## S3 method for class 'numeric'
huberloss(actual, predicted, delta = 1, ...)

## S3 method for class 'numeric'
weighted.huberloss(actual, predicted, w, delta = 1, ...)

## Generic S3 method
huberloss(
 actual,
 predicted,
 delta = 1,
 ...
)

## Generic S3 method
weighted.huberloss(
 actual,
 predicted,
 w,
 delta = 1,
 ...
)

Arguments

Value

A <numeric> vector of length 1.

Definition

The metric is calculated as follows,

$\frac{1}{2} (y - \upsilon)^2 ~for~ |y - \upsilon| \leq \delta$

and

$\delta |y-\upsilon|-\frac{1}{2} \delta^2 ~for~ \text{otherwise}$

where $y$ and $\upsilon$ are the actual and predicted values respectively. If w is not NULL, then all values are aggregated using the weights.

See Also

Other Regression: ccc.numeric(), mae.numeric(), mape.numeric(), mpe.numeric(), mse.numeric(), pinball.numeric(), rae.numeric(), rmse.numeric(), rmsle.numeric(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric()

Other Supervised Learning: ROC.factor(), accuracy.factor(), baccuracy.factor(), ccc.numeric(), ckappa.factor(), cmatrix.factor(), dor.factor(), entropy.matrix(), fbeta.factor(), fdr.factor(), fer.factor(), fpr.factor(), jaccard.factor(), logloss.factor(), mae.numeric(), mape.numeric(), mcc.factor(), mpe.numeric(), mse.numeric(), nlr.factor(), npv.factor(), pinball.numeric(), plr.factor(), pr.auc.matrix(), prROC.factor(), precision.factor(), rae.numeric(), recall.factor(), rmse.numeric(), rmsle.numeric(), roc.auc.matrix(), rrmse.numeric(), rrse.numeric(), rsq.numeric(), smape.numeric(), specificity.factor(), zerooneloss.factor()

Examples

# 1) fit a linear
# regression
model <- lm(
  mpg ~ .,
  data = mtcars
)

# 1.1) define actual
# and predicted values
# to measure performance
actual    <- mtcars$mpg
predicted <- fitted(model)


# 2) calculate the metric
# with delta 0.5
huberloss(
  actual = actual,
  predicted = predicted,
  delta = 0.5
)

# 3) caclulate weighted
# metric using arbitrary weights
w <- rbeta(
  n = 1e3,
  shape1 = 10,
  shape2 = 2
)

huberloss(
  actual = actual,
  predicted = predicted,
  delta = 0.5,
  w     = w
)

Description

The jaccard()-function computes the Jaccard Index, also known as the Intersection over Union, between two vectors of predicted and observed factor() values. The weighted.jaccard() function computes the weighted Jaccard Index.

Usage

## S3 method for class 'factor'
jaccard(actual, predicted, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.jaccard(actual, predicted, w, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
jaccard(x, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
csi(actual, predicted, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.csi(actual, predicted, w, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
csi(x, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
tscore(actual, predicted, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'factor'
weighted.tscore(actual, predicted, w, micro = NULL, na.rm = TRUE, ...)

## S3 method for class 'cmatrix'
tscore(x, micro = NULL, na.rm = TRUE, ...)

## Generic S3 method
jaccard(
 ...,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
csi(
 ...,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
tscore(
 ...,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.jaccard(
 ...,
 w,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.csi(
 ...,
 w,
 micro = NULL,
 na.rm = TRUE
)

## Generic S3 method
weighted.tscore(
 ...,
 w,
 micro = NULL,
 na.rm = TRUE
)

Arguments

Value

If micro is NULL (the default), a named <numeric>-vector of length k

If micro is TRUE or FALSE, a <numeric>-vector of length 1

Definition

The metric is calculated for each class $k$ as follows,

$\frac{\#TP_k}{\#TP_k + \#FP_k + \#FN_k}$

Where $\#TP_k$, $\#FP_k$, and $\#FN_k$ represent the number of true positives, false positives, and false negatives for each class $k$, respectively.

Creating <factor>

Consider a classification problem with three classes: A, B, and C. The actual vector of factor() values is defined as follows: