19 The Naive Bayes Classifier

The Naive Bayes Assumption

• The Naive Bayes classifier is a simple probabilistic classifier based on Bayes’ Theorem.
• It assumes that the features (attributes) are conditionally independent given the class label, hence the term “naive”.

$$P(\mathbf{x}\mid C)=P(x_1,x_2,\dots ,x_d\mid C)=\prod _{i=1}^{d}P(x_i\mid C)$$

• This assumption simplifies the computation, making it efficient for classification tasks.

The Naive Bayes Classifier

Training a Naive Bayes model involves calculating:

• Prior probability of each class $P(C)$.
• Conditional probabilities $P(x_i|C)$ for each feature $x_i$ given class $C$.
• For a given class $C$, the posterior probability is:

$$P(C\mid \mathbf{x})=\frac{P(C){\prod }_{i=1}^{n}P(x_i\mid C)}{P(\mathbf{x})}$$

• To classify a new instance, we compute the posterior probability for each class using Bayes’ Theorem and choose the class with the highest probability. Note : The denominator is the same for class posteriors!.

$$\hat{C}=\arg \underset{C}{\max }\left[P(C)\prod _{i=1}^{n}P(X_i\mid C)\right]$$

Advantages & Limitations of Naive Bayes

Naive Bayes classifier have several advantages :

• Simple and easy to implement.
• Works well with large datasets.
• Performs well with categorical and continuous features.
• Fast training and prediction.

It has some limitations:

• The “naive” assumption of feature independence is rarely true in practice.
• It may perform poorly with highly correlated features.
• Sensitive to imbalance datasets.