Exercise 01

1 Probabilities

Given a Doctor’s Patient Record, where each cell $n(x_r,{\theta }_c)$ represents the number of patients with symptom $x_r$ and disease ${\theta }_c$ :

$$\begin{array}{cccccccccccc}x\backslash \theta & {\theta }_1& {\theta }_2& {\theta }_3& {\theta }_4& {\theta }_5& {\theta }_6& {\theta }_7& {\theta }_8& {\theta }_9& {\theta }_{10}& \text{Sum}\\ x_1& 8& 9& 9& 5& 4& 1& 1& 0& 0& 0& 37\\ x_2& 3& 5& 8& 9& 14& 10& 3& 3& 0& 0& 55\\ x_3& 0& 1& 1& 10& 16& 11& 12& 7& 8& 5& 71\\ x_4& 0& 0& 1& 0& 3& 5& 10& 7& 7& 4& 37\\ \text{Sum}& 11& 15& 19& 24& 37& 27& 26& 17& 15& 9& 200\end{array}$$

Table 1 : Joint observations of symptoms $x_r$ (rows) and disease ${\theta }_c$ (columns). Questions from the doctor :
Q1 : What is the joint probability $P(x_3,{\theta }_2)$ that a patient has the symptom $x_3$ and the disease ${\theta }_2$ ? Ans :

$$\begin{array}{rl}P(x_3,{\theta }_2)& =\frac{n(x_3,{\theta }_2)}{N}\\ & =\frac{1}{200}\\ & =0.005\end{array}$$

Q2 : What is the probability $P(x_3)$ that a patient has the symptom $x_3$ ? Ans :

$$\begin{array}{rl}P(x_3)& =\frac{n(x_3)}{N}\\ & =\frac{71}{200}\\ & =0.355\end{array}$$

Q3 : What is the probability $P({\theta }_2)$ that a patient has the disease ${\theta }_2$ ? Ans :

$$\begin{array}{rl}P({\theta }_2)& =\frac{n({\theta }_2)}{N}\\ & =\frac{15}{200}\\ & =0.075\end{array}$$

Q4 : What is the conditional probability $P(x_3|{\theta }_2)$ that a patient has the symptom $x_3$ given they have the disease ${\theta }_2$ ? Ans :

$$\begin{array}{rl}P(x_3|{\theta }_2)& =\frac{P(x_3,{\theta }_2)}{P({\theta }_2)}\\ & =\frac{0.005}{0.075}\\ & =\frac{1}{15}\end{array}$$

*Q5 : What is the conditional probability $P({\theta }_2|x_3)$ that a patient has the disease ${\theta }_2$ given that they have the symptom $x_3$ ? Ans :

$$\begin{array}{rl}P({\theta }_2|x_3)& =\frac{P(x_3,{\theta }_2)}{P(x_3)}\\ & =\frac{0.005}{0.355}\\ & =\frac{1}{71}\end{array}$$

2 Maximum Likelihood

Q1. We flipped a coin 100 times. Given that there were 55 heads, use maximum likelihood estimation to find the probability $p$ of heads on a single toss. Ans : Binomial Distribution

$$\begin{array}{rl}P(55\text{ heads}\mid p)& =\left(\genfrac{}{}{0ex}{}{100}{55}\right)p^{55}(1-p{)}^{45}\\ \log P(55\text{ heads}\mid p)& =\log \left(\genfrac{}{}{0ex}{}{100}{55}\right)+55\log p+45\log (1-p)\end{array}$$

Differentiate and Solve for $p$

$$\begin{array}{rl}\frac{d}{dp}\log P(55\text{ heads}\mid p)& =\frac{55}{p}-\frac{45}{1-p}\end{array}$$

Set derivative to zero and solve:

$$\frac{55}{p}-\frac{45}{1-p}=0$$

Multiply both sides by $p(1-p)$:

$$\begin{array}{rl}55(1-p)-45p& =0\\ 55-55p-45p& =0\\ 55& =100p\\ p& =\frac{55}{100}=0.55\end{array}$$

Q2 : Let $X$ be independent and identically distributed (i.i.d.) Poisson ($\lambda$) distributed. Find the maximum likelihood estimator for $\lambda$, $\hat{\lambda }$. Calculate an estimate using the estimator when : $x_1=1,x_2=2,x_3=5,x_4=3$ Ans : PMF for Poisson

$$P(X=x)=\frac{{\lambda }^x{e}^{-\lambda }}{x!}$$

Likelihood Function

$$L(\lambda \mid x_1,\dots ,x_n)=\prod _{j=1}^n\frac{{\lambda }^{x_j}{e}^{-\lambda }}{x_j!}$$

And for the log-likelihood:

$$\log L(\lambda )=\sum _{j=1}^n{x}_j\log \lambda -\lambda -\sum _{j=1}^n\log x_j!$$

Differentiate and solve for $\lambda$

$$\begin{array}{rl}\frac{d}{d\lambda }\left(\sum _{j=1}^n{x}_j\log \lambda -\lambda \right)& =\frac{{\sum }_{j=1}^n{x}_j}{\lambda }-n=0\\ \lambda & =\frac{1}{n}\sum _{j=1}^n{x}_j\end{array}$$

For the given data :

$$\lambda =\frac{1+2+5+3}{4}=2.75$$

Logistic Regression

A company wants to predict whether a customer will purchase a product based on their income and age. The logistic regression model is given by :

$$P(Y=1\mid X)=\frac{1}{1+e^{-({\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2)}}$$

where :

• $Y=1$ if the customer buys the product, $Y=0$ otherwise.
• $X_1$ = Age of the customer.
• $X_2$ = Monthly income of the customer (in £1000s).
• ${\beta }_0=-3$, ${\beta }_1=0.05$, ${\beta }_2=0.1$.
  

Question 1 : Compute the probability that a customer aged 30 with a monthly income of £5000 will purchase the product. Ans :

$$\begin{array}{rl}P(Y=1\mid X_1=30,X_2=5)& =\frac{1}{1+e^{-(-3+0.05\times 30+0.1\times 5)}}\\ & =\frac{1}{1+e^{-(-3+1.5+0.5)}}\\ & =\frac{1}{1+e^{-(-1)}}\\ & =\frac{1}{1+e^1}\\ & \approx \frac{1}{1+2.718}\\ & \approx 0.269\end{array}$$

Question 2 : Interpret the meaning of ${\beta }_1=0.05$ in the context of this problem. Ans : A one-year increase in age increases the log-odds of purchasing by 0.05, meaning age has a positive effect on purchase probability.

Question 3 : If a customer’s probability of purchasing is 0.7, should the company predict that they will buy the product? Justify your answer. Ans : $P(Y=1|X)=0.7$ : Since 0.7 > 0.5, the model predicts that the customer will buy the product.

Question 4 : Briefly explain one advantage and one limitation of using logistic regression for this problem. Ans : Advantage : Interpretability - coefficients show how features affect probability. Limitation : Assumes a linear relationship in log-odds, which may not always hold.

Naive Bayes - Disease Diagnosis

A medical test is used to diagnose a rare disease based on two symptoms: Fever and Cough. The model uses a Naive Bayes classifier. The probabilities of symptoms appearing given the disease status are:

$$\begin{array}{ccc}\text{Symptom}& P(\text{symptom}\mid \text{disease})& P(\text{symptom}\mid \text{no disease})\\ \text{Fever}& 0.8& 0.2\\ \text{Cough}& 0.6& 0.3\end{array}$$

Additionally, we know that only 5% of the population has the disease (i.e., $P(disease)=0.05$).

Question 1 : Compute the probability that a patient with both Fever and Cough has the disease, and classify the patient as having or not having the disease. Ans :

$$\begin{array}{rl}P(\text{Fever},\text{Cough}\mid \text{Disease})\cdot P(\text{Disease})& =(0.8\times 0.6)\times 0.05\\ & =0.48\times 0.05\\ & =0.024\end{array}$$

For no disease :

$$\begin{array}{rl}P(\text{Fever},\text{Cough}\mid \text{No Disease})\cdot P(\text{No Disease})& =(0.2\times 0.3)\times 0.95\\ & =0.06\times 0.95\\ & =0.057\end{array}$$

Posterior probability :

$$\begin{array}{rl}P(\text{Disease}\mid \text{Symptoms})& =\frac{0.024}{0.024+0.057}\\ & =\frac{0.024}{0.081}\\ & \approx 0.296\end{array}$$

Since $0.296<0.5$, the model does not classify the patient as having the disease.

Question 2 : What is the ’naive Bayes’ assumption and why is it important in Naive Bayes classification? Ans : The assumption simplifies computation by treating symptoms as independent given the disease. This makes the model fast and scalable, though symptoms in reality may be correlated.