Classification Models: Logistic Regression, k-NN, Decision Trees, and Naive Bayes

2026 GitHub

Python scikit-learn pandas

In the previous article I introduced supervised learning: learning a mapping from inputs to outputs from labeled examples. When the output is a number, that's regression. When the output is a category (spam or not spam, fraud or legitimate, malignant or benign), that's classification, and it's the subject of this article.

This is adapted from the classification week I taught in the VADSTI program. We'll build four classic models on a real medical dataset, compare them, and then do the most important part, which is to look past "accuracy" and ask what kinds of mistakes each model makes.

Where Classification Shows Up

Classification models predict discrete labels from input features. They're trained on labeled data and then used to label new, unseen cases. A few examples:

Healthcare: is a tumor malignant or benign?
Finance: is this transaction fraudulent?
Marketing: will this customer churn?
Email: spam or not spam?
Social media: does this post contain hate speech or bullying?
NLP: is this review positive, negative, or neutral?
Autonomous vehicles: is that object a car, a pedestrian, or a bicycle?
Lending: approve or decline this loan?

The Classification Workflow

Every project I've worked on follows roughly the same loop:

Define the problem. Make sure the target really is a set of discrete classes.
Gather labeled data. Each row is a set of features plus a known class.
Preprocess. Handle missing values, encode categorical variables, and scale numeric features.
Split into training and test sets. The test set is the "new basket of fruit" the model has never seen.
Choose a model.
Train it on the training set.
Evaluate it on the test set.
Tune hyperparameters (cross-validation, grid search).
Predict on new data.
Monitor and retrain as the data drifts.

It's iterative. You'll bounce between steps 3 and 8 more than once.

The Dataset: Breast Cancer Wisconsin (Diagnostic)

We'll use the Breast Cancer Wisconsin (Diagnostic) dataset, collected by Dr. William H. Wolberg at the University of Wisconsin Hospitals in the 1990s and bundled with scikit-learn. Each of its 569 rows describes a breast mass from a digitized image of a fine-needle aspirate, using 30 numeric features of the cell nuclei: mean radius, texture, perimeter, area, smoothness, compactness, concavity, symmetry, and so on.

The goal is to predict whether the tumor is malignant or benign, a binary classification problem.

One detail that matters later: in scikit-learn's version, the target is coded 0 = malignant and 1 = benign. Keep that in mind when we read the confusion matrix.

This dataset is already clean. Real clinical data rarely is, so expect much more preprocessing in practice.

load_data.py

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

cancer = load_breast_cancer()
df = pd.DataFrame(cancer.data, columns=cancer.feature_names)
df["target"] = cancer.target          # 0 = malignant, 1 = benign

# Always look at the class balance first
df["target"].value_counts().plot(kind="bar")

X = df.drop("target", axis=1)
y = df["target"]

Why standardize?

The features live on wildly different scales. Mean area is in the hundreds or thousands, while mean smoothness is around 0.1. Standardization rescales each feature to mean 0 and standard deviation 1 (it's just a z-score per column):

$$z = \frac{x - \mu}{\sigma}$$

This matters for any algorithm that uses distances (k-NN, SVMs) or gradient descent (logistic regression), where a large-scale feature would otherwise dominate. Tree-based models such as decision trees and random forests don't care about scale. Don't confuse standardization (StandardScaler) with normalization (MinMaxScaler), which squeezes values into a fixed range like [0, 1].

Then we split, using an 80/20 train/test split (75/25 is also common):

preprocess.py

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42
)

A subtle leak: fitting the scaler on all the data before splitting lets a little information about the test set seep into training. The cleaner pattern is to split first, then fit the scaler on the training data only, which a scikit-learn Pipeline does for you. I'll show that below.

Model 1: Logistic Regression

Despite the name, logistic regression is a classifier. It computes a weighted sum of the features, exactly like linear regression, and then squashes it through the sigmoid function so the output lands between 0 and 1 and can be read as a probability:

$$P(y = 1 \mid x) = \frac{1}{1 + e^{-(w \cdot x + b)}}$$

The decision boundary is where that probability crosses 0.5. With two features it's a straight line, and with 30 it's a flat hyperplane through feature space. The weights are fit by maximum likelihood: choose the parameters that make the observed labels most probable.

Its assumptions are a binary (or ordinal) target, independent observations, little multicollinearity, a linear relationship between features and the log-odds, and a reasonably large sample.

logistic_regression.py

from sklearn.linear_model import LogisticRegression
from sklearn import metrics

model_logistic = LogisticRegression()
model_logistic.fit(X_train, y_train)

predictions = model_logistic.predict(X_test)
print(f"Accuracy: {metrics.accuracy_score(y_test, predictions):.2%}")
# Accuracy: 97.37%

Model 2: k-Nearest Neighbors

k-NN is the most intuitive classifier there is: to label a new point, look at the k closest training points and take a majority vote.

Instance-based: it doesn't really "learn" a model. It memorizes the training set.
Distance metric: Euclidean is typical, with Manhattan and Minkowski as alternatives.
Choosing k: too small (1 or 2) and it's jittery and sensitive to noise. Too large and it blurs everything into the majority class. k = 5 is a common starting point, and cross-validation finds a better one.

Pros: simple, makes no assumptions about the data's shape, and works for both classification and regression. Cons: slow on large datasets, struggles with many features, and is very sensitive to feature scale and irrelevant features.

That last point showed up in our class notebook. The k-NN cell computed the scaled features, but then split and trained on the raw X:

knn_unscaled.py

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

# As run in class: trained on UNSCALED features
X_train_raw, X_test_raw, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model_knn = KNeighborsClassifier(n_neighbors=3)
model_knn.fit(X_train_raw, y_train)
print(accuracy_score(y_test, model_knn.predict(X_test_raw)))
# 0.9298...

That gives 93.0%, the lowest score of the four models. Without scaling, "nearest" mostly means "closest in area," because area's numbers are hundreds of times larger than everything else's. The other 29 features barely count. It's a nice accidental demonstration of why scaling matters for distance-based models. The fix is a pipeline, which also takes care of the leakage issue:

knn_pipeline.py

from sklearn.pipeline import make_pipeline

X_train_raw, X_test_raw, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

knn_scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=3))
knn_scaled.fit(X_train_raw, y_train)          # scaler is fit on training data only
print(accuracy_score(y_test, knn_scaled.predict(X_test_raw)))

Run it and compare. It's a good exercise in seeing how much preprocessing can matter.

Model 3: Decision Trees

A decision tree is a flowchart learned from data. At each internal node it asks a yes/no question about one feature ("is worst radius above some threshold?"), and each branch leads to another question until it reaches a leaf, which holds a class label. Follow the path from the root to a leaf and you've read off a classification rule.

Root node: the full dataset.
Splitting: dividing a node into purer sub-nodes, using the feature and threshold that best separate the classes.
Decision node: a node that splits further.
Leaf / terminal node: a node that doesn't split; it carries the prediction.

Pros: easy to explain to non-technical stakeholders, handles numeric and categorical data, and needs little preprocessing (no scaling required). Cons: unstable, since a small change in the data can produce a very different tree, and prone to overfitting if it's allowed to grow deep without pruning.

decision_tree.py

from sklearn.tree import DecisionTreeClassifier

# Trees don't need scaling, so we use the raw features
X_train_raw, X_test_raw, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model_tree = DecisionTreeClassifier()          # tip: set random_state for reproducibility
model_tree.fit(X_train_raw, y_train)
print(accuracy_score(y_test, model_tree.predict(X_test_raw)))
# 0.947... (this run; varies slightly between runs without a fixed random_state)

Model 4: Naive Bayes

Naive Bayes is a probabilistic classifier built on Bayes' theorem:

$$P(\text{class} \mid \text{features}) = \frac{P(\text{features} \mid \text{class})\; P(\text{class})}{P(\text{features})}$$

To choose between two labels, it compares their posterior probabilities and picks the larger. The hard part is estimating $P(\text{features} \mid \text{class})$ when there are 30 correlated features. The "naive" trick is to assume the features are independent given the class, so that probability becomes a simple product of one-feature probabilities. This is a generative model: it describes how each class could have generated the data.

The independence assumption is almost never true, yet Naive Bayes often works surprisingly well, and it's very fast. scikit-learn offers GaussianNB (continuous features assumed normal), MultinomialNB (counts, a classic for text), and BernoulliNB (binary features). It's a workhorse for spam filtering, text classification, and sentiment analysis.

naive_bayes.py

from sklearn.naive_bayes import GaussianNB

X_train_raw, X_test_raw, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model_nb = GaussianNB()
model_nb.fit(X_train_raw, y_train)
y_pred_nb = model_nb.predict(X_test_raw)
print(accuracy_score(y_test, y_pred_nb))
# 0.9736...

Scoreboard

Model	Test accuracy
Logistic regression (scaled features)	97.4%
Gaussian Naive Bayes	97.4%
Decision tree	94.7%
k-NN, k = 3 (unscaled features)	93.0%

On a clean dataset like this, the simple models do extremely well. But a single accuracy number hides a lot. Let's look closer.

Evaluating Classifiers: Beyond Accuracy

Every prediction falls into one of four buckets:

	Predicted positive	Predicted negative
Actually positive	True Positive (TP)	False Negative (FN)
Actually negative	False Positive (FP)	True Negative (TN)

From those come the standard metrics:

Accuracy = (TP + TN) / total. It's the simplest, and it can be misleading on imbalanced data: a model that always predicts "not fraud" is 99.9% accurate and useless.
Precision = TP / (TP + FP). When the model says positive, how often is it right?
Recall (sensitivity, TPR) = TP / (TP + FN). Of all the actual positives, how many did it find?
F1 score = 2 · (precision · recall) / (precision + recall). The harmonic mean, which balances the two and is especially useful with uneven classes.
Confusion matrix: the full 2×2 table, showing which errors were made.
ROC curve / AUC: the true positive rate plotted against the false positive rate across all thresholds. AUC is the area under that curve, a threshold-free measure of how well the model separates the classes.

Here are the full metrics for our Naive Bayes model:

evaluate.py

from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                             f1_score, confusion_matrix, roc_auc_score)

print("Accuracy: ", accuracy_score(y_test, y_pred_nb))
print("Precision:", precision_score(y_test, y_pred_nb))
print("Recall:   ", recall_score(y_test, y_pred_nb))
print("F1-Score: ", f1_score(y_test, y_pred_nb))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred_nb))
print("ROC-AUC:  ", roc_auc_score(y_test, y_pred_nb))

# Accuracy:  0.9736842105263158
# Precision: 0.9594594594594594
# Recall:    1.0
# F1-Score:  0.9793103448275862
# Confusion Matrix:
#  [[40  3]
#   [ 0 71]]
# ROC-AUC:   0.9651162790697674

Reading the confusion matrix carefully

Recall = 1.0 looks perfect. But remember the coding: class 1 is benign, so scikit-learn's metrics treat "benign" as the positive class. Reading the matrix with that in mind:

	Predicted malignant	Predicted benign
Actually malignant (43)	40	3
Actually benign (71)	0	71

All 71 benign tumors were correctly called benign, which is the perfect "recall."
3 of the 43 malignant tumors were predicted benign. In a screening context, that's the error you least want: a cancer the model would have sent home.

Flip the perspective and treat malignant as the positive class, as a clinician would, and the model's recall for cancer is 40/43, about 93%. Same predictions, very different story. So I always ask two things: which class is "positive" in your metrics, and which error costs more? In medicine, a false negative usually costs far more than a false positive, so you might lower the decision threshold to catch more malignant cases, even if it means more follow-up biopsies.

One more technical note: the ROC-AUC above was computed from hard 0/1 predictions. AUC is meant to be computed from predicted probabilities, e.g. roc_auc_score(y_test, model_nb.predict_proba(X_test_raw)[:, 1]), which uses the model's full ranking instead of one threshold.

No single metric tells the whole story. Look at several together, and always look at the confusion matrix.

Overfitting and Underfitting

Overfitting is studying for an exam by memorizing last year's answers. The model captures the noise along with the signal:

It does great on training data and poorly on unseen data.
It's typical of overly complex models, such as unpruned deep decision trees or high-degree polynomials.
Remedies: cross-validation, regularization, pruning, and more training data.

Underfitting is studying only the chapter titles. The model is too simple to capture the real pattern:

It does poorly on both training and test data.
It's typical of, say, a straight line fit to curved data.
Remedies: a more expressive model and better features.

The goal sits in between: low bias (capture the real trend) and low variance (ignore the noise). That's the bias–variance tradeoff. A simple diagnostic is to compare training accuracy to test accuracy. A big gap signals overfitting, and both being low signals underfitting.

Try It Yourself

The in-class exercise asked students to repeat the full workflow on a diabetes dataset: explore the classes, standardize, split, train any two of the four models, evaluate with accuracy, precision, recall, and F1, and then decide whether the model over- or underfits.

A heads-up if you try it: scikit-learn's built-in load_diabetes() is actually a regression dataset (its target is a continuous disease-progression score), so you'd need to binarize the target first, for example "above vs. below the median." A more natural choice is a dataset with a true class label, such as the Pima Indians Diabetes data on UCI or the BRFSS diabetes health-indicators file in the course repo.

Key Takeaways

Classification predicts categories. The four classic models draw their boundaries in different ways: a sigmoid over a linear score, a neighborhood vote, a sequence of yes/no splits, or Bayes' rule with a simplifying assumption.
Preprocessing matters. Scale features for distance- and gradient-based models, and fit the scaler on training data only (use a Pipeline).
Simple models can be excellent. Logistic regression and Naive Bayes both hit about 97% here.
Accuracy isn't enough. Know which class is "positive," read the confusion matrix, and weigh what each type of error costs, especially in healthcare.
Watch the train/test gap to diagnose over- and underfitting.

Next, I put all of this to work on a larger, messier, real-world health problem: predicting 30-day hospital readmission for diabetic patients.