Part 5 - Implementing K-Nearest Neighbors (KNN) in Python

Machine Learning Algorithms Series - Classification with scikit-learn

Subtitle:

This article explains how to implement the K-Nearest Neighbors (KNN) algorithm in Python for classification problems. It covers importing necessary libraries, preparing data, training the model, making predictions, and evaluating performance using accuracy scores and confusion matrices.

Introduction to K-Nearest Neighbors (KNN)

The K-Nearest Neighbors (KNN) algorithm is a simple, non-parametric classification and regression algorithm. It classifies new data points based on the majority class of the K nearest points in the feature space. It is particularly useful for smaller datasets where the relationships among data points can be easily visualized.

Step-by-Step Implementation

1. Importing Libraries:

   • Import the KNeighborsClassifier class from sklearn.neighbors.

   • Import train_test_split from sklearn.model_selection to split the dataset into training and testing sets.

   • Import accuracy_score and confusion_matrix from sklearn.metrics to evaluate the model.

   • Import numpy for numerical operations.

2. Preparing Data:

   • Create a NumPy array X representing the hours studied and prior grades of students.

   • Create a NumPy array y representing the outcomes (0 for fail, 1 for pass).

3. Splitting the Data:

   • Use train_test_split to divide the data into training and testing sets.

   • Specify test_size=0.2 to use 20% of the data for testing and 80% for training.

   • Set random_state=42 for reproducibility.

4. Initializing and Training the Model:

   • Create an instance of the KNeighborsClassifier class, specifying the number of neighbors (n_neighbors). For example, n_neighbors=3 means the model will classify a new data point based on the majority class among its three nearest neighbors.

   • Train the model using the training data (X_train, y_train). KNN is a non-parametric model, meaning it stores the training data to make predictions based on the nearest neighbors.

5. Making Predictions:

   • Use the trained KNN model to make predictions on the test data X_test.

   • The output y_pred contains the model's predictions (0 or 1).

6. Evaluating the Model:

   • Calculate the accuracy of the model by comparing the actual values y_test to the predicted values y_pred using accuracy_score.

   • Compute the confusion matrix to understand true positives, true negatives, false positives, and false negatives.

   • Print the accuracy and confusion matrix.

Complete Code Example

# Import libraries
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
import numpy as np

# Prepare data
X = np.array([[,], [,], [,], [,], [,], [,], [,], [,], [,], [,]])
y = np.array()

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize and train the model
model = KNeighborsClassifier(n_neighbors=3)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
confusion_matrix = confusion_matrix(y_test, y_pred)

# Print results
print("Accuracy:", accuracy)
print("Confusion Matrix:\n", confusion_matrix)

Conclusion

This article demonstrates a full workflow for training and evaluating a K-Nearest Neighbor classifier. The KNN model predicts binary outcomes based on hours studied and prior grades, showing how it classifies each test data point by looking at the classes of the nearest neighbors.