🚧 Under construction — VoTechWorks is being built. A free tech program — complimentary, as our commitment to get America working.
voTechWorksGrade 13 · a vocational technology learning center, by urbanicity
newSchoolWorks.com › VoTechWorks › Technology & Design › Data Science & Analytics › Introduction to Machine Learning for Data Analytics › Lesson 3
TEC-DATA85-08 · Lesson 3 · 12 clock hours · demo

Linear and Logistic Regression: Theory and Implementation

Objective. Students will train linear regression and logistic regression models, interpret coefficients, and evaluate results using RMSE/R² and accuracy/confusion matrix respectively.

⚠ Pending professional review

This lesson, its worked examples and its practice questions has not yet been verified by an industry-certified professional or by a certification or approval body. It is a study draft and may contain errors. Do not use it as instructions for work on a job site or on live equipment. Always follow your instructor, your supervising professional, and current official references and regulations.

VoTechWorks program approval and accreditation are still in progress. Credentials are issued only by the certifying bodies named in each program, after their own exams.

Linear and Logistic Regression: Theory and Implementation

Regression models are among the most widely used tools in data analytics. Understanding how they work mathematically—and how to explain them clearly to a non-technical audience—is a core competency for any data professional.

Ordinary Least Squares and Linear Regression

Linear regression models the relationship between one or more input features and a continuous numeric target. The simplest case, simple linear regression, fits a straight line: ŷ = β₀ + β₁x, where β₀ is the intercept and β₁ is the slope. With multiple features you extend this to ŷ = β₀ + β₁x₁ + β₂x₂ + … + βₙxₙ.

Ordinary Least Squares (OLS) finds the coefficients that minimize the sum of squared residuals—the squared differences between each actual value y and the model's prediction ŷ. Intuitively, you are finding the line (or hyperplane) that sits as close as possible to all the data points at once. Each coefficient βᵢ tells you: holding all other features constant, a one-unit increase in xᵢ is associated with a βᵢ-unit change in the predicted target. This is the sentence you should be able to say to a client.

Evaluating Linear Regression: RMSE and R²

Root Mean Squared Error (RMSE) measures average prediction error in the same units as the target. Lower is better. R² (coefficient of determination) measures the proportion of variance in the target explained by the model, ranging from 0 to 1 (or negative if the model is worse than predicting the mean). An R² of 0.75 means the model explains 75% of the variability in the outcome.

Always inspect a residual plot (residuals vs. fitted values). Residuals should scatter randomly around zero. Patterns—curves, funnels—signal that the linear model is missing something: a non-linear relationship, or non-constant variance (heteroscedasticity).

The Logistic Sigmoid Function and Logistic Regression

When the target is a category (yes/no, survived/did not survive), you need a model that outputs a probability between 0 and 1, not an unbounded number. Logistic regression does this by applying the sigmoid function to a linear combination of features:

P(y=1) = 1 / (1 + e^(−z)), where z = β₀ + β₁x₁ + … + βₙxₙ

The sigmoid squashes any real number into the range (0, 1). You choose a threshold—commonly 0.5—to convert that probability into a class prediction. Coefficients in logistic regression are interpreted in terms of log-odds: a one-unit increase in xᵢ multiplies the odds of the positive outcome by e^βᵢ. For a client briefing, it is clearest to say something like: passengers in first class had a substantially higher estimated probability of survival than those in third class, even after accounting for age and sex.

Evaluating Logistic Regression: Accuracy and Confusion Matrix

Accuracy is the fraction of predictions the model got right. It is easy to understand but misleading when classes are imbalanced—if 95% of cases are negative, a model that always predicts negative has 95% accuracy but is useless.

The confusion matrix breaks results into four cells: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). From these you can compute precision (TP / (TP + FP))—how often a positive prediction is correct—and recall (TP / (TP + FN))—how many actual positives the model caught. Always report at least one metric beyond raw accuracy.

Practical Implementation with scikit-learn

In scikit-learn, both models follow the same pattern: import the class, instantiate it, call `.fit(X_train, y_train)`, then `.predict(X_test)`. For linear regression use `LinearRegression`; for logistic regression use `LogisticRegression`. Metrics live in `sklearn.metrics`: `mean_squared_error` (take the square root for RMSE), `r2_score`, `accuracy_score`, and `confusion_matrix`. Visualize the confusion matrix as a heatmap with `seaborn.heatmap` for instant readability.

Split your data before fitting with `train_test_split`. Never evaluate a model on the same data it was trained on—this inflates performance metrics and misleads everyone, including yourself.

Data and Workspace Safety

When working with real datasets that may contain personally identifiable information (PII)—such as names, survival records, or location data—handle them according to your organization's data governance policy. Do not upload sensitive data to public repositories. On shared lab machines, clear notebook outputs containing PII before logging off. These are professional habits that protect both the individuals in the data and your own reputation.

Communicating Results to Non-Technical Readers

A strong five-sentence interpretation covers: (1) what question the model answers, (2) what the key metric means in plain language, (3) what the model's biggest limitation is, (4) one actionable insight, and (5) a caution about what the model cannot tell us. Clients make decisions based on your summaries—precision and honesty matter as much here as they do in the code.

Key terms10

Ordinary Least Squares (OLS)A method for fitting a linear regression model by minimizing the sum of squared differences between observed values and model predictions.
ResidualThe difference between an actual observed value and the value predicted by a model (residual = y − ŷ); used to diagnose model fit.
RMSE (Root Mean Squared Error)The square root of the average squared residuals; expresses average prediction error in the same units as the target variable. Lower values indicate better fit.
R² (Coefficient of Determination)A metric from 0 to 1 that represents the proportion of variance in the target variable explained by the model; 1 indicates a perfect fit.
Sigmoid FunctionA mathematical function, 1 / (1 + e^(−z)), that maps any real number to a value between 0 and 1; used in logistic regression to output class probabilities.
Confusion MatrixA table showing the counts of true positives, true negatives, false positives, and false negatives produced by a classification model, allowing detailed evaluation beyond simple accuracy.
PrecisionThe proportion of positive predictions that are actually correct: TP / (TP + FP). Relevant when false positives are costly.
Recall (Sensitivity)The proportion of actual positive cases that the model correctly identified: TP / (TP + FN). Relevant when false negatives are costly.
OverfittingWhen a model learns the training data too closely, including its noise, and therefore performs worse on new, unseen data than its training metrics suggest.
Train-Test SplitThe practice of dividing a dataset into separate subsets for training the model and evaluating it, ensuring that performance metrics reflect behavior on unseen data.

Worked examples4

1. A linear regression model predicts house prices. On a 200-sample test set, the sum of squared residuals is 18,000,000 (dollars²). Compute the RMSE.
  1. Step 1 – Compute Mean Squared Error (MSE): MSE = Sum of Squared Residuals / n = 18,000,000 / 200 = 90,000.
  2. Step 2 – Take the square root: RMSE = √90,000 = 300.
  3. Step 3 – Attach units: because the target is in dollars, RMSE = $300. On average, the model's predictions are off by $300.
Answer: RMSE = $300
2. A model's predictions and actual values for 5 test samples are: Actual = [200, 250, 180, 300, 220], Predicted = [210, 240, 190, 280, 230] (all in $000s). Compute R² using the formula R² = 1 − SS_res / SS_tot.
  1. Step 1 – Compute residuals: [200−210, 250−240, 180−190, 300−280, 220−230] = [−10, 10, −10, 20, −10].
  2. Step 2 – Compute SS_res (sum of squared residuals): (−10)²+(10)²+(−10)²+(20)²+(−10)² = 100+100+100+400+100 = 800.
  3. Step 3 – Compute the mean of actuals: (200+250+180+300+220)/5 = 1150/5 = 230.
  4. Step 4 – Compute SS_tot (total sum of squares): (200−230)²+(250−230)²+(180−230)²+(300−230)²+(220−230)² = 900+400+2500+4900+100 = 8800.
  5. Step 5 – Compute R²: R² = 1 − 800/8800 = 1 − 0.0909 ≈ 0.909.
Answer: R² ≈ 0.91, meaning the model explains about 91% of the variance in the test samples.
3. A logistic regression model is tested on 100 Titanic passengers. The confusion matrix shows: TP = 30, TN = 50, FP = 10, FN = 10. Compute accuracy, precision, and recall.
  1. Step 1 – Accuracy: (TP + TN) / Total = (30 + 50) / 100 = 80 / 100 = 0.80 (80%).
  2. Step 2 – Precision: TP / (TP + FP) = 30 / (30 + 10) = 30 / 40 = 0.75 (75%).
  3. Step 3 – Recall: TP / (TP + FN) = 30 / (30 + 10) = 30 / 40 = 0.75 (75%).
  4. Step 4 – Interpret: The model correctly classifies 80% of passengers overall, and when it predicts survival it is right 75% of the time; it also catches 75% of actual survivors.
Answer: Accuracy = 80%, Precision = 75%, Recall = 75%.
4. A logistic regression coefficient for the feature 'Pclass' (passenger class, coded 1/2/3) is β = −0.85. Compute the odds ratio and give a plain-language interpretation.
  1. Step 1 – Compute the odds ratio: e^β = e^(−0.85) ≈ 0.427.
  2. Step 2 – Interpret the odds ratio: For each one-unit increase in passenger class (e.g., moving from class 1 to class 2, or class 2 to class 3), the odds of survival are multiplied by approximately 0.43—meaning they are more than halved.
  3. Step 3 – Plain-language version for a client: 'Each step down in passenger class is associated with roughly a 57% reduction in the odds of survival, holding other factors constant.'
Answer: Odds ratio ≈ 0.43; moving to a higher class number is associated with substantially lower estimated survival odds.

The math in this lesson2

Learners take a square root of mean squared error to get RMSE in dollars.
N-RN.2S-ID.6
Brush up in the math halls at newSchoolWorks.com:
In class: Lecture with whiteboard derivation of ordinary least squares intuition and the logistic sigmoid function; students annotate slides with plain-language explanations they could use with a client. · Guided coding demo: fit LinearRegression on a California housing sample, plot residuals, compute RMSE and R²; then fit LogisticRegression on the Titanic dataset, display a confusion matrix heatmap using seaborn. · Individual mini-project: students choose one regression target and one classification target from a provided dataset list, fit both models, and write a five-sentence plain-language interpretation of each model's performance for a non-technical reader.
Materials: Jupyter Lab, scikit-learn, seaborn, matplotlib, statsmodels optional, California Housing and Titanic datasets (sklearn.datasets / Kaggle), Géron ch. 4
Check: Mini-project notebook must show correct metric computation and a plain-language summary that accurately describes model limitations; instructor rubric requires ≥ 70% on a 10-point rubric to pass.

Practice check8

Answer each question, then check it. Nothing is recorded; this is your own practice.

Data Analysis and Statistics
1. Which loss function does Ordinary Least Squares minimize when fitting a linear regression model?
Data Analysis and Statistics
2. A linear regression model produces an R² of 0.12 on the test set. What does this indicate?
Machine Learning Concepts
3. What is the primary purpose of the sigmoid function in logistic regression?
Data Preparation and Model Validation
4. A data analyst trains a logistic regression model without splitting the data, then reports 97% accuracy. What is the most likely problem?
Data Analysis and Statistics
5. A confusion matrix for a binary classifier shows TP = 40, TN = 30, FP = 20, FN = 10. What is the model's precision?
Data Analysis and Statistics
6. When a residual plot of a linear regression model shows a clear curved (U-shaped) pattern, what does this most likely indicate?
Machine Learning Concepts
7. In the context of the Titanic dataset, if recall for the 'survived' class is much lower than precision, what does this mean for the model's practical use?
Machine Learning Concepts
8. A logistic regression coefficient for a binary feature 'is_first_class' is β = 1.20. Which interpretation is most accurate?
Record this lesson on my transcript. Sign in with your key and enrollment email to record lessons (enroll free if you have no key). This page stays open to read and practise.
← Previous lessonProgram planNext lesson →