🚧 Under construction — VoTechWorks is being built. A free tech program — complimentary, as our commitment to get America working.
voTechWorksGrade 13 · a vocational technology learning center, by urbanicity
newSchoolWorks.com › VoTechWorks › Technology & Design › Data Science & Analytics › Introduction to Machine Learning for Data Analytics › Lesson 9
TEC-DATA85-08 · Lesson 9 · 9 clock hours · lab

Capstone ML Project: End-to-End Build and Defense

Objective. Students will independently complete a documented end-to-end ML pipeline on an unseen dataset, present findings in a 10-minute defense, and respond to evaluator questions.

⚠ Pending professional review

This lesson, its worked examples and its practice questions has not yet been verified by an industry-certified professional or by a certification or approval body. It is a study draft and may contain errors. Do not use it as instructions for work on a job site or on live equipment. Always follow your instructor, your supervising professional, and current official references and regulations.

VoTechWorks program approval and accreditation are still in progress. Credentials are issued only by the certifying bodies named in each program, after their own exams.

Capstone ML Project: End-to-End Build and Defense

What This Lesson Asks of You

This capstone is your opportunity to demonstrate every skill you have built in this course. You will receive a dataset you have never seen before, work through a complete machine learning pipeline on your own, and defend your decisions in front of a panel. The goal is not perfection — it is evidence of sound, professional practice.

Step 1: Exploratory Data Analysis (EDA)

Before touching a model, spend real time understanding your data. Compute descriptive statistics, check distributions, identify missing values, and look for outliers. Plot relationships between features and the target variable. Document every observation in your notebook with Markdown cells — a panel evaluator should be able to read your notebook like a lab report and understand your reasoning without asking you a single question.

Common EDA tasks: shape and data types, null counts, class balance for classification targets, correlation heatmaps, and pairplots for small feature sets. Note any data quality issues you find. These notes protect you during the defense because they show you were paying attention, not just running code.

Step 2: Feature Engineering

Raw data rarely enters a model as-is. Feature engineering includes encoding categorical variables (one-hot encoding, ordinal encoding), scaling numerical features (StandardScaler, MinMaxScaler), handling missing data (imputation or removal with justification), and creating new features when domain logic supports it. Document every transformation. Your model card must describe each feature used in the final model.

Step 3: Model Selection and Justification

Choose at least two candidate algorithms. Compare a simpler baseline (logistic regression, decision tree) with a more complex approach (random forest, gradient boosting). Justify your choices in writing: why is this algorithm appropriate for the problem type and the data characteristics? Complexity for its own sake is not a virtue — a simpler model that generalizes better is the correct choice.

Use a train/validation/test split or cross-validation. Never tune hyperparameters on the test set. Evaluate on the held-out test set exactly once, after all tuning is complete.

Step 4: Hyperparameter Tuning

Use grid search, random search, or another systematic method. Document the parameter space you searched and why. Report both training and validation performance across your search to show you monitored for overfitting. scikit-learn's `GridSearchCV` and `RandomizedSearchCV` are standard tools for this step.

Step 5: Final Evaluation and the Model Card

Report at least one primary metric appropriate to the task (accuracy is rarely sufficient alone — include precision, recall, F1, or AUC-ROC for classification; RMSE or MAE for regression). Explain why you chose each metric. Discuss at least one failure mode: a pattern the model got wrong, a subgroup where performance dropped, or a data quality issue that limited results. Identifying failure modes is a sign of professional maturity, not weakness.

Complete the model card template provided. A model card is a short structured document that summarizes what a model does, how it was trained, what its performance is, and where it may not be appropriate to use. Model cards are now standard practice in responsible AI development.

Step 6: The 10-Minute Oral Defense

Structure your presentation around four areas: (1) problem framing and dataset overview, (2) your methodology and key decisions, (3) metric results with honest interpretation, and (4) at least one failure mode and how you addressed or would address it. Practice to time — ten minutes passes quickly. Speak to the slide, not at it. The panel will ask follow-up questions; answer what you know, and say clearly if you are uncertain rather than guessing.

Step 7: Written Reflection

Within 24 hours of your defense, submit a minimum 200-word reflection. Describe one concrete thing you would do differently with more time. Then map at least two steps from your project to specific exam domains from CompTIA Data+, IBM Data Science Professional, or Google Data Analytics. This reflection is not optional — it is part of your passing score.

Professional and Ethical Considerations

Data safety applies even in a classroom setting. Do not share the instructor-curated dataset outside the class or post it publicly, as datasets may contain synthetic representations of sensitive record types. When working with any dataset that resembles personal information (names, IDs, health indicators), apply the same care you would in a professional environment: do not attempt to re-identify individuals, and flag any concerns to your instructor immediately.

Report your results honestly. Inflating a metric by leaking the test set into training, or omitting a poor result, is a form of data misrepresentation. Employers and credentialing bodies treat this as a serious ethics violation. Your rubric rewards accurate reporting, not the highest number.

Rubric Summary

The 20-point panel rubric covers: data preparation quality (5 points), model selection justification (5 points), metric reporting accuracy (5 points), and clarity of defense communication (5 points). You must score at least 14 out of 20 to pass. Submit your Jupyter notebook and model card at least 48 hours before your scheduled defense.

Key terms8

End-to-end ML pipelineThe complete sequence of steps to build and deploy a machine learning solution, from raw data ingestion through EDA, feature engineering, model training, evaluation, and documentation.
Model cardA short structured document that describes a trained model's purpose, training data, performance metrics, intended uses, and known limitations, following responsible AI disclosure practices.
Hyperparameter tuningThe process of systematically searching for the best configuration settings for a machine learning algorithm — settings that are set before training, not learned from data.
Train/validation/test splitA data partitioning strategy in which data is divided into three non-overlapping sets: one for training the model, one for tuning and comparing models, and one held out for final unbiased evaluation.
Feature engineeringThe process of transforming, combining, or creating input variables from raw data to improve a model's ability to learn meaningful patterns.
Failure modeA specific condition, subgroup, or pattern for which a model produces incorrect or unreliable predictions; identifying failure modes is a required part of responsible model reporting.
Cross-validationA resampling technique that divides data into multiple folds, training and evaluating the model on different subsets in turn, to produce a more reliable estimate of generalization performance.
AUC-ROCArea Under the Receiver Operating Characteristic Curve; a metric for binary classification models that measures the model's ability to distinguish between classes across all classification thresholds.

Worked examples4

1. Your binary classification model on a test set of 200 samples produces 150 true positives, 30 true negatives, 10 false positives, and 10 false negatives. Calculate accuracy, precision, and recall.
  1. Identify counts: TP = 150, TN = 30, FP = 10, FN = 10. Total samples = 150 + 30 + 10 + 10 = 200.
  2. Accuracy = (TP + TN) / Total = (150 + 30) / 200 = 180 / 200 = 0.90, or 90%.
  3. Precision = TP / (TP + FP) = 150 / (150 + 10) = 150 / 160 = 0.9375, or approximately 93.8%.
  4. Recall = TP / (TP + FN) = 150 / (150 + 10) = 150 / 160 = 0.9375, or approximately 93.8%.
Answer: Accuracy = 90%, Precision ≈ 93.8%, Recall ≈ 93.8%. Even though accuracy is high, always report precision and recall separately — in a different class-balance scenario these three metrics can diverge significantly.
2. You run a 5-fold cross-validation and get F1 scores of 0.82, 0.79, 0.85, 0.81, and 0.83. What is the mean F1 score and the range (max minus min)?
  1. Sum the five F1 scores: 0.82 + 0.79 + 0.85 + 0.81 + 0.83 = 4.10.
  2. Divide by 5 to get the mean: 4.10 / 5 = 0.82.
  3. Identify the maximum (0.85) and minimum (0.79) scores.
  4. Range = 0.85 − 0.79 = 0.06.
Answer: Mean F1 = 0.82; Range = 0.06. A small range indicates consistent performance across folds, which is a positive sign. Report both values in your model card.
3. Your regression model produces the following absolute errors on five test samples: 3.2, 1.5, 4.8, 2.1, and 3.4. Calculate the Mean Absolute Error (MAE).
  1. Confirm all values are already absolute (non-negative): 3.2, 1.5, 4.8, 2.1, 3.4.
  2. Sum the absolute errors: 3.2 + 1.5 + 4.8 + 2.1 + 3.4 = 15.0.
  3. Divide by the number of samples (5): MAE = 15.0 / 5 = 3.0.
Answer: MAE = 3.0 in the same units as your target variable. MAE is interpretable and robust to outliers compared to RMSE, making it a useful primary metric to report alongside RMSE.
4. You split 1,000 labeled samples into 70% train, 15% validation, and 15% test. After hyperparameter tuning, your best model scores 0.88 F1 on validation and 0.81 F1 on test. Calculate the size of each partition and explain what the gap between validation and test performance suggests.
  1. Training set size: 1,000 × 0.70 = 700 samples.
  2. Validation set size: 1,000 × 0.15 = 150 samples.
  3. Test set size: 1,000 × 0.15 = 150 samples.
  4. Performance gap: 0.88 − 0.81 = 0.07 F1. A gap of this size suggests mild overfitting to the validation set during tuning — the model learned some characteristics specific to the validation distribution.
Answer: Train = 700, Validation = 150, Test = 150 samples. The 0.07 gap indicates the reported test F1 of 0.81 is the honest generalization estimate and should be the figure presented in your defense, not the higher validation score.

The math in this lesson1

Learners compute accuracy, precision and recall from confusion-matrix counts.
S-CP.4
Brush up in the math halls at newSchoolWorks.com:
In class: Students receive an instructor-curated dataset on Day 1 of this lesson block; independently perform EDA, feature engineering, model selection, hyperparameter tuning, and final evaluation; submit a complete Jupyter notebook and model card 48 hours before defense. · Ten-minute oral defense before a panel of at least two evaluators (instructor plus one peer or external reviewer): student presents problem framing, methodology, metric results, and at least one failure mode encountered and how it was addressed. · Post-defense written reflection (minimum 200 words): student describes one thing they would do differently if given more time and maps their project steps to at least two credential exam domains (CompTIA Data+, IBM Data Science Professional, or Google Data Analytics).
Materials: Instructor-curated dataset (held out until lesson day), Jupyter Lab, scikit-learn, presentation slides, model card template, rubric distributed in advance
Check: Panel uses a 20-point rubric covering: data preparation quality (5), model selection justification (5), metric reporting accuracy (5), and clarity of defense communication (5); minimum 14/20 to pass; written reflection must be submitted within 24 hours of defense.

Practice check8

Answer each question, then check it. Nothing is recorded; this is your own practice.

IBM Data Science Professional – Model Evaluation and Deployment
1. A student tunes hyperparameters by repeatedly evaluating candidate configurations on the test set until performance improves. What is the primary problem with this approach?
Google Data Analytics – Data Presentation and Communication
2. Which document, now standard in responsible AI practice, summarizes a model's intended use, training data, performance metrics, and known limitations?
CompTIA Data+ – Data Analysis and Statistics
3. A binary classifier achieves 95% accuracy on a dataset where 95% of samples belong to the negative class. What does this reveal about using accuracy as the sole metric?
IBM Data Science Professional – Machine Learning Algorithms
4. During an oral defense, an evaluator asks why the student chose a random forest over logistic regression for their problem. Which response best demonstrates professional justification?
Google Data Analytics – Data Preparation and Processing
5. What is the correct order of steps in a well-structured end-to-end ML pipeline?
CompTIA Data+ – Data Analysis and Statistics
6. A student's 5-fold cross-validation F1 scores are 0.74, 0.76, 0.75, 0.73, and 0.75. Their test set F1 is 0.74. What do these results most strongly suggest?
IBM Data Science Professional – AI Ethics and Responsible Data Science
7. Why is it important to identify and disclose at least one failure mode in your model card and defense?
CompTIA Data+ – Data Preparation and Transformation
8. A student applies StandardScaler to the entire dataset before splitting it into train and test sets. What error does this introduce?
Record this lesson on my transcript. Sign in with your key and enrollment email to record lessons (enroll free if you have no key). This page stays open to read and practise.
← Previous lessonProgram plan