Module Snapshot
Module Title:
Module 5 – Model Training & Quality Control (No-Code)
AI Lifecycle Stage:
Model Training, Evaluation & Deployment
Target Level:
Intermediate
Learning Pathway:
No-code (Operational / Management Focus)
Estimated Length:
~90 minutes
Primary Tools:
Hugging Face AutoTrain, Vertex AI, Amazon Q
Why No-Code Model Training & Quality Control?
Once library datasets are cleaned (Module 2) and annotated (Module 4), librarians can move beyond general-purpose commercial chatbots to train custom, domain-specific AI models. Modern Automated Machine Learning (AutoML) platforms allow library staff to build, evaluate, deploy, and monitor custom classifiers without writing code.
Module Goal: Learners practice training a custom "Library Ticket Classifier" using Hugging Face AutoTrain, evaluating model performance with quantitative metrics and a Confusion Matrix, testing deployed endpoints on Vertex AI / Hugging Face, and exploring AI governance and model drift monitoring.
This module provides academic librarians and library managers with the strategic and operational skills needed to oversee AI training pipelines, interpret quality metrics, and enforce ongoing AI governance in library systems.
The AutoML & Quality Control Workflow
This module follows a 6-step operational lifecycle for non-coding library managers and specialists:
1. Upload Data
Import clean, labeled dataset (e.g., 200 labeled Library Support Tickets) into the AutoML platform.
2. Auto-Train
Configure task parameters; the platform automatically tests, tunes, and selects the optimal algorithms.
3. Evaluate Metrics
Analyze quantitative metrics: Accuracy, Precision, Recall, and F1-Score to assess model quality.
4. Diagnose Errors
Inspect the Confusion Matrix to identify where the AI gets confused between categories.
5. Deploy Endpoint
Publish the model endpoint to Vertex AI or Hugging Face Spaces for real-time testing and inference.
6. Govern & Monitor
Establish dashboards to track performance over time, detect model drift, and prevent bias.
Task 1: Auto-Training a Text Classifier with Hugging Face AutoTrain
Automated Machine Learning (AutoML) automates the complex process of model architecture selection, hyperparameter tuning, and validation. In this task, learners use Hugging Face AutoTrain to build a custom text classification model without writing a single line of code.
Hands-on Auto-Training Steps
- Log into Hugging Face AutoTrain and create a new Text Classification project.
- Upload the labeled CSV dataset of 200 Library Support Tickets.
- Map the columns: set
textas input andtarget(category label) as target. - Click Start AutoTrain to launch automatic candidate training.
- Review candidate model rankings based on validation scores.
Teaching Point for Librarians
AutoML allows librarians to focus on domain data quality and task definition rather than hyperparameter tuning. The platform tests multiple transformer architectures (e.g., BERT, RoBERTa, DistilBERT) automatically.
Sample Dataset Structure (200 Library Support Tickets)
| ID | Ticket Text (Input) | Assigned Category (Target Label) |
|---|---|---|
| T-001 | "I cannot access IEEE Xplore from home using EZproxy." | Technical Issue |
| T-002 | "How do I renew my overdue interlibrary loan books online?" | Account Inquiry |
| T-003 | "Need peer-reviewed articles on microplastics in freshwater systems." | Reference Question |
Task 2: Understanding Evaluation Metrics & Dashboard Analysis
Evaluating AI models requires moving beyond simple qualitative impressions ("it looks right") to hard quantitative data. AutoTrain and Vertex AI present dashboards with core evaluation metrics.
Accuracy
Overall percentage of tickets correctly classified across all categories.
Correct Predictions / Total Tickets
Precision
Of all tickets flagged as "Technical Issue", how many were actually technical issues? (Measures exactness / low false alarms).
True Positives / (True Positives + False Positives)
Recall
Of all actual "Technical Issue" tickets, how many did the AI successfully catch? (Measures completeness / low missed tickets).
True Positives / (True Positives + False Negatives)
F1-Score
The harmonic mean of Precision and Recall, providing a single balanced quality metric.
2 * (Precision * Recall) / (Precision + Recall)
Dashboard Diagnostics: The Confusion Matrix
A Confusion Matrix is a diagnostic grid showing where the model succeeds and where it confuses labels. Below is the evaluation result for our 200 Library Support Tickets:
| Actual Label \ Predicted | Pred: Technical Issue | Pred: Account Inquiry | Pred: Reference Question |
|---|---|---|---|
| Actual: Technical Issue | 64 (Correct) | 5 (Mistake) | 1 |
| Actual: Account Inquiry | 8 (Confusion!) | 54 (Correct) | 3 |
| Actual: Reference Question | 2 | 2 | 61 (Correct) |
Diagnostic Insight: Notice that the model frequently mistakes Account Inquiries for Technical Issues (8 cases). For example, a ticket saying "I can't log in to see my loans" mentions "log in" (technical keyword) but relates to patron accounts. This insight tells librarians to add more specific training examples clarifying login-related account inquiries!
Task 3: Model Deployment & Interactive Testing
Once model training and evaluation are complete, the model must be deployed to an endpoint so library staff and automated workflows can query it in real time.
Hugging Face Spaces & Inference API
AutoTrain models can be deployed directly to a Hugging Face Space or Inference Endpoint with one click, providing an interactive web widget to test ticket classification.
Google Cloud Vertex AI
For enterprise library IT environments, models deployed on Vertex AI offer scalable API endpoints, low latency, role-based access control, and SLA monitoring.
Interactive Endpoint Testing Demonstration
- Open the deployed model endpoint interface.
- Enter a new unlabelled ticket: "The proxy popup says invalid password when accessing JSTOR off-campus."
- Click Predict Class.
- Observe the output confidence scores:
- Technical Issue: 92.4%
- Account Inquiry: 6.1%
- Reference Question: 1.5%
Task 4: AI Governance, Model Drift & Quality Control
Deploying a model is not the end of the AI lifecycle. Over time, AI models can suffer from Model Drift—a decline in accuracy caused by changing user terminology, new library services, or shifting vocabulary.
Data & Concept Drift
When the library introduces a new authentication system (e.g. switching from EZproxy to OpenAthens), patrons submit new terms. Without retraining, the AI's accuracy degrades over time.
Bias & Fairness Auditing
Ensuring the ticket classifier treats patron inquiries equitably regardless of phrasing, language background, or academic department.
Library AI Governance Checklist
- Monthly Performance Audit: Sample 50 live tickets to recalculate Precision, Recall, and F1 scores.
- Human-in-the-Loop Safeguard: Automatically route low-confidence predictions (< 75% score) to human staff.
- Retraining Schedule: Update training datasets quarterly with new vocabulary and newly resolved tickets.
- Privacy Compliance: Ensure patron names, ID numbers, and sensitive text are scrubbed before model training.
Task 5 (Exploratory): Agentic AI & Task Automation with Amazon Q
In addition to model training, modern library managers can explore Agentic AI and Auto-Programming tools like Amazon Q Developer to automate repetitive operational tasks via natural language prompts.
Automation Experiment: Learners enter natural language instructions into an AI assistant to auto-generate library workflow scripts (e.g., "Write a workflow to parse incoming CSV book lists by ISBN and highlight missing catalog entries" or "Trigger an alert when support ticket backlog exceeds 20 items").
This connects traditional model classification with agentic workflow automation, preparing librarians for end-to-end service integration (Module 6).
Data & Materials
Data Source
A curated dataset of 200 "Library Support Tickets" evenly distributed across 3 operational categories:
- Technical Issue: Proxy errors, e-resource access, website glitches.
- Account Inquiry: Fines, borrowing limits, card renewals.
- Reference Question: Research help, citation assistance, database guidance.
Expected Learner Outputs
- Deployed Model Endpoint: A functioning text classifier hosted on Hugging Face or Vertex AI.
- Model Evaluation Report: A concise document summarizing accuracy, precision/recall trade-offs, and confusion matrix insights.
Core Skills in This Module
AutoML Training
Configuring and running code-free model training pipelines on Hugging Face AutoTrain and Vertex AI.
Quantitative Metric Analysis
Evaluating model performance using Accuracy, Precision, Recall, and F1-Score instead of intuitive guesses.
Confusion Matrix Diagnostics
Reading confusion matrices to pinpoint classification errors and target data cleaning efforts.
AI Governance & Monitoring
Tracking model drift over time, auditing bias, and enforcing human-in-the-loop safeguards.
Library and GLAM Applications
- Automated Support Triage: Automatically routing patron tickets to IT, Circulation, or Reference subject specialists.
- Catalog Subject Auto-Tagging: Training custom text classifiers to assign subject tags to digital book descriptions.
- E-Resource Access Monitoring: Detecting recurring access failures from ticket text before system outages worsen.
- AI Policy & Governance: Developing institutional guidelines for model evaluation, retraining, and patron privacy.
Video Tutorials
Replace each YouTube link placeholder with the final video URL after uploading.
Demonstrates how to set up an AutoTrain project, upload labeled library support tickets, configure parameters, and train a text classifier without code.
Explains how to interpret evaluation dashboards, calculate Precision, Recall, and F1-Scores, and diagnose error patterns using a confusion matrix.
Walks through deploying models to Vertex AI / Hugging Face endpoints, conducting real-time inference tests, and monitoring model drift in library operations.
Learning Outcomes
- Explain the core principles of Automated Machine Learning (AutoML) and how platforms select models.
- Upload labeled datasets to Hugging Face AutoTrain and execute code-free text classifier training.
- Evaluate machine learning models using quantitative metrics: Accuracy, Precision, Recall, and F1-Score.
- Analyze a Confusion Matrix to pinpoint specific misclassification patterns in library data.
- Deploy trained models to cloud endpoints (Vertex AI or Hugging Face) and test live inference queries.
- Formulate an AI governance checklist to monitor model drift, prevent algorithmic bias, and protect patron privacy.