Skip to content

ML_RETRAINING.md

Latest commit

 

History

History
239 lines (192 loc) · 7.99 KB

ML_RETRAINING.md

File metadata and controls

239 lines (192 loc) · 7.99 KB
Raw

VaxTrust ML Model Retraining & Real-Time Trust Scores

Overview

This document describes the automatic ML model retraining system that has been implemented to automatically retrain the anomaly detection model and regenerate trust scores every time a user submits an adverse event report.

Architecture

1. Backend Services

Model Retraining Service (backend/services/model_retraining_service.py)

  • Purpose: Automatically retrain the Isolation Forest ML model with new user reports
  • Key Features:
    • Converts user symptom reports into AEFI-formatted features
    • Incrementally retrains the anomaly detection model
    • Updates and saves model artifacts (model.pkl, scaler.pkl, features.pkl)
    • Symptom-to-diagnosis mapping for consistent feature engineering

Report Service (backend/services/report_service.py)

  • Updated: Now returns the saved report as a dict for model retraining
  • Purpose: Stores user reports in CSV and provides data for retraining

Flask App (backend/app.py)

  • New Components:
    • ModelRetrainingService initialization on startup
    • Async background thread for model retraining after report submission
    • New endpoint /api/trust-scores/live with model status info
  • Flow:
    1. User submits report → POST /api/report-symptom
    2. Report saved to CSV
    3. Background thread triggers retrain_with_user_reports()
    4. AEFI data reloaded with updated trust scores
    5. Frontend polls for updated scores

2. Frontend Updates

Web Frontend (frontend/src/)

API Service (services/api.js):

  • New function: getTrustScoresLive() - fetches trust scores with model status

Dashboard Page (pages/DashboardPage.js):

  • Uses getTrustScoresLive() instead of getTrustScores()
  • Auto-refresh every 3 seconds when retraining_in_progress = true
  • Displays "Updating trust scores..." alert with spinner

Feedback Page (pages/FeedbackPage.js):

  • Success message now mentions trust scores are being updated
  • Blue info box: "Trust scores are being updated with your report. Check the dashboard to see the latest scores!"

Mobile App (vaxtrust-mobile/src/)

API Service (services/api.js):

  • New function: getTrustScoresLive() - same as web

Dashboard Screen (screens/DashboardScreen.js):

  • Uses getTrustScoresLive()
  • Auto-refresh while retraining is in progress
  • Displays warning alert: "Updating trust scores with latest user reports"

Feedback Screen (screens/FeedbackScreen.js):

  • Success screen includes blue info box mentioning trust scores update

API Endpoints

1. /api/report-symptom (POST) - Enhanced

Request:

{
  "batch_id": "COV_AN_202312_00070",
  "symptoms": ["fever", "headache"],
  "severity": "moderate",
  "onset_days": 2,
  "notes": "Optional custom description"
}

Response:

{
  "success": true,
  "report_id": "RPT_1707406943517",
  "message": "Report submitted successfully. Trust scores are being updated.",
  "retraining_started": true
}

2. /api/trust-scores (GET) - Existing

Returns the aggregated trust scores for all vaccines.

3. /api/trust-scores/live (GET) - NEW

Response:

{
  "vaccines": [
    {
      "vaccine_name": "COVAXIN",
      "total_batches": 150,
      "trust_score": {
        "score": 0.92,
        "total_cases": 854,
        "causal_cases": 68,
        "classification_breakdown": {...}
      },
      "common_side_effects": [...]
    }
  ],
  "model_status": {
    "retraining_in_progress": true,
    "last_retrain_time": 1707406943.517,
    "timestamp": 1707406945.123
  }
}

How It Works

User Report Flow

  1. User submits symptoms through FeedbackPage/FeedbackScreen
  2. Backend receives POST request to /api/report-symptom
  3. Report is saved to user_reports.csv
  4. Async retraining thread starts:
    • Converts user report to AEFI features
    • Combines with existing processed AEFI data
    • Trains new Isolation Forest model
    • Saves updated model artifacts
    • Reloads AEFI_DF in memory
  5. Backend returns "retraining_started": true
  6. Frontend/Mobile shows success message with note about updates
  7. Frontend/Mobile starts auto-polling /api/trust-scores/live
  8. When retraining_in_progress = false, polling stops
  9. User sees updated trust scores on Dashboard

Trust Score Calculation

  • Based on: WHO causality assessment classifications (A4, B1, B2, C)
  • Formula: Trust Score = 1 - (Causal Cases / Total Cases)
  • Range: 0.0 (no trust) to 1.0 (complete trust)

Feature Mapping for Retraining

User reports are converted to AEFI features:

  • Symptoms → Diagnosis codes (e.g., fever → DIAGNOSIS_FEVER)
  • Severity → Outcomes & Hospitalization (mild → home, severe → hospitalized)
  • Onset days → Temporal features
  • Vaccine name → One-hot encoded vaccine columns

Configuration

Auto-Refresh Polling

  • Interval: 3 seconds (configurable)
  • Trigger: When modelStatus.retraining_in_progress === true
  • Stop: Automatically stops when retraining completes

Model Training Parameters

Located in ModelRetrainingService._train_isolation_forest():

  • Algorithm: Isolation Forest (sklearn)
  • Estimators: 300 trees
  • Contamination: 0.1 (10% anomalies)
  • Max Jobs: -1 (use all CPU cores)

Files Modified/Created

Created

  • backend/services/model_retraining_service.py - Complete retraining service
  • test_model_retraining.py - Test script for verification

Modified

  • backend/app.py - Added retraining logic and new endpoint
  • backend/services/report_service.py - Returns report dict
  • frontend/src/services/api.js - Added getTrustScoresLive()
  • frontend/src/pages/DashboardPage.js - Auto-refresh logic
  • frontend/src/pages/FeedbackPage.js - Success message update
  • vaxtrust-mobile/src/services/api.js - Added getTrustScoresLive()
  • vaxtrust-mobile/src/screens/DashboardScreen.js - Auto-refresh + status alert
  • vaxtrust-mobile/src/screens/FeedbackScreen.js - Success message update

Data Files Updated

  • vaxtrust_covid_data/user_reports.csv - Fixed header with "other" column
  • backend/data/user_reports.csv - Fixed header with "other" column

Testing

Run the test script to verify model retraining:

cd /home/frosfres/Documents/project/Vaxtrust
./.venv/bin/python test_model_retraining.py

Expected output:

Testing model retraining service...
==================================================

Retraining Result:
Success: True
Message: Model retrained successfully
Records Processed: 2
Total Records: 3
Timestamp: 2026-02-08T15:43:03.517381

✓ Model retraining completed successfully!
==================================================

Performance Considerations

  • Retraining Time: ~1-2 seconds per report (background thread)
  • Memory Usage: Loads AEFI data into memory (~100MB for full dataset)
  • Scalability: Isolation Forest scales well with feature count, not sample count
  • Polling: 3-second interval strikes balance between freshness and server load

Future Enhancements

  1. WebSocket Integration: Real-time push updates instead of polling
  2. Batch Processing: Accumulate N reports before retraining
  3. Model Versioning: Save historical model versions
  4. Performance Metrics: Track model accuracy metrics over time
  5. Distributed Training: Use multi-processing for faster retraining
  6. Database Integration: Move from CSV to SQL database for analytics

Troubleshooting

Retraining not triggering?

  • Check Flask app logs for errors during retraining
  • Verify RETRAINING_SERVICE is initialized
  • Ensure report_service.py returns dict from save_user_report()

Trust scores not updating on frontend?

  • Check browser console for API errors
  • Verify /api/trust-scores/live endpoint returns model_status
  • Check auto-refresh interval (should be 3 seconds)

Model retraining failing?

  • Check logs for feature column mismatches
  • Verify AEFI CSV columns exist
  • Ensure scikit-learn and pandas are installed

Support

For questions or issues, refer to the main README.md and project documentation.