Model & evaluation

A transparent plan, not a performance claim

This page documents the planned modeling and evaluation approach. The model has not been trained or evaluated in this prototype, so no performance numbers are shown.

System architecture

User upload

Browser

Web UI

This app

Inference API

VITE_INFERENCE_API_URL

Planned · not connected

Preprocessing

Crop · resize · normalize

Planned · not connected

Classifier

CNN, 5 classes

Planned · not connected

Quality / uncertainty

Abstain if unreliable

Planned · not connected

Output + safety notice

Model output, not diagnosis

Differentiator: the workflow is quality-aware and uncertainty-aware. When the image or model cannot support a reliable output, no grade is shown — only a recommendation for professional image review.

Modeling approach

Task: Five-class supervised image classification of fundus photographs, grades 0–4.
Backbone: Transfer learning from an ImageNet-pretrained CNN (e.g., EfficientNet or ResNet), fine-tuned end-to-end after a frozen warm-up.
Preprocessing: Crop to the retinal disc, resize to a fixed resolution, channel normalization. Optional contrast enhancement only if validated to help on held-out data.
Class imbalance: Class-weighted loss and/or balanced sampling; monitored via per-class recall rather than overall accuracy.
Splitting: Stratified split by grade; patient-level grouping wherever metadata supports it to prevent leakage between eyes/images.
Tuning: Hyperparameters and referral threshold chosen on the validation split only.
Hold-out: A locked test set evaluated once, after all design choices are frozen.

Dataset candidate

APTOS 2019 Blindness Detection

Retinal fundus photographs with diabetic retinopathy severity grades 0–4, released for a Kaggle competition.

View on Kaggle

Access is subject to Kaggle competition rules. Users must accept the terms and verify permitted use before downloading or redistributing. No dataset images are bundled with or redistributed by this app.

0 — No apparent DR signs
1 — Mild
2 — Moderate
3 — Severe
4 — Proliferative DR

Why accuracy is not enough

Screening datasets are heavily imbalanced toward grade 0. A model predicting “no signs” for nearly every image can look accurate while missing the cases that matter most. Evaluation therefore prioritizes:

Sensitivity

Of images truly at or above the referral grade, the share the model flags. Missing these is the costliest error in screening.

Value: Not evaluated yet

Specificity

Of images truly below the referral grade, the share correctly not flagged. Affects unnecessary referrals.

Value: Not evaluated yet

Macro F1

F1 averaged equally across all five grades, so rare severe grades count as much as common ones.

Value: Not evaluated yet

Per-class recall

Recall reported for each grade individually, exposing classes the model neglects.

Value: Not evaluated yet

Quadratic weighted kappa

Agreement with reference grades that penalizes errors more the further apart grades are.

Value: Not evaluated yet

Calibration

Whether predicted probabilities match observed frequencies (e.g., reliability diagram, ECE).

Value: Not evaluated yet

Confusion matrix + error analysis

Where and why errors happen — image quality, grade boundaries, subgroups.

Value: Not evaluated yet

Confusion matrix
Not evaluated yet

Evaluation checklist

Dataset terms reviewed and accepted
Pending
Patient-level leakage check
Pending
Locked hold-out set defined before training
Pending
Referral threshold prespecified
Pending
Per-class metrics computed
Pending
Calibration assessed
Pending
Quality/uncertainty abstention evaluated
Pending
Subgroup checks where metadata allows
Pending
Error review of false negatives
Pending
Model card completed and versioned
Pending

Model card (skeleton)

Intended use: Research and education on AI-assisted DR screening triage. Not for diagnosis or treatment decisions.
Data sources: Candidate: APTOS 2019 (subject to Kaggle terms). Not yet acquired.
Split strategy: Stratified train / validation / locked hold-out; patient-level where possible.
Metrics: Macro F1, per-class recall, sensitivity/specificity at referral threshold, QWK, calibration — not evaluated yet.
Subgroup checks: Planned where metadata allows (e.g., camera/site). Not performed.
Known limitations: Single-source data, label noise, unknown generalization to other devices and populations, no clinical validation.
Version / date: v0.0 — untrained · documentation draft