Helena Real-World Performance Framework

1.1. Purpose

This document establishes the methodology under which Helena Bioinformatics conducts Real-World Performance Studies of the Helena variant classification platform. It defines study design, ground truth construction, performance metrics, quality controls, conflict of interest management, and reporting standards applicable to Helena current organizational stage.

This Framework supersedes the prior Helena Clinical Validation Framework documents (v1.0 and v1.1, now archived). It reflects a deliberate methodological reframing: from a regulatory-validation paradigm calibrated for organizations with mature quality infrastructure, to a real-world performance characterization paradigm appropriate to an early-stage deep-technology startup pursuing clinical credibility through scientifically rigorous evidence accumulation.

1.2. Why Reframing

The prior framework treated a single peer reference laboratory as ground truth and structured study activities around regulatory validation conventions. Three observations during early use motivated reframing:

• Single-laboratory reference is not scientifically defensible as ground truth. Inter-laboratory concordance studies (Amendola 2016, Harrison 2017, Bergquist 2025) document 25 to 35 percent legitimate methodological disagreement between qualified laboratories. Treating any single peer laboratory as definitive truth misrepresents the state of the field.

• Helena classifier applies stricter ClinGen SVI 2020+ specifications than many peer laboratories applying historical 2015 ACMG. Where Helena and the comparator laboratory differ, the difference frequently reflects methodological evolution rather than Helena defect. Framing such cases as Helena failures was scientifically misleading.

• Helena Bioinformatics operates in Phase 1 (current organizational stage) with no immediate Notified Body engagement. Heavy regulatory-validation paperwork serves no current submission. The need is for credible, defensible, peer-publishable evidence supporting Helena classification claims, not for premature regulatory dossiers.

This Framework therefore positions Helena near-term work as Real-World Performance Studies producing scientifically rigorous, peer-reviewable evidence. Path to formal regulatory submission is preserved as a downstream goal but not pursued through current activities.

1.3. Helena Bioinformatics Current Position

Helena Bioinformatics currently operates within Phase 1 of an explicitly phased pathway (Section 9.1). Phase 1 activities reflect the organizational scope appropriate to this stage. The platform processes clinical genomic data through partnership arrangements with established peer reference laboratory partners.

The current stage admits the following considerations, which this Framework explicitly accommodates:

• Phase 1 organizational scope - activities reliant on external collaboration for clinical and reference functions where appropriate.

• Partnership-based patient data access - VCF files originating from collaborating peer reference laboratories under appropriate agreements.

• Phased pathway to Notified Body engagement - IVDR submission anticipated Q3 2027 or later, conditional on funding and organizational growth.

Within Phase 1 scope, scientifically rigorous performance characterization remains achievable. The methodologies in this Framework are designed to deliver defensible evidence under such scope, while preserving an unobstructed path to regulatory-grade work in subsequent organizational stages.

1.4. Intended Audience for Performance Study Outputs

Outputs of studies conducted under this Framework are intended for the following audiences:

• Scientific community: Peer-reviewed publications. Conference abstracts. Methodology demonstration.

• Investors and strategic partners: Clinical credibility evidence. Capability demonstration. Due diligence material.

• Potential collaborators: Platform capability evidence supporting partnership discussions with laboratories and clinical centers.

• Future Notified Body (downstream): Foundational evidence retained for citation in eventual IVDR submission. Not pivotal evidence under current Framework - designated as foundational methodology and capability demonstration.

• Internal Helena Bioinformatics R&D: Defect identification, classifier improvement targets, methodology refinement.

3.1. Family A - Analytical Performance

3.2. Family B - Classification Performance

3.3. Family C - Clinical Performance

3.4. Reporting Disposition Logic

Real-World Performance Studies do not produce binary pass/fail dispositions. Instead, study outputs report:

• Performance values across all metric families with reference to published benchmarks.

• Methodological characterization of all PARTIAL and DISCORDANT cases.

• Identified platform defects with linked Change Requests and remediation status.

• Limitations and caveats explicitly acknowledged.

• Conclusions appropriate to study scope: descriptive performance characterization, not regulatory clearance.

Future Pivotal Cohorts under expanded Framework versions will introduce binary pass/fail thresholds. Such thresholds are not applicable at the current organizational stage.

5.1. Detection Assessment (Layer 1)

Detection assessment evaluates whether Helena correctly identifies variants present in input VCF.

• 1.

  Input VCF processed through Helena pipeline using production classifier version under study.

• 2.

  For each target variant, the classified_variants DuckDB queried by genomic position (GRCh38), HGVS coding change, protein change, and ClinVar identifier. Detection confirmed if any query returns a match.

• 3.

  For variants not detected, root cause determined by examining: input VCF presence, liftover output, quality filter status, annotation pipeline logs.

• 4.

  Detection failures categorized: (a) Helena pipeline failure - counts against Detection Sensitivity; (b) upstream pipeline failure - reported separately as upstream failure rate; (c) quality filter failure - recorded as quality-flagged detection.

5.2. Classification Assessment (Layer 2)

Classification assessment evaluates Helena ACMG class output against multi-source ground truth.

• 1.

  For each target variant, all available reference sources are documented (peer laboratory class, ClinVar entry, VCEP curation if any, functional studies in literature, population evidence).

• 2.

  Helena ACMG class extracted from classified_variants database for the target variant.

• 3.

  Concordance computed independently against each reference source per Section 2.3 levels.

• 4.

  For PARTIAL and DISCORDANT cases (Helena vs any reference), ACMG criteria differences documented in detail. Methodology root cause categorized: (a) ClinGen SVI 2020+ stricter application by Helena, (b) evidence access difference, (c) classifier defect, (d) reference data limitation.

• 5.

  Cases categorized as classifier defect escalated as Change Requests with full traceability.

• 6.

  Multi-source agreement determined: where Helena disagrees with peer laboratory but agrees with multi-source evidence (VCEP, ClinVar consensus, functional), this is documented as Helena-aligned-with-broader-evidence.

5.3. Clinical Performance Assessment (Layer 3)

Clinical Performance assessment evaluates platform behavior on clinical workflow dimensions. Conducted by clinical reviewer (with COI disclosure per Section 8 if applicable).

• Q1 (Tier Placement): Reviewer records Phenotype Matching tier assignment for target variant. Rated PASS (Tier 1), ACCEPTABLE (Tier 2), FAIL (Tier 3 or 4).

• Q2 (Phenotype Match Score): Reviewer records percentage match score. Rated HIGH (at least 50 percent), MODERATE (20 to 49 percent), LOW (less than 20 percent).

• Q3 (Screening Rank): Reviewer records ordinal position of target variant in screening service output. Rated TOP-5, TOP-10, TOP-20, OUTSIDE-20.

• Q4 (AI Report Quality): Reviewer reads AI-generated clinical report. Rated Q4a (variant mentioned: YES/NO), Q4b (interpretation correct: YES/PARTIAL/NO), Q4c (overall quality: GOOD/ACCEPTABLE/POOR). Any factual hallucination triggers AI Faithfulness flag.

5.4. Methodological Characterization of Divergences

For all PARTIAL and DISCORDANT cases, the following characterization is mandatory and constitutes the substantive scientific output of the study:

• Side-by-side ACMG criteria table (Helena vs each reference source).

• Identification of which Helena-applied or Helena-withheld criteria reflect ClinGen SVI 2020+ specifications (PP5 deprecation, PS4 single-case caution, calibrated computational thresholds, BS2 inheritance-aware thresholds, PVS1 pext/expression evidence requirement, etc.).

• Independent multi-source evidence summary (VCEP, ClinVar consensus, functional, population, segregation).

• Statement on whether disagreement falls within published inter-laboratory disagreement benchmarks.

• Disposition: methodological divergence (Helena and reference apply different but defensible methodologies), classifier defect (Helena error to be remediated), reference limitation (Helena correct, reference applies less stringent standard), or evidence-genuinely-uncertain (current evidence does not unambiguously support one classification).

7.1. Per-Case Performance Report

Every case in every Real-World Performance Study produces a Per-Case Performance Report following the structure below. This structure is consistent with the Per-Case Concordance Reports already produced for Cohort 1 and Cohort 2; section additions reflect this Framework emphasis on multi-source characterization.

• 1.

  Case Header and Performance Summary: Patient ID (anonymized), gene, variant identifiers, clinical diagnosis, HPO terms, session ID, classifier version, report date. Performance summary statement and concordance outcomes against each reference source.

• 2.

  Multi-Source Comparison Table: Side-by-side: Helena classification vs each available reference (peer laboratory, ClinVar with star rating, VCEP if any, published literature). ACMG criteria, HGVS, consequence, zygosity, gnomAD AF documented.

• 3.

  Evidence Review: ClinVar entries with review_status detail, in silico predictions with calibrated thresholds, population frequency analysis, gene constraint metrics, sequencing quality metrics, published case reports if any, functional evidence if any.

• 4.

  Layer 1 Assessment (Detection): P1, P2 results. Detection failure root cause if applicable.

• 5.

  Layer 2 Assessment (Classification): P3 result against each reference source. Methodological characterization for any PARTIAL/DISCORDANT outcome. Specific identification of ClinGen SVI 2020+ application differences.

• 6.

  Layer 3 Assessment (Clinical Performance): Q1, Q2, Q3, Q4 results with rationale. AI Faithfulness check explicitly noted.

• 7.

  Diagnostic Relevance Assessment: Comparative clinical utility analysis: Helena versus reference. Documented per intended use.

• 8.

  Platform Issues Identified: Numbered list of any defects, limitations, or improvement opportunities. Each linked to existing or proposed Change Request.

• 9.

  COI Statement: Disclosure of any conflict of interest applicable to operator, reviewer, or other contributor to this case. Mitigation applied.

• 10.

  Conclusions and Methodological Disposition: Final disposition: methodological divergence / classifier defect / reference limitation / evidence-genuinely-uncertain. Action items if any. Responsible authority sign-off.

7.2. Cohort-Level Real-World Performance Study Report

At cohort completion, Helena Bioinformatics produces a Real-World Performance Study Report. This is the primary scientific output of the study and is structured to be peer-publishable.

• Title and Abstract: Cohort identifier, classifier version, key findings.

• Introduction: Background on Helena, classifier methodology (ACMG/AMP 2015 with ClinGen SVI specifications), study purpose.

• Methods: Cohort selection criteria, processing pipeline, classifier version, reference sources used, comparison methodology, COI disclosure.

• Results - Layer 1: Detection performance with attention to upstream pipeline failures.

• Results - Layer 2: Classification concordance against each reference source. Inter-laboratory concordance with peer laboratory benchmarked against published literature. Multi-source agreement analysis.

• Results - Layer 3: Clinical workflow metrics.

• Discussion: Methodological characterization of PARTIAL/DISCORDANT cases. Identified defects and remediation status. Limitations. Comparison to published benchmarks.

• Conclusions: Study scope and findings stated honestly. Foundational evidence framing for downstream regulatory work.

• Appendices: All Per-Case Performance Reports, classifier version manifest, reference database snapshots, COI Disclosure Registry.

7.3. Identified Defects and Change Request Tracking

All identified defects, methodology gaps, and improvement opportunities are entered into a centralized Change Request log. Log entries contain: discovery date, source case(s), description, severity, assigned Change Request identifier, status, target resolution, post-remediation re-validation requirement.

7.4. Optional Academic Peer Review

Real-World Performance Study Reports may optionally be submitted to one or more independent academic clinical geneticists for pre-publication peer review. This is not formal regulatory adjudication but standard scientific peer review practice. Reviewer feedback is documented and addressed in the final report. Reviewers acknowledged in the report. Reviewer COI disclosed in line with Section 8.

9.1. Phased Pathway

This Framework operates within Phase 1 of an explicitly phased pathway toward eventual regulatory submission. Phase progression is conditional on organizational growth and is not pursued at the expense of current activities.

9.2. Differences in Future Phases

Activities reserved for Phase 2 and Phase 3 (and not pursued in current Phase 1) include:

• Formal external adjudication panels (multi-adjudicator, paid engagement)

• Independent operator structure (no peer laboratory affiliations)

• Pivotal Cohort composition (60/30/10 positive/negative/edge)

• Multi-site validation across independent laboratories

• Pre-submission Notified Body consultations

• Risk Management File at full ISO 14971 detail

• Quality Management System at full ISO 13485 detail

9.3. Foundational Evidence Carryforward

Real-World Performance Studies conducted under this Framework produce foundational evidence that is carried forward to future Pivotal submissions. Specifically:

• Performance characterization data is cited as preliminary evidence supporting platform readiness.

• Identified defects and remediation history demonstrate active quality management posture.

• Methodology rationale established in published literature provides scientific defense for classifier choices.

• Multi-source agreement analysis demonstrates that Helena classifications align with broader expert consensus where peer-reference disagreements occur.

This Framework outputs are therefore not consolation work for premature regulatory submission. They are appropriate-stage scientific evidence with permanent value across the regulatory pathway.