Description

Subject of the Challenge:

The National Institutes of Health (NIH), through the National Center for Advancing Translational Sciences (NCATS), is launching the Real-World Evidence (RWE) Innovation Challenge, a national prize competition designed to establish validated benchmarks for analytic methods operating on real-world clinical data.

Over the past decade, NIH has made substantial investments in building national-scale real-world data infrastructure, including the Clinical and Translational Science Award (CTSA) Program and the National Clinical Cohort Collaborative (N3C). The National Clinical Cohort Collaborative (N3C) is a secure, centralized data enclave that aggregates and harmonizes longitudinal electronic health record data from participating health systems across the United States into a common data model. It supports large-scale, multi-institutional research while preserving real-world variation across sites, including differences in patient populations, clinical workflows, and data capture practices. By enabling analytic methods to be executed consistently across multiple independent health systems within a single, governed environment, N3C provides a unique capability to conduct structured, repeatable multi-site validation, albeit, at national scale. While these efforts have enabled access to harmonized, multi-institutional data, a critical gap remains: analytic methods developed in one environment frequently fail to demonstrate stable and reliable performance across different health systems.

Structural harmonization of data alone does not ensure analytic interoperability. Predictive models and causal inference methods often exhibit degradation in discrimination and calibration, instability across patient subgroups, and reduced generalizability when applied across heterogeneous clinical environments. Without systematic validation across multiple health systems, analytic outputs may not be sufficiently reliable for clinical, research, or regulatory use.

The RWE Innovation Challenge directly addresses this gap by shifting the focus from model development to validated analytic performance. Using the N3C secure enclave as a national-scale validation environment, this Challenge will evaluate analytic methods across multiple independent health systems to measure reproducibility, transportability, and stability under real-world conditions. The goal is to define and operationalize Good Algorithmic Practice (GAP) for real-world data; establishing measurable expectations for analytic methods to demonstrate consistent, reliable performance across heterogeneous environments prior to broad deployment.

Objectives of the Challenge

The Real-World Evidence (RWE) Innovation Challenge is designed to address a fundamental gap in the biomedical data ecosystem: the lack of validated standards for analytic performance across heterogeneous real-world clinical environments. While access to harmonized, multi-site data has expanded significantly, the ability to ensure that analytic methods produce stable, reliable, and generalizable results across health systems has not kept pace.

The primary objective of this Challenge is to define and operationalize Good Algorithmic Practice (GAP) for real-world data by establishing measurable expectations for analytic methods to demonstrate reproducibility, transportability, and stability under real-world conditions. Rather than prioritizing methodological novelty alone, the Challenge emphasizes validated performance, ensuring that analytic approaches can maintain reliability across dissimilar patient populations, institutional workflows, and data capture practices. Through a structured, multi-site evaluation framework, this Challenge seeks to:

• Establish benchmarks for reproducibility, transportability, and stability of analytic approaches across health systems;
• Evaluate how analytic performance varies across patient populations, institutional workflows, and data capture practices;
• Identify common failure modes, including calibration drift, subgroup instability, cross-site inconsistency, and sensitivity to missingness or measurement variation;
• Require transparent, reproducible, and containerized workflows suitable for secure execution; and
• Generate evidence to inform future NIH initiatives, peer-reviewed publications, and best practices for reliable real-world evidence generation.

More broadly, this Challenge aims to shift the paradigm from evaluating analytic methods based on performance within a single dataset to evaluating their ability to generate reliable evidence across heterogeneous real-world clinical environments. In doing so, it positions validated transportability, not just accuracy, as a core requirement for real-world evidence generation. A central outcome of this effort will be the establishment of a foundational evidence base to support future NIH initiatives, peer-reviewed publications, and broader adoption of validated analytic standards across the biomedical ecosystem. By defining what constitutes acceptable and robust performance in real-world data, this Challenge will help ensure that analytic methods are not only innovative, but trustworthy and suitable for use at scale.

Priority Areas of Interest

While the Real-World Evidence Innovation Challenge is open to a broad range of clinically meaningful use cases, NIH has identified several high-priority areas where validated analytic transportability is of particular importance. Only one priority area can be addressed by an entry; however, an individual, team or entity may submit more than one entry. The high priority areas include, but are not limited to:

• Cancer: where variability in treatment pathways, biomarker testing, and patient populations can introduce substantial heterogeneity across health systems, affecting both predictive performance and treatment effect estimation
• Kidney Disease: where longitudinal disease progression and reliance on consistent laboratory measurements make analytic approaches sensitive to differences in data capture, measurement frequency, and clinical workflows
• Pediatrics: where developmental variability, smaller sample sizes, and differences in care delivery across institutions present unique challenges for generalizable analytic performance

Submissions addressing these or similar high-impact domains are encouraged. However, participation is not restricted to these areas, and all submissions that meet the requirements of the Challenge and address clinically meaningful real-world evidence questions will be considered.

NIH Early Stage Investigator Recognition Component

NIH anticipates including an NIH Early Stage Investigators Recognition component, supported by HHS/ASPE, to encourage participation by investigators who are early in their independent research careers and to strengthen the future workforce in real-world evidence methods and validation science. For purposes of this Challenge, Early Stage Investigator eligibility will be determined using NIH criteria (see NIH Early Stage Investigators Policies), unless otherwise specified in the final Challenge rules. Early Stage Investigator submissions will be evaluated using the same scientific, reproducibility, and validation-readiness criteria as all other submissions. ESI status will not substitute for methodological rigor or feasibility for multi-site validation. This component is intended to provide special recognition for meritorious submissions led by eligible early career investigators while maintaining the same scientific and validation standards applied to all submissions.

Structure of the Challenge

The Real-World Evidence Innovation Challenge is structured as a two-phase prize competition designed to balance broad participation with rigorous, controlled multi-site validation. The structure reflects a deliberate progression from methodological readiness to demonstrated analytic performance under real-world conditions. Specifically, the structure reflects a deliberate separation between methodological readiness and validated performance. Phase 1– Methodology ensures that only methods capable of rigorous evaluation are advanced, while Phase 2 – Validation and Benchmarking provides a controlled environment to assess transportability at scale.

This design enables the Challenge to move beyond traditional benchmarking approaches and instead generate a systematic, evidence-based understanding of analytic performance under real-world conditions. By preserving site-level variation and explicitly measuring performance across heterogeneous environments, this Challenge structure treats real-world complexity not as noise to be eliminated, but as a signal to be quantified.

Phase 1: Methodology

Phase 1 is designed to identify analytic methods that are technically rigorous, reproducible, and suitable for structured multi-site validation. Participants will submit fully specified analytic approaches addressing clinically meaningful real-world evidence questions. These may include predictive models, causal inference methods, phenotyping algorithms, or other analytic workflows designed to generate clinically or translationally relevant insights from real-world data.

Submissions must go beyond conceptual proposals and demonstrate readiness for execution. Each submission is expected to include:

• Clearly Defined Analytic Method
  A complete description of the analytic approach, including model type (e.g., predictive, causal, phenotyping), underlying assumptions, and intended clinical or translational use case. Submissions should articulate how the method addresses a meaningful real-world evidence question and why the chosen approach is appropriate for that context.
• Feature Definitions and Data Representation
  Detailed specification of input features and variables, including how they are derived from real-world data sources and their alignment with a common data model (e.g., OMOP). Submissions should demonstrate that required data elements are broadly available across health systems and not dependent on site-specific or proprietary inputs.
• Methodological Transparency and Documentation
  Comprehensive documentation describing preprocessing steps, cohort definitions, inclusion/exclusion criteria, handling of missing data, and any transformations applied. The goal is to ensure that an independent reviewer could fully understand, interpret, and reproduce the analytic workflow.
• Evidence of Internal Validation
  Results from internal validation procedures (e.g., cross-validation, temporal validation, or evaluation on an independent dataset where available) demonstrating that the method performs as intended in its development setting. Submissions should include appropriate performance metrics and a discussion of limitations or potential sources of bias.
• Reproducible Code and Workflow Artifacts
  Submission of code and associated workflow components necessary to execute the analytic method. Code should be organized, well-documented, and structured in a way that supports reproducibility and eventual deployment in a controlled environment.
• Readiness for Containerization and Secure Execution
  Evidence that the analytic workflow can be packaged into a containerized format (e.g., Docker or equivalent) and executed within a secure enclave environment. Submissions should avoid dependencies on external systems, APIs, or data sources that would not be accessible within N3C.
• Feasibility for Multi-Site Validation
  Demonstration that the method can be executed consistently across multiple independent health systems without site-specific modification. Submissions should be designed to operate on harmonized data while remaining robust to variation in patient populations, data completeness, and clinical workflows.
• Potential for Transportability and Generalizability
  Discussion of how the method is expected to perform across heterogeneous environments, including any design features intended to support stability (e.g., feature selection strategies, regularization, handling of distributional shift). Submissions should acknowledge potential limitations and describe how the method may behave under real-world variation.

Evaluation in Phase 1 will focus on methodological quality and implementation readiness, rather than peak performance. Reviewers will assess the scientific rigor of the approach, the completeness and clarity of documentation, and the extent to which the method is positioned to support multi-site validation without reliance on site-specific customization.

Up to twelve (12) individuals, teams, and/or entities will be selected to advance to Phase 2 based on their ability to meet these criteria. Phase 1 serves as a critical filtering stage to ensure that only methods capable of reproducible execution and meaningful evaluation under real-world conditions proceed to the validation phase.

Phase 2: Validation and Benchmarking

Phase 2 represents the core of the Challenge and is designed to evaluate analytic methods under conditions that reflect real-world clinical heterogeneity. Finalists will deploy their analytic workflows as containerized applications within the secure N3C enclave, where all evaluation will be conducted under a standardized, controlled framework.

Unlike traditional challenges that evaluate performance within a single dataset, Phase 2 is structured to assess how analytic methods behave across multiple independent health systems and as data heterogeneity increases. Evaluation will occur across three complementary dimensions:

1. Independent Site-Level Validation
   Each method will be executed across multiple individual health systems, with performance assessed independently at each site. This allows for direct measurement of cross-site variability and identification of environments where performance degrades.
2. Progressively Pooled Multi-Site Evaluation
   Methods will be evaluated on pooled datasets that incrementally combine data from multiple health systems. Beginning with a baseline pooled configuration, additional sites will be introduced in a structured manner to assess how performance changes as heterogeneity increases. This enables quantification of performance degradation under increasingly complex, real-world conditions.
3. Permutation-Based Evaluation Framework
   To ensure robustness of results, pooled evaluations will be conducted across predefined permutations of site inclusion. This prevents dependence on any single dataset configuration and provides a stable estimate of performance under varying compositions of real-world data.

Across these evaluation settings, methods will be assessed using predefined metrics appropriate to their analytic type. For predictive models, this includes discrimination, calibration, and subgroup stability. For causal inference methods, evaluation will focus on the consistency and robustness of estimated treatment effects across sites and pooled datasets. Importantly, all methods will be evaluated under strict reproducibility constraints

• Workflows must execute consistently across all environments.
• Methods will be trained once and optimized within a predefined training data tenant (partition) in the N3C enclave and then evaluated across separate, predefined site-level and pooled data tenants. Training and evaluation environments will be logically isolated, and evaluation data will not be accessible for iterative tuning, parameter adjustment, or model re-training.
• All results must be reproducible within the secure enclave.

The emphasis of Phase 2 is not on identifying the highest-performing method in a single context, but on identifying methods that demonstrate stable, interpretable, and reproducible performance across heterogeneous real-world environments.

Non-Responsive Submissions

Submissions will be deemed non-responsive if they do not align with the objectives of the Challenge or cannot be evaluated within the N3C validation framework; such submissions will not be evaluated. This includes, but is not limited to:

• Submissions lacking sufficient methodological specification or readiness for rigorous evaluation
  Proposals that are incomplete, insufficiently specified, lack methodological transparency, or do not provide reproducible workflows and supporting documentation necessary for independent evaluation.
• Submissions not feasible for execution within the N3C environment
  Methods that cannot be containerized, rely on external data or systems, or depend on infrastructure incompatible with secure enclave execution.
• Submissions relying on non-generalizable or site-specific approaches
  Approaches that depend on proprietary variables, institution-specific workflows, or data elements not broadly available across participating health systems are not suitable for multi-site validation.
• Submissions misaligned with the purpose of the Challenge
  Proposals focused on model development without validation readiness, clinical implementation or deployment, data platform or infrastructure development, or policy/regulatory frameworks without a testable analytic method.

Partners

This Challenge is led by NCATS in coordination with the NIH Office of the Director and in collaboration with federal partners and stakeholders across the biomedical research ecosystem. Additional partners include the National Cancer Institute (NCI), the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the Food and Drug Administration (FDA), the HHS Office of the Assistant Secretary for Planning and Evaluation (ASPE) and the National Institute of Standards and Technology (NIST).

Statutory Authority to Conduct the Challenge

NCATS is conducting this Challenge under the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Reauthorization Act of 2010, as amended [15 U.S.C. § 3719]. NCATS, part of the NIH, has the mission of turning research observations into health solutions through translational science. Its statutory authority highlights the purpose of the Center to advance translational science, including by (1) coordinating and developing resources that leverage basic research in support of translational science and (2) developing partnerships and working cooperatively to foster synergy in ways that do not create duplication, redundancy, and competition with industry activities. This Challenge is consistent with and promotes NCATS’ mission by catalyzing the goal-driven development of innovative tools and technologies.

Timeline

Challenge Launch (Registration Opens)	September 22, 2026
RWE Innovation Challenge Technical Assistance Webinar	February 16, 2027
Challenge Phase 1 – Analytic Approach and Methodology Submission Deadline	March 26, 2027

 

Challenge Phase 1 – Announcement of Top Submissions	May 2027
Challenge Phase 2 – Validation and Benchmarking Submission Deadline	September 2027
Challenge Phase 2 – Announcement of Winners	January 2028
RWE Innovation Challenge Winners’ Presentations	February 2028

Prizes

Award Approving Official:

The Award Approving Official will be Dr. Joni Rutter, NCATS Director.

Payment of the Prize

Prizes awarded under this Challenge will be paid by electronic funds transfer and may be subject to federal income taxes. HHS/NIH will comply with the Internal Revenue Service withholding and reporting requirements, where applicable. Entities participating in this Challenge are encouraged, but not required, to request and obtain a free Unique Entity ID (UEI), if they have not already done so, via SAM.gov as this will expedite prize payment. Additional information can be found at https://sam.gov/content/entity-registration.

NIH reserves the right, in its sole discretion, to (a) cancel, suspend, or modify the Challenge, or any part of it, for any reason, and/or (b) not award any prizes if no submissions are deemed worthy.

How to Enter

Registration and Submission Process

Although the Challenge announcement will be posted at nih.gov/challenges, registration forms and complete Phase 1 submissions must be obtained by and submitted by email to [email protected]. Submissions will not be accepted through nih.gov/challenges. Only complete registration forms and submission packages received by email by the applicable deadline will be considered.

Registration Requirements

To be eligible to submit an entry to this prize competition, interested Participants, whether participating as an individual, group of individuals (i.e., team), or entity, must complete the Challenge Registration Form and submit the completed form as an email attachment to [email protected] no later than February 2, 2027, at 5:00 PM Eastern Time. The registration form will be available on nih.gov/challenges.

Participants may register as individuals, groups of individuals (i.e., team), or entities. For a group of individuals, one individual must be identified as the Team Lead and will be responsible for submitting registration and submission materials on behalf of the group. For an entity, one individual must be identified as the Point of Contact and will be responsible for submitting registration and submission materials on behalf of the entity.

For a group of individuals (i.e., team), the Team Lead is responsible for all communications with Challenge sponsors. To be eligible to receive a cash prize, the Team Lead must be a citizen or permanent resident of the United States. Non-U.S. citizens and non-permanent residents may participate as members of a team that otherwise satisfies the eligibility criteria but are not eligible to win a monetary prize, in whole or in part.

For an entity, the designated point of contact is responsible for all communications with Challenge sponsors. In the event of winning a cash prize, the prize will be paid directly to the entity, not to the point of contact. To be eligible to receive a cash prize, the entity must be incorporated in and maintain a primary place of business in the United States.

As part of registration, Participants must indicate whether they intend to use Federal funds from a grant, cooperative agreement, or other transaction award to develop or support their Challenge submission. Participants intending to use such funds, where permissible, must register for and participate in the Challenge as an entity on behalf of the awardee institution, organization, or entity.

Registration must include the following information:

• Whether the Participant is registering as an individual, group of individuals (team), or entity;
• Name and contact information for the individual Participant, Team Lead, or entity point of contact (POC);
• Name of Designated Scientific Lead for a group of individuals or entity, if different from the POC;
• Names and affiliations of all participating members or contributors;
• Confirmation of citizenship or permanent resident status, where applicable for prize eligibility;
• Whether the Participant is seeking consideration for the NIH Early Stage Investigators Recognition component, if applicable;
• Whether Federal funds will be used to develop their challenge submissions or to fund efforts in support of their challenge submissions;
• Certification that the Participant has read, understands, and agrees to comply with all eligibility criteria, rules, and requirements of the Challenge.

NIH Early Stage Investigator Recognition

Participants seeking consideration for the NIH Early Stage Investigators Recognition component must indicate this request at registration and provide information sufficient for NIH to determine eligibility. For individual submissions, the individual Participant must meet the Early Stage Investigator criteria. For group submissions, the designated scientific lead must meet the Early Stage Investigator criteria. For entity submissions, the designated scientific lead for the submitted analytic approach must meet the Early Stage Investigator criteria. Early Stage Investigator status does not alter the submission requirements, review criteria, or Phase 2 validation expectations.

Submission Requirements

Complete Phase 1 submission packages must be submitted as PDF files by email to [email protected] no later than March 26, 2027, at 5:00 PM Eastern Time. Complete submissions must include all required materials described below. Incomplete submissions, submissions received after the deadline, or submissions that do not comply with the instructions in this announcement may be deemed non-responsive and will not be evaluated.

Each submission must address one clinically meaningful real-world evidence question and identify the priority area or use case addressed by the submission. Priority areas include, but are not limited to, cancer, kidney disease, and pediatrics. Participants may submit more than one entry; however, each entry must be submitted as a complete and separate submission package.

Submissions may include predictive models, causal inference approaches, phenotyping algorithms, survival or time-to-event models, risk scores, or other analytic workflows designed to generate clinically or translationally meaningful evidence from real-world data. Underlying methods may include traditional statistical approaches, machine learning, artificial intelligence, propensity score methods, inverse probability weighting, target trial emulation, instrumental variable approaches, or hybrid analytic strategies.

Phase 1: Analytic Approach and Methodology Submission

Phase 1 submissions must describe a fully specified analytic approach suitable for structured multi-site validation within the N3C environment. Phase 1 focuses on methodological rigor, transparency, reproducibility, feasibility, and readiness for secure multi-site execution. Phase 1 does not require Participants to demonstrate multi-site performance within N3C at the time of submission.

Each Phase 1 submission package must include the following components:

1. Cover Page
   
   The Cover Page must include:

   • Challenge title;
   • Submission title;
   • Participant type: individual, group of individuals (team), or entity;
   • Name and contact information for the individual Participant, Team Lead, or entity point of contact;
   • Names, affiliations, and roles of all participating members;
   • Priority area or use case addressed;
   • Analytic type, such as predictive modeling, causal inference, phenotyping, survival modeling, or other analytic workflow;
   • A statement agreeing to the Challenge terms and conditions.
     
     The Cover Page must include the following statement:
     
     “I understand and agree that each Participant for the Challenge must comply with all terms and conditions of the Challenge rules, and participation in this Challenge constitutes each such Participant’s full and unconditional agreement to abide by these rules.”
     
     Submissions that do not include this statement may be deemed non-compliant and may not be evaluated.
      
2. Executive Summary
   Participants must provide an executive summary describing the proposed analytic approach, the real-world evidence question being addressed, the intended clinical or translational relevance, and why the approach is appropriate for multi-site validation. The summary should clearly state whether the submission is predictive, causal, phenotyping-focused, survival/time-to-event, or another analytic type.
   
   At the end of the Executive Summary, include a Plain Language Summary. Provide 2-3 sentences that explain how the submission relates to public health. Use language understandable to the general public, and focus on the potential benefits to public health, not technical details.
    
3. Analytic Approach Description
   
   Participants must provide a complete description of the analytic approach, including the clinical or translational question addressed; cohort definition; inclusion and exclusion criteria; outcome definition; model type or analytic framework; underlying assumptions; intended use of the analytic output; rationale for the selected approach; and description of how the approach is expected to perform across health systems.
   
   For predictive models, this section should describe the prediction target, prediction window, candidate predictors, model structure, and intended performance measures. For causal inference submissions, this section should describe the exposure or intervention, comparator, outcome, estimand, analytic framework, assumptions, and approach to confounding adjustment.
    
4. Feature Definitions and Data Representation
   
   Participants must provide detailed definitions of all required input variables, features, exposures, outcomes, and covariates. This section should describe how these elements are derived from real-world data and how they align with a common data model, such as OMOP, where applicable.
   
   Submissions should demonstrate that required data elements are likely to be available across multiple health systems and are not dependent on proprietary, site-specific, or non-generalizable data inputs. Participants should describe anticipated issues related to missingness, measurement timing, coding variability, laboratory measurement, workflow variation, or other site-level differences that may affect transportability.
    
5. Internal Validation and Evidence of Performance
   
   Participants must provide evidence that the analytic approach performs as intended in its development setting. This may include cross-validation, temporal validation, validation on an independent dataset where available, simulation, sensitivity analyses, or other relevant validation procedures.
   
   For predictive models, Participants should report appropriate performance metrics, such as AUROC, AUPRC, calibration slope/intercept, Brier score, sensitivity/specificity, positive predictive value, or other task-relevant metrics. For causal inference approaches, Participants should report relevant evidence supporting the plausibility and robustness of effect estimates, including sensitivity analyses, covariate balance diagnostics, positivity assessment, uncertainty estimates, or other appropriate measures.
   
   Participants must describe limitations, potential sources of bias, and steps taken to reduce overfitting, data leakage, confounding, or other threats to validity.
    
6. Reproducibility Package
   
   Participants must provide a reproducibility package sufficient for independent review and eventual secure execution. This should include code or pseudocode necessary to execute the analytic workflow; description of software, packages, dependencies, and computational environment; documentation of preprocessing steps, feature construction, model training, and inference procedures; information on random seeds or stochastic elements, where applicable; and description of how the workflow can be reproduced by an independent evaluator.
   
   Participants are encouraged to provide code structured for eventual containerization, such as Docker or equivalent, but Phase 1 submissions do not need to include a fully operational container unless specified in the final announcement instructions.
    
7. Analytic Approach Disclosure and Reproducibility Appendix
   
   Participants must submit sufficient technical detail to allow NIH, reviewers, and Challenge evaluators to assess reproducibility, feasibility for secure execution, and readiness for multi-site validation. This appendix should include the analytic approach, model class or causal framework, cohort and outcome definitions, feature derivation logic, preprocessing steps, model-selection procedures, hyperparameters or tuning strategy, software dependencies, random seeds where applicable, and internal validation results.
   
   For predictive models, Participants should provide model parameters or equivalent executable representations, thresholds, calibration procedures, and trained model artifacts or a complete training recipe sufficient for execution in Phase 2. For causal inference approaches, Participants should provide the estimand, exposure, comparator, covariates, confounding adjustment strategy, balance diagnostics, sensitivity analyses, and assumptions. Participants must also disclose any proprietary components, external dependencies, or licensing restrictions that could affect execution, review, reproducibility, or publication of Challenge findings.
   
   Submission of this information does not require Participants to transfer intellectual property rights to NIH or to make proprietary model components publicly available. However, sufficient information must be provided to NIH and Challenge evaluators to permit independent review, secure execution, and validation within N3C.
    
8. Secure Execution and Containerization Readiness
   
   Participants must describe how the analytic workflow can be packaged and executed within a secure enclave environment. Submissions should avoid reliance on external APIs, external systems, external data sources, manual site-specific processes, or infrastructure that would not be available within N3C.
   
   This section should include anticipated computational requirements, software dependencies, and any known barriers to secure execution.
    
9. Transportability and Generalizability Statement
   
   Participants must include a statement describing why the analytic approach is expected to remain stable across different health systems. This statement should address how the analytic approach accounts for variation in patient populations, coding practices, data completeness, measurement frequency, workflows, treatment patterns, or other real-world differences.
   
   Participants should also describe potential failure modes and how these would be detected or assessed during multi-site validation.
    
10. References
    
    References may be included separately and do not count against the page limit.

Phase 1 Page Limits and Format

The Phase 1 submission narrative must be no more than 12 pages, excluding the Cover Page and references. Any required appendices, code documentation, or supplemental materials should follow instructions provided in the submission template.

All written submissions must:

• Be submitted as PDF files;
• Use 8.5” x 11” page size;
• Use at least 1-inch margins, font no smaller than 11-point Arial, Georgia, Helvetica, or Palatino Linotype, and line spacing no less than 1.0;
• Be written in English;
• Not use the HHS logo, NIH logo, NCATS logo, or official seal;
• Not claim federal government endorsement.

Phase 2: Validation and Benchmarking in N3C

Only Participants selected as winners in Phase 1 will be invited to participate in Phase 2. Phase 2 represents the core validation phase of the Challenge. Phase 2 finalists will deploy their analytic workflows within the secure N3C enclave for standardized evaluation across predefined site-level and pooled data partitions.

Detailed Phase 2 submission instructions will be provided to Phase 1 winners before the start of Phase 2. At a minimum, Phase 2 finalists should expect to provide a locked, containerized analytic workflow capable of execution within the N3C secure enclave. This package must include all code necessary to execute the submitted workflow within N3C, trained artifacts or training scripts, dependency manifests, configuration files, feature-generation logic, finalized cohort, feature, outcome, and model specifications, model parameters or equivalent executable representations, training and inference scripts, expected input/output schemas, and documentation necessary to reproduce the workflow. Finalists must also confirm that the workflow does not require external data access, external APIs, or site-specific modification during evaluation. Analytic approaches that cannot be executed reproducibly or inspected sufficiently to support validation and interpretation may be deemed non-responsive or ineligible for Phase 2 awards.

During Phase 2, analytic workflows will be trained or optimized only within predefined training partitions and evaluated separately across predefined site-level and pooled evaluation partitions. Training and evaluation environments will be logically separated. Evaluation data will not be available for iterative tuning, parameter adjustment, or model retraining.

Performance will be assessed using metrics appropriate to the analytic type. Predictive models may be assessed on discrimination, calibration, subgroup stability, site-level consistency, and degradation under pooled evaluation. Causal inference approaches may be assessed on the consistency and robustness of estimated effects across sites and pooled datasets, including effect direction, magnitude, uncertainty, sensitivity to confounding structure, and robustness to differences in treatment patterns and data capture.

Final awards will be based on demonstrated reproducibility, transportability, and stability under real-world heterogeneity, rather than peak performance in a single dataset.

Rules

Eligibility Rules

To be eligible to win a prize under this Challenge, a Participant (whether an individual, group of individuals, or entity) —

1. Shall have registered to participate in the Challenge under the rules promulgated by the National Institutes of Health (NIH) as published in this announcement;
2. Shall have complied with all the requirements set forth in this announcement;
3. In the case of a private entity, shall be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States. However, non-U.S. citizens and non-permanent residents can participate as a member of a team that otherwise satisfies the eligibility criteria. Non-U.S. citizens and non-permanent residents are not eligible to win a monetary prize (in whole or in part). Their participation as part of a winning team, if applicable, may be recognized when the results are announced.
4. Shall not be a federal entity or federal employee acting within the scope of their employment;
5. Shall not be an employee of the Department of Health and Human Services (HHS, or any other component of HHS) acting in their personal capacity;
6. Who is employed by a federal agency or entity other than HHS (or any component of HHS), should consult with an agency ethics official to determine whether the federal ethics rules will limit or prohibit the acceptance of a prize under this Challenge;
7. Shall not be a judge of the Challenge, or any other party involved with the design, production, execution, or distribution of the Challenge or the immediate family of such a party (i.e., spouse, parent, stepparent, child, or stepchild).
8. Shall be 18 years of age or older at the time of submission.

Participation Rules

1. Federal grantees and recipients of cooperative agreements or other transaction (OT) awards are eligible to participate in the Challenge, but may not use Federal funds from a grant award, cooperative agreement, or OT award to develop their Challenge submission or to fund efforts in support of their Challenge submission unless use of such funds is consistent with the purpose, terms, and conditions of the grant award, cooperative agreement, or OT award. Each Participant intending to use Federal grant, cooperative agreement, or OT award funds must register for and participate in the Challenge as an Entity on behalf of the awardee institution, organization, or entity. If a Participant uses Federal grant, cooperative agreement, or OT award funds to win the Challenge, the prize must be treated as program income for purposes of the original grant, cooperative agreement, or OT award in accordance with applicable Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards [2 CFR § 200].
2. Federal contractors may not use federal funds from a contract to develop their Challenge submissions or to fund efforts in support of their Challenge submissions.
3. By participating in this Challenge, each Participant (whether an Individual, Team, or Entity) agrees to assume any and all risks and waive claims against the federal government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from participation in this Challenge, whether the injury, death, damage, or loss arises through negligence or otherwise.
4. Based on the subject matter of the Challenge, the type of work that it will possibly require, as well as an analysis of the likelihood of any claims for death, bodily injury, property damage, or loss potentially resulting from Challenge participation, no Participant (whether an Individual, Team or Entity) participating in the Challenge is required to obtain liability insurance, or demonstrate financial responsibility, or agree to indemnify the federal government against third party claims for damages arising from or related to Challenge activities in order to participate in this Challenge.
5. A Participant (whether an Individual Team or Entity) shall not be deemed ineligible because the Participant used federal facilities or consulted with federal employees during the Challenge if the facilities and employees are made available to all Participants participating in the Challenge on an equitable basis.
6. By participating in this Challenge, each Participant (whether an Individual, Team or Entity) warrants that they are sole author or owner of, or has the right to use, any copyrightable works that the submission comprises, that the works are wholly original with the Participant (or is an improved version of an existing work that the Participant has sufficient rights to use and improve), and that the submission does not infringe any copyright or any other rights of any third party of which the Participant is aware.
7. As a condition for winning a cash prize in this Challenge, each Participant (whether an Individual, a Team or Entity) grants to the NIH an irrevocable, paid-up, royalty-free nonexclusive worldwide license to reproduce, publish, post, link to, s cleahare, and display publicly the Individual/Team/Entity Name, Title, and Executive Summary, including the Plain Language Summary, components of the submission on the web or elsewhere. Each Participant will retain all other intellectual property rights in their submissions, as applicable. To participate in the Challenge, each Participant must warrant that there are no legal obstacles to providing the above-referenced nonexclusive licenses of the Participant’s rights to the federal government. To receive an award, Participants will not be required to transfer their intellectual property rights to NIH, but Participants must grant to the federal government the nonexclusive licenses recited herein.
8. Each Participant (whether an Individual, Team or Entity) agrees to follow all applicable federal, state, and local laws, regulations, and policies.
9. Each Participant (whether an Individual, Team or Entity) participating in this Challenge must comply with all terms and conditions of these rules, and participation in this Challenge constitutes each such Participant’s full and unconditional agreement to abide by these rules. Winning is contingent upon fulfilling all requirements herein.
10. As a condition for winning a cash prize in this Challenge, each Participant (whether an Individual, Team or Entity) that has been selected as a winner must complete and submit all requested winner verification and payment documents to NIH within 7 business days of formal notification. Failure to return all required verification documents by the date specified in the notification may be a basis for disqualification of a cash prize winning submission.
11. As a condition for winning a cash prize in this Challenge, each Participant (whether an Individual, Team or Entity) irrevocably grants to NIH the right to the use of their name, affiliation, city and state, and likeness or image for the purposes of publicity releases and any other promotion of this Challenge.

Judging Criteria

Basis Upon Which a Winner Will be Selected.

Submissions will be evaluated by a panel of external technical reviewers who hold subject matter expertise in predictive modeling, causal inference, real-world data, and translational science.

Evaluation will be conducted in two stages, with distinct criteria applied at each stage to reflect the progression from methodological readiness to validated analytic performance under real-world conditions.

Final prize recommendations will be made by NIH-designated federal employees serving as judges and will be based on reviewer input and performance against the evaluation framework described below.

Phase 1: Methodology (100 total points)

Phase 1 evaluation is designed to identify analytic methods that are scientifically sound, reproducible, and suitable for multi-site validation. Reviewers will assess submissions across the following domains:

1. Technical Rigor and Methodological Soundness (25 points)
   Reviewers will evaluate whether the analytic approach is appropriate for the stated research question and grounded in sound methodological principles. Considerations include:
   • Clarity and completeness of the analytic design, including model structure, assumptions, and rationale
   • Appropriateness of the method for the intended real-world evidence use case
   • Evidence that the approach avoids common sources of bias (e.g., data leakage, improper validation design)
   • Robustness of the analytical framework given the complexity of real-world data
     
     Submissions should demonstrate methodological credibility and internal consistency, rather than rely solely on reported performance metrics.
      
2. Reproducibility and Transparency (20 points)
   Submissions must demonstrate that the analytic workflow can be independently understood and reproduced. Reviewers will assess:
   • Completeness and clarity of documentation describing preprocessing, feature construction, and model execution
   • Availability and organization of code and workflow components
   • Extent to which the method can be executed consistently given the materials provided
   • Transparency of assumptions, parameterization, and implementation choices
     
     Priority will be given to submissions that enable full reproducibility without reliance on implicit knowledge or undocumented processes.
      
3. Internal Validation and Evidence of Performance (20 points)
   Reviewers will evaluate the quality and rigor of internal validation:
   • Appropriateness of validation strategy (e.g., cross-validation, temporal validation, external dataset where applicable)
   • Use of relevant performance metrics aligned with the analytic task
   • Evidence that reported results are credible and not driven by overfitting or data leakage
   • Acknowledgment of limitations and uncertainty in reported performance
     
     Evaluation will emphasize the credibility of the validation approach rather than peak performance alone.
      
4. Feasibility for Multi-Site Execution (20 points) 
   Submissions must be suitable for deployment within the N3C environment. Reviewers will assess:
   • Readiness for containerization and secure execution
   • Independence from external data sources or infrastructure
   • Compatibility with harmonized data models (e.g., OMOP)
   • Ability to execute consistently across multiple health systems
     
     Methods that require substantial re-engineering or site-specific adaptation will not be considered competitive.
      
5. Applicability and Potential for Generalizable Impact (15 points)
   Reviewers will assess the relevance and broader utility of the submission:
   • Whether the use case addresses a clinically or translationally meaningful question
   • Potential for the analytic approach to be applied across multiple settings or domains
   • Degree to which the method relies on broadly available versus highly specialized data
   • Likelihood that the approach could contribute to generalizable real-world evidence generation
     
     Submissions that demonstrate broader applicability and generalizable analytic value will be prioritized over narrowly scoped solutions.

Phase 2: Validation and Benchmarking (100 total points)

Phase 2 evaluation is designed to assess whether analytic methods demonstrate stable, reproducible performance across heterogeneous real-world environments, rather than peak performance in any single setting. All methods will be evaluated within the N3C enclave using a standardized, multi-site evaluation framework.

1. Reproducibility and Execution Consistency (15 points)
   Across all evaluation settings, reviewers will assess:
   • Whether workflows execute consistently across all sites without modification
   • Stability of outputs across repeated runs and evaluation conditions
   • Absence of site-specific tuning or adaptation during evaluation
     
     Methods must demonstrate reliable and consistent execution as a baseline requirement.
      
2. Baseline Performance (15 points)
   At the initial pooled multi-site configuration, reviewers will assess:
   • For predictive models: discrimination (e.g., AUROC), calibration, and overall predictive utility
   • For causal methods: plausibility, stability, and interpretability of estimated treatment effects
     
     Baseline performance establishes whether the method produces meaningful results under controlled multi-site conditions.
      
3. Stability Under Increasing Heterogeneity (20 points)
   A central component of evaluation is how performance changes as additional health systems are introduced:
   • Degree of performance degradation as data heterogeneity increases
   • Whether changes in performance are gradual and bounded versus abrupt or unstable
   • Sensitivity of results to differences in patient populations, data completeness, and workflows
     
     Methods that maintain stable performance under increasing heterogeneity will be prioritized.
      
4. Consistency Across Independent Sites (20 points)
   Reviewers will evaluate variability in performance across individual health systems:
   • Consistency of key performance metrics across sites
   • Identification of sites where performance deviates substantially
   • Evidence that variability is explainable rather than unpredictable Excessive variability across sites will be interpreted as limited transportability.
      
5. Subgroup Stability and Equity Considerations (10 points)
   Submissions will be evaluated for consistency across clinically relevant subgroups:
   • Stability of performance across demographic and clinical strata
   • Absence of systematic degradation in specific populations
   • Interpretability of subgroup-level differences
     
     Methods should demonstrate robustness across dissimilar patient populations.
      
6. Transportability and Generalizability (15 points)
   Reviewers will assess the overall ability of the method to generalize across environments:
   • Whether the method maintains performance without site-specific customization
   • Evidence that the approach is resilient to differences in data structure and clinical practice
   • Ability to produce consistent and interpretable results across evaluation conditions
     
     Transportability is a primary determinant of final award consideration.
7. Holistic Assessment of Performance (5 points)
   Final evaluation will consider performance across all dimensions rather than any single metric:
   • Balance between baseline performance and stability under heterogeneity
   • Consistency across sites and pooled datasets
   • Interpretability and reliability of results
     
     Methods that demonstrate strong overall performance profiles (characterized by stability, reproducibility, and consistency) will be prioritized over those optimized for isolated conditions.

Final Winner Selection

Finalists will be determined based on performance in Phase 2. criteria. Phase 2 finalists will receive a final Phase 2 score out of 100 points based on the criteria described above. Final winners will be determined by NIH-designated federal employees serving as judges based on final Phase 2 score, rank order, reviewer input, and confirmation that all Challenge requirements have been satisfied.

Phase 2 prize amounts will be determined according to the prize structure described in the Details of the Prize section. NIH may establish minimum score thresholds for any Phase 2 prize category and may decline to award a prize to any submission that does not meet the required standards for execution, reproducibility, transportability, stability, eligibility, or overall performance.

NIH may adjust the number, distribution, and amounts of prizes based on available funds, final scores, rank order, the number and quality of eligible submissions, and programmatic priorities. NIH reserves the right to award fewer prizes than anticipated or not award all of the prize purse.

Additional Information

Supplementary Information:

A technical assistance webinar for registrants for the RWE Innovation Challenge is planned for February 16, 2027. NIH staff will be available to answer questions related to the Challenge.

Information about the webinar will be made available to registered participants. Registrants interested in the technical assistance webinar are encouraged to submit questions and/or comments by February 9, 2027 to [email protected].

Webinar information will be available on the NCATS website at ncats.nih.gov. The slides and recorded webinar will be available approximately 7 days following the completion of the webinar. Participation in the webinar, although encouraged, is optional and is not required for the submission of an entry.

For Further Information Contact:

NIH POC: Christopher M. Hartshorn, PhD [email protected].