Model

GLIDES Evaluation Plan

I. Evaluation Of Transformation Of Text Guidelines Into Decision Support Plan Overview

• Evaluation focuses on the feasibility and replicability of guideline knowledge transformation, with particular attention on recommendations for improving the quality of guideline writing

• Compare Yale and Nemours: two separate approaches to implementing the same guidelines, whose knowledge was transformed using the same tools.

Activities

• Collect, organize, and report knowledge transformation artifacts

• Evaluate effectiveness of the GuideLine Implementability Appraisal (GLIA)

• Identify and document information that is vital for accurate and efficient implementation

• Feed lessons learned back to guideline developers through our contacts at professional societies and via scientific publications.
  
  

II. Evaluation Of CDS Tool Development And Implementation

• Evaluate: design and development process/activities; quantity and quality of end-user input; degree to which each decision support system meets requirements; implementation process; barriers encountered (technical, cultural, design, workflow); and solutions to barriers. 

Activities

• Record, at each site, the technical barriers encountered in the codification of recommendations in each EHR system

• Collect, categorize and report problems in codifying guideline concepts and embedding them in the vendors’ EHR products

• Record lessons learned related to the design of clinical decision support tools

• Feed lessons learned back to guideline developers through our contacts at professional societies and via scientific publications.

• Retain and evaluate training manuals and materials

• Prepare recommendations for the implementation community.

III. Evaluation Of Usability And Clinician Use Of CDS

• Evaluate use of the clinical decision support systems by clinicians in practice, including frequency and timing of use, workflow context, clinician workarounds and avoidance, clinician feedback and recommendations, and overall satisfaction with the systems.

Activities

• Obtain information through structured queries of CIS, clinician surveys, direct observation, and structured interviews with key stakeholders

• Observe clinicians in practice and interview clinicians about their use of the system

• Survey clinicians to determine overall perceptions of guidelines and CDS (one year after implementation)

• Query clinical information system to obtain data about usage by clinicians

IV. Evaluation Of Effect Of CDS On Guideline-Adherent Care

• Evaluate the effect of the CDS tools on guideline-adherent care, statistically comparing before- and after-intervention results.

Activities

• Assess quantitative metrics of guideline adherence

• Derive qualitative data through chart reviews, surveys, interviews and direct observation

• Obtain post-intervention metrics at least 6 months after intervention

• Apply contingency tables, multivariable models, hierarchical modeling, mixed effects logistic models,

• Explore the rate of racial/ethnic disparity in these outcomes

V. Evaluation of Patient Outcomes

• Evaluate the effect of the system on patient outcomes, comparing one year of pre-intervention data to one year of post-intervention data

• This portion of the evaluation is not slated to begin until 1 full year post-implementation at each site and may therefore not take place if the contract ends as specified.

Activities

• Assess rate of asthma-related hospitalizations and ED visits to the study institution (for asthma cohort only)

• Assess average asthma control level (asthma cohort only)

• Assess number of visits/year per patient (asthma cohort only)

• Assess Number of oral steroid courses per patient per year (asthma cohort only)

• Assess BMI (for obesity cohort only)

• Assess “5-2-1-0” improvement (i.e., counseling re: ≥5 portions fruit and vegetables daily, ≤2 hours screen time, ≥1 hrs activity/d, and near 0 sugar-sweetened beverages collected as yes/no