Automated evaluation of guideline quality using GEM-Q

Agrawal A, Karras BT, Shiffman RN

Yale Center for Medical Informatics, New Haven, Connecticut

Purpose: To facilitate evaluation of the quality of practice guidelines using Extensible Stylesheet Language  (XSL) and the Guideline Elements Model (GEM).

Background:  Many clinical practice guidelines published in the peer-reviewed medical literature do not adhere well to established quality standards. GEM is an XML-based knowledge representation for guidelines that can facilitate evaluation of guideline quality. Using GEM, guideline text is "marked-up" with tags that denote semantic content in a format that is both human-readable and accessible for computer processing. A 25-item rating instrument has been published recently to evaluate guideline quality based on format, method of development, evidence summation, and formulation of recommendations (1). This instrument was validated using a test set of six guidelines that were ranked by experts in critical appraisal.

Method: We marked-up these guidelines as XML documents according to a pre-defined Document Type Definition using a GEM template. We then devised an XSL stylesheet to display components of the guidelines that are relevant to quality rating using XSL pattern match, template rules, and conditional processing. This stylesheet was applied to the six GEM-tagged guidelines. The resulting HTML document was used to evaluate the concordance of guideline quality scores with those of the experts.

Result:  The XSL system ranked guidelines in order of quality score concordant with the expert reviewers. Two guidelines scored as "good" quality, two as "intermediate", and two as "poor". The stylesheet also explicitly displayed specific components of the guideline text that are relevant to quality evaluation, thereby allowing a better understanding of the quality score.   

Conclusion: Using a GEM-derived XSL stylesheet can facilitate evaluation of guideline quality. Various stakeholders can use this tool to evaluate (and possibly improve) guideline quality during development, dissemination, and implementation phases of the guideline life-cycle.

1.  Shaneyfelt TM, Mayo-Smith MF, Rothwangl J.  Are guidelines following guidelines?  The methodological quality of clinical practice guidelines in the peer-reviewed medical literature.  JAMA. 1999;281:1900-1905