Automated evaluation of guideline quality using GEM-Q
Agrawal
A, Karras BT, Shiffman RN
Purpose:
To facilitate evaluation of the quality of practice guidelines using
Extensible Stylesheet Language
(XSL) and the Guideline Elements Model (GEM).
Background:
Many clinical practice guidelines published in the peer-reviewed medical
literature do not adhere well to established quality standards. GEM is an XML-based
knowledge representation for guidelines that can facilitate evaluation of
guideline quality. Using GEM, guideline text is "marked-up" with tags
that denote semantic content in a format that is both human-readable and
accessible for computer processing. A 25-item rating instrument has been
published recently to evaluate guideline quality based on format, method of
development, evidence summation, and formulation of recommendations (1). This
instrument was validated using a test set of six guidelines that were ranked by
experts in critical appraisal.
Method:
We marked-up these guidelines as XML documents according to a pre-defined
Document Type Definition using a GEM template. We then devised an XSL stylesheet
to display components of the guidelines that are relevant to quality rating
using XSL pattern match, template rules, and conditional processing. This
stylesheet was applied to the six GEM-tagged guidelines. The resulting HTML
document was used to evaluate the concordance of guideline quality scores with
those of the experts.
Result:
The XSL system ranked guidelines in order of quality score concordant
with the expert reviewers. Two guidelines scored as "good" quality,
two as "intermediate", and two as "poor". The stylesheet
also explicitly displayed specific components of the guideline text that are
relevant to quality evaluation, thereby allowing a better understanding of the
quality score.
Conclusion:
Using a GEM-derived XSL stylesheet can facilitate evaluation of
guideline quality. Various stakeholders can use this tool to evaluate (and
possibly improve) guideline quality during development, dissemination, and
implementation phases of the guideline life-cycle.