Найдено 46
Ничего не найдено
Design, Conduct, and Analysis of Demonstration Studies
Friedman C.P., Wyatt J.C.
Demonstration studies answer questions about an information resource, exploring such issues as the resource’s value to a certain professional group or its impact on the processes and outcomes of health care.1 Recall from Chapter 4 that measurement studies are required to test, refine and validate measurement processes before they can be used to answer questions about a resource or its impact. Chapters 5 and 6 explained these ideas and how to conduct measurement studies in more detail. In this chapter we assume that measurement methods are available and have been verified by appropriate measurement studies. To answer questions via a demonstration study, appropriate evaluation strategies and study designs must be formulated, the sample of subjects and tasks defined, any threats to validity identified and either eliminated or controlled for, and the results analyzed.2, 3 These issues are discussed in this chapter.
Introduction to the PACE Project
Evans S.
The first goal of this book is to utilize the case study of the development of the PACE (Patient Care Expert) system to describe and explain fundamental requirements that enabled a 20-year, large-scale project not only to survive but also to achieve its overall mission in the face of significant challenges (Evans, 1988b). The challenges facing a major initiative differ crucially from those confronting a focused and cohesive limited project. Contrast the task of obtaining lunch for oneself with the goal of preparing lunch for several battalions of an army. For yourself, you can find a convenience store, buy a ready-made sandwich and a drink, eat the lunch, and discard the packaging as you leave. Feeding several battalions requires obtaining massive amounts of ingredients, storing them, identifying recipes commensurate with huge quantities, preparing the different components of the meal in parallel with numerous personnel, distributing the food in a short time to large numbers, deploying support facilities to pick up (and store) the leftovers, completing the cleanup, and so on.
Structure of Objectivist Studies
Friedman C.P., Wyatt J.C.
This chapter begins to explore objectivist studies in detail. In Chapters 4–7 we address how to design studies, how to develop measurement procedures to collect data, and how subsequently to analyze the data. The methods introduced relate directly to the comparison-based, objectives-based, and decision-facilitation approaches to evaluation as introduced in Chapter 2. They are useful for addressing most of the purposes of evaluation in informatics, the specific questions that can be explored, and the types of studies that can be undertaken—all as introduced in Chapter 3.
Proposing, Reporting, and Refereeing Evaluation Studies; Study Ethics
Friedman C.P., Wyatt J.C.
This final chapter addresses a set of issues focusing on communication. These are the often “hidden” but important considerations that can determine if a study receives the resources that make its conduct possible, if a study in progress encounters procedural difficulties, and if a completed study leads to improvement or adoption of an information resource. Whether a study is funded depends on how well the plan for the study is represented in a proposal; whether a study encounters procedural difficulties depends on the investigator’s adherence to general ethical standards as well as more specific stipulations built into an evaluation contract; whether a study leads to improvement or adoption of a resource depends on how well the study findings are represented in various reports.
Evaluation as a Field
Friedman C.P., Wyatt J.C.
The previous chapter should have succeeded in convincing the reader that evaluation in medical informatics, for all its potential benefits, is difficult in the real world. The informatics community can take some comfort in the fact that it is not alone. Evaluation is difficult in any field of endeavor. Fortunately, many good minds—representing an array of philosophical orientations, methodological perspectives, and domains of application—have explored ways to address these difficulties. Many of the resulting approaches to evaluation have met with substantial success. The resulting range of solutions, the field of evaluation itself, is the focus of this chapter.
Developing Measurement Technique
Friedman C.P., Wyatt J.C.
This chapter applies the theoretical material of Chapter 5 to actual practice. In Chapter 5 we introduced theories of measurement that were, in effect, theories of error. In this chapter we address specific procedures for estimating and minimizing error. We discuss the structure of measurement studies, the mechanics of conducting them, and how to use the results of these studies to improve measurement techniques. We consider how to develop measurement methods that yield results that are acceptably reliable and valid. We discuss in detail three specific situations that arise frequently in informatics: first, when the repeated observations in a measurement process are tasks completed by either persons or information resources; second, when the repeated observations are the opinions of judges about clinical cases; and third, when the repeated observations are items or questions on forms. Although the same overall measurement issues apply in all three instances, there are issues of implementation and technique specific to each.
Studying Clinical Information Resources
Friedman C.P., Wyatt J.C.
In Chapter 1 we introduced the challenge of conducting evaluations in medical informatics and discussed specific sources of complexity that give rise to these challenges. In Chapter 2 we introduced the range of approaches that can be used to conduct evaluations in medical informatics and across many areas of human endeavor. Chapter 2 also stressed that the evaluator can address many of these challenges by viewing each evaluation as anchored by specific purposes. Each study is conducted for some identifiable client group, often to inform specific decisions that must be made by members of that group. The work of the evaluator is made possible by focusing on the specific purposes the particular study is designed to address, often framing them as a set of questions and choosing the approach or approaches best suited to those purposes. A study is successful if it provides credible information to help members of an identified audience make decisions.
Challenges of Evaluation in Medical Informatics
Friedman C.P., Wyatt J.C.
This chapter develops in a general and intuitive way many issues that are explored in more detail in later chapters of this book. It gives a first definition of evaluation, describes why evaluation is needed, and notes some of the problems of evaluation in medical informatics that distinguish it from evaluation in other areas. In addition, it lists some of the many clinical information systems and resources, questions that can be asked about them, and the various perspectives of those concerned.
Example Knowledge Bases
Warner H.R., Sorenson D.K., Bouhaddou O.
The popularity of medical informatics has been on the upswing for a number of years as the role of computers has increased in importance in health care. The Department of Medical Informatics at the University of Utah has taught a graduate level course in Knowledge Engineering for the past 6 years.
Iliad: The Model Used for This Text
Warner H.R., Sorenson D.K., Bouhaddou O.
In the case of a diagnostic expert system, each disease or diagnostic decision can be represented as a list or table called a frame. The decision frame contains the relevant findings (i.e., disease manifestations), associated logic, and probabilities in the case of a probabilistic (e.g., Baye-sian) frame. Frames can be represented in different ways, e.g., Bayesian, Boolean, and value-type Boolean frames are used in the Iliad expert system.
The Data Dictionary: Limiting the Domain of the Model
Warner H.R., Sorenson D.K., Bouhaddou O.
The data dictionary of an expert system is the component that describes all the terms known to the system. In a diagnostic expert system, the data dictionary is the set of diseases and disease manifestations (i.e., symptoms, signs, laboratory results, radiological results, and special procedures) across all the diseases represented in the knowledge base.
Evaluation of the Model
Warner H.R., Sorenson D.K., Bouhaddou O.
In the evaluation phase, the expert system is tested by comparing the differential diagnosis at each stage of the patient workup with the expert’s opinion and determining where the knowledge base might be revised to improve performance. This process is carried out first. If the system appears to be overconfident about a particular diagnosis (i.e., assigns a higher probability to that diagnosis than the expert thinks appropriate), suspicion is raised that the false positive rate assigned to some data item already entered for the patient being diagnosed is too low. Another explanation might be that highly associated findings are being treated as independent; i.e., they need to be represented as an “or” group or built into a cluster.
Knowledge Engineering Tools
Warner H.R., Sorenson D.K., Bouhaddou O.
Knowledge engineering (KE) tools (i.e., software applications defined for a specific purpose) facilitate a quick cycle time from acquiring information from domain experts, the literature, and patient databases, to restructuring that information in the system and testing its performance against expert judgment and real cases. The process of developing a working knowledge base is iterative, and multiple cycles are necessary before a satisfactory behavior evolves.
The Expert System Model
Warner H.R., Sorenson D.K., Bouhaddou O.
Medical informatics is the branch of science concerned with the use of computers and communication technology to acquire, store, analyze, communicate, and display medical information and knowledge to facilitate understanding and improve the accuracy, timeliness, and reliability of decision making.39 Understanding an observation is defined as the recognition of the relationship between what is observed and some prior observations or communication. A generalization that describes the relationships among such a set of observations is called a model of the set of observations or system. Decision making frequently involves the use of models. The subject matter in this volume is designed to assist the reader in learning to build a model of a system and implement the model using a computer so as to perform some useful intellectual task. Medical diagnosis is used as the principal example.
Future Challenges
Warner H.R., Sorenson D.K., Bouhaddou O.
Although many expert systems or decision support systems have been developed, used, and evaluated to some extent, the full benefits of this advanced computer technology await their wide use in the health care system. Major obstacles (see also Chapter 1) to the wide dissemination of decision support systems exist, e.g.:
Background and Legacy
Warner H.R., Sorenson D.K., Bouhaddou O.
Lessons Learned
Warner H.R., Sorenson D.K., Bouhaddou O.
Iliad was initially developed as a tool for use by third-year medical students at the University of Utah during their clerkship on the internal medicine wards.56-58 It was intended to facilitate the learning that occurs during the student’s first real contact with patients. Four specific goals influenced the design of this expert system.
System Implementation
Evans S.
Installing a system in a clinical setting presents a wide variety of issues. One may at first believe that prior pilot projects will adequately reflect the installation as a whole. It might seem that the evaluation from the limited experience could be extrapolated to the full clinical use of the system. We have consistently found this belief to be completely incorrect. The controlled environment in which a pilot project is initiated inevitably is highly skewed to the particularity of that specific setting.
PACE—Then and Now
Evans S.
The extensive transformation of the PACE delivery system is a pertinent issue in the developmental history of this particular product. Over its 20-year history, the various evolutionary stages were fairly well anticipated as part of the natural growth path of the services and product. As a conseqence, a reasonably smooth evolution has unfolded from the origins of the first system to the current status of the product.
Issues in Semantic Network Development and Utilization
Evans S.
The origins of the semantic network methodology for PACE arose from the author’s doctoral studies on the development of Semantic Operators Producing Heuristically Interesting Sentential Types (SOPHIST) system (Evans, 1969). This 1969 development was in turn based on prior research in related studies begun in 1967 (Evans, 1967). Such studies were forerunners of what were later described as semantic trees by Quillian (Quillian, 1966). Since both Evans and Quillian were later in the same doctoral program at the same university, it is not surprising that the conceptual environment should foster commonality in this research area.
Initial Knowledge Base Development
Evans S.
In this chapter we identify and explore issues that must be addressed in the initial stages of developing a knowledge base. It might appear to the naive developer that the acquisition of knowledge is primarily an issue of access to experts. Actually, the availability of experts is the easiest component in the entire picture. The real issue is to ascertain what it is that one wishes to acquire from each expert. The information desired must be described so that there is compatibility between elements acquired from one expert in one area and another expert in some other area, as well as from future experts who supply information in the same domain in later years (Giuse et al., 1993; Giuse et al., 1993; Giuse et al., 1989; Suwa et al., 1982).
Synthesis of Principles and Lessons Learned
Evans S.
Several design principles can now be more easily recognized The original PACE design had a rich and interactive software system that involved the users extensively to ascertain their needs and fine-tune their specific interests. It is extremely difficult, however, to take full advantage of such a dynamic interactive software “front end,” in the present health care environment (Ball & Douglas, 1995). Users today rarely have the time to interact, and they preferred the system to be responsive in terms of a guess at what their needs would be. They preferred that the system produce a highly accurate first guess that was most often correct, requiring them only to fine-tune it to a specific patient’s needs (Metzger & Teich, 1995). As a result, the investment in the capacity to explore, tune, and permit the user to reflect on a wide variety of intellectual interests represented an academic orientation that had less applicability in the actual clinical practice setting.
Acquiring, Maintaining, and Managing a Knowledge Base
Evans S.
Selecting experts is much like selecting a “significant other”—an extremely subtle and idiosyncratic process. Which individuals will prove to be useful contributors is hard to predict simply by their clinical credentials and performance alone. The personality of the experts becomes a significant factor. It is important that they be willing to follow the established rules by which information is collected. The taxonomy for all specific entries that must be honored has been discussed. There are, in addition, individual aspects of each taxonomic entry that require information from the expert, as well as careful review of what the expert has actually provided.
