Evaluation Methods
Working group in the DeGEval - Evaluation Society
Starting point and problem situation
More so than in university research - where questions of content can be dealt with according to methodological preferences - the process of methodological work in evaluation must be strictly oriented towards the requirements of the respective object of investigation. This means that appropriate methodological skills and the openness required for their application must be constitutive elements of the evaluation process.
These are two elementary reasons for establishing a methods working group. In this context, "openness" means that an exchange should take place in the group not only about research methods, but also about other types of evaluative procedures (for example: decision-theoretical or participation-oriented evaluation procedures). In turn, "methodological competence" is understood here to mean that - unlike in basic research - a perfect solution often cannot be implemented in mission-based evaluation under the pressure to act in practice; instead, economically justifiable and pragmatically manageable shortcut procedures must be found. The justifiability of such shortcut procedures, which are increasingly being discussed in the literature, and their differentiation from poor, pseudo-scientific evaluation practice must be just as important a concern for those working in evaluation and those commissioning it as the application of generally accepted, scientifically based quality criteria in the field of evaluation.
Overall, rapid developments and significant debates have taken place in the very broad and interdisciplinary field of empirical research methods and the methodology of empirical social and behavioral research in recent decades. These concern, on the one hand, the development of advanced and sometimes quite complex methods of data collection and data analysis and, on the other hand, conceptual and fundamental issues such as the relationship between qualitative and quantitative methods. Many of these developments are of direct relevance to evaluation as a professional practice and scientifically based activity: This applies, for example, to the increasing availability of computerized, statistically sophisticated, but often easy-to-implement data analysis techniques (whose application requirements can, however, be very complex), this continues to concern debates about "hierarchies of evidence" and "evidence-based practice", among others.
Inspired by these developments, or even independently of them, intensive debates on scientifically founded standards for good and justifiable research are taking place time and time again in individual scientific disciplines. The "DeGEval - Gesellschaft fΓΌr Evaluation" must itself become active here in order to participate as a visible actor in debates on methods in evaluation and to discuss and represent scientifically justifiable and pragmatically practicable standards and quality criteria in the professional community. The main task of a "Research Methods in Evaluation" working group would therefore be the open exchange of evaluation methods across individual content-related evaluation fields.
Aims of the working group
The working group will review and discuss the current status of the discussion on methods, the development of methods and other evaluation procedures in relevant disciplines (psychology, statistics, sociology, economics, medicine, nursing sciences, etc.), insofar as this status is relevant to evaluation, and thus also bring the various disciplines into an exchange with each other. The focus will be on quality standards and quality criteria for evaluation methods. In line with the "hierarchy of evidence" known from evidence-based medicine, best practices and acceptable good practices will be identified and the scope of validity of statements that can be formulated using methods of varying methodological rigor and complexity will be determined by comparison. In this way, the range of scientifically acceptable evaluation methods is to be delineated in two directions: on the one hand, against time-consuming and labor-intensive, highly complex methodological procedures, which have a place primarily in basic scientific research, and on the other hand, against "folklore methods" of practice, which largely lack methodological control. In this way, the usefulness and knowledge limits of different methodological approaches are to be clarified for those working in evaluation research as well as for those commissioning and those interested in the evaluation results.
A detailed description of the work of the Methods Working Group has been published in the Zeitschrift fΓΌr Evaluation issue 02/2017 and can be viewed at this link.