Validation of Reconstructed Program Theory

Abstract

Background: The focus of this dissertation is on methods associated with evaluating a program's merit and worth. There are many approaches documented in the literature for evaluating merit and worth. The focus is only on theory driven evaluation (TDE). The premise of TDE is the program theory (PT) must be understood before being able to evaluate the merit and worth of a program. One of the early limitations in the TDE literature was a lack of methods for deriving PT. Renger has recently published methodology describing how existing source documentation could be to develop a program theory. A key component of Renger's methodology is the validation of the PT. Renger suggested using subject matter experts (SME) and program staff to validate the PT. However, it is uncertain whether relying on SMEs to validate a PT is sufficient. Objectives/Methods: Thus the current work focuses on whether there is empirical (i.e., research) and/or statistical (i.e., correlation analyses) support for a PT generated by SMEs. Results: Findings of the correlation analysis provide some evidence of the effectiveness of SMEs validation process. Specifically, weak or very weak statistical support was found for 56.25% (N=9) of relationships between mechanisms of change depicted in the model from Aim 5 (N=16). The results of targeted literature review indicate a strong relationship between the PT generated by SMEs and targeted literature search. Specifically, research evidence was found for 13 (81%) relationships between mechanisms of change identified in the model from Aim 5. Conclusion: PT can be reconstructed from source documentation. Reconstructed PT should be validated. Validation by SME appears to be time a fast, cost-effective way of getting feedback on the initial draft of PT. However, due to the limited scope of targeted literature search and correlational analysis, it is not possible to conclusively determine whether relying on subject matter experts is sufficient to validate reconstructed Program Theory. More research on TDE validation methods is needed.