Assignment 1

            The program evaluation (PE) that is being reviewed is entitled Process and outcome evaluation of an emergency department intervention for persons with mental health concerns using a population health approach. In this article the authors sought to evaluate the introduction of a mental health triage and a mental health counsellor on an emergency department’s (ED) dealings with people with mental health concerns. This review will outline the models that were used in formulating this evaluation, will explore various strengths and weaknesses, and I will mention some personal thoughts on this PE.

            Rarely only one model is used when formulating a PE; four models were used in the current PE. Stake’s countenance model states that PEs should include both anecdotal data and descriptive data. In the present PE qualitative anecdotal evidence was obtained from various stakeholders and descriptive data was also obtained through a quantitative component. Another component of the countenance model that is relevant to the current PE is the question of if the intervention is being implemented in relation to the objectives. This directly relates to the question posed by the countenance model: is there congruence between what is intended and what is observed? A third component of the countenance model that is applicable to the current PE is the question of if there are logical connections between an event and purpose. This is seen in the current PE through the observation of the connection between the mental health triage and a mental health counsellor and the PE goals, or purpose.

            An element of Stufflebeams’s CIPP model was also used in the current PE. A premise of the CIPP model is that PEs can look at any of the four components (C-context, I-input, P-process, and/or P–product) that are part of a program. The current PE focused on the process and outcome, or product of the program. In this sense, the CIPP model was utilized.

            Scriven’s model is also apparent in the current PE. Scriven stated that PEs can be goal or role focused. The current evaluation is goal focused in that, part of the focus is on the outcomes of the program. Furthermore, there appeared to be a focus on what the goals of the program were when identifying the measures and research questions that were used in this PE.

            Lastly, Rippey’s transactional model has appeared to influence the current PE. Part of the transactional model is to include all those who are affected by an issue. The current PE included many of the individuals who are affected by the program, from psychiatric consultants, to the families of individuals with mental health concerns. Another component of the current PE that falls in the realm of the transactional model is that the program was implemented because of the potentially damaging situation of misuse/non-use of resources for people with mental health concerns in the ED.

Even though this PE used multiple models to inform its implementation, there are still weaknesses of both the PE and the models that were used.  In terms of weaknesses of the PE, a number of the pre and post intervention participant characteristics were significantly different from each other and therefore, the results of this PE need to be seen through this lens. Secondly, although long-term data were reported, the authors caution relying on this data since not enough time has elapsed to truly measure long-term outcomes. In terms of weaknesses of the models that were used for the PE, although there was mention that the program was implemented partially due to the fact that treating people with mental health concerns the same as those without was creating a strain financially, there was no analysis of the change in cost with the implementation of the new program. This may have been analyzed had Provus’s discrepancy model been used. Furthermore, the goals of the program were known a head of time and therefore, there is the possibility that the authors may have been affected by ‘tunnel vision’.

Strengths were also present both in terms of the PE and the models that were used. One of the strengths of the PE was the use of mixed methods. The inclusion of both qualitative and quantitative data allows for triangulation and therefore more validity. Another strength includes utilizing components of many models, allowing for a more inclusive PE. One of the strengths of the models that were used was to include many stakeholders in the evaluation. By including many stakeholders, a truer picture of the program is obtained. Lastly, the connection that was made between the intended use of the program and what was observed in the PE can be seen as a strength.

Personally, although I think that the program that was evaluated was a valuable contribution to the mental health care field in Ontario, Canada, I thought that this PE could have been written more clearly. I found that I had to re-read parts of the evaluation to truly understand what the authors were trying to say. Furthermore, I think that the PE would have been more valuable had it been conducted during the program implementation as opposed to after. This would have allowed the people who were involved in the program to respond in real time to the needs of the clients. Lastly, although the authors reported the results of the PE, and the results appeared to be positive in terms of the usefulness of the new program, the authors did not mention if this PE was used in changing the current program, or if this PE had any effect on the program at all. It is my view that PEs should be conducted with the intent of having an actual impact on the program, and it would have been beneficial if the authors would have explored this. On a positive note, I thought that the author’s use of five different datasets to be valuable because this allowed for more rich data and results. Furthermore, I found the background section of the program very helpful as setting up why this program was needed, and I enjoyed that the PE was Canadian research. This PE is applicable to me because of my thesis topic and work that I am doing with Community-University Institute for Social Research. It was great to be able to read an article and have it be utilized in a few areas of my life.

Many PEs make use of different models for implementation. The current PE utilized Stake’s countenance model, Stufflebeam’s CIPP model, Scriven’s model, and Rippey’s transactional model. Although this program evaluation had some weaknesses, not only in terms of the PE, but also in terms of the models that were used, this PE also had many strengths, including using mixed methods and mixed models.

Reference

Vingilis, E., Hartford, K., Diaz, K., Maitchell, B., Velamoor, R., Wedlake, M., & White, D. (2007). Process and outcome evaluation of an emergency department intervention for persons with mental health concerns using a population health approach. Administration and Policy in Mental Health and Mental Health Service Research, 34(2), 160-171.