Program Evaluation of Crisis Management Service (CMS)
Engage Stakeholders:
Who should be involved?
For this program evaluation the stakeholders that should be involved include CMS, their staff, and the clients that CMS serves.
How might they be engaged?
The CMS staff will be engaged through focus groups and meetings to determine their goals and ideas for this program evaluation. The information that I obtained though these meetings will be used to guide the direction of the program evaluation. The CMS staff can also provide information about the organization as they see it, and help to formulate a list of potential participants for this program evaluation. The clients of CMS will be engaged through interviews. These interviews will be used to determine the strengths and weaknesses of CMS through the view of its clients. The final data will be summarized in a report.
Focus the Evaluation:
What are you going to evaluate? Describe the program (logic model).
CMS would not be able to operate without the inputs that it receives. CMS receives funding from many agencies. CMS is part of the Saskatoon Crisis Management Service along with Mobile Crisis. CMS shares some of its funding inputs with Mobile Crisis. The funding that is shared is from the United Way, Social Services, and the City of Saskatoon. Other funding agencies contribute to CMS, these include, Mental Health and Addiction Services, Community Correction, and private donations. Other inputs that are necessary for CMS to operate are the 6 staff that CMS has, the vehicles that they use to take their clients to appointments and services, and relationships with other agencies.
These inputs feed into activities that CMS engage in. CMS is an organization that provides their clients with crisis intervention, assertive outreach to organizations, behaviour shaping and management, to help with basic needs, such as food, clothes, and shelter, to provide service coordination, to screen, assess, and consult for/with their clients, to built a network of support for their clients, to advocate, inform, educate, and train their clients, to assist other frontline workers, and to make necessary referrals to other agencies.
These various activities feed to different participants, of which the CMS services reach. Therefore, participation in these activities include the clients and their families, other agencies, society, and family physicians. These levels of participation then feed into short-term goals of CMS.
The short term goals of CMS include providing necessities for their clients such as food, clothes, and shelter, to connect them with medical professionals, to connect them with legal aid, and to connect them with addiction services. Medium goals are goals which CMS hopes to accomplish through the short term goals. The medium term goals include teaching their clients through modeling how to manage their money, to help them learn life skills, to help them learn about addictions, and to help their clients be better able to manage their own lives. Finally, the final goal of CMS is to have their clients be able to live at their own optimal level of independence in the community.
What is the purpose of the evaluation?
The purpose of the evaluation is to determine client’s views on the programs strengths and weaknesses so that CMS can improve their services to benefit their clientele.
Who will use the evaluation? How will they use it?
CMS staff - To determine where their clients think improvements need to be made in their services, and to make the improvements they feel are necessary
CMS clientele - To have a say in the quality of the services they access
Community-University Institute for Social Research (CUISR) - To adhere to their mandate of bridging university research and community needs. To add to their publications for public knowledge
The researcher (Terra Quaife) - To generate a report for CMS and CUISR, and to generate data for her thesis to add to the research literature
What questions will the evaluation seek to answer?
1. Are the goals of CMS consistent with the services that their clients receive?
2. Are the CMS clients that are being served the population that needs to be served?
3. Is the process of client engagement with CMS contributing to the desired goals of CMS?
4. What are the strengths of CMS?
5. What are the weaknesses of CMS?
6. Are clients generally satisfied with CMS?
What information do you need to answer the questions?(what I wish to know and how will I know it?)
Are the services that clients receive in line with CMS’s goals? If I notice similarities between services that the clients say they obtain through the data and the goals of CMS (outlined in the logic model)
Are the clients using the services? If clients say through the interviews the different services they use that are facilitated through CMS
The clients perception of CMS strengths - Clients will be directly asked about the perceived strengths of CMS
The clients perception if CMS weaknesses - Clients will be directly asked about the perceived weaknesses of CMS
Client general satisfaction - Clients will be asked to indicate anything about CMS that they would change and if they are generally satisfied with the services they receive
When is the evaluation needed?
The deadline for the final program evaluation has been negotiated to be May 31, 2012.
What evaluation design will you use?
The evaluation will be a formative evaluation since the program is ongoing. It will use the ‘process’ component of Stufflebeam’s CIPP model as a framework for this evaluation.
Collect the Information
What sources of information will you use?
Existing information: Research on crisis management programs, research on program evaluation. Furthermore, CMS had a program evaluation completed in 1988. This program evaluation, although it had a different focus (financial), will be used to help inform the current evaluation.
People: CMS staff, CMS clients.
What data collection method will you use?
The data collection methods that will be used are document review, interviews, and focus group meetings. Document reviews will be conducted to have a foundation of what crisis management is, and to explore what the program evaluation found that was completed in 1988. Interviews will be conducted with the CMS clientele to understand what they want out of the services they receive, and the focus groups and meetings will be conducted with CMS staff in order to understand what CMS wants out of the program evaluation. This will allow them to participate in the initial stages of the program evaluation.
Instrumentation: What is needed to record the information?
Tape recorder, computer. Notes will be taken during the focus group and staff meetings. A tape recorder will be used during participant interviews to ensure accuracy of transcription, notes will also be taken as a backup.
When will you collect data for each method you’ve used?
Document review – this data will be collected during the program evaluation (mostly prior to) and during the program itself.
Focus groups/meetings – this data will be collected throughout the program evaluation and the program itself.
Interviews – this data will be collected throughout the program evaluation and the program itself.
Will a sample be used?
A sample will be used. For the focus group and meetings all 6 CMS staff will be present. The procedure used for sampling the CMS clientele will be purposeful sampling. The CMS staff will compile a list of active and assessment clients (those who are currently using their services) and 10-15 participants will be chosen (depending on theoretical saturation). To obtain a purposeful sample, client’s sex and age will be used.
Pilot Testing: when, where, how?
No pilot testing will be done for the questions that will be asked in the interview with clients. Due to time and money constrains and due to the vulnerable nature of the clientele, it is not feasible to conduct a pilot study first.
Analyze and Interpret
How will the data be analyzed?
The researcher (Terra Quaife) will be responsible for data analysis. The data analysis method that will be used is the general inductive approach. The general inductive approach is a method used to find significant themes from raw data in order to answer the research questions without the restraints of one methodology in particular.
How will the information be interpreted – by whom?
The information will be interpreted independently by two researchers to ensure trustworthiness of the data.
What did you learn? What are the limitations?
It is hard to say what the limitations are currently; however, one could speculate that a limitation may be that only a couple stakeholders were engaged in this program evaluation. The program evaluation would be more comprehensive if multiple stakeholders were engaged in the program evaluation. Furthermore, it would be a more comprehensive program evaluation if all elements of the CIPP model were used instead of only the ‘process’ component. Lastly, due to the voluntary nature of the program evaluation some data may not be obtained if participants do not wish to participate.
Use the Information
How will the evaluations be communicated and shared?
The evaluation will be made into a publication through CUISR, and therefore, will be made available to the public. A copy of the publication will be given to CMS and they have agreed to give access to the publication to all of their clients. Furthermore, the researcher (Terra Quaife) will be analyzing the data for her thesis. Once the thesis is complete, it will be available online through the University of Saskatchewan library.
Next steps?
CMS and CUISR would also like the researcher to complete a phase 2 program evaluation including stakeholders that are involved with CMS. These stakeholders will include Mental Health and Addictions Services, a psychiatrist, Client Patient Access Services, and the Provincial Court.
Manage the evaluation
Human subjects protection
Steps have been taking to ensure that the participant’s identity will remain confidential. The staff at CMS will not know which clients were chosen to participate. Furthermore, ethics have been received through the University of Saskatchewan Behavioural Research Ethics Board, thus ensuring the protection of this vulnerable population.
Management chart and timeline – see picture 1
Responsibilities
The researcher (Terra Quaife) has the responsibility to perform the literature review, to set up meetings with CMS staff, to complete the ethics application, to perform the client interview, to transcribe and analyze the data, and to write up the final report for dissemination. The CMS staff hold the responsibility of funding the program evaluation, attending meetings, disseminating the finding to the CMS clients, and making the changes from the results of the program evaluation as they see fit.
Standards
Utility
The utility of this program evaluation will be ensured due to how the request for this program evaluation came about. The CMS staff asked for an evaluation of their program, and furthermore, they will be involved in many stages of the evaluation. It therefore stands to reason that the results of the program evaluation will be utilized by the CMS staff in the betterment of their program.
Feasibility
This program evaluation is realistic, prudent, and frugal. As this outline suggests, the program evaluation is realistic: due to time restrains and financial restraints, the program evaluation was tailored to meet these requirements. There are no elements listed above that are unnecessary to carry this program evaluation out, it is therefore prudent and frugal.
Propriety
Care will be taken by the CMS staff to ensure that the list of potential participants the researcher received are all capable of giving informed consent. Furthermore, there is no obligation to participate in this study, and ethics approval was obtained from the University of Saskatchewan Behavioural Research Ethics Board.
Accuracy
The use of an external researcher will help to ensure accuracy of the reported findings. Furthermore, the methods that are used to collect and analyze the data are shown to be effective and true through research that has been done on these methods previously.
