Program Evaluation: An Introduction 5th Edition Padgett, Deborah K.;Royse, David Daniel;Thyer, Bruce A Publisher: Wadsworth - Solutions

5. Discuss the most common reactions to being evaluated and what an evaluator could do to make evaluation efforts less threatening.
4. Is it possible to conceive of a situation in which program evaluation would not be threatening? Think about the characteristics of a program and its staff that would completely welcome and embrace evaluation activities.
3. Discuss ways in which the political pressures identified in question 2 might be neutralized or negated.
2. From the class’s knowledge of local agencies, discuss political pressures that might surface and influence (either positively or negatively) any evaluation effort in selected agencies.
1. Brainstorm ways that treatment fidelity could be compromised in a program with loose management.
e. Ownership of the data and editorial authority. (At the end of the evaluation, who keeps the completed questionnaires and other data? You may need the data, or at least access to it, should you decide to write an article for a professional journal.Further, it should be clear who will write and
d. The budget needed. (Not only is agreement on the consultant’s fee important, but also there should be a definite budget for such items as travel, supplies, scales or instruments, printing, secretarial services, and so on. It is also important to specify the payment schedule—when the
c. The evaluation methodology: data, staff, and facilities needed. (Will the evaluation design necessitate the use of control groups? Will random assignment of clients be necessary? Will it be necessary to obtain sensitive information from clients? What other data will be necessary to access? How
b. The beginning and ending dates of the evaluation. (At a minimum, it is important to specify deadlines when the evaluation products must be finished.)On some occasions it may also be advisable to describe the sponsor’s expectations of a final product—in terms of appearance, the amount of
a. The purpose or focus of the evaluation. (Incorporate a list of the questions that the evaluation sponsor wants answered or hypotheses that will be investigated.)
. Using a computer database or Internet search engine, search for a scale that you could use in a project of your choosing. Be sure to identify the concept the scale should measure and the population with which it would be used. List your search efforts: What key terms or concepts did you search
3. Using either a scale that you have developed or one with known psychometrics, collect a small sample of data from your classmates or others. Enter the data into a computer with statistical analysis software so that you can compute the scale’s internal consistency. Discuss what you found and
2. With a partner, attempt to create a scale measuring some concept of your choice. Present the scale to the class for a discussion of its face and content validity.
1. Identify an instrument and read one or more articles on the instrument. Write a short paper on its development, as well as the reliability and validity that you would expect it to have if you were to use it in a program evaluation. (Be sure to identify the study’s population and how you intend
2. Read one of the articles on the development and validation of a scale referenced at the end of this chapter. Summarize, in a short paper, all the steps the author went through.
1. Skim one of the books containing rapid assessment instruments noted in this chapter. Make a list of at least five scales that you might be able to use in your future practice. Explain how each might be used.
6. Suggest one or more variables measured at the interval level that could be converted to nominal or ordinal level variables.Mini-Projects: Experiencing Evaluation Firsthand
5. For various client groups, brainstorm what behaviors might be good indicators of treatment success or failure. Why might they work better than measurements of attitudes or knowledge?
4. If you were asked to evaluate an instrument measuring hyperactivity described in a journal, what information about the instrument would you want to find in the article?
3. Discuss the following item taken from the evaluation instrument Barbara designed for adolescents: When you are an adult, what are the chances that you will be a drinker?________ I am certain I will never drink ________ I don’t think I will drink ________ I am not sure ________ I think I will
2. Barbara Daydreamer designed a three-item questionnaire to be used as a preand posttest instrument to measure adolescents’ knowledge of alcoholism as a disease. Later she was surprised to find that there were no significant differences between pre- and posttest scores. How would you explain
1. What is wrong with the following questionnaire items?a. Describe your mother’s condition during her pregnancy with you.b. Yes or No: Have you ever been involved in any accidents?c. Have you been called names and had your life threatened?
6. Performance ratings (Do performance ratings of supervisors improve?)
5. Productivity (Does productivity increase?)
4. Accident rates (Are there fewer accidents?)
3. Documentation of records (Are a greater percentage of records in compliance with quality assurance standards?)
2. Operating costs (Are costs lower?)
1. Absenteeism (Is there less absenteeism after the workshop than before?)
In the past 30 days, how many times have you taken a drink the first thing in the morning?
Once you start drinking, do you find it difficult to stop before becoming completely intoxicated?
On how many occasions have you tried to quit drinking but were unable to do so?
On how many days of the past 30 days have you had an alcoholic drink?
10. Are you an alcoholic?a. yesb. noc. undecided
9. Wouldn’t you agree that clients should keep their accounts current with the agency?a. yesb. noc. undecided
8. Approximately how many minutes do you dream each evening?a. under 15b. 16 to 30c. more than 31 minutes
7. Do you have a male relative and a female relative over 55 years of age living at home with you?a. yesb. noc. undecided
6. Do you not make a practice of shopping only on weekends?a. yesb. noc. undecided
5. What is your income?a. $10,000 to $20,000b. $20,000 to $30,000c. $30,000 or more
4. How long have you been a client with us?a. 6 months or lessb. under a yearc. 1 year or longer
. What is your marital status?a. singleb. marriedc. divorced
2. Do you come here often for help?a. yesb. noc. don’t know
1. Please rate the quality of our services:a. excellentb. goodc. poor
6. Discuss a proposal made to Congress to allow a $3,000 tax deduction for expenses incurred by parents who adopt a handicapped or older child. How would you evaluate such legislation using either a cost-effectiveness or a costbenefit approach?
5. During periods of great flux, as when there is a dramatic increase in inflation, sensitivity analyses are important. What are some other situations or factors over which an evaluator has no control and which could change over a 2- or 3-year period that could significantly affect the value of
4. Discuss whether it is more important to know a program’s cost or its effectiveness.
3. An outpatient counseling program has a long waiting list of potential clients. As a new manager, what actions could you take to reduce the waiting list? Make a list of these and discuss each relative to a cost-benefit analysis.
2. Discuss whether the following desired outcomes are likely to be monetary or nonmonetary. Would your evaluation be a cost-benefit analysis or a costeffectiveness study?a. Increased employee morale in a social service agencyb. Improved client satisfaction in an after-school tutoring programc.
1. Identify local, state, or national programs for which you would like to conduct cost-effectiveness evaluations. Discuss the program outcomes for these. Is there a choice of more than one outcome indicator per program? List all of the relevant indicators.
2. Locate a published, randomized controlled trial in a field of your choice, and read it. Assess your ability to understand the terminology and methods used in this report.
1. Browse through professional journals in a field of your choice to find the report of an evaluation using a nonequivalent control group design. Read the article and critique it.
e. Do the effects of this treatment persist over time?Mini-Projects: Experiencing Evaluation Firsthand
. Do clients who receive this therapy improve more than they would, compared to receiving standard accepted treatment?
c. Do clients who receive this therapy improve more than they would, compared to receiving a placebo treatment?
b. Do clients who receive this therapy improve more than they could with the simple passage of time?
a. Do clients who receive this therapy improve?
7. Imagine that you have been trained in a new form of individual therapy which seems very exciting and is reported to produce dramatic improvements in traumatized clients very rapidly. Discuss how you might design and carry out a series of evaluation studies of increasing complexity to answer the
Would it be ethical to offer group therapy of unknown efficacy to clients, without placing such services in the context of a systematic program evaluation?
They did get offered group therapy some months later, when the study was concluded. Why did the authors see the need to have such a control group?
6. Discuss the ethics of the study of Cohen and Fried (2007) described in this chapter, which involved randomly assigning breast cancer survivors to a notreatment condition, instead of providing them with group therapy right way.
5. Discuss for any specific program the various outcome indicators that might be chosen in a program evaluation. Are some more valuable than others for showing the impact the program is having on the lives of clients?
Who would be the subjects?
What would be the primary outcome variable? What would be the threats to the internal validity? How would you control for these? What would be the data collection procedures? Who would need to assist with the evaluation?
4. Have the class identify a local program that would be interesting to evaluate using one of the designs in this chapter. What evaluation design would be used?
3. Discuss the problems of using a self-report questionnaire with smokers in a smoking cessation clinic. What percentage might be motivated to indicate that they had stopped when in fact they were still smoking? What would constitute“hard evidence”?
d. Did the agency staff seem committed to evaluating the outcome of their program?____ YES ____ NO____ Cannot be determined
8. Solomon four-group R O X O R O O R X O R O Note: O ¼ Observation or assessment period X ¼ Intervention R ¼ Randomly assigned group research designs 249
7. Posttest-only control group R X O R O
6. Pretest–posttest control group R O X O R O O
Table 9.2 Selected Types of Group Research Designs Commonly Used in Program Evaluation Studies±±±±±±±±±±±±Name of Design Notation Pre-experimental Designs 1. One-group posttest-only X O 2. One-group pretest–posttest O X O Quasi-experimental Designs 3. Nonequivalent control group O OX
c. Did the agency conduct an outcome evaluation of this program?____ YES ____ NO ____ Cannot be determined
b. Did the program serve as many clients as projected?____ YES ____ NO ____ Cannot be determined
a. Did this program have objectives derived from the goals for the program?____ YES ____ NO ____ Cannot be determined
2. Discuss the advantages and disadvantages of an evaluator monitoring similar programs in different agencies using these four questions:
1. Discuss conducting program evaluations in various community agencies. Do some of the designs in this chapter seem to “fit” some programs better than others? Why?
4. Were these improvements maintained over a fairly long period of time, after the intervention was completed?
3. Were these improvements caused by the group work? (Yes!)
2. Were these improvements due to simply being assessed? (No!)
1. Did kids who received group work evidence improvements in self-esteem, selfcontrol, and behavior? (Yes!)
5. With what kinds of problems or programs would you feel it necessary to have a larger sample that would give you a margin of error of 5 percent and a confidence level of 95 percent, as opposed to a smaller sample giving you a margin of error of 10 percent and a level of confidence of 90 percent?
4. Give an example of a situation when a purposeful sample would be needed.
3. Under what conditions would it be important to have a level of confidence of 99 percent as opposed to 95 or 90 percent?
2. Describe at least one project best investigated by probability sampling.
1. List some evaluation projects for which nonprobability sampling might be appropriate.
3. Design a client satisfaction study for a program of your choosing. Locate or create a suitable instrument, develop the data collection procedures, and draw up a budget and projected timeline. Be sure to identify the program and the group from whom you will be drawing data.
2. Ask the agency where you are working, interning, or volunteering if any client satisfaction studies have been conducted in the past 7 years. If so, review the instrument used, the data collection procedures, and the findings. What would you have changed or done differently? Write a brief paper
1. Collect as many client satisfaction forms as you can from school, work, fastfood restaurants, or other sources. Examine them and determine how many questionnaires have similar or identical items. After you review several questionnaires, construct one of your own. Be sure to identify the program
8. What are the pros and cons of using a convenience sample of clients for program evaluation activities?
7. What are the arguments for and against using standardized instruments to measure client satisfaction?
6. What open-ended questions would you want to ask consumers participating in a client satisfaction study of a community mental health psychoeducational intervention?Would you think it would be best to use mailed questionnaires or telephone interviews?
5. Knowing what you now know about client satisfaction studies, argue for and against their place in a comprehensive program evaluation.
4. Although it is known that client satisfaction studies typically are positively biased, discuss why this form of evaluation is used so often in evaluating university faculty.
3. What variables could affect response rates of a client satisfaction study besides the survey administration procedures?
2. How would you tackle the problem of variability in response rates within the same agency (as discussed by Baker, Zucker, and Gross, 1998)?
1. Can the problem of positive bias with client satisfaction studies be “fixed”?What could be done to minimize this form of bias in a consumer feedback study?
6. Under what conditions could an SSRD be considered to represent an “experimental”design?
5. Distinguish between the functions of needs assessments, formative evaluations, quality assurance studies, and summative evaluations.
4. Locate an example of a single system design published in a professional journal during the past 5 years. Identify the type of design, the outcome measure, and how well the intervention is described. Judge for yourself whether the author’s conclusions are justified by the data.
3. Could SSRDs be used in conjunction with consumer satisfaction studies? How?
5. How many cancellations (or no-shows) were there last month? What were the characteristics of those who canceled and gave no notification? (How many were single mothers with small children or unemployed persons with no transportation?)
4. Of those clients referred for services last month, how many were referred by the criminal justice system?

Showing 400 - 500 of 564

Program Evaluation: An Introduction 5th Edition Padgett, Deborah K.;Royse, David Daniel;Thyer, Bruce A Publisher: Wadsworth - Solutions

Step by Step Answers