Chi Yan Lam, PhD(c), CE, is a program evaluator, researcher, and educator. He works closely with social innovators and public sector leaders across Canada to bring analysis and strategy to bear on program development, evaluation, and decision-making.
View all posts by Chi Yan Lam →
“The term inputs refers to the resources made available to a program, policy, or curriculum to enable its operation. More precisely, inputs provide the antecedent conditions from which some programmatic activities are to occur and, as a consequence, achieve some predetermined objectives. Put simply, inputs are what get invested to do the work.
Inputs are important to make explicit because they play a limiting function in the implementation program, policy, or curriculum. For instance, the reach of a program is dependent on its inputs, such as the funding allocated to the program, the size of the venue in which the program is delivered, or the availability of program staff with expertise in the area.Without sufficient input, the efficacy and/or the effectiveness of a program may suffer. Yet, the opposite is not necessarily true. Overinvesting in a program, policy, or curriculum does not necessarily yield greater or better outcomes, if processes are unable to take advantage of abundant inputs. Hence an accurate accounting of inputs is important to understanding the effects of a program, policy, or curriculum.”
The Program Design Method series profiles design methods and exercises from professional design communities that program designers and evaluators can use to work more creatively and innovatively with clients to develop more impactful programs. The first in this series is User Persona.
From evaluating programs we know that effective programs are those that respond to some genuine need(s) or perceived problem(s) for some targeted users in some purposeful ways over a period of time which can lead to meaningful change. Program developers typically have a good sense about what meaningful results they wish to promote with the program. And, program evaluators can help program developers to clarify, measure, and interpret what meaningful, measurable results might be. Where there can be more ambiguity is uncovering constitutes purposeful ways to working with those targeted users to respond to their genuine needs.
Take, for instance, a hospital’s initiative to promote hand sanitation ( the desired behaviour change) to reduce iatrogenic, hospital-acquired infections (impact). What might be effective means toward that end, given a constrained budget? Would installing hand sanitizer dispenser at doorways suffice? Should the hospital, instead, equip its staff with mini sanitizer bottles to be worn on the body? Or, should the initiative focus on producing public health educational messaging to remind its health care staff the potential risk and harm to patients that improper hand sanitation can cause? Or, should it instead highlight the benefits and protection to handwashing for the clinicians? Should the program target both health care staff and visitors?
More often than not, we do know what can be done to remedy a problem or address a user need; we just don’t know which of many plausible solutions would prove effective for a particular situation. Deciding on which course of action to take depends a great deal on how we construe the problem and who the end-users are. This is important because the more specific we understand our users, the more targeted and tailored a program intervention can be designed, and the higher the likelihood that we can make a real contribution.
So, how can program designer design programs with users in mind? One design method is called User Persona.
What is User Persona?
A User Persona is a composite character constructed for the purpose of program design to represent a significant user group. A user persona is typically presented as a profile, detailing the constructed user’s background, histories, needs, aspirations, and values, among other attributes.
It helps if you can see some examples. Here’s one.
The substance of User Persona should be empirical grounded, i.e., the designer should go out there and meet with people who could shed light on the program’s target users; the process to creating and refining a user persona is not fiction-writing.
A good User Persona gives its reader a good sense about the profiled user. This means that the information presented in User persona has to be highly specific and contextual. It is unlike the kind of information that can typically be learned from survey research (e.g., demographics), which tends to be summaries of particular measures. The most useful User Personas are often constructed from the extreme cases, the outliers, and fringes. This is because even users who are alike in their demographics can differ on important dimensions. In a recent developmental evaluation, the distinction technology-welcoming vs tech-adverse was helpful in guiding program decisions for even Millennial teachers.
User Persona is an important program design method for at least two reasons:
The process to creating and refining a user persona brings program designers closer to its users. In fact, designers are encouraged to develop an emphatic understanding, i.e., first-hand experiences of the particular needs and situations user find themselves in. The old adage rings true here: to walk a mile in someone else’ shoes. Such experiential understanding may be the source of some important insights and further inform program directions
A User Persona provides a reference point for evaluating how (and how well) a program is expected to work for particular users.
When to use/do it?
The User Persona method can be initiated during all phases of program development. It is especially helpful during initial planning or as a part of ongoing development:
Initial Planning: User Personas allow us to make some projection about how the studied user might access, benefit from, and participate in the program. Insights generated from the exercise would inform decisions to be made of the program.
Ongoing Development: User Personas allow us to evaluate how the studied user would access, benefit from, and participate in the existing program.
In either case, User Personas provide a means for clients and program designers to engage in evaluative discussion. For a particular user, consider the following:
How might he/she experience the programs?
How might his/her experience differ from the assumed theory of change?
What implication might this person’s needs/values/assumptions/characteristics have on realizing program outcomes?
Might this person experience any barriers or ‘hiccups’ as they participate in the program?
What could be meaningful adaptation/modifications for this user (and others like him/her)?
Okay, so how do I create a User Persona?
User Persona should be constructed from empirical data, i.e., they should not be works of fiction. And you should get as close as you can to targeted users. After all, it is not helpful to design a program for hypothetical users.
Christof Zürn of Creative Companion has constructed an excellent poster to illustrate the kinds of information that can be helpful to capture and represent in a User Persona.
Such information can be learned through interviews, observation, focus group, and even surveying. Usability.gov has a great section on which questions to ask during persona development here.
It is helpful as you can imagine to work with a number of different User Persona during program design.
User Persona is an user-centered method in program design. It allows program evaluators and developers to interrogate a program from a particular user’s perspective. The User Persona should tell a story about the user’s history, needs, values, and aspirations; the substance of which should be grounded empirically.
Are you curious about program design? Have you any particular questions about its methods and methodologies that you’d like us to write about? Drop me a note below or find me on Twitter @chiyanlam, where I curate tweets on evaluation, design, social innovation, and creativity. My thanks to Brian @StrongRoots_SK for inspiring this post.
AEA 2015 was a special one for me for it was the first time the concept of program design got traction. I presented with fellow design-minded evaluators in two sessions.
In the first one, I reported on my experience of embedding design principles into a developmental evaluation. The presentation was entitled, Lessons-learned from embedding design into a developmental evaluation: The significance of power, ownership, and organizational culture. And, here’s the abstract:
Recent attempts at developmental evaluation (DE) are incorporating human-centered design (HCD) principles (Dorst, 2011; IDEO, n.d.) to facilitate program development. HCD promotes a design-oriented stance toward program development and articulates a set of values that focuses the evaluation beyond those ideals expressed by stakeholders. Embedding design into DE promises to offer a more powerful means to promoting program development beyond either approach alone. Yet, embedding design into DE introduces additional challenges. Drawing on a case study into a design-informed DE, this panelist discusses the tensions and challenges that arose as one developmental evaluator attempted to introduce design into a DE. Insights from the case study point to the importance of:
– Attending to power dynamics that could stifle or promote design integration; and,
– Evaluator sensitivity over the deep attachment program developers had over program decisions
These findings allude to the significance of organizational culture in enabling a design-informed DE.
In the second presentation, Chithra Adams (@ChithraAdams), John Nash (@jnash), Beth Rous (@bethrous), and I discussed how principles of human-centered design could be applied to the development of programs.
Specifically, we introduced two design exercises–Journey Mapping, and User Archetyping–as means to bringing human-centered design principles into program design and evaluation.
In an upcoming post, we’ll take a deep dive into these design exercises and examine their application to program design.
Are you curious about program design? Have you any particular questions about its methods and methodologies that you’d like us to write about? Drop me a note below or find me on Twitter @chiyanlam, where I curate tweets on evaluation, design, social innovation, and creativity.
I don’t know about you, but after every conference, I go into hermit mode. This often means that I fail to follow up with dear colleagues whom I meet only once a year or act on important ideas and learning. But, slack no more.
Here are five things I’m resolving to doing post-conference #eval15.
1. Follow up with contacts e-mails.
2. Upload my slides to TIG libraries.
3. Consolidate and review my conference notes. Follow up on new resources.
4. Declare that I am a part of a global eval community
Proud to launch Blue Marble Evaluators at #eval15 – Think globally, act globally, evaluate globally
The panelists been asked to consider what program design could mean in the context of evaluation theory and practice. The goal of the session is to attempt to arrive at an initial articulation of what program design could mean in terms of theory and practice.
Here’s the abstract: Notions of design have entered the mainstream in both public and private sectors. Underpinning this shift is an emerging realization that the once-professionalized approaches and mindsets designers employ to solve complex problems may be applied to other contexts. Bridging evaluation with design holds potential to reconceptualize both the theories and practices of evaluation, and as a consequence, enhance evaluation influence. This panel of expert evaluators draws on their theoretical and practical experiences to explore what ‘program design’ could mean for evaluators and evaluation practice.
Without giving too much of the plan away, the speakers will be responding to the following prompts:
How have you come to ‘program design’? What do you mean by it?
What potential do you see in program design in enhancing evaluation, if at all? What hazards do you see in evaluators engaging in program design?
Are there any dangers in evaluators assuming the role of a program designer? Is there not a risk of cooptation?
What competencies or skills do you see as critical to doing PD work? How might newcomers go about learning these skills?
If there is potential in program design, what might be next step toward growing or legitimizing its practice? What should we strive to understand better? What might this body of knowledge be comprised of?
It promises to be an exciting panel. This session has been scheduled for November 12th, 2015 (3:00PM – 4:30PM) in “Field”. Come for the panel and stay for the business meeting, which will be short!
Who hasn’t screwed up in grad school? Been there, done that.
Professors Kevin Haggerty (Professor of Sociology and Criminology, University of Alberta) and Aaron Doyle (Associate Professor in the Department of Sociology and Anthropology, Carleton University) recently published a book on the many ways one could screw up in grad school.
An Introduction to Screwing Up
Who are I?
Thesis vs. Dissertation
THE SCREW UPS Starting Out
1. Do Not Think about Why You Are Applying
2. Ignore the Market
3. Stay at the Same University
4. Follow the Money Blindly
5. Do an Unfunded PhD
6. Do an Interdisciplinary PhD
7. Believe Advertised Completion Times
8. Ignore the Information the University Provides You
9. Expect the Money to Take Care of Itself
10. Go it Alone and Stay Quiet
11. Choose the Coolest Supervisor
12. Have Co-Supervisors
13. Do Not Clarify Your Supervisor’s (or Your Own) Expectations
14. Avoid Your Supervisor and Committee
15. Stay in a Bad Relationship
16. Expect People to Hold Your Hand
Managing Your Program
17. Concentrate Only on Your Thesis
18. Expect to Write the Perfect Comprehensive Exam
19. Select a Topic Entirely for Strategic Reasons
20. Do Not Teach, or Teach a Ton of Courses
21. Do Not Seek Teaching Instruction
22. Move Away from the University Before Finishing Your Degree
23. Postpone Those Tedious Approval Processes
24. Organize Everything Only in Your Head
25. Do Not Attend Conferences, or Attend Droves of Conferences
Your Work and Social Life
26. Concentrate Solely on school
27. Expect Friends and Family to Understand
28. Socialize Only With Your Cliques
29. Get a Job!
30. Write Only your PhD Thesis
31. Postpone Publishing
32. Cover Everything
33. Do Not Position Yourself
34. Write Only to Deadlines
35. Abuse Your Audience
Your Attitude and Actions
36. Expect to be Judged Only on Your Work
37. Have a Thin Skin
38. Be Inconsiderate
39. Become “That” Student
40. Never Compromise
42. Say Whatever Pops Into Your Head on Social Media
43. Assume That the University is More Inclusive Than Other Institutions
44. Rush into a Legal Battle
45. Get Romantically Involved with Faculty
46. Cheat and Plagiarize
Am I Done Yet? On Finishing
47. Skip Job Talks
48. Expect to Land a Job in a Specific University
49. Expect People to Hire You to Teach Your Thesis
50. Turn Down Opportunities to Participate in Job Searches
51. Neglect Other People’s Theses
52. Get an Unknown External Examiner
53. Do Not Understand the Endgame
54. Be Blasé about Your Defense
55. Do Not Plan for Your Job Interview
56. Persevere at All Costs
57. Consider a Non-Academic Career a Form of Failure
Final Thoughts Appendix: A Sketch of Grad School
The People Acknowledgements Index