Category Archives: 5×52

Spotlight on Productivity

Theming qualitative data
Theming qualitative data.

It’s been a week since I posted on this blog. During this time I have made significant progress on several projects. I analyzed data, wrote up findings, and planted seeds for new projects. Needless to say, I haven’t had the time and space to think about my evaluation and design! Since I have committed to posting once every workday (my 5×52 project), I’m going to be doing a bit of catch up in the next few days. Before returning to discussing evaluation, let’s turn to the topic of productivity.

The next series of post will feature  productivity tools that works well for those of us leading the researcher/evaluator lifestyle.

(List will be updated with links when content becomes available.)

Post 1) Productivity challenges among grad students, researchers, and evaluators

Update. Post 1.5) Getting Things Done: Mindset and Approaches

Post 2) Project Dashboard: Kanban Style

Post 3) How to: Track your time and progress using Task Progress Tracker

Post 4) Day Planning: Emergent Time Planning

Post 5) 5 principles to jumpstarting productivity


Evaluation Lessons from The Stanford $5 Dollars Challeng

English: Many dollar banknotes.

Yesterday, I introduced the Stanford $5 Challenge. Today, I look at what evaluators doing design / developmental evaluation work could learn from this.

If you have $5 dollars in seed funding and only 2 hours to make it happen, what would you do to make the most money?

This is known as the Stanford $5 Challenge. Tina Seelig asks this of her students at Stanford University enrolled in the Stanford Technology Ventures Program. Most students, she explains, would use the money towards a lottery ticket or gamble away the money at Las Vegas. These students assume that the $5 is too little money to do much with, and engaging in high risk/high reward activity is the way to go to net the most profit.

Surprisingly, the teams that made the most money kept their $5. Instead, they reframed the problem and challenged assumptions, and looked to opportunities beyond the initial framing of the problem. Focusing on the $5 seed money framed the problem too tightly.

So, what could design-informed evaluators learn from this?

There are two questions we must raise in working with any innovative program at any phase of our engagement:

  • Does the program serve a real and significant (i.e. meaningful) need, and;
  • Is the program design optimal for effecting the intended change.

Raising the question of whether the program can serve a real and significant need is analogous to asking whether there is a market for a product/service in the business world. A program may be mounted in response to some perceived needs on the part of the implementers (e.g. government, funders, etc.), but not from the perspective of the program targeted recipients. For instance, universities may feel the need to introduce educational programming for students living in residences out of a sense of social purpose, but the program may be deemed  ineffective and flawed, because students see little reasons to be ‘educated’ in their living spaces. Some might view such intervention as an intrusion of their down time, while others might actually resent such attempts on the University’s part. In other words, our job is to raise the question of whether the program serve some real and significant need from the perspectives of the program recipients.

However, raising such a question of program recipients can sometimes be problematic. Recipients may very well perceive that a program is unwarranted, when in fact they could very well benefit from participation (and in some cases, they should participate in the program in spite of feeling no particular need for it). Those of us who have worked with children know this:  few children would volunteer to sit patiently and practice at the piano or voluntarily sign up for swimming lessons at their own will. What good parents do is that they expose their children to these opportunities, build their confidence, and help them persist despite initial resistance. Why? It’s because they know that some activities are good for the kids in the end. In other words, misinterpreting that there are no extant needs in a program situation can be equally dangerous, as the $5 challenge illustrated; those students who focused too narrowly on the problem saw $5 as too little money to do anything meaningful and subsequently gave up.  Evaluators can help their clients by raising questions and questioning assumptions. One way to do is to problematize the situation to promote discourse. “Is it really the case that… ” On to the second question.

The question of whether the program design is optimal for effecting the intended change is about the linkage between the theory of change and the theory of action within a particular program. We saw in the $5 dollar challenge who made the most money thought outside of the box and turned to different ways to make money.  In program evaluation, we can ask the following questions of the theory of change: is the way we currently conceptualize change appropriate? Might there be other ways to effect change? What blinders might we have on? Where else can we learn more and think differently about this program? If this is where we want to end up (and see these kind of changes happening in the program recipients), how else could we facilitate these changes?

Assuming that we are satisfied with the theory of change, we can begin to consider the theory of action, i.e. how a program marshal its resources to operationalize its theory of change. Ask yourself , might there be other ways of achieving the same intended change, given how we think change could be realized? This is a challenge that the business world is especially well-adept at tackling due to

competition. Let’s take the example of a fueling station franchise.  While the business model to turn a product (gasoline or diesel) into profit is essentially the same across different companies, each theory of action differs in where companies place their refueling stations, loyalty programs, pricing of products, and other convenience items (e.g. coffee, car washes). These different operations (activities) influence the purchasing decision, and so companies develop strategies in hopes of gaining a competitive advantage over one another.  In the social space, these inevitably will be questions of the comparative sort, and will have to be answered empirically.

  • To sum up, the take-home lesson here is to think hard about whether the program model fits the program context. When it does, we have a viable program that serve a real need (and therefore stand to make a real difference in people’s lives.)
  • When we are designing program models, i.e. when we are trying to come up with a program, the focus is about optimizing the program model to fit the program context.

The Stanford $5 Dollars Challenge

If you have $5 dollars in seed funding and only 2 hours to make it happen, what would you do to make the most money?

English: Many dollar banknotes.
English: Many dollar banknotes. (Photo credit: Wikipedia)

This is known as the Stanford $5 Challenge. Tina Seelig asks this of her students enrolled in the Stanford Technology Ventures Program at Stanford University.

Most students, she explains, would use the money towards buying a lottery ticket or gamble away the money at Las Vegas. These students assume that the $5 is too little money to do much with, and engaging in high risk/high reward activity is the way to go to net the most profit.

Surprisingly, the teams that make the most money kept the $5. Instead, they reframed the problem and challenged assumptions. Focusing on the $5 seed money framed the problem too tightly. Seelig tells of students who looked for opportunities around them. One team set up a free bike tire pressure check-up service outside of Stanford Student Union. They charged a few dollars to re-inflate tires. Stanford students were appreciative of the service, so much so that they generated a higher profit when they switched to a by-donation model. Another team secured reservations in local restaurants for diners and sold them for a profit. The team that made the most money did something even more inventive: they sold their three-minute slot, when teams were to present to their classmates on their strategy, to the very same companies that wanted to recruit the program’s graduates.

Lessons learned: The take-away here is one of learning to think innovatively and creatively. Identify what the perceived problem to be. Identify what assumptions are at play that frame the initial problem formulation. Then, question those assumptions at play. Finally, reframe the problem.

You can watch Tina Seelig talking about  the $5 Challenge here. Tomorrow, I’ll explore the implications of this challenge for developmental evaluators and design-minded evaluators.

The Design-Informed Program Evaluation Manifesto

the mastery within.

Consider the lyrical lines
from a Verdi’s bel canto aria,
or the highly evolved
from Picasso’s bull.

Consider the artful sentence,
or a poet’s communication through
white space.

the injustice of genocide
on a photograph;

the peaceful, pulsating
indicator light
on an Apple computer;

the transformation of an apprentice
under the tutelage of a master.

the catharsis from a Shakespearian tragedy
(or, say, a modern-day Steven Sondheim Sweeney Todd);

to inspire, to catalyze, to set in motion.

rhythm, sound, harmony, syntax,
colour, shade, composition.
This gestalt generates creative tension.

the ingenious thinking that lies within springs to life.

To the trained eyes,
simplicity may reveal complexity
and from chaos reveal order;
the resolution is one of beauty and wholeness.
To the untrained eyes,
the sophistication remains,
though unnoticed;

An experience is nevertheless shaped,

experienced, and inspired by the


To design is to render our intentions into the active voice.



On Writing and Program Development

My first opportunity to seriously consider the intersections between evaluation and design was in a class on writing. The instructor, a poet herself, had us developing our craft as mature writers would. She introduced us to how writers think about writing and how writers approach the task of writing. The formulaic, straight-through write-once approach learned and honed in grade school made way for a more organic approach—a practice I continue to this day.

The approach goes something like this: Start with flow-writing, an interrupted 10-min session of brain dump, to pen thoughts onto paper. (This is the creative phase of the writing process.)  Then, return to the writing and edit ruthlessly. Focus on clarity and precision of language. Finally, copyedit the writing after all the heavy-lifting is done. (See Peter Elbow’s work, Writing without Teachers; video clip included below).

What struck me about this approach to writing was how it mirrors the developmental approach advocated in developmental evaluation. Both approaches focus on promoting purposeful and intentional changes made the object of development, be it a piece of writing or a program. However, the kind of change desired is not one of incremental changes, but more of changes in form and function. In program evaluation, we understand this to be changes to the program model.

Like writing, there has been a strong emphasis and reliance on utilizing a linear approach to developing programs (needs assessment –> program planning –> program implementation –> program evaluation).  It would seem, though, that this linear approach has limited utility and is only appropriate for few conditions meeting strict conditions.  More on this thought in the future.

My Evaluation Origin Story: How I discovered evaluation and design.

Innovation (Photo credit: masondan)

A young program evaluator was casted into the deep waters of innovation and discovered the potentials of developmental evaluation and design. 

One afternoon during the winter term of my first year in graduate school, my advisor asked if I had a second to spare. Standing in a darkened, narrow corridor,  in one of those serendipitous moments, she asked if I could help her re-think the undergraduate training of preservice teachers. For five years, she and a colleague had been working on developing a  course in classroom assessment that was mandatory for all 700 students enrolled each year.  Despite their efforts, the a different quality of learning was desired. Their instruction was constrained by limited financial resources, restrictive lecture-style instruction, and a meagre allotment of 7 hours for instruction. They had wanted to experiment with alternative ways of structuring learning in classroom assessment. I was intrigued by the problem and the prospects of making a small contribution to a practical problem. I  agreed.  As was typical of conversations with my advisor, I left with a sense of intellectual bewilderment and  stimulation.

My curiosity in using evaluation as a vehicle for problem-solving and social change thus began. What I had originally thought to be a simple project involving some literature review, analysis into our particular program context, and maybe some coaching sessions, blossomed into a full-fledged developmental evaluation project. A few months later from that first interaction I found myself helping to pilot a blended learning mini-course that saw the integration of microblogging to connect teacher candidates, who were by then on field placement, with their peers and with faculty mentors. Just when they were grappling with doing assessment in their respective classroom, the instructional team had the opportunity to guide and inject new   thinking about assessment. Assessment suddenly sprung to life became a practical and situated professional practice.

What was remarkable about that project was the role that evaluation–specifically, developmental evaluation–played. Developmental evaluation not only provided a means towards anchoring our decision-making through ‘best-available data’, it also helped us to continually develop the program through incremental learning, itself a developmental process. It also occurred to us that incorporating some sort of evaluation exercise as a way of reality-testing was the prudent and responsible way to go about piloting an unknown, untested way of teaching; DE offered that.

Emerging from this substantial experience myself, I subsequently went back, analyzed the project through a researcher lens, and wrote it up in the form of a case study. This project proved not only intellectually gratifying but also viscerally draining. During this project,  I experienced anxiety.  I experienced emotions like feeling lost and feeling stuck. In the write-up, I explained that these emotions resulted from the uncertainty associated with innovating. Often we see and talk about evaluation as if it’s a systematic, clinical procedure, and far too often we neglect the emotional component of leading and participating in evaluations.

In trying to unpack the evaluation for the case study, I contended with the notion that what I had participated in was indeed not an evaluation. Indeed, the project (the evaluation) and the evaluand (the program) lacked many of the hallmarks of a program evaluation: clear, specific, and measurable goals; an operational program; or program participants. But a closer look at some of the processes and activities told a different story. What had transpired wasindeed  an evaluative exercise and evaluative thinking played a prominent part in moving the project forward.

But there was something more…

the way we systematically explored options and made decisions about what might had been appropriate for use in our particular context… the aim to ‘do different’… and the permission to be vision-driven and participant-driven…oh, and the way in which we went from ‘nothing to something’…. how we embraced uncertainty, how I led and coached in service to my clients and participants… and creating the space to think creatively and innovatively…

… and that something was design.