Category Archives: Conference

5 things I’m resolving to doing post-conference #eval15.

I don’t know about you, but after every conference, I go into hermit mode. This often means that I fail to follow up with dear colleagues whom I meet only once a year or act on important ideas and learning. But, slack no more.

Here are five things I’m resolving to doing post-conference #eval15.

1. Follow up with contacts e-mails.

2. Upload my slides to TIG libraries.

3. Consolidate and review my conference notes. Follow up on new resources.

4. Declare that I am a part of a global eval community

5. Heed the call to embrace and report on failure in evaluation, and to learn from failure. To that end, I’m committing to writing up a less-than-successful developmental evaluation.

So, who’s with me?

PD-TIG #EVAL15 Panel: Huey, Gargani, Stead, & Norman on Program Design: Evaluation’s New Frontier?

I’m really looking forward to #EVAL15 because this will be the first year that the conference will feature a program track in program design. Here’s a look at the full agenda.
I am especially looking forward to the PD TIG-sponsored panel,  “Program Design: Evaluation’s New Frontier?”. The session will feature:
The panelists been asked to consider what program design could mean in the context of evaluation theory and practice. The goal of the session is to attempt to arrive at an initial articulation of what program design could mean in terms of theory and practice.
Here’s the abstract: Notions of design have entered the mainstream in both public and private sectors. Underpinning this shift is an emerging realization that the once-professionalized approaches and mindsets designers employ to solve complex problems may be applied to other contexts. Bridging evaluation with design holds potential to reconceptualize both the theories and practices of evaluation, and as a consequence, enhance evaluation influence.  This panel of expert evaluators draws on their theoretical and practical experiences to explore what ‘program design’ could mean for evaluators and evaluation practice.
 Without giving too much of the plan away, the speakers will be responding to the following prompts:
  • How have you come to ‘program design’? What do you mean by it?
  • What potential do you see in program design in enhancing evaluation, if at all? What hazards do you see in evaluators engaging in program design?
  • Are there any dangers in evaluators assuming the role of a program designer? Is there not a risk of cooptation?
  • What competencies or skills do you see as critical to doing PD work? How might newcomers go about learning these skills?
  • If there is potential in program design, what might be next step toward growing or legitimizing its practice? What should we strive to understand better? What might this body of knowledge be comprised of?
 It promises to be an exciting panel. This session has been scheduled for November 12th, 2015 (3:00PM – 4:30PM) in “Field”. Come for the panel and stay for the business meeting, which will be short!
See you there!

On launching the Program Design Topical Interest Group

I recently wrote about our motivation behind starting a Topical Interest Group (TIG) on Program Design on AEA365.

Our interest in organizing the PD-TIG grew out of a casual conversation. We (Karen Widmer, Terence Fitzgerald, and I) realized that we each held responsibilities for program design in our respective practice. We were inspired by the potential for infusing evaluative thinking and evidence into program development, and in doing so, evaluators might further contribute to clients’ goals of developing robust, impactful programs. However, even among ourselves, we had differing perspectives on what this might look like in practice. As a group, we were inspired by Gargani and Donaldson’s work on program design, Patton’s work on developmental evaluation, and more generally, writing on theory-driven evaluation. We said to ourselves: Wouldn’t it be great if we could get together with others who might share our passion and curiosity about program design?

It’s been over two years since that initial conversation, and it has taken a lot of work behind-the-scene to get the TIG up and running.

I am most excited by the idea that the TIG can engage the broader evaluation community on program design than individuals alone.

New this year to the annual AEA conference is the program design track to the program. A program schedule can be found on the PD-TIG web site.

In an upcoming post, I’ll profile an all-star panel session being organized during the PD-TIG Business Meeting. It features Dr. Huey Chen, Dr. John Gargani, Brenda Stead, and Dr. Cam Norman as panelists.

Until then, onwards!

Highlights from Michael Quinn Patton’s #eval13 talk on the ‘State of Developmental Evaluation’

Michael Patton gave a great talk today at AEA13 on the State of Developmental Evaluation.  Here are some highlights.

1. The ‘Doors’ to Discovering Developmental Evaluation.

Patton observed that developmental evaluators and clients typically arrive at DE through multiple doors. One door through which people arrive at DE are those engaged in innovation. The second door through which people arrive at DE are those seeking systems change. The third door through which people arrive at DE are those dealing with complexity. The final door through which people arrive at DE are those working with unstable, changing context.

Driving this  ‘search for the alternative’ are evaluation users’ desire for a compatible evaluation framework.

2. DE is becoming a bonafide approach. 

AEA 13 features over 30+ sessions on developmental evaluation.

The Australasian Evaluation Society recently awarded their Best Policy and Evaluation Award to a crew of developmental evaluators.

(The CES awarded its  best student essay to an empirical research on understanding the capacity of DE for developing innovative program.)

3. DE is best enabled by clients who are willing to explore and experiment.

4. DE is methods-agnostic, and in fact, defies prescription.

Patton emphasized the importance of operating from the principles of DE and applying and adapting them when conducting DE. (Another way of looking this is to frame DE as engaging in inquiry… this might actually make a nice blog post).

Some observations…

Participants raised some great questions during the Q&A session.  Part of the confusion, it seems to me, lies in the more subtle aspects  to how and why Developmental Evaluation might be more appropriate/useful in some contexts. This confusion arises because of how necessarily responsive developmental evaluation is by design. The on-ramping for someone who hasn’t done DE, but wants to do it, can be difficult. So,  I wonder if there might be a place for a clearinghouse of sort for frequently asked questions—i.e. the sort often asked by newcomers.

3 Quick Survival Tips for Attending AEA13 #eval13

Here are three quick ‘survival tips’ if you are attending AEA 13 in Washington.

1) Free wifi is provided in the lobby of the Washington Hilton. There is no free wifi coverage in the conference areas. There is no free wifi in rooms if you are staying at the Washington Hilton. Alternatively, the Office Depot across the street offers free wifi.

2) Buy a jug or two of water from the nearest pharmacy or convenience store. If you are staying at the Washington Hilton, the nearest pharmacy is called Rite-Aid. It’s 350 ft, or one minute of walking, away, from the Hilton.

3) There are many reasonable lunch and dinner options along Connecticut if you go south. Aim for the Dupont Circle area.

Bonus: There are two Starbucks within close walking distance if you go south on Connecticut. They have cheap sandwiches for around $5. Possible lunch options if you are on a (grad student) budget.

Key Takeaways from Tom Chapel’s AEA13 Workshop: Logic Models for Program Evaluation and Planning

Learning is never an easy task, but, boy, is it worth it. One of the best aspects of the American Evaluation Association annual conference is actually what precedes it — the preconference workshops. More than 60(!) workshops are being offered this year. It is a great opportunity to hear some of our field’s luminaries, thinkers, theorists, practitioners, and innovators share what they know and love doing. It’s also a chance to ‘stay close to the ground’ and learn about the very real concerns and challenges practitioners are experiencing.

TomChapelI just finished Tom Chapel’s (Chief Evaluation Officer, Centre for Disease Control)  2-day workshop on “Logic Model for Program Evaluation and Planning”. In this blog post, I share some of the more salient insights gathered from his session.  Rarely  can one abstract evaluation issues so clearly from a practitioner perspective and be able to teach it so succinctly. He draws in great case example; they are rich, sufficiently complex, yet simple enough to carry great educational value. Kudos to Tom.

My interest in this is two-fold. I am interested in the practical aspects of logic modeling. I am also interested on a theoretical level how he argues for its role in evaluation practices. So, in no particular order, here are nine key insights from the session.  Some are basic and obvious, while others are deceivingly simple but not.

Some foundational ideas:

1)   At the most basic level, a logic model is concerned with the relationship between activities and outcomes. It follows the logic: if we do this, then we can expect this to occur.

2)   Program outcomes—more appropriately, a series of outcomes—drive at a “need”, i.e. the social problem that the program aspires to change.

3)   A logic model is aspirational in nature. It captures the intentions of a program. It is not a representation of truth or how the program actually is (that’s the role of evaluation).

4)   Constructing a logic model often exposes gaps in logic (e.g. how do we get from this step to this step…??). Bringing clarity to a logic model often requires clarification from stakeholders (drawing on practical wisdom) or  empirical evidence (drawing from substantive knowledge underlying the field). It also sets up the case to collect certain evidence in the evaluation if it proves meaningful in an evaluation to do so.

5)   And in talking with program folks about their conceptions of a program, differing logic about why and how the program works is often exposed. These differing views are not trivial matters because they influence the evaluation design and the resulting values judgment we make as evaluators.

6)   And indeed, explicating that logic can surface assumptions about how change is expected to occur, the sequencing of activities through which change is expected to occur, and the chain of outcomes through which change progresses towards ameliorating the social problem. Some of these assumptions can be so critical that unless attended to could lead to critical failure in the program (e.g. community readiness to engage in certain potentially taboo topics; cultural norms, necessary relationships between service agencies, etc…).

7)   Employing logic modeling, thus, avoids the business of engaging in black-box evaluation (a causal-attribution orientation)  which can be of limited value in most program situation. I like the way Tom puts it: Increasingly evaluation are engaged in the improving business, not just the proving business. Logic modeling permits you to open the black box and look at how change is expected to flow from action, and more importantly, where potential pitfalls might lie.

But here’s the real take-away.

8)   These kinds of observations generated from logic modeling could be raised not only at the evaluation stage, but also during planning and implementation. These process use (an idea usually attributed to Michael Patton) insights could prove tremendously useful even at these early stages.

9)   Indeed, problems with the program logic is especially problematic when raised at the end. Imagine telling the funder at year 5 that there is little evidence that the money made any real impact on the problem it set out to address. Early identification of where problematics could lie and the negotiations that ensue can be valuable to the program.

The Design Argument inherent in using Logic Modelling for Planning

First, what Tom is essentially suggesting here is that attention paid to the program logic is worthwhile for evaluators and program staff at any point during the program life cycle.

Where these conversations stand to make a real, meaningful contribution is before the “program is let out of the barn”.  This is important because the intentions inherent in the logic underlying a program gives rise/governs/promotes the emergence of certain program behaviour and activities (in much the same way that DNA or language syntax gives rise to complex behaviour). The logic both defines what IS and IS NOT within the program, doesn’t it.

So, if we accept  the premise that a program can be an object of design (i.e. that we can indeed design a program), then we could argue that the program logic constitutes a major aspect of the design. And because we can evaluate the design itself, as can we with any design objects, evaluating the program design becomes a plausible focus within program evaluation.