31165-apple

Merit/Worth/Significance Explained in Plain Language

I recently received an e-mail from a fellow doctoral student asking me to explain Scriven’s notion of merit/worth/significance. One part of her dissertation is around determining the value of test preparation training (e.g. MCAT/GMAT/LSAT prep courses) among language learners. One of her committee members suggested that she use M/W/S as a framework for tackling this aspect of her work. So, I wrote back to her, saying, why don’t we Skype and talk about this.

I’ve been thinking about this problem since. As an evaluator, I am reminded that one of the basic purposes in evaluation is the determination of merit/worth/significance of something. And, we typically refer to whatever we are evaluating (the ‘something’) as the evaluand. This classical definition of evaluation constitutes a part of what Scriven (1991) refers to as the logic of evaluation in a paper by the same name in the Evaluation Thesaurus. The logic of evaluation is a seminal contribution to the field as it gets at the core of what makes evaluation unique as compared to, say, research–evaluation allows us to make evaluative claims. The distinction between M/W/S and its application in evaluation is an important one, but finding accessible writing on this topic is difficult. Perhaps, m/w/s is so obvious to everyone else but me :). Hopefully not.

So… what’s merit, worth, and significance?

Merit, worth, and significance can be easily explained by reference to evaluating anapple. Say you’re at a grocery store. The decision you’ll have to make is to buy an apple. 

Merit

Merit has to do with the intrinsic properties, characteristics, or attributes of an evaluand. When buying an apple, most people would prefer an apple that is not rotten, is sweet to taste, and is not otherwise damaged or deformed. That’s typically what people would look for if the apple were to be eaten on its own. But, what if you were buying the apple to make an apple pie? Then, you may wish to buy an apple that is not sweet but  tart. So, as we can see, what we value to be desirable attributes of an object depends on other contextual factors. 

Here is another example. A car has merit if it is reliable (i.e. does not break down while you’re driving down the highway; predictable), is safe (i.e. has adequate safety features and operates as intended), and is powerful relative to its intended application (i.e. say, a commuter car vs a pick-up truck to haul construction material). Now, you may say, a car has merit only if it has an integrated air conditioning unit or a stereo system. A design-conscious person may insist that a car be visually appealing. Increasingly, drivers want good fuel consumption. Different people may hold different views of what constitutes merit. In other words, an evaluand may be evaluated against different dimensions of quality, i.e. criteria. Part of what makes evaluation  fun is surfacing the criteria that one might use to evaluate an evaluand. What’s ‘good’ to you is not necessarily ‘good’ to me. That’s why there are so many kinds of cars out there. 

In a program evaluation, we typically think of a program as having merit if: 1) it does what it sets out to do, i.e. achieves its intended outcomes, and that 2) it makes a meaningful difference as a consequence to its operation.

Worth

Now, worth is a trickier concept. In everyday parlance, we might say that an apple (assuming that is ‘good’) is worth something; that ‘something’ is typically expressed in some monetary value (e.g. this apple is worth $2.00; that car is worth $24,999.) So, worth is the value of an evaluand that is expressed as an equivalence to something else. We may say… that this activity is worth ‘my time’. Whereas merit can be difficult to measure, worth is usually expressed in some more easily measurable unit.

Another way to think about worth is in a comparative situation. Let say you’re evaluating two instances of the same program: Program Breakfast-for-all at Site A and Site B. While they may both have merits, the worth of the program at Site A may be different from Site B depending on its impact on the constituents. Worth between two comparable, but different programs may also differ if one is cheaper to run (so one is worth more than the other).

Finally, significance.

Significance is the fuzziest of the three. Significance refers to the values and meanings that one ascribe to an evaluand. Typically, one can learn about the significance of something by asking questions about: What makes this evaluand special? What meaning does it hold for particular individuals?

Ask any young bride about her diamond ring. While it may not feature a big diamond (so, the ring is of limited worth), it probably holds great significance. A young college graduate may be driving a high-mileage car that is nearing the end of its service life. We might speculate that the car has limited merit (i.e. the transmission is wonky, the body is rusting, but the car is still roadworthy), and as a result is of limited worth to any body, but to the college graduate it may hold significance for his/her livelihood depends on it to get him to work everyday.

Notice that significance often have little to do with merit. Indeed, a program may be shown to have limited impact on a community, but it may hold great significance for its symbolic value. We may say that “it matters! Even if it is to a few.” As another example, a program may be shown to be inefficacious, but if it is the only program of its kind that serves a specific need for a vulnerable population, that’s significance to know, isn’t it?

So what?

Knowing m/w/s well enables us not only to unpack what others mean by ‘good’, but it also helps in raising questions around understanding quality, say, when designing an interview guide or constructing survey questions.

Question for you: Is this how you understand merit/worth/significance? Might you have other powerful ways of explaining m/w/s to others? Comment below.  Thanks for reading!

PS: For all you educators out there, is a grade an indication of merit, worth, or significance, or any/all of three?

About these ads

4 thoughts on “Merit/Worth/Significance Explained in Plain Language

  1. Chi, great post! I’d enjoy reading a future post with your perspectives on how developmental evaluation fits into all this (along with other evaluation activities, like needs assessments, evaluability assessments, etc. where the primary goal isn’t to determine M/W/S). (Yikes we work in a tricky field!)

  2. Great explanation! I use some of these examples myself too.

    Another way to think of it is that something can be very meritorious in the sense of being a high quality program of its type. But if it is not actually what was needed it has low value or worth to the organization, community, etc. Might be valuable in another context because it is high quality stuff, but just not here.

    Significance can have multiple meanings. A program could be a real breakthrough as a way of addressing an issue, i.e. a significant advancement or innovation.

    The more pedestrian sense of significance is around the importance of, say, outcomes or aspects of process.

    It’s relatively easy to say a program has these strengths and those weaknesses. But to make your evaluation truly actionable you need to say which strengths and which weaknesses are most important and, say, urgent. From the client’s perspective … ok, I have 57 weaknesses, but which should I address first and fast? For that you need to work out what’s most important, i.e. significant.

    Ann, even I’m developmental evaluation we are still trying to work out how good the design is based on what we know, which considerations are most important to info further development, how good the emerging outcomes are even at this early stage, whether the initiative is worth pursuing or just fundamentally flawed. And, whether it’s a breakthrough approach for addressing messy problem!

    Developmental evaluation is still real evaluation in the sense that it’s asking and answering evaluative questions. It’s just that those questions are geared to a developmental process and an emergent context, not a static or stable environment.

    Where is #omgMQP when we need him? :)

    Jane

    1. Hi Jane! Thank you for taking the time to drop by and comment on this. Your feedback is valuable given your deep understanding of Scriven’s works. Your illustrations have given me the idea to extent this post and talk about how we may think about m/w/s within programs.

      Jane and Ann — Jane’s discussion around m/w/s & DE resonated with me. Whether we do classical evaluation or developmental evaluation, we’re still asking evaluative questions and soliciting evaluative responses—one might even say we’re engaging in evaluative thinking within DE. Part of why Scriven’s work appeals to me so greatly is his notion of what it means to make evaluative claims. Indeed, the surfacing of evaluative claims admist uncertainty and turbulence (e.g. why one program option is more appropriate than another, or some emergent phenomenon is ‘good or ‘bad’) is perhaps one of the most critical contributions the developmental evaluation process can make to program development.

  3. Great post and wonderful topic Chi! I think Anne’s question is important with regard to developmental evaluation. I’m working on an evaluation of a newly designed, complex, dynamic program and taking a developmental approach, but I think the goal is still to determine m/w/s along the way – not as an endpoint, but rather ongoing determination of m/w/s as the program grows and changes. As I read the post, I was thinking of the way I heard Jane Davidson (in her Eval 13 workshop with Scriven) describe m/w/s as quality, value, and importance and I really like those terms. I’m thrilled to see Jane explain it herself here! :-)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s