Category: 5×52

Categories
5x52 Program Evaluation

Merit/Worth/Significance Explained in Plain Language

I recently received an e-mail from a fellow doctoral student asking me to explain Scriven’s notion of merit/worth/significance. One part of her dissertation is around determining the value of test preparation training (e.g. MCAT/GMAT/LSAT prep courses) among language learners. One of her committee members suggested that she use M/W/S as a framework for tackling this aspect of her work. So, I wrote back to her, saying, why don’t we Skype and talk about this.

I’ve been thinking about this problem since. As an evaluator, I am reminded that one of the basic purposes in evaluation is the determination of merit/worth/significance of something. And, we typically refer to whatever we are evaluating (the ‘something’) as the evaluand. This classical definition of evaluation constitutes a part of what Scriven (1991) refers to as the logic of evaluation in a paper by the same name in the Evaluation Thesaurus. The logic of evaluation is a seminal contribution to the field as it gets at the core of what makes evaluation unique as compared to, say, research–evaluation allows us to make evaluative claims. The distinction between M/W/S and its application in evaluation is an important one, but finding accessible writing on this topic is difficult. Perhaps, m/w/s is so obvious to everyone else but me :). Hopefully not.

So… what’s merit, worth, and significance?

Merit, worth, and significance can be easily explained by reference to evaluating anapple. Say you’re at a grocery store. The decision you’ll have to make is to buy an apple. 

Merit

Merit has to do with the intrinsic properties, characteristics, or attributes of an evaluand. When buying an apple, most people would prefer an apple that is not rotten, is sweet to taste, and is not otherwise damaged or deformed. That’s typically what people would look for if the apple were to be eaten on its own. But, what if you were buying the apple to make an apple pie? Then, you may wish to buy an apple that is not sweet but  tart. So, as we can see, what we value to be desirable attributes of an object depends on other contextual factors. 

Here is another example. A car has merit if it is reliable (i.e. does not break down while you’re driving down the highway; predictable), is safe (i.e. has adequate safety features and operates as intended), and is powerful relative to its intended application (i.e. say, a commuter car vs a pick-up truck to haul construction material). Now, you may say, a car has merit only if it has an integrated air conditioning unit or a stereo system. A design-conscious person may insist that a car be visually appealing. Increasingly, drivers want good fuel consumption. Different people may hold different views of what constitutes merit. In other words, an evaluand may be evaluated against different dimensions of quality, i.e. criteria. Part of what makes evaluation  fun is surfacing the criteria that one might use to evaluate an evaluand. What’s ‘good’ to you is not necessarily ‘good’ to me. That’s why there are so many kinds of cars out there. 

In a program evaluation, we typically think of a program as having merit if: 1) it does what it sets out to do, i.e. achieves its intended outcomes, and that 2) it makes a meaningful difference as a consequence to its operation.

Worth

Now, worth is a trickier concept. In everyday parlance, we might say that an apple (assuming that is ‘good’) is worth something; that ‘something’ is typically expressed in some monetary value (e.g. this apple is worth $2.00; that car is worth $24,999.) So, worth is the value of an evaluand that is expressed as an equivalence to something else. We may say… that this activity is worth ‘my time’. Whereas merit can be difficult to measure, worth is usually expressed in some more easily measurable unit.

Another way to think about worth is in a comparative situation. Let say you’re evaluating two instances of the same program: Program Breakfast-for-all at Site A and Site B. While they may both have merits, the worth of the program at Site A may be different from Site B depending on its impact on the constituents. Worth between two comparable, but different programs may also differ if one is cheaper to run (so one is worth more than the other).

Finally, significance.

Significance is the fuzziest of the three. Significance refers to the values and meanings that one ascribe to an evaluand. Typically, one can learn about the significance of something by asking questions about: What makes this evaluand special? What meaning does it hold for particular individuals?

Ask any young bride about her diamond ring. While it may not feature a big diamond (so, the ring is of limited worth), it probably holds great significance. A young college graduate may be driving a high-mileage car that is nearing the end of its service life. We might speculate that the car has limited merit (i.e. the transmission is wonky, the body is rusting, but the car is still roadworthy), and as a result is of limited worth to any body, but to the college graduate it may hold significance for his/her livelihood depends on it to get him to work everyday.

Notice that significance often have little to do with merit. Indeed, a program may be shown to have limited impact on a community, but it may hold great significance for its symbolic value. We may say that “it matters! Even if it is to a few.” As another example, a program may be shown to be inefficacious, but if it is the only program of its kind that serves a specific need for a vulnerable population, that’s significance to know, isn’t it?

So what?

Knowing m/w/s well enables us not only to unpack what others mean by ‘good’, but it also helps in raising questions around understanding quality, say, when designing an interview guide or constructing survey questions.

Question for you: Is this how you understand merit/worth/significance? Might you have other powerful ways of explaining m/w/s to others? Comment below.  Thanks for reading!

PS: For all you educators out there, is a grade an indication of merit, worth, or significance, or any/all of three?

Advertisement
Categories
5x52 Lifehacking Productivity

Spotlight on Productivity: 5 Productivity Tricks for Researchers/Evaluators/Graduate Students

This is the sixth and final post  of the Spotlight on Productivity series, in which I examine productivity challenges associated with academic/knowledge work and take stock of current thinking and tools to help us get things done.

5. Mise en place

Everything Ready
(via Flickr, wickenden, http://www.flickr.com/photos/wickenden/3629186048/)

Mise en place is French for ‘put in place’. It describes a practice by chefs preparing all the necessary ingredients in advance of service. All ingredients are prepared for use, organize, and within reach. Taken to the context of productivity, it means  gaining as much clarity around the nature of the problem you’re solving, the tasks that need to be performed, and having the necessary pieces to execute a task. Execution is not the time to fumble around with getting things ready. Because knowledge work is often emergent,  take  preparation as far as you can.

4. Workflow

Adobe Photoshop Lightroom presents a workflow-based solution to photographers. (via Flickr, devar, http://www.flickr.com/photos/59874422@N00/253450773)

Professional photographers rely on a well-rehearsed workflow to maximize  productivity. (After all, any time not spent behind a camera is time wasted not making money.) A workflow refers to the general sequence of tasks that need to be performed for any projects. Associated with each step of a workflow are inputs, processing, and outputs.

For research projects, chances are you need to: 1) define the scope and context of a study, 2) design the study, 3) apply for ethics clearance, 4) collect data, 5) analyze data, 6) interpret data, 7) write-up the data, and 8) disseminate the findings. That constitutes a generalized workflow for researching/evaluating. Practicing and adhering to a workflow means less thinking and planning. The GTD workflow I wrote about here is another example.

3.  Define your top 3 tasks to complete for each day.

583-the-emergent-task-planner-01

Identify and limit your day to completing only 3 tasks. Do them when your are mentally charged and refreshed (i.e. soon after you wake up).

2. Pomodoro

Italiano: Autore: Francesco Cirillo rilasciata...
(Photo credit: Wikipedia)

Pomodoro is a timing technique for maximizing productivity.  Pomodoro is Italian for tomato and the technique makes reference to those manual kitchen 30-minute timers. ///CHECK To use the pomodoro technique, simply work in bursts of 25 minutes, followed by a 5-minute break. Each 30-minute burst consistute a pomodoro.  During each pomodoro, avoid any distraction and work ONLY on your task. Pomodoro aficionados would tell you to do 4 pomodoros, totalling 2 hours, and take a longer break.

1. Apply OHIO — only handle it once — to your e-mails.

For each piece of correspondence, only handle it once. Act on it immediately. Then file it, or delete it. Apply David’s GTD workflow.  (via FastCompany, http://www.fastcompany.com/3004136/11-productivity-hacks-super-productive-people#2)
There you have it. I hope you found this series helpful in enhancing your productivity!
Categories
5x52 Graduate Student Development Program Evaluation

Jennifer Ann Morrow on 12 Steps on cleaning and prepping dataset

Jennifer Ann Morrow, faculty member in Evaluation, Statistics, and Measurement at the University of Tennessee, recently blogged about data cleaning and data set preparation at AEA365. She describes 12 steps in her post here, and excerpted below. This is a skill that all quantitative (and qualitative!) researchers should know how to do.

She’ll be running a Professional Development workshop  on the same topic at the upcoming Evaluation 2013 conference in Washington, DC.

1. Create a data codebook
a. Datafile names, variable names and labels, value labels, citations for instrument sources, and a project diary
2. Create a data analysis plan
a. General instructions, list of datasets, evaluation questions, variables used, and specific analyses and visuals for each evaluation question
3. Perform initial frequencies – Round 1
a. Conduct frequency analyses on every variable
4. Check for coding mistakes
a. Use the frequencies from Step 3 to compare all values with what is in your codebook. Double check to make sure you have specified missing values
5. Modify and create variables
a. Reverse code (e.g., from 1 to 5 to 5 to 1) any variables that need it, recode any variable values to match your codebook, and create any new variables (e.g., total score) that you will use in future analyses
6. Frequencies and descriptives – Round 2
a. Rerun frequencies on every variable and conduct descriptives (e.g., mean, standard deviation, skewness, kurtosis) on every continuous variable
7. Search for outliers
a. Define what an outlying score is and then decide whether or not to delete, transform, or modify outliers
8. Assess for normality
a. Check to ensure that your values for skewness and kurtosis are not too high and then decide on whether or not to transform your variable, use a non-parametric equivalent, or modify your alpha level for your analysis
9. Dealing with missing data
a. Check for patterns of missing data and then decide if you are going to delete cases/variables or estimate missing data
10. Examine cell sample size
a. Check for equal sample sizes in your grouping variables
11. Frequencies and descriptives – The finale
a. Run your final versions of frequencies and descriptives
12. Assumption testing
a. Conduct the appropriate assumption analyses based on the specific inferential statistics that you will be conducting.

 

Categories
5x52 Graduate Student Development Lifehacking Productivity

Spotlight on Productivity: 5 Steps on Managing your Projects Kanban Style

This post is number three in the Spotlight on Productivity series, in which I examine productivity challenges associated with academic/knowledge work and take stock of current thinking and tools to help us get things done.

In the first post of this series, I characterized academic / knowledge work as having to juggle multiple project. In the second, I argued that the cognitive toll on managing productivity can be alleviated through relegating planning to a productivity system. This is where I introduced David Allen’s Getting Things Done (GTD) system. GTD is useful to shortcut decision-making on tasks. Today, we look at how to manage decision-making with projects.

(By decision-making with projects, I don’t mean project management. Project management has more to do with a set of technical skills involved in managing within projects. My focus here is simply how to juggle between multiple projects.)

Kanban Style

Kanban is a process originating from Toyota manufacturing that now sees adaptation and application to software development and personal productivity. Kanban, Japanese for “billboard” or “signboard”, is a visual method for managing logistics. The way Toyota uses it goes like this: Suppose you’re assembling dashboards for cars. To assemble this dashboard, you need various components that are already pre-assembled upstream: odometer, RPM meters, clocks, car audio panels, navigation systems, etc. As each dashboard gets assembled, components get used up. This drop in component level triggers a signal to upstream manufacturers to manufacture more components just in time for use in the next round of manufacturing. How this differs from typical manufacturing processes is avoiding having large batches of inventory sitting around. By keeping track of the rate at which components are used up, the quality of the components, you benefit from improved product quality and increased productivity through working in small batches, hence “just in time”.

Applying Kanban to Personal Productivity

What we can take away from Kanban are two principles. The first principle is the importance of visualizing productivity. By visualizing productivity, you have a more powerful way of understanding the complexity and demands of your work. Here are five steps to jumpstart the process.

1. Make a list of all the project you are involved in.

2. Make a list of all the publications you are currently working on. Identify the status of each of these writing projects (e.g. conceptualization, initial drafting, waiting for review, editing, copy-editing, submission to journal, revision, etc… )

3. Identify which of the projects/publications are inactive at the moment. (In GTD language, this is your Someday-Maybe/Waiting list.)

4. Identify tasks within each project.

5. (optional) Flag tasks with impending deadlines.

Mapping out your projects this way give you a high-level view of that various projects you’re involved in, and the complexity of the tasks involved.

Here is the second principle. According to Personal Kanban, a site dedicated to applying kanban to personal productivity, it is also important to limit ‘work in progress’. Their reasoning is simple. You can only do so much in a day.

Here is my own Kanban Board that I have been using for some time to help visualize my projects. Here’s a link to a PDF that you can print and use on your own.

personal kanban

In the next post of this series, we’ll complete this personal kanban process by identifying tasks for work-in-progress. We’ll do this through looking at David Seah‘s incredibly useful tool, Emergent Task Planner. Stay tuned. 🙂

Categories
5x52 Lifehacking Productivity

Spotlight on Productivity: Getting things done with David Allen’s Getting Things Done (GTD) system

Before diving into the specifics of productivity challenges, we should start with the concept of ” Getting Things Done“.

For me getting things done is a mindset to approaching productivity. By relegating the decision-making associated with each piece of task to the GTD system, we can move through our work more efficiently.

David Allen is the celebrated productivity guru whose book on productivity–Getting Things Done– revolutionizes how we think and about productivity.

There are LOTS of primer and discussions on GTD… so I’ll only highlight a few of the key principles.

GTD boils down to this according to Gina Trapani on Lifehacker: Make three lists. Revise them daily and weekly. (She’s referring to a to-do list, project list, and a someday-maybe list).

GTD is premised on the idea that the brain is best for high-level cognitive activities, as far as productivity is concerned. Brain resources are not meant for and should not be wasted on low-level tasks. But what do we typically do?  We clutter our mind with having to remember what needs to be done. By freeing up cognitive resources through prioritizing, we can work faster, better, and with less effort.

GTD has several key concepts and key activities to it.

1)  For every task that enters your work-queue (GTD calls it an inbox), decide on what to do about it. Is it actionable? Is it Best tip: Act immediately on whatever task that can be completed in 2 minutes. Here’s a flow-chart cheat-sheet.

gtd-workflow

2) Scour your office/home and collect all that needs to be processed and acted upon. Loose pieces of paper. Receipts. Papers to be filed.  Make a big list of all the things you have to do. These tasks now form your work queue.

3) Also, empty and unclutter your mind of all the things you want to accomplish in the next while. That project idea you have lingering at the back of your mind. That paper you want to write “when you have time”.

4) Finally, tackle your work queue systematically. Realize that some items receive immediate attentions, while others don’t. Create a “Someday/Maybe” for projects and tasks that you don’t need to attend to immediately.

If this post on Getting Things Done piques your interests, the GTD Cheatsheet series at LifeDev.net gives as an excellent overview to the system.

These four steps form the basis of the GTD system. While some adhere strictly to the system, you could also see it more as a guiding framework. I find it more helpful to adapt it to the nature of knowledge work —- which is what I’ll be discussing in the upcoming posts.

Thanks for reading!

 

What does your personal GTD system look like? Do you have experience with using Allen’s GTD system?  Let me know.

 

 

Categories
5x52 Lifehacking Productivity

Spotlight on Productivity: A discussion on productivity challenges among grad students/researchers/evaluators/academics

Graduate students, researchers, evaluators, and academics (“knowledge workers“) encounter productivity challenges unlike other fields. Productivity strategies that work for most often do not transfer well for us. One primary reason is because knowledge workers are expected to contribute original knowledge. Lots of it. Boosting productivity means learning how to do more of it, in the least amount of time possible. By gaining greater clarity  into the nature of knowledge work and principles of productivity, and insights into one’s productivity habits, you can expect to produce more at a high quality, reclaim lost time, while keeping  stress down.

Knowledge work requires one to digest a high volume of information, and increasingly, requires close collaboration. Complicating matter is that knowledge work is time-bound: a few months for a project, 2 years  for a masters, 4 years for a Phd; 5 years until tenure review for asst. profs…  The pressure to produce is real.

At the most basic level, engaging in knowledge work requires one to do three things:  Reading + Thinking + Writing.

Productivity matters because performance is hinged upon successes in executing tasks (both in quantity and quality). Grad students are increasingly expected to publish in today’s competitive climate. Tenure decisions are made based on producing a substantial body of works. And of course, for starving grad students,  pay is dependent on productivity

Doing more in less time, while maintaining a high quality is not enough. Knowledge work also requires a high degree of creativity and integrative thinking. Being able to free the mind to think in novel ways is as important as being able to play to the rules.

Having finished four years of graduate studies and working in the roles of researcher/evaluator, what have I learned about the nature and demands of productivity?

1. Project work

I’ve learned that much of knowledge work is project-based. Projects have definite start-dates, end-dates, and key deliverables. Some projects are low-stakes (e.g. class assignments), some are medium stakes (RA work), and some are high stakes (e.g. scholarship/grants). It’s important to be able to deliver quality work in all situations.

To be high functioning, we must be able to juggle between multiple projects at once.  This requires high-level planning to keep track of project statuses. This means being able to switch between projects. This means recognizing the rhythm to when things tend to get busy in a project, and when things tend to slow down. This means recognizing project bottle-necks.

2. Substantial time horizon

Many of our projects span some significant time. And projects are often put in waiting patterns, like when planes are queuing up to land. Developing a tolerance for waiting is key. Know when to follow-up on projects to give it that little nudge to move things along.

3. Thinking is hard work. 

Thinking is a crucial, and obligatory part of our work, but rarely do we give it enough attention.   It occurred to me some time ago that there are different kinds of task and each require a different level of engagement.

  • Repetitive tasks  require little cognitive demand, but demands high accuracy. Tasks like data entry, processing e-mails, searching for literature are of this type.
  • Intellectual tasks require some creativity. They  usually require some integration or synthesis of information. Most acts of writing are of this type. Qualitative coding is also another example.
  • Finally, creative tasks are those that require a high level of creativity, high level of integrative thinking, and high engagement on task. Theming qualitative data, planning writing pieces, and synthesizing literature are examples of creative tasks.

Now, why do we care about these distinctions?  We care because creative tasks are those that  matter most in knowledge work, but that’s those are the tasks that are most draining. It’s really hard to sustain creative bouts. We  have about two hours of golden productivity each day—- allocate these time towards creative tasks!

4. Balancing obligations

Complicating productivity are those obligations that get in the way. Groceries, car troubles, emails, administrative paperwork, etc… Balancing obligations requires one to be mindful about what really counts and what can wait.  There’s increasing pushback against checking e-mails first thing in the morning. Create time intentionally to allow you to do tasks that really matter. For grad students and academics, that usually means one thing: publications.

5. Tracking success

Finally, I’ve found that tracking productivity successes to be one of the most important, but least obvious ways of boosting productivity. Tracking successes require us to clarify what success looks like. When we being to track success, we can begin to assess how we actually spent time, how much work got done, and how better to optimize our work. At the simplest level,  checking off  to-do lists is one way to track success. But what about higher-level success tracking? Productivity tools are really good at planning, but not at evaluating successes. One of the tools that I’ll be introducing tackles this problem.

Now that I have laid the foundation to how I understand productivity, in the posts that follow, I’ll look at how we can tackle each of these challenges. I’ll be introducing some tools and concepts that I have found to have transformed how I work. 

How are you finding this article? What productivity challenges are you experiencing? Does any of this resonate with you? Share below. I would love to hear from you.