ID Process: Deconstructing Kirkpatrick

Models are great. There’s nothing wrong with models. Models are helpful when working in a team, so everyone knows what is expected, or in large organizations, where accountability is important, or when someone is just learning the ropes, so they have something to build upon.

But sometimes, often, a model is extrapolated or interpreted in a way that limits or even harms the project it’s meant to help.

James and Wendy Kirkpatrick, of the Famed Kirkpatrick 4 Levels of Evaluation, seemed to want to ‘right the ship,’ so to speak, in their virtual ATD session called “An Essential Training Evaluation Question That Changes Everything.”

In this session, they offered 5 clear actions for L&D professionals to take away, and not all of the pointers were explicitly about any particular level. I would say that it was simply good course design planning.

1. The “question that changes everything” is a question we should all be asking when approached about developing a course:

• “Is this purely a training solution?” or
• “Can you tell me more about this situation?”

Many times, we as developers are hung up on selling existing trainings or talking about what we do or have done, but we need to be more like physicians or counselors who are helping a patient to pin-point the problem and then guide the process of finding a solution to that problem.

• Ask open-ended questions.
• Learn what others are doing in the organization.
• Create value by being relevant.

Find out how you can help, but remember that you can only find out how you can help if everyone fully understands the nature of the issue.

2. Once you start asking that question, you’ll start getting to the end goal or to your purpose for training. The Kirkpatricks said that it’s not important who initiates the conversation, but that it’s to our benefit if we do. We can demonstrate our value to the organization by being the person who gets to the heart of any issue, and who wouldn’t want that?

3. As I gain more experience and insight into L&D, it’s becoming clear that SME lack of involvement is a frequent issue. I have had the question put to me a couple of times. Kirkpatrick, and myself, believes that ensuring SME involvement starts when the whole project starts. They emphasize that any approach to training has to be coordinated and that SMEs should know their involvement is compulsory. Compelling anyone to do anything requires accountability and consequence, so involvement also must come from above. If we can demonstrate our value by identifying an issue that can be mitigated with training and also document how SME involvement is necessary, then all the pressure points to make that involvement happen should be activated.

4. “Begin with the end in mind” is the mantra of UbD or Backward Design, so it was great to have Kirkpatrick draw a line between UbD and the necessity of gaining all 4 of the levels, even the ones L&D teams tend to ignore or put off: Behavior and Results. Kirkpatrick offered sound, routine advice: do a decent TNA (training needs analysis) and gain stakeholder buy-in early on. A well researched TNA will tell you exactly what needs to be happening in the end, and stakeholders will support you through the evaluation “ordeal” by letting you access more data. Maybe some L&D professionals shy away from Results because it’s “not my problem,” but remember — we want to constantly demonstrate our value to the organization. Their bottom-line is our bottom-line, too!

5. Kirkpatrick calls the plan to collect all 4 levels a “blended” plan, and he emphasized that this must be decided from the start. This is because we’re going to be using a battery of tools to get even a minimal amount of feedback from all 4 levels, so while we’re designing the program, we need to account for that data collection. Again, Kirkpatrick is cautioning us to not leave it to chance. Don’t assume that eventually maybe we can collect some 3rd and 4th level feedback somehow. Put it in the plan so the stakeholders can facilitate it.

What has kept you from gathering Level 3 Behavioral or 4 Results data?

How are you gaining buy-in from reluctant SMEs?

How have you created value for your department in the organization?