According to the latest Training Industry Report, US companies have spent over $82.5 billion on training in 2020. Throughout the pandemic, companies have continued to invest heavily in the learning and development of employees. With this investment comes the expectation of measuring its impact, which is why training evaluation is so important.
Every year, there has been a steady increase in spending on training to improve employees’ skills. The ability to evaluate the effectiveness of this training had been proven equally important. Due to the COVID-19 pandemic, there is also an increase in remote working, resulting in a substantial amount of remote training to replace in-person learning strategies. Subsequently, the need to evaluate the effectiveness of training has never been greater.
How can you make sure that you’re evaluating training the best way possible?
Training evaluation is a systematic process to analyze the effects of a training program and to assess its effectiveness. It is also used to gain feedback on how to improve a training program. Furthermore, it measures whether a training program meets the objectives as to why it was created.
Here’s an example of how you might use a training evaluation at your workplace. Consider you’re a software company and released version 2.0 of your product. In order to ensure that all engineers and IT employees understand the new product, you deploy a training program. A training evaluation would look at this program holistically, and evaluate it based on the business needs, performance, learning needs, impact of training, and finally, the return on investment.
The outcome of an evaluation of training should give management all the information they need as to what worked, what did not work, and improvements for the future. It should also provide a clear indication if the money spent on the training program produced the desired results.
In other words, training evaluation is key to a successful training development process. It’s also a fundamental component of instructional design models, including the ADDIE model.
A good and practical method of evaluating training is to follow the Phillips V-model, also known as Phillips ROI model. The model, which was developed by Jack Phillips, broadly follows the Kirkpatrick model. Therefore, it is important that you understand the Kirkpatrick model, which you can read more about here: The Kirkpatrick model.
In short, the Phillips model has five levels, compared to Kirkpatricks’ four levels of evaluation. Phillips added a fifth level, which is return on investment (ROI) – a metric that is incredibly valuable for HR leaders to demonstrate to decision-makers the importance of training. The ROI Institute (Canada) provides a good graphic to illustrate this model:
The Phillips V-model follows the below sequence of five levels of training evaluation:
Level 1: Reaction
The model begins with evaluating participants’ experience of the training program they’ve gone through. The most common technique is to use a post-training survey. However, as employees tend to feel ‘over surveyed’, other techniques include pulse surveys (pop-ups), AI technology to understand emotional reactions, suggestion box and review sites. This is the step to gather data for evaluation.
Level 2: Learning
This is to assess if learning actually took place. To put it another way, it’s a great way to measure if the training program’s learning objectives have been met and if there was attendance and attentiveness. A way of doing this is to do a pre and post-test. For example, if the training objective was to teach a group of coders a new coding language, Python, for instance, it would be good to test the learners before they start their python skills before and after the program with an assessment.
Level 3: Application and Implementation
At this stage, the Phillips V-model tries to distinguish if the problem is with the application of learning or if it is in implementation. Let’s, for example, say that the information collected in level 2 confirmed that the training was not successful; then level 3 would be able to determine what the issue is as to why there is no application or implementation. For instance, if the employees understood Python, but were unable to apply it because of software requirements, then the issue to be resolved is for the organization to invest in better software.
Level 4: Impact
At the fourth level, the Phillips V-model looks at the total impact the training has had on the organization. It not only attributes impact to the training effectiveness but also considers organizational factors that might hinder the successful implementation of the skills gained during training. It takes into consideration external factors as well, and whether that has any impact on the performance of training.
As an example, let’s say you ran a training program for sales agents on the skill of door-to-door sales training. Due to COVID-19, your sales agents are unable to approach customers at their doorstep. Therefore, it can be surmised that applying the new skills gained during the training is hindered due to not being able to use them.
Level 5: Return on investment (ROI)
This is where the Phillips V-model distinguishes itself from the Kirkpatrick model. It measures the return on investment of the training. Although there is no direct way to attribute training to business performance, the Phillips V-model uses specific metrics and measures to give as close a viewpoint as possible.
For example, using cost-benefit analysis for a training program can determine if the money invested in training has had any impact. Are there any measurable results? In other words, if the training cost $50,000 to train sales agents to sell an organization’s products online, and the net profit from this is $500,000, then that would be considered a good investment. The benefits of training are clear in this case.
Of course, it would be important to take other considerations in place specific to your organization and program.
Now that you understand the theory behind the Phillips-V model, let’s take a look at how to implement this in practice.
Generally, not all of your training efforts will be evaluated at all five levels of the Phillips V-model. That would be a waste of resources. In fact, only about 5% of training programs are evaluated up to level 5.
As an example, you would evaluate a two-year executive leadership training program at all five levels. This is because it is high-impact, high complexity, and of high value.
However, you may not evaluate, for example, a training program that only focuses on three employees in developing a new way of replying to customer emails.
Phillips recommends evaluating all learning projects on the first level and then assess based on need and importance which levels the training program needs to be evaluated. For example, you may determine that you need to evaluate learning up until level 3. As a reminder of the evaluation targets of the Phillips V-model, look at the diagram below:
Again, your determination of which level to measure should be based on budget and business need. The levels of evaluation happen at different times, as demonstrated in the following table:
| Reaction (level 1) | Learning (level 2) | Application (level 3) | Impact (level 4) | Return on investment (level 5) | |
| Moment of evaluation | During or directly after the learning program | During or directly after the learning program | 2-6 months after the learning program | 6-12 months after the learning program | 12-24 months after the learning program |
As an example, let’s say you were to measure a training program up until level 4. The examples of tangible and intangible outcomes of learning would look something like this (Phillips 2008):
– Revenue, profit, market share
– Production or uptime
– Number of sent invoices
– Number of customer complaints
