Measuring the Five Levels with a Case Study
ROI Methodology® is a credible process designed to improve the monetary value of your projects and programs.
With it, you can easily improve the initial planning and execution of your programs and initiatives.
There are Five Levels of Measuring Program Outcome Data
LEVEL 0: Input
This level helps you to create the initial planning for your program. It helps in determining the participants, costs, efficiencies of the programs.
LEVEL 1: Reaction & Planned Action
This level measures participant satisfaction with the program and helps you to capture or plan your actions.
LEVEL 2: Learning
Every program or initiative has a learning component. The learning measurement takes place during the program and helps in measuring changes in knowledge and skills.
LEVEL 3: Application & Implementation
Measuring application and implementation provide evidence about what exactly the participants and employees have learned from the program, and how well they are implementing it in their workplace.
LEVEL 4: Business Impact
This Level helps you to connect your program into important business impact measures. To achieve the organizational goals this level helps in improving the business productivity, quality coast and more.
LEVEL 5: Return on Investment (ROI)
The ultimate measure for the success of a program or an initiative. This level compares the monetary benefits to the costs.
In the final level, you can calculate your profits into a monetary value with the cost-benefit analysis formula. The profit is the net value created by the event minus the program costs.
The standard formula to calculate ROI as a percentage is:
ROI = (Monetary benefits – Program costs)/Program Costs x 100.
Sexual Harassment Prevention – Case Study
Most organizations have sexual harassment prevention programs, but few are subjected to accountability up to and including a return-on-investment (ROI) analysis.
In the setting, in this case, a large healthcare chain conductsed a sexual harassment prevention workshop involving first-level managers and supervisors. Workshops were followed by meetings with all employees, conducted by the same managers and supervisors.
In all, seventeen workshops were presented, and the monetary impact was developed. Several unique issues are involved in this case, including the techniques to isolate the effects of training and convert data to monetary values.
The analysis used a traditional ROI model and yielded significant and impressive results that surprised the evaluation team and senior managers.
The Program: Design, Development, and Implementation
Armed with input from ten interviews and with detailed input from the EEO/AA staff, the major causes of the problem were identified. There was an apparent lack of understanding of
- the company’s sexual harassment policy and
- what constituted inappropriate and illegal behaviour.
Also, there was an apparent insensitivity to the issue. As a result, a one-day sexual harassment prevention workshop was designed for all first- and second-level supervisors and managers.
LEVEL 0: Input
The program had the following objectives. After attending this program, participants should be able to:
- Understand and administer the company’s policy on sexual harassment
- Identify inappropriate and illegal behaviour related to sexual harassment
- Investigate and discuss sexual harassment issues
- Conduct a meeting with all direct reports to discuss policy and expected behaviour
- Ensure that the workplace is free from sexual harassment
- Reduce the number of sexual harassment complaints
LEVEL 1: Reaction & Planned Action
A pre/post-test was administered to measure knowledge of HI’s sexual harassment policy and inappropriate and illegal behaviour. The 20-item questionnaire was evenly split on policy and behaviour issues.
LEVEL 2: Learning
To measure the success of the program application, two data collection methods were used.
First, a meeting record was required of each supervisor and manager to document the actual meeting with employees, recording the time, duration, topics, and participants. Although this form did not address the quality of the meeting, it provided evidence that the meeting was conducted.
Second, a data collection method was implemented which is a survey of the non-supervisory employees, the typical target group for harassment activity.
Although the entire team could have been surveyed, it was felt that it was more important to examine behaviour change from the perspective of those who were more likely to be victims of harassment.
The survey was planned for administration six months after the program was completed.
LEVEL 3: Application & Implementation
To measure the success of the program application, three data collection methods were used.
First, a meeting record was required of each supervisor and manager to document the actual meeting with employees, recording the time, duration, topics, and participants.
Although this form did not address the quality of the meeting, it provided evidence that the meeting was conducted.
The second data collection method was a survey of the non-supervisory employees, the typical target group for harassment activity. Although the entire team could have been surveyed, it was felt that it was more important to examine behaviour change from the perspective of those who were more likely to be victims of harassment.
The survey was planned for administration six months after the program was completed. It provided post-program data only and therefore each questionnaire had to be worded to capture change since the training was conducted.
The 15-item survey examined specific behaviour changes and environmental changes related to harassment activity, including actions that might be considered inappropriate or offensive.
The third data collection method was a self-assessment questionnaire completed by supervisors and managers. This questionnaire captured actions, behaviour change, and results linked to the program.
Although there were a variety of other data collection possibilities, including focus groups, interviews, and third party observation, it was felt that, given the time and cost considerations, these three methods provided sufficient data to capture behaviour change and show that the program had succeeded.
LEVEL 4: Business Impact
Business results measures included several items. Initially, it was planned that internal complaints, lodged formally with the HR division, would be monitored along with external charges filed with various agencies (primarily the EEOC).
Because of the lag time between changes in behaviour and a reduction in complaints, data would be collected for one year after the program and compared with data from one year before the program to determine specific improvements.
Also, as alternative information, litigated complaints would be tracked along with the direct costs, including legal fees, settlements, and losses. Also, because of the perceived link between a hostile work environment and turnover, annual employee turnover would be examined for the same period.
LEVEL 5: ROI Analysis
To develop the ROI calculation, several other issues needed to be addressed to plan for the Level 5 evaluation. The specific method(s) to isolate the effects of the program from other influences would have to be selected and implemented.
The various ways in which data are converted to monetary values would need to be pinpointed. Each specific program cost element would have to be identified along with other issues that might influence the reduction in complaints and litigation expenses.
Finally, intangible benefits expected from the program needed to be itemized and the communication targets established.
