Step 1: Business Alignment & Project Objectives

Project objectives are developed based on needs. The needs assessment begins with a review of the opportunity to be realized if the proposed project is implemented.

• Is there an opportunity for a positive payoff?
• Is this a problem worth solving or an opportunity worth exploring?

Next, the business needs are identified.

• What specific business measures will be influenced with this project? The business needs are met by changing individual performance.
• What should the participants do or stop doing to change the business measure? Performance needs are met by implementing new knowledge/skills presented to the participants.
• What specific knowledge, skills, or information do participants need for the new performance?

Finally, preference needs focus on how the stakeholders, including the participants, should perceive the project in terms of value and need.

The project is developed to achieve the objectives at each level. The project can be evaluated at each level, based on the objectives. The V Model illustrates this process as it presents the relationship between needs assessment, objectives, and evaluation.

Developing the Alignment of your program, project or initiative to the business is critical for a success.

Evaluation begins with the objectives of the program, project or solution. The objectives must go beyond typical learning objectives and include up to five levels of data.

Step 2: Plan for Evaluation

Planning begins as soon as it is decided that an impact/ROI study should be conducted and typically involves key stakeholders. All important decisions for the study are made early through evaluation planning. This step involves completing three documents: data collection plan, ROI analysis plan, and evaluation project plan.

Step 3: Two types of data are collected during a project’s implementation (1) Reaction and (2) Learning

Methods to collect Reaction & Learning Data

• Feedback questionnaires
• Written tests/exercises
• Demonstrations
• Interviews
• Focus Groups

Step 4: Two types of data are collected after a project is implemented (3) Application and (4) Impact

Methods to collect Application and Impact Data

• Follow up surveys
• Follow up questionnaires
• Observation on the job
• Interviews
• Focus groups
• Action planning
• Performance contracting
• Performance records

Improving response rates is a critical issue for post-project collection. When used consistently, the techniques can achieve 70-80% response rate for questionnaires, surveys, or action plans.

Step 5: Isolate the effects of your program

 One of the most critical steps in the process is to isolate the effects of the project or program.

Techniques used to Isolate the Effects of Programs:

• Use of a control group arrangement
• Trend line analysis of performance data
• Use of forecasting methods of performance data
• Participant’s estimate of impact (percent)
• Supervisor’s estimate of impact (percent)
• Management’s estimate of impact (percent)
• Use of experts/previous studies
• Calculating/estimating the impact of other factors
• Use of customer input

While isolating the effects of the project with other influences is sometimes difficult, it is necessary for credibility of the study. Without this step, there is no proof that the project is connected to a business measure.

Step 6: Convert data to monetary values

To calculate the ROI, improvement in business measures must be converted to money.

This step develops a monetary benefit for one or more impact measures linked to the project. It usually follows the step to isolate the impact of the project.

Methods to convert data to money:

• Standard values
• Historical costs
• Expert input
• External Studies
• Linking with other measures
• Participant estimates
• Supervisor/manager estimates

Step 7: Intangibles

Intangible benefits are project benefits that we choose not to convert to money. They are measures that cannot be converted to money credibly with minimal resources. Intangible data should be collected for all projects.

Step 8: Tabulate the Costs

When impact studies are conducted, the total costs of the project are needed for the ROI calculation. The costs must be fully loaded, i.e., must include all direct and indirect costs.

Step 9: Calculate ROI

Return on Investment (ROI) is a financial metric, representing the ultimate measure of project success. ROI is calculated using the project benefits and costs.

Sample ROI Calculation

Benefits (isolated) = $400,000

Cost of Program (fully loaded) = $220,000

Calculation:

(Net Benefits ÷ Cost) x 100 = ROI

([$400,000 – $220,000] ÷ $220,000) x 100 = 82%

Step 10: Reporting the results to key stakeholders

Reporting the results of the study is an important final step in the ROI Methodology. Properly identifying the audience and providing appropriate information is essential.

By default, four audiences are always essential:

1. The participants directly involved in the project who provide data to the evaluators
2. The immediate managers of the participants who need evidence of the success of the project

3. The sponsors of the project who need to understand the project’s value to the organization

4. The staff team members who need to understand how the study was developed

A variety of media can be used to communicate the project’s success, for example:

• Face to Face Meetings
• Executive Summary
• Panel Discussions

What is included in an ROI Study: Impact Study Outline:

• Background information
• Objectives
• Evaluation Strategy
• Data Collection & Analysis:
  • Reaction
  • Learning
  • Application
  • Business impact
  • Program costs
  • Return on investment
  • Intangible measures
  • Conclusions & Recommendations