When we implement change, we want verification of improvement to ensure the problem was solved at the root.
You may hear potential solutions called "counter-measures" until the project team has verified that the problem has been fully resolved. As your team makes improvements, validate how the process changed using "before" metrics vs. "after" metrics.
If the metrics remain constant or decline, you need to re-evaluate your potential solutions. Analyze your process and data to determine why the process did not improve. In the spirit of continuous improvement, you may need to try multiple options before you find the best way to solve the problem.
If your metrics improve, you will want to continually monitor them to ensure the process does not revert back.
A pilot test is testing your potential solution on a small scale before full implementation. A pilot test is an excellent risk mitigation strategy.
With a pilot test, you can:
- Evaluate actual implementation costs
- Measure impact and estimate total impact of the project
- Identify unintentional consequences
- Collect data to better analyze the problem
- Improve upon the potential solution before full implementation
- Gain valuable feedback from your impacted stakeholders.
Depending on your process, your pilot may consist of one person, one department, one location or just one part of the process.
Conducting a Pilot Test
Step 1: Planning the Pilot
Scope the pilot
Determine which parts of the process will be involved in the pilot. If you have a major, large-scale project, you may want to pilot specific processes or sub-processes.
If you have a smaller process, you may be able to pilot the entire new process or project.
Determine pilot test group
Identify the group or department that will pilot the process. You can conduct a pilot by certain location, a specified department or group of employees.
You want to ensure your test group is open-minded and flexible. They need to understand that this is a pilot, so perfect is not anticipated. They need to be ready and willing to document the challenges they encounter and provide you with feedback.
Next, determine when and for how long the pilot test will be run. Meet with your test group and find out if they have other high priority tasks or deadlines coming soon. You want to resolve any conflicts before you begin your pilot. Give yourself enough time to reasonably capture data and tester feedback.
Step 2: Design the Pilot
Structure of the Pilot
Determine how the pilot will be conducted. The pilot may be conducted:
- At the worksite
- In a training or classroom setting
- At an employee's desk or office
Where possible, conduct the pilot where the work is going to happen. However, if you need quick, real-time feedback, you may want to conduct your pilot in a training room or classroom. This allows you to gather multiple testers and have them work through the new process together.
Complete all process documentation and review it with your project team. This may include process maps, checklists, job aids, RACI(D) to define roles and responsibilities, testing scripts, etc.
This is the documentation that you will provide to your testers so make it it is near final.
Establish a method to measure your Impacted Metric. The pilot test will give you an opporunity to see total impact that you can make with this improvement. Assign a team member to be responsible for gathering and tracking data to support the metrics.
If you need to validate implementation costs, ensure you have a tracking mechanism for this too!
Design a method to collect feedback from your users. Give them a simple survey or spreadsheet to quickly and easily track challenges or obstacles they encounter.
Assign a team member to follow up with each of the testers to gather their personal feedback. As the tester progresses through the process documentation, they may not thoroughly capture all of their challenges, so a personal follow-up is a great way to collect even more ideas for improvement.
Step 3: Conduct the Pilot
Provide training and documentation for your testers.
Clearly set expectations:
- Purpose of the pilot test
- Deadlines for testing
- How to document and submit feedback
- Who to contact for help
Make your project team members readily available during the test. If the pilot is being conducted at a worksite or at an employee's desk, go walk the processes. Observe the tester and document when the tester encounters issues.
Sep 4: Verify results
Re-measure Impacted Metric
Did your test have the desired result? Was the magnitude of the improvement as expected? Did you experience any unintended consequences?
Evaluate and integrate tester feedback and survey scores. If possible, incorporate their ideas into the process and process documentation.
Finally, meet with your sponsor to determine readiness to implement the new process on its full scale.
If you're not quite ready, create an action plan to get you there. This may involved conducting another pilot. It's okay if this happens. Keep in mind - it's better to fail in a pilot than with full implementation.
If you're ready, congratulations on a job well done!!