Pilots are a great way to try and test ideas, learn from implementation, and refine your approach before scaling up. However, first pilots are rarely about the results. They’re about learning about how what you’ve designed in the conference or Zoom room actually feels in real life: what works, what doesn’t, and what it takes to implement effectively. In this summary, we’ll explore how to structure pilots for meaningful learning, highlight key considerations, and share practical tips for getting the most out of your pilot program.
Key Takeaways
- Pilots are about learning, not perfection. Focus on implementation and what it teaches you. Set reasonable learning goals to evaluate what went well and what didn’t.
- Leverage existing data. Use data you already collect and supplement with pilot-specific data points. This keeps your energy focused on execution and set you up to draw meaningful comparisons.
- Learning happens at the speed of implementation. Be realistic about when you’ll have actionable insights and their quality. Early learnings will focus more on launching and initial implementation feedback rather than effectiveness.
Designing for Learning: Assumptions, Constraints, and Goals
Before you dive into your pilot, clarify why you’re piloting in the first place. What assumptions or constraints are influencing the design? What outcomes are you hoping to achieve? Documenting these factors upfront ensures everyone is aligned and sets a clear framework for evaluating progress.
Document Program Design Basics
- What are you implementing? Be specific about the program design, including intended processes and outcomes.
- What are you trying to learn? Identify the key questions your pilot should answer, whether they relate to execution, user experience, or performance.
Implementation vs. Design
No pilot goes exactly as planned. That’s okay! The goal is to understand what happened and why. Did what you implemented align with your design? If not, what drove the differences? Was it a logistical hiccup you can correct next time or a flawed assumption that needs to be reexamined? Structured feedback from staff and participants/customers, as well as observational data, will be critical for answering these questions.
Measuring Performance: Focus on What You Have
When evaluating your pilot, start with metrics you already measure in your day-to-day operations. This minimizes extra work while laying a strong foundation for comparing pilot performance to existing practices. Supplement this with pilot-specific metrics only when necessary.
- Mind the gap. Expect the pilot to perform below standard targets in its early iterations, and focus on understanding why. Was it an implementation challenge? A design issue? Or is this a signal that this pilot may not actually solve the business challenge as well as we hoped?
- Remember, pilots are about learning. Reinforce this mindset with all stakeholders, and avoid setting arbitrary performance targets. Instead, use the pilot to dig into why certain gaps exist and how you can address them moving forward.