Partnering with governments to co-design pilot interventions and embed them in local bureaucratic systems is increasingly seen as ‘best practice’ on grounds of scalability and sustainability. This paper reports on a pilot program that was co-designed with, and embedded within, the Elementary and Secondary Education (E&SE) Department in Pakistan's Khyber Pakhtunkhwa province that offers a cautionary tale. The over-arching desire to work within existing bureaucratic systems, while laudable, constrained the design of the randomized controlled trial. This paper presents findings on some of the institutional factors which resulted in failed implementation of the RCT and a lost opportunity to learn about the efficacy of key design features. The paper briefly outlines the design of the pilot—promotion-based incentives for educators—and summarises the largely null results. It then turns to implementation, discussing what went wrong, how this was uncovered, and lessons learned for co-designing and embedding future pilot studies with(in) government.