Cloudflare's Code Orange: Fail Small Initiative

After two recent network failures, Cloudflare has launched a comprehensive resilience plan called "Code Orange: Fail Small." This initiative aims to prevent major outages by making their network more resilient to errors or mistakes. The plan is organized into three main areas:

1. Controlled Rollouts: Cloudflare will implement controlled rollouts for any configuration change that is propagated to the network, similar to their software binary release process. This will ensure that new changes are thoroughly tested and monitored before being deployed.
2. Failure Mode Review and Testing: Cloudflare will review, improve, and test failure modes of all systems handling network traffic to ensure they exhibit well-defined behavior under all conditions, including unexpected error states.
3. Internal Process Improvements: Cloudflare will change their internal "break glass" procedures and remove any circular dependencies to enable faster access to all systems during an incident.

Key Takeaways for Developers

1. Cloudflare's recent network failures were caused by instantaneous deployment of configuration changes, which exposed a gap in their deployment process.
2. Cloudflare will implement controlled rollouts for configuration changes, similar to their software binary release process.
3. The Code Orange initiative will deliver iterative improvements, rather than a single "big bang" change, to ensure continuous resiliency.

Practical Implications

Developers using Cloudflare services can expect improved network resilience and reduced likelihood of major outages. By implementing controlled rollouts and failure mode review and testing, Cloudflare aims to deliver a more stable and reliable experience for their customers.