Cloudflare Reduces Core Unit Boot Time from Hours to Minutes

Cloudflare's core units, which run the control plane, billing, and analytics, were experiencing boot times of up to four hours after a routine firmware update. This was causing significant issues with maintenance windows, engineering teams, and overall system performance. To resolve this issue, Cloudflare's engineers investigated the root cause and discovered a firmware quirk and an over-eager linear search through every available network boot interface.

Key Technical Details

• Cloudflare's core units use UEFI firmware, which initializes hardware and hands off control to the operating system.
• The boot sequence is orchestrated by UEFI, but small quirks in the handoff can have significant consequences.
• The issue was caused by a linear search through every available network boot interface, including IPv4 HTTPS, IPv4 iPXE, and IPv6 HTTPS.
• The system was attempting to boot over the network instead of from local storage, which is critical for centralized, automated, and scalable control over how machines start up.

Practical Implications for Developers

• Developers can use this knowledge to troubleshoot similar issues with their own systems.
• Understanding the UEFI firmware and network boot interfaces can help developers optimize their system's boot process.
• Using open-source tools like iPXE can provide advanced scripting capabilities for automating complex deployments.
• Developers can also use this knowledge to improve the reliability and speed of their system's boot process.