Monzo, the UK digital bank known for its microservice architecture, runs a custom build system that has evolved organically over the past decade. At the heart of that system lies a TOML parser that, until early 2025, silently dropped any key it did not recognize. That design choice, initially harmless, eventually swallowed three teams' deployment scripts whole. The incident, documented in a recent post-mortem, offers a cautionary tale about config parsing assumptions that can quietly accumulate technical debt.

When a Config Parser Became a Deployment Gatekeeper

Monzo's build system uses a custom TOML parser originally written in 2019 to handle deployment configuration files. The parser was designed to be lenient: any unknown keys in a config file would be ignored without warning. This allowed teams to add custom keys for their own workflows, such as deploy_env or rollback_strategy, without needing to modify the core parser. For years, this flexibility was seen as a feature. Teams could extend their configs freely, and the parser would simply skip what it didn't understand.

In late 2024, the parser maintainers decided to add a strict mode to support new features that required validating every key. The strict mode was intended to catch typos and deprecated keys. However, the transition was handled as a standard update: the parser was changed to reject unknown keys by default, with a migration flag to opt out. The deployment system was updated, and the change was merged without a cross-team review. No one noticed that three teams had been relying on keys that the parser had always ignored.

The first signs of trouble emerged in early 2025. The platform team, responsible for deployment orchestration, saw their automated rollouts fail silently. Their scripts read the deploy_env key to determine which environment to target. After the parser update, that key was no longer passed downstream. The deployment script defaulted to a fallback value, causing services to be deployed to the wrong environment. The team spent roughly 4 hours debugging before they traced the issue to the parser change.

The SRE team encountered a similar failure. Their rollback logic depended on a rollback_strategy key that defined whether to revert to a previous version or apply a hotfix. Without that key, the system fell back to a default strategy that sometimes triggered full rollbacks unnecessarily. The team lost about 5 hours of investigation time, during which production incidents were handled manually. The data pipeline team, using a custom key for schema versioning, saw their ETL jobs fail intermittently because the version key was missing.

The TOML Parser's Design Choice That Backfired

The parser's original leniency was an explicit design decision. The maintainers believed that ignoring unknown keys would make the system more resilient to config drift. If a team added a temporary key during development, the parser wouldn't break the build. This approach worked well when the config files were small and the team count was low. But as Monzo grew to dozens of teams, the config files became sprawling, and the custom keys multiplied.

The strict mode was introduced to enforce a schema for a new feature: dynamic environment resolution. The parser needed to know every valid key to interpolate variables correctly. The maintainers added a strict = true flag to the TOML config files that opted into validation. However, the default was changed to strict for all new configs, and existing configs without the flag were treated as strict by default after a migration script ran. The migration script did not check for unknown keys; it simply added the flag and assumed all keys were valid.

No warning logs were emitted when keys were ignored in the old mode. This was a deliberate choice to avoid log noise. The maintainers assumed that if a key was unknown, it was either a typo or a deprecated key that the team should remove. They did not anticipate that teams would rely on custom keys for critical logic. Internal documentation for the parser never mentioned that unknown keys were silently dropped. Teams that read the docs saw only the parsing rules for known keys.

The parser's codebase had a comment that read: "Unknown keys are ignored to allow forward compatibility." That forward compatibility became backward incompatibility when the strict mode shipped. The teams that had been using custom keys for years had no reason to check whether those keys were officially supported. The parser had accepted them without complaint, so they assumed the system was designed to pass them through.

Three Teams, One Silent Failure Mode

Platform Team: The deploy_env Key

The platform team maintained a set of shell scripts that automated deployment workflows. These scripts read a TOML config file that included a deploy_env key specifying the target environment (staging, production, or canary). The key had been in use since 2020, added by a former engineer who needed a way to override the default environment for a specific service. The parser never validated it, but the scripts depended on it. After the strict mode update, the key was silently dropped, and the scripts used a hardcoded default of "staging." Deployments to production failed because the scripts attempted to deploy to staging instead.

The team noticed the issue when a routine production deployment resulted in a staging rollout. The deployment dashboard showed green, but the services were not accessible in production. The team spent roughly 4 hours cross-referencing logs and configs before they identified the missing key. They had no visibility into the parser's behavior because the parser did not log ignored keys. The fix involved reverting the parser change temporarily and adding the key to the allowed list.

SRE Team: The rollback_strategy Key

The SRE team's rollback scripts used a rollback_strategy key to determine how to recover from a failed deployment. The key could be set to "full" (revert to previous version) or "hotfix" (apply a patch on top). Without the key, the scripts defaulted to "full," which caused unnecessary full rollbacks for minor issues. The team saw an increase in rollback times and initially suspected a network issue. It took roughly 5 hours to correlate the problem with the parser update, during which several deployments were delayed.

The SRE team had explicitly tested their rollback scripts in a staging environment, but the staging config files still had the old parser version. The parser update had not been rolled out to staging yet. This mismatch between environments masked the issue until production deployments triggered the new parser. The post-mortem noted that the team's tests did not include a config validation step that would have caught the missing key.

Data Pipeline Team: Schema Versioning Key

The data pipeline team used a custom key called schema_version to track which version of their data schema a particular deployment should use. This key was read by a separate service that transformed data before loading it into the warehouse. After the parser update, the key was missing, and the transformation service defaulted to an older schema version. Data quality checks failed, and the team spent roughly 6 hours investigating before they found the root cause. The incident caused a delay in data availability for analytics.

The data pipeline team's config files were generated by an internal tool that automatically added the schema_version key. The tool had been written before the parser change and assumed the key would be passed through. The team had no integration tests that verified the full pipeline with the actual build system. The post-mortem recommended adding a schema registry with per-team ownership to prevent such silent failures.

Post-Mortem: How the Parser Escaped Review

The post-mortem, held roughly two weeks after the incidents, identified several systemic failures. The parser change was merged without a cross-team review because it was classified as a minor internal refactor. The maintainers assumed that the strict mode would only affect teams that had opted in by adding the flag. They did not realize that the migration script had effectively opted all teams in. The change was also not tested against real deployment configs; unit tests only covered new keys and ignored the behavior of existing configs.

Monzo's config schema was never formally defined. Each team used a subset of keys that had grown organically over time. There was no central registry of valid keys, so the parser maintainers had no way to know which keys were in use. The post-mortem recommended creating a schema definition that listed all known keys and their owners. The parser would then validate against this schema and emit warnings for unknown keys, giving teams a migration window before enforcing strict mode.

The incident also highlighted a lack of visibility into parser behavior changes. Teams had no way to see what keys the parser was ignoring. The parser did not log ignored keys, even in debug mode. The maintainers had considered logging but decided against it to avoid log bloat. The post-mortem concluded that the cost of log bloat was outweighed by the cost of silent failures. They implemented a warning log for ignored keys in the development environment, with a rate limit to prevent flooding.

Another factor was the absence of integration tests that combined the parser with real deployment scripts. Each team tested their configs in isolation, but no test verified that the configs worked end-to-end with the build system. The post-mortem recommended adding a CI step that ran a config validation against a schema definition for every change to the parser. This step would catch unknown keys before they reached production.

Lessons for Build System Maintainers

The Monzo incident offers several lessons that apply to any build system that processes configuration files. First, strict parsing should log warnings for unknown keys before enforcing a block. This gives teams time to either add their keys to the schema or remove them. A deprecation window of at least two weeks is reasonable for most internal systems. The parser should emit warnings in a structured format that teams can monitor via their usual logging infrastructure.

Second, config schemas must be versioned and validated. A schema registry that maps keys to owning teams allows the parser to know which keys are legitimate. When a new key is added, the registry should be updated. The schema itself should be validated against the parser's capabilities to avoid mismatches. Monzo now runs a config linter in CI that checks every config file against the schema before allowing a build. The linter is maintained by a central platform team but accepts contributions from all teams via pull requests.

Third, cross-team integration tests catch silent regressions. The parser change had no integration test that used a real deployment config from another team. A simple test that loads a sample config from each team and checks that all expected keys are present would have caught the issue. The post-mortem recommended that any change to the parser must include a test that runs against a representative set of configs from across the organization. This test should be run in a staging environment that mirrors production as closely as possible.

Fourth, parser changes require a migration window with deprecation warnings. The maintainers should have communicated the change to all teams at least two weeks in advance, with instructions on how to check if their configs were affected. A deprecation warning in the parser logs for any unknown key would have alerted teams to update their configs before the strict mode took effect. Monzo now uses a phased rollout: first, warnings are emitted for one month; then, strict mode is enabled with a fallback to lenient mode for configs that still have unknown keys; finally, strict mode is enforced.

Finally, teams should have visibility into parser behavior changes. A changelog that documents every modification to the parser, including the rationale and impact, helps teams anticipate issues. Monzo now maintains a public (internal) changelog that is posted to a Slack channel with a bot that tags relevant teams based on the keys they own. This ensures that no team is caught off guard by a parser update.

Practical Takeaway: Hedge Your Parser's Ignorance

The Monzo incident is a reminder that a parser's default behavior—silently ignoring unknown keys—can create hidden dependencies. The fix that Monzo implemented is surprisingly simple: a 20-line schema definition that lists all known keys and their owners. The parser now validates every config against this schema and rejects unknown keys with a clear error message. But the real lesson is about process, not code. The schema definition alone would not have prevented the incident if the parser had still been lenient. The change required a cultural shift toward treating config as a first-class artifact with formal validation.

Other organizations can learn from Monzo's experience by adopting a few practical measures. First, if your build system uses a custom parser, audit it for lenient behavior. Check whether unknown keys are silently ignored. If they are, decide whether that behavior is intentional or an oversight. If it's intentional, document it clearly and ensure that teams know the parser will ignore their custom keys. If it's an oversight, plan a migration to strict mode with appropriate warnings.

Second, implement a schema registry as a single source of truth for config keys. This registry should be version-controlled and reviewed like any other code change. Each key should have an owner team that is responsible for its definition and usage. The registry can be a simple YAML file or a database, depending on your scale. The key is that the parser reads this registry to know which keys are valid.

Third, run a config linter in CI that checks every config file against the schema. This linter should be run as part of the build process for any service that uses the build system. If a config file contains an unknown key, the linter should fail the build with a clear error message pointing to the schema registry. This prevents teams from accidentally relying on keys that the parser will ignore.

Fourth, monitor parser logs for ignored key patterns. Even if you have a schema, there will be edge cases where keys are inadvertently ignored. A monitoring dashboard that tracks the rate of ignored keys can alert you to potential issues. Monzo now has a Grafana dashboard that shows the number of ignored keys per team per day. Any spike triggers an investigation.

The incident also underscores the value of canary deployments with config validation. Monzo now runs a canary deployment for every parser change that deploys to a small subset of services before rolling out to the entire fleet. The canary checks that all config keys are correctly propagated and that no service fails. This adds roughly 30 minutes to the deployment pipeline but has already caught two potential regressions since the incident.

Ultimately, the Monzo story is not about a broken parser. It is about the assumptions we make about our tools. A parser that silently ignores unknown keys is a parser that allows teams to build invisible dependencies. Those dependencies will eventually break when the parser changes. The fix is not just to make the parser stricter, but to create a system where config is transparent, validated, and owned. The 20-line schema definition that Monzo adopted is a small price to pay for avoiding the kind of silent failure that swallowed three teams' deployment scripts.

For more on how small changes can have outsized impact, see One Maintainer's Two-Line CSS Fix Cut Load Times by Forty Percent. And for a cautionary tale about API contracts, read One Platform Constraint Forces an API Contract That Both Stores Reject. For a perspective on compensation disparities in distributed systems, see A Distributed Systems Role Pays Less Than Monolith Work at Equivalent Scale.