Explicit mention of UTF-8 BOM in the ABNF?

[DecimalTurn]

Currently, there is no mention of UTF-8 BOM in the ABNF. However, since BOM support is now part of the official test suite (toml-lang/toml-test@542746b), perhaps it should be made more explicit that TOML supports optional UTF-8 BOM.

For comparison, RFC 5424 does mention the UTF-8 BOM in its ABNF:

      MSG-UTF8        = BOM UTF-8-STRING
      BOM             = %xEF.BB.BF

For TOML, the change could simply be:

;; Overall Structure

toml = [ utf8-bom ] expression *( newline expression )

utf8-bom = %xEF.BB.BF  ; U+FEFF

[arp242]

Does this solve a practical problem?

Keeping the ABNF encoding-neutral would be my preference in absence of a practical issue.

[DecimalTurn]

Well, in practice 4/10 of the TOML parsers mentioned here don't support the UTF-8 BOM at the moment. My implementation also didn't support BOM until I saw the new tests added for it in the test suite. So clearly, unless implementers deliberately chose to ignore it, the BOM support requirement isn't communicated well enough.

Also, if we make that change, a parser that is generated directly from the ABNF by using a tool such as declaresub/abnf, would get BOM support for free without having to deal with encoding directly. I don't personally use it, but @pradyunsg used in in the past based on this issue, however I can't confirm if it was for TOML specifically.

[arp242]

Well, in practice 4/10 of the TOML parsers mentioned toml-lang/toml-test#199 don't support the UTF-8 BOM at the moment.

I'd say that's actually a good indication that it's not really a practical problem, because otherwise people would have reported and fixed it. I'm somewhat ambivalent on even adding it to toml-test in the first place.

[DecimalTurn]

I'm somewhat ambivalent on even adding it to toml-test in the first place.

May I ask why?

[arp242]

That it's not supported in some very widely used libraries like tomli or go-toml is a good indication that no one is really encountering problems with it. In a quick check in some other libraries it was only supported as a theoretical problem rather than a practical one.

[DecimalTurn]

To me, it looks like this issue is starting to come up in practice. Perhaps that comes from the fact that PowerShell 5 still adds the BOM by default and people that are using coding agent on Windows might run PowerShell 5 commands to edit TOML files.

For instance, here are some issues related to TOML UTF-8 BOM I could find from a quick search:

• Codex Desktop hides automation when automation.toml is UTF-8 with BOM openai/codex#13967
• bug: Parsing of zebrad.toml on Windows fails due to Byte Order Mark (BOM) added to the beginning of the file ZcashFoundation/zebra#10561
• Config file with UTF-8 BOM causes Invalid TOML error on startup MoonshotAI/kimi-cli#2043
• [Bug]: CLI does not handle UTF-8 BOM in input files aether-framework/aether-datafixers#198
• installer: Set-Content -Encoding UTF8 writes BOM on config.json and frpc.toml IT-Dev-Group-6/dcs-bullseye#2
• [BUG] [v0.0.7] cortex-hooks, cortex-engine, cortex-plugins: UTF-8 BOM in config files causes serde_json/toml parse failure — 4 unhandled BOM instances across 3 crates PlatformNetwork/bounty-challenge#11530

Also, these issues might not always surface to the parser library, since a lot of people/agent might just decide to strip the BOM themselves instead of raising an issue.

[ChristianSi]

The fact that different parsers treat this differently – some allowing a BOM and others rejecting it – strongly suggests that the spec should address the issue explicitly. Otherwise, a TOML document starting with a BOM is left in the unpleasant state of being valid or invalid depending on the parser.

RFC 3629, the IETF specification for UTF-8, says: "A protocol SHOULD forbid use of U+FEFF as a signature for those textual protocol elements that the protocol mandates to be always UTF-8, the signature function being totally useless in those cases." (sec. 6)

This applies to TOML, since all TOML documents are required to be UTF-8. Hence I'd consider it best to follow the IETF recommendation and forbid the use of a BOM at the start of TOML documents.

This doesn't require a change to the ABNF spec, which effectively already forbids it since U+FEFF is not allowed outside quoted strings and comments. However, I'd suggest adding a short note to the text spec to make explicit that no BOM is allowed.

[arp242]

U+FEFF is the UTF-16 BOM. This is about the UTF-8 BOM.

[DecimalTurn]

U+FEFF is the UTF-16 BOM. This is about the UTF-8 BOM.

The character U+FEFF is also the UTF-8 BOM, it's just that it's encoded with the bytes EF BB BF as opposed to FE FF for UTF-16 (BE). Basically, the BOM is always the character U+FEFF and the way it's encoded tells you the encoding of the document. It just happens that for UTF-16 (BE), all characters in the Basic Multilingual Plane (BMP) are represented as their exact codepoints, so the byte representation in hexadecimal matches perfectly.

[DecimalTurn]

RFC 3629, the IETF specification for UTF-8, says: "A protocol SHOULD forbid use of U+FEFF as a signature for those textual protocol elements that the protocol mandates to be always UTF-8, the signature function being totally useless in those cases." (sec. 6)

This applies to TOML, since all TOML documents are required to be UTF-8. Hence I'd consider it best to follow the IETF recommendation and forbid the use of a BOM at the start of TOML documents.

This is indeed what should have been decided initially when the question was raised in #437 (comment). However, from that issue we have that:

• The sentence "A TOML file must be a valid UTF-8 encoded Unicode document." was meant to allow an optional UTF-8 BOM when it was introduced.
• People (at least one) have interpreted this issue as an indication that the UTF-8 BOM is valid in a TOML document.

For those reasons, forbiding the BOM at this point would constitute a breaking change since previously valid TOML files would no longer be valid. Having a major version bump just for a BOM question seems excessive, but at least it would make TOML document more minimalist™️. Luckily, RFC 3629 only uses "SHOULD" and not "MUST", so the pragmatic choice of keeping the BOM remains a valid option in this context.

[ChristianSi]

@DecimalTurn Thanks for finding that earlier conversation! Personally, I wouldn't interpret "A TOML file must be a valid UTF-8 encoded Unicode document" as necessarily allowing a BOM. However, since the earlier discussion clearly shows that this was the consensus view at the time, I agree that we should stick to it now.

So it does make sense to add a rule for it to the ABNF. However, utf8-bom = %xEF.BB.BF would be wrong, since those are bytes, while the ABNF operates at the level of code points (see the initial comment that "this grammar refers to the Unicode Code Points you get after you decode the UTF-8 input"). Hence a reasonable rule would be:

toml = [ bom ] expression *( newline expression )
bom = %xFEFF

We should also explicitly mention this in the textual spec, since interpreting "a valid UTF-8 encoded Unicode document" as implying that a BOM is allowed is clearly stretching what that wording actually says. For clarity, I'd suggest adding something like this after that sentence:

A BOM at the start of the file is allowed, but not recommended.

[DecimalTurn]

However, utf8-bom = %xEF.BB.BF would be wrong, since those are bytes, while the ABNF operates at the level of code points

Ah yes, that right, bom = %xFEFF is the right way to do it in this context and it has the benefit of being encoding-neutral.