Update to Unicode 16.0.0

[rmisev]

What is the issue with the URL Standard?

Version 16.0.0 (2024-08-30) of Unicode Technical Standard #46 has been released. It fixes some previously reported issues:

1. More IDNA roundtrippability issues #760
   The Processing 4.3. step (after Punycode decode) fixes this issue:

   If the label is empty, or if the label contains only ASCII code points, record that there was an error.

2. IdnaTestV2.json "xn--xn--a--gua.pt" test case problem #803
   The test in question is now correctly labeled in IdnaTestV2.txt:

   xn--xn--a--gua.pt; xn--a-ä.pt; [V2, V4]; xn--xn--a--gua.pt; ; ;  # xn--a-ä.pt
   

So I think it's worth upgrading to that standard:

1. Reference the new 16.0.0 Unicode Technical Standard #46 in the Normative References section.
2. In the WPT update the IdnaTestV2-parser.py tool and the IdnaTestV2.json test file. I have opened a PR for this: URL: Update IdnaTestV2 to UTS46 16.0.0 web-platform-tests/wpt#48301

[annevk]

@markusicu @macchiati could you provide some context for these changes? While we submitted a bunch of feedback (as recorded in #744) it seems there's quite a few other changes as well.

E.g., Ⅎ is an invalid domain name today, but with Unicode 16 would be valid?

(We are seeing this in WebKit as well now: WebKit/WebKit#37104.)

[markusicu]

We considered several issues and made recommendations that the UTC approved.

For example, there were complicated and unnecessary differences in processing with UseSTD3ASCIIRules=true vs. false, and some characters were disallowed based on differences between IDNA2003 and IDNA2008, while (a) IDNA2003 has not been relevant in a long time and (b) transitional processing had been deprecated in Unicode 15.1.

For details see

• Unicode 16 Modifications: https://www.unicode.org/reports/tr46/tr46-33.html#Modifications
• UTC #178 PAG report pages 23-24
• UTC #179 PAG report pages 40-44
  • This includes the reduction of the base exclusion set. See the feedback on pages 41-42 and the proposal starting on page 43.
  • Page 44 has a list of “Changing from "disallowed" to "mapped"” which includes U+2132 TURNED CAPITAL F.
• UTC #180 PAG report pages 33-35

[annevk]

@markusicu thank you, that was very helpful. We'll update things here within the next couple of months.

Edit: The URL Standard references the latest version of UTS46 now, but the test situation will take a bit longer to resolve.