gh-109559: Update unicodedata for Unicode 15.1#109560
Merged
benjaminp merged 12 commits intopython:mainfrom Sep 20, 2023
Merged
gh-109559: Update unicodedata for Unicode 15.1#109560benjaminp merged 12 commits intopython:mainfrom
unicodedata for Unicode 15.1#109560benjaminp merged 12 commits intopython:mainfrom
Conversation
Closed
SnoopJ
commented
Sep 19, 2023
| ('2B740', '2B81D'), # CJK Ideograph Extension D | ||
| ('2B820', '2CEA1'), # CJK Ideograph Extension E | ||
| ('2CEB0', '2EBE0'), # CJK Ideograph Extension F | ||
| ('2EBF0', '2EE5D'), # CJK Ideograph Extension I |
Contributor
Author
There was a problem hiding this comment.
The range check that occurs later in this file implicitly assumes this list is in sorted order. It seems simpler to have an idiosyncratic order here than to try to introduce sorted() or somesuch.
SnoopJ
commented
Sep 19, 2023
Comment on lines
+1109
to
+1112
| for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded(): | ||
| if propinfo: | ||
| # this is not a binary property, ignore it | ||
| continue |
Contributor
Author
There was a problem hiding this comment.
All the properties defined in DerivedCoreProperties.txt happened to be binary until the latest update, so this tool was getting away with the assumption that they are always binary.
As of Unicode 15.1, this file also includes definitions that use the Indict_Conjunct_Break (InCB) property, which is an enumeration.
With this change, the loop skips over any non-binary properties, since we have nothing to do with them.
Contributor
There was a problem hiding this comment.
It seems like it would be safer to explicitly ignore Indict_Conjunct_Break rather than throw out everything with a second column.
Contributor
Author
There was a problem hiding this comment.
Is there a particular failure mode you have in mind? My rationale here was that the current internalized DB only cares about binary properties in this file, but in practice any of the property types enumerated by UAX#44 could appear in a future revision.
I'm not strongly opposed to ignoring the specific property that breaks the tool against the current revision, but my rationale was that it seems safer to prevent this class of failure in the future if/when additional non-binary properties are added.
benjaminp
reviewed
Sep 19, 2023
Contributor
benjaminp
left a comment
There was a problem hiding this comment.
There are various parts in the documentation which reference the Unicode standard version, which should be updated.
Comment on lines
+1109
to
+1112
| for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded(): | ||
| if propinfo: | ||
| # this is not a binary property, ignore it | ||
| continue |
Contributor
There was a problem hiding this comment.
It seems like it would be safer to explicitly ignore Indict_Conjunct_Break rather than throw out everything with a second column.
Contributor
Author
Good catch! The only reference I could find to Unicode 15.0.0 was in |
Contributor
That's the one you shouldn't touch because it's generated. :) See references in |
Contributor
Author
|
Whoops! Guess my initial grep was poorly-formed. Thanks for the guidance. |
|
All commit authors signed the Contributor License Agreement. |
ca72e1c to
44f6770
Compare
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Contributor
Author
|
Whoops, sorry about the breakage, I forgot to include |
csm10495
pushed a commit
to csm10495/cpython
that referenced
this pull request
Sep 28, 2023
…nGH-109560) --------- Co-authored-by: Benjamin Peterson <benjamin@python.org>
Glyphack
pushed a commit
to Glyphack/cpython
that referenced
this pull request
Sep 2, 2024
…nGH-109560) --------- Co-authored-by: Benjamin Peterson <benjamin@python.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This changeset implements #109559, adding Unicode 15.1 support to the internal databases that support the
unicodedatamodule. The bulk of this Unicode update is the addition of a new CJK Ideograph Extension, which requires updating the CJK ranges baked into themakeunicodedata.pytool andunicodedata.c.I have also taken the liberty of making the documentation of CJK Ideograph Extension ranges consistent between
makeunicodedata.pyandunicodedata.c(i.e. each range includes a comment naming the range).