* [Python3] Fix "undefined symbol 'unicode'" from python_lint
This is a little tricky.
Python 2 "unicode" was renamed to "str" in Python 3.
For Python 2 compatibility, we need to use "unicode" in a couple
of places, but that's not defined on Python 3, which causes
python_lint errors (even if the reference is never actually executed).
To workaround this, when running in Python 3, define "unicode"
as a synonym for "str". This defines the symbol (avoiding the
"undefined symbol" error from python lint) while preserving
the correct functionality on both Python 2 and Python 3.
When we drop Python 2 support (which we should do as soon as
possible), we can drop this workaround and globally replace
"unicode" with "str" to get the right Python 3-only functionality.
Hacking sys.modules here was added unconditionally to fix an import problem on
Windows (with Python 2.7???).
This script works fine on Python 2.7 on macOS either with or without this hack.
This script breaks badly on Python 3.8 on macOS with this hack, so I've disabled
it here for all Python 3.
* [Parser] Update 'Confusables.def' file to include confusable and base character names
* [Parser] Add a new utility method to return the names of the confusable and base characters for a given confusable codepoint
* [Parser] Update diagnostic for confusable character during lexing to mention confusable and base character names
* [Sema] If there is just a single confusable character, emit a tailored diagnostic that also mentions the character names
* [Diagnostics] Add new diagnostic messages to the localization file
* [Test] Update confusables test
* [Utils] Update unicode confusables txt file and update script to regenerate confusables def file
* [Parse] Regenerate 'Confusables.def' using updated script
* [Utils] Adjust generate_confusables script based on review feedback
Fix a mistake with name mapping. Updated header comment. Fix a couple of linting issues.
* [Parse] Regenerate 'Confusables.def' file once again after script changes
* [Parse] Add the newline after end of 'getConfusableAndBaseCodepointNames' method
* [Test] Update diagnostic message in 'Syntax/Parser/diags.swift'
Clang's driver started linking with libclang_rt.<os>sim.a when building for simulator.
Swift's build need to adapt to this clang change, by ensuring that the libclang_rt.<os>sim.a libraries are created during the build
* Python3 compatibility for Benchmarks
Three issues addressed here:
1. Dependency on dictionary iteration order
CharacterProperties.swift.gyb iterated a dictionary to produce its output.
Changed this to a list of tuples to ensure the order is predictable.
2. Python3 `map` returns an iterator, not a list
Changed a bunch of `map` calls to `list(map(...))` to ensure the result is a list
3. Python3 `int()` expects a string, won't accept a list of characters
Added a concatenation step that is effectively a no-op on Python2
* Make update-checkout work with Python3
Python3 doesn't like to pickle local functions, so make the multiprocessing
initialization function be global.
Python3 multiprocessing also seems to require conditional execution of the
top-level code. TBH, I'm not entirely sure why Python 2 didn't need this.
* Adjust whitespace per python-lint
In workflows featuring git worktrees, it is common for the same branch
to be in use by multiple multiple checkouts. For example, at the
moment, the "master" branches of swift-format, swift-tensorflow-apis,
and pythonkit (and the "release" branch of ninja) are indicated, in
update-checkout-config.json, by both "master" and "release/5.3". If one
has a workflow featuring git worktrees, that means that when one runs
<<update_checkout --scheme master>> in one's mainline directory and
<<update_checkout --scheme release/5.3>> in one's release/5.3 directory,
the latter will encounter failures for each of those four projects
because the branch "master" will already be checked out in the mainline
directory's worktrees and so it cannot be checked out in the
release/5.3's directory's worktrees. The error looks something like:
/path/to/swift-container/release53/swift-format failed
(ret=128): ['git', 'checkout', u'master']
fatal: 'master' is already checked out at
'/path/to/swift-container/mainline/swift-format'
/path/to/swift-container/release53/tensorflow-swift-apis failed
(ret=128): ['git', 'checkout', u'master']
fatal: 'master' is already checked out at
'/path/to/swift-container/mainline/tensorflow-swift-apis'
/path/to/swift-container/release53/pythonkit failed (ret=128):
['git', 'checkout', u'master']
fatal: 'master' is already checked out at
'/path/to/swift-container/mainline/pythonkit'
/path/to/swift-container/release53/ninja failed (ret=128):
['git', 'checkout', u'release']
fatal: 'release' is already checked out at
'/path/to/swift-container/mainline/ninja'
Here, that workflow is enabled. If <<git checkout branch_name>> fails,
for one of the projects, update_checkout falls back to getting the SHA
for the indicated branch via <<git rev-parse branch_name>> and then
checking out the SHA directly.
During the Python 2 to Python 3 conversion, the difference in encoding
became apparent. Explicitly handle the encoding by opening the file
with an explicit encoding. This prevents falling back to the `C` locale
which will use ASCII for UTF-8 which fails.
In order to deal with encoding conversions for Python 2 and Python 3,
the string had to be passed through CStringIO for Python 2. On Python
3, the strings are unicode.
Treat the I/O data as UTF-8 rather than the system encoding, which can
fallback to `C`, treating unicode data as ASCII. This improves the test
coverage on Linux with Python 3.
We would rely on the system encoding, which if unspecified would fall
back to `C` treating the encoding as ASCII. This fails with unicode
data in Python3. Thanks to @tbkka for the brilliant idea of forcing the
encoding when reading the file. This improves the test pass rate on
Linux with Python 3.
* More Python3 lint fixes
Some of the issues addressed include:
* Don't use `l` as a variable name (confusable with `1` or `I`)
* `print` statement does not exist in Py3, use `print` function instead
* Implicit tuple deconstruction in function args is no longer supported,
use explicit splat `*` at the call site instead
* `xrange` does not exist in Py3, use `range` instead
* Better name per review feedback
* Rework a couple of benchmarks to work with both Python2 and Python3
* Adjust relative import statements to use relative syntax
Importing files from "the same directory as this file" was
implicitly supported in Python2.7 but not Python3. The
syntax here works for both.
The custom `shell` module would coerce the byte-string representation
from `subprocess` into a text string representation in the happy path.
However, the same handling was not applied to the error case. This
would result in a byte-string representation being returned in Python 3
rather than the text string. Perform the conversion in both cases to
ensure that we can handle the strings without having to do the
conversion.
`zip` will return a generator in Python 3 rather than the zipped list.
This results in the Nelder-Mead Simplex to fail as it does not actually
perform the optimization of the data. Explicitly convert the data to a
form which can be consumed.
`ModuleNotFoundError` was introduced in python 3.6 as a child of
`ImportError`. However, some of the CI hosts use python 3.5. Use
`ImportError` as the filter type to be compatible with earlier python
versions.
The file-like object's `write` method in Python 3 returns the number of
bytes that are written. This was not previously checked and is not
particularly interesting. Simply blackhole the bytes written, repairing
the test on Python 3.
Today unchecked_bitwise_cast returns a value with ObjCUnowned ownership. This is
important to do since the instruction can truncate memory meaning we want to
treat it as a new object that must be copied before use.
This means that in OSSA we do not have a purely ossa forwarding unchecked
layout-compatible assuming cast. This role is filled by unchecked_value_cast.
