A number of gyb tests (gyb --test) were failing on Windows. It appears it was
introduced by commit d57b41e which altered the behaviour of line
directives to print paths with forward-slashes even on Windows.
This commit changes two areas of gyb.py:
1. Doctests use normalized paths where applicable.
2. Path normalization is done early in ParseContext initialization
instead of when formatting line directives in ExecutionContext.append_text.
Fixes: SR-9644
Most people don't know what this is for and set it to "", robbing themselves of usability benefits. I'm going to add more help text to line-directive to explain that.
When building on Windows, the filepath contains `\` as the path
separator. Substitute `/` for the path separator prior to replacement.
This allows clang to properly compile the source file. Windows supports
both as a path separator and this matches what clang does by default for
the filepaths when preprocessing. This allows gyb generated files to
build on Windows.
This patch reworks the line-directive command line argument for gyb to
take in a python-style format string with `line` and `file` variables
to be substituted.
Because utils/gyb is just a launcher, `doctest.testmod()` without
explicit module doesn't perform doctest of `gyb` module.
It used to perform doctest of `util/gyb`, the `__main__` module.
From the Python documentation:
* NotImplemented: "Special value which can be returned by the 'rich comparison'
special methods (__eq__(), __lt__(), and friends), to indicate that the
comparison is not implemented with respect to the other type."
* NotImplementedError: "This exception is derived from RuntimeError. In user
defined base classes, abstract methods should raise this exception when they
require derived classes to override the method."
* E101: indentation contains mixed spaces and tabs
* E111: indentation is not a multiple of four
* E128: continuation line under-indented for visual indent
* E302: expected 2 blank lines, found 1
* W191: indentation contains tabs
The repo contains roughly 80 Python scripts. "snake_case" naming is used for
local variables in all those scripts. This is the form recommended by the PEP 8
naming recommendations (Python Software Foundation) and typically associated
with idiomatic Python code.
However, in nine of the 80 scripts there were at least one instance of
"camelCase" naming prior to this commit.
This commit improves consistency in the Python code base by making sure that
these nine remaining files follow the variable naming convention used for
Python code in the project.
References:
* PEP 8: https://www.python.org/dev/peps/pep-0008/
* pep8-naming: https://pypi.python.org/pypi/pep8-naming
Fixes:
* multiple statements on one line (colon) (E701)
* missing whitespace around arithmetic operator (E226)
* missing whitespace around operator (E225)
* closing bracket does not match visual indentation (E124)
* blank line contains whitespace (W293)
* continuation line missing indentation or outdented (E122)
* continuation line over-indented for hanging indent (E126)
* missing expected blank line (E301)
* trailing whitespace (W291)
* unexpected spaces around keyword / parameter equals (E251)
* whitespace after '(', '[' or '{' (E201)
* whitespace before ')', ']' or '}' (E202)
* whitespace before ',' or ':' (E203)
Python 3 on Linux reads the system locale information to determine what
it should use as the default encoding for strings read from files (this
is different from OS X which is always UTF-8 by default [1]).
Since all the Swift gyb templates are UTF-8 encoded there is effectively
no reason to parse them as anything else. This patch forces the gyb
template parser to read the template using UTF-8 encoding. It accounts
for both reading and writing to a file as well as reading from stdin and
writing to stdout.
Two changes of note are that it now includes a __future__ import that
should make Python 2 behave a little closer to Python 3 in terms of
unicode support. Additionally Python 2 can no longer use cStringIO
because it does not support unicode [2].
To test this patch I ran these commands before and after the patch.
Note: that before the patch if the locale was set to something other
than UTF-8, ASCII for instance, the Python 3 runs would fail. See [3]
for example failure message.
Without stdin/stdout:
$ python2 utils/gyb -o Arrays.2.7.swift stdlib/public/core/Arrays.swift.gyb
$ python3 utils/gyb -o Arrays.3.5.swift stdlib/public/core/Arrays.swift.gyb
$ diff -u Arrays.2.7.swift Arrays.3.5.swift
With stdin/stdout:
$ cat stdlib/public/core/Arrays.swift.gyb | python2 utils/gyb > Arrays.2.7.stdin.stdout.swift
$ cat stdlib/public/core/Arrays.swift.gyb | python3 utils/gyb > Arrays.3.5.stdin.stdout.swift
$ diff -u Arrays.2.7.stdin.stdout.swift Arrays.3.5.stdin.stdout.swift
[1] https://docs.python.org/3/howto/unicode.html#unicode-filenames
[2] https://docs.python.org/2/library/stringio.html#cStringIO.StringIO
[3] https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160111/000780.html
