test: explicitly encode arguments for Python 3

The argument handling for `subprocess.check_output` differs between
Python 2.7, [3.0, 3.6], [3.7).

With Python 2.7, the list is implicitly decoded and we cannot use the
`encode` method as it will attempt to decode it as ASCII which will fail
with unicode data.

Python [3.0, 3.6] will attempt to encode the arguments before executing,
but will do so based on the system locale.  In the case that the locale
is `POSIX`, the encoding will be attempted in ASCII, which again fails
when unicode data is used.

Python [3.7) will accept the encoded strings and pass them through.  It
will implicitly encode the arguments using the file system encoding
(normally UTF-8), which will allow it to actually work with the content
encoded or decoded.

In order to provide maximal compatibility, pre-encode the arguments as
UTF-8 when using python 3.  Rely on the interpreter encoding the
argument implicitly on Python 2.  This allows running the
incrParse.simple test with python 2.7, python 3.6, python 3.7, python
3.8 with `LC_ALL=POSIX` as is the default on Ubuntu Server.

Thanks to @tbkka for the details on how to reproduce this issue and the
help with the python conversion!
This commit is contained in:
Saleem Abdulrasool
2020-07-31 15:43:24 +00:00
parent fb364ae2b7
commit 162c5e1bc7

View File

@@ -25,7 +25,11 @@ def run_command(cmd):
if sys.version_info[0] < 3:
cmd = list(map(lambda s: s.encode('utf-8'), cmd))
print(' '.join([escapeCmdArg(arg) for arg in cmd]))
return subprocess.check_output(cmd, stderr=subprocess.STDOUT)
if sys.version_info[0] < 3:
return subprocess.check_output(cmd, stderr=subprocess.STDOUT)
else:
return subprocess.check_output(list(map(lambda s: s.encode('utf-8'), cmd)),
stderr=subprocess.STDOUT)
def parseLine(line, line_no, test_case, incremental_edit_args, reparse_args,