The builtin iconv in macOS has been quite buggy since macOS 14, when
Apple replaced GNU iconv with a bespoke version. It introduced
backwards-incompatible changes, and behaves oddly in certain character
substitutions.
As such, build the official binary release using GNU iconv instead of
system iconv. This means we have to compile/cache it manually in our CI
just like gettext/libsodium in order to have a universal x86/arm64
binary with the correct deployment target set. We also need to modify
gettext to be built against GNU iconv as well to avoid link-time errors.
Note that this does not affect the Homebrew release of MacVim. The
standard Homebrew gettext is still linked against system iconv, and as
such we can't make an unilateral change without modifying Homebrew's
gettext as well.
This will result in the Vim binary being larger by 2 MB. It's not ideal
but tolerable. If Apple fixes their implementation of iconv we could
revert this in the future.
Related: macvim-dev/macvim#1624
The test_artifacts action (which we use to upload artifacts when a test
has failed) relies on parsing the CI matrix values to generate a unique
artifact name. However, in #1559 we switched to using a reusable
workflow instead, which no longer uses a matrix inside the composable
workflow. We could simply make a matrix with only one item in it to
satisfy the test_artifact action but that seems a bit overkill. Instead,
just modify it so we manually pass in the preferred artifact name
instead which also gives us more flexibility in the naming. It does mean
future upstream merges may cause conflicts.
And include "extra" Linux elements to further disambiguate
archive names.
The current naming of artifacts is inadequate when it comes
to files whose differing name-parts only come from array
values, as arrays are not automatically converted to string.
For example, both artifacts for failing "socketserver" and
"no_x11" CI jobs will claim the same name, and whichever job
finishes last is allowed to overwrite another matching name
artifact.
Reference:
https://docs.github.com/en/actions/reference/workflows-and-actions/expressions#functionscloses: #18556
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
When we don't trigger CI for more than 1 week, or if the
gettext/libsodium packages were updated, CI needs to re-generate the
artifacts and cache them. However, if the overall build fails later due
to unrelated test failures, previously it would stop the cache from
being uploaded at all, making iterating on fixing the test failure quite
painful as we need to rebuild the artifact every run until it passes.
This change now makes it so we always upload the cache as long as the
package was built correctly, so that CI runs after the first failure
will be much smoother.
The universal-package action tries to build a custom formula and
installs it by calling `brew install` on a raw .rb file. This recently
started to fail as Homebrew added a change to consider this use case as
an error. Change the action to use the officially sanctioned method
which is to create a tap and then place the formula in said tap.
The currently given names to the uploaded archives are too
common and require (often manual) renaming for downloaded
archives that belong to different CI runs/attempts of a PR
and/or different PRs. Let's automatically disambiguate such
archives from one another by giving them more unique names
for convenience and future reference.
related: #17704
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Previously when we try to set up a cached package, Homebrew generates an
annoying "already installed, it's just not linked" warning which is
distracting when parsing CI logs. Just make sure to run `brew install`
if we don't have the package cached.
Note that when a package is not cached and we have to rebuild it,
Homebrew will still warn needlessly because we have
HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK set. This is mostly ok because
most of the time our packages should be cached in CI.
Add new step to upload MacVim test results when we have failed tests.
The xcresult folder can be opened in Xcode for inspection to help
understand what went wrong. Also, fix the ordering so failed Vim GUI
test artifacts can be uploaded as well.
When configuring Vim, be explicit to not build with sodium/gettext for
the non-publish builds. This makes sure that even if the CI environment
somehow has them installed by default, we won't use them by mistake.
Those installed packages would be built for a different OS target anyway
and throw out warnings (the ones we bundle with MacVim are custom built
to target the proper OS versions).
Problem: CI: tests can be improved
Solution: collect failed indent tests, raise timeout for search()
functions when using ASAN/Valgrind (Aliaksei Budavei)
ASan-instrumented Vim builds tend to run slower (x2) than
non-instrumented Vim builds and occasionally make indent
tests fail when "search*()" functions time out and give up
further execution.
Reference:
https://github.com/google/sanitizers/wiki/AddressSanitizercloses: #15974
Co-authored-by: Christian Brabandt <cb@256bit.org>
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
When building gettext, Homebrew tries to upgrade all dependencies, and
Python is somehow a downstream dependency and it fails link in certain
binaries like 2to3 which GitHub's CI environment already installed
separately. This is a messy situation in general, but to fix it, just
set `HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK` so Homebrew doesn't go
helpfully upgrade our dependencies when all we want is to build the
package itself. For the most part it should not cause any issues.
With the submitted "viewdumps.vim" script, a few manual
steps in typical workflows (see below) can be automated.
The updated "README.txt" contains additional information.
============================================================
Reviewing LOCAL failed syntax tests can be arranged as
follows:
1) Run tests and generate screendumps:
------------------------------------------------------------
cd /path/to/fork/runtime/syntax
make clean test
------------------------------------------------------------
2) Examine the screendumps from the "failed" directory:
------------------------------------------------------------
../../src/vim --clean -S testdir/viewdumps.vim
------------------------------------------------------------
============================================================
Reviewing UPLOADED failed syntax tests can be arranged as
follows (it can be further locally scripted):
1) Fetch an artifact with failed screendumps from
"github.com/vim/vim/actions/runs/A_ID/artifacts/B_ID".
2) Extract the archived files:
------------------------------------------------------------
unzip /tmp/failed-tests.zip -d /tmp
------------------------------------------------------------
3) Set up the "dumps" directory. Create a symlink to
"/path/to/fork/dirs/dumps" in the extracted directories so
that term_dumpdiff() can be used. (The lookup algorithm
resolves "dumps" for every loaded filename. So, with
"/tmp/runtime/syntax/testdir/failed/*.dump" files passed
as script arguments, the algorithm will make the files in
"/tmp/runtime/syntax/testdir/dumps" queried.)
------------------------------------------------------------
cd /path/to/fork
ln -s $(pwd)/runtime/syntax/testdir/dumps \
/tmp/runtime/syntax/testdir/dumps
------------------------------------------------------------
4) Examine the extracted screendumps:
------------------------------------------------------------
./src/vim --clean -S runtime/syntax/testdir/viewdumps.vim \
/tmp/runtime/syntax/testdir/failed/*.dump
------------------------------------------------------------
5) Clean up:
------------------------------------------------------------
unlink /tmp/runtime/syntax/testdir/dumps
rm -rf /tmp/runtime
------------------------------------------------------------
============================================================
Reviewing SUBMITTED FOR PULL REQUEST syntax tests can be
arranged as follows (it can be further locally scripted):
1) List the fetched changeset and write the changed "dumps"
filenames to "/tmp/filelist":
------------------------------------------------------------
cd /path/to/fork
git switch prs/1234
git diff-index --relative=runtime/syntax/testdir/dumps/ \
--name-only prs/1234~1 > /tmp/filelist
------------------------------------------------------------
2) Reconcile relative filepaths, and copy next-to-be-updated
"dumps" files in the "failed" directory (note the missing
new screendumps, if any):
------------------------------------------------------------
git switch master
cd runtime/syntax/testdir/dumps
cp -t ../failed $(cat /tmp/filelist)
------------------------------------------------------------
3) Remember about the introduced INVERTED relation between
"dumps" and "failed", i.e. the files to be committed are in
"dumps" already and their previous versions are in "failed";
therefore, copy the missing new screendumps from "dumps" to
"failed" (otherwise these won't be shown):
------------------------------------------------------------
git switch prs/1234
cp -t ../failed foo_10.dump foo_11.dump foo_12.dump
------------------------------------------------------------
4) Examine the screendumps from the "failed" directory (new
screendumps will be shown with no difference between their
versions):
------------------------------------------------------------
cd ..
../../../src/vim --clean -S viewdumps.vim
------------------------------------------------------------
closes: #15476
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
It's a bit of a pain to debug failing screendump tests without knowing
exactly what went wrong. Therefore include actions/upload-artifact for
the Github CI runners and have them uploaded those failing screen dump
tests automatically.
Let's add this step to each of the Linux/MacOS/Windows workflows but do
not duplicate the code, factor it out to a single file
.github/actions/screendump/action.yml and reference this one from the
main ci.yml file
Example:
https://github.com/chrisbra/vim/actions/runs/9085493619closes: #14771
Co-authored-by: dundargoc <gocdundar@gmail.com>
Co-authored-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
GitHub Actions has recently added support for macos-14 runners which
run on top of Apple Silicon. Switch CI to using it for publishing binary
builds. Keep running the other OSes for now to test compatibility.
Close#1263
When Apple Silicon came out, we needed to get universal binaries of
gettext/libsodium to link against, and solved it in a somewhat hacky
temporary solution by just downloading the bottles from Homebrew and
patching them with the x86_64 version. However, Homebrew only maintains
bottles for 3 recent OSes, and with macOS 14's release, they no longer
have bottles for macOS 11, which we still want to support as it's a
recent OS. As such, we need to build the arm64 version of the packages
in CI as well instead of just downloading.
When installing from source, Homebrew uses a custom "clang" script that
injects compiler flags including "-march" which will cause clang to fail
to work when building universal binaries (since it doesn't make sense
to specify Intel architectures when specifying `-arch arm64`). Just
force it to use system clang instead to avoid inject unwanted flags.
In the CI setup, xcode-select is only called after the packages have
been set up, but during setup we actually need to build the packages
from source to have the correct deployment target linked correctly. This
means we could potentially be building with the wrong configuration
(e.g. when building for 10.9 legacy builds, we could be using an Xcode
version that's too new for that) when building gettext/libsodium. This
is compounded by the fact that our GitHub Action cache key for the
libraries only include the formula, but not the active Xcode version
itself so a cache could be propagated for a while until suddenly things
break.
To fix this, first make sure we do Xcode configuration early on in the
build, and to properly use the Xcode version as part of the cache key.
This way, when Xcode version changes, we will invalidate the cache and
rebuild gettext / libsodium with the correct configuration and if that's
wrong we will immediately fail the build instead of using old stale
caches.
Also, bump Xcode from 14.1 to 14.2 in CI.
With the release of Homebrew 4, auto-update is supposed to be more
efficient now. By removing the HOMEBREW_NO_AUTO_UPDATE env var from CI,
this will make sure that all the packages that we link to in CI will be
up to date instead of whatever just happens to be installed on the CI
environment which can sometimes fluctuate.
dependabot[bot]
Yee Cheng Chin
ichizok
Aliaksei Budavei
Christian Brabandt