* [CMake] Only build SwiftSyntax if building SDK overlays
SwiftSyntax depends on Foundation, which depends on the SDK overlays
being built. However, the existing build configuration tried to build
SwiftSyntax even if the SDK overlays were not built. Ensure we're
building overlays before building SwiftSyntax, and guard tests with an
sdk_overlay test.
* Remove TODO comment
* Create Swift libSyntax API
This patch is an initial implementation of the Swift libSyntax API. It
aims to provide all features of the C++ API but exposed to Swift.
It currently resides in SwiftExperimental and will likely exist in a
molten state for a while.
* Only build SwiftSyntax on macOS
* Generate libSyntax API
This patch removes the hand-rolled libSyntax API and replaces it with an
API that's entirely automatically generated. This means the API is
guaranteed to be internally stylistically and functionally consistent.
Add an option to the lexer to go back and get a list of "full"
tokens, which include their leading and trailing trivia, which
we can index into from SourceLocs in the current AST.
This starts the Syntax sublibrary, which will support structured
editing APIs. Some skeleton support and basic implementations are
in place for types and generics in the grammar. Yes, it's slightly
redundant with what we have right now. lib/AST conflates syntax
and semantics in the same place(s); this is a first step in changing
that to separate the two concepts for clarity and also to get closer
to incremental parsing and type-checking. The goal is to eventually
extract all of the syntactic information from lib/AST and change that
to be more of a semantic/symbolic model.
Stub out a Semantics manager. This ought to eventually be used as a hub
for encapsulating lazily computed semantic information for syntax nodes.
For the time being, it can serve as a temporary place for mapping from
Syntax nodes to semantically full lib/AST nodes.
This is still in a molten state - don't get too close, wear appropriate
proximity suits, etc.
I am going to use this in bug reducer for debugging runtime crashes. I just
found the branch and cleaned it up, so I fugred I would commit it sooner rather
than after I lost the branch again.
This enables one to dump the various passpipelines in a yaml format. Other
pretty print formats can be added in the future as well if desired. Its intended
usage is to provide a source of pass pipeline information for external python
bug-reducing tools. By integrating this as a compiler-tool, we are guaranteed to
never have to update any of these tools in the face of passpipeline changes.
This will allow for modules to be split from the command line using a script.
The one thing that is missing from this still is that it does not handle shared
functions in IMO a satisfactory way. Given that we are splitting a module, my
feeling that the correct way to do this is to create a public shim for the
shared function in the module that the shared function gets put in and let all
other users use that entry point.
But I need to think about this a bit more.
[Tools] Add a tool to detect source-breaking API changes introduced from libraries.
swift-api-digester is a test utility to detect source-breaking API changes
during the evolution of a swift library. The tool works on two phases:
(1) dumping library contents as a json file, and (2) comparing two json
files textually to report interesting changes.
During phase (1), the api-digester looks up every declarations inside
a module and outputs a singly-rooted tree that encloses interesting
details of the API level.
During phase (2), api-digester applies structure-information comparision
algorithms on two given singly root trees, trying to figure out, as
precise as possible, the branches/leaves in the trees that differ from
each other. Further analysis decides whether the changed leaves/branches
can be reflected as source-breaking changes for API users. If they are,
the output of api-digester will include such changes.
Also, this commit includes a regression test that make sure API changes
from the Swift stdlib are expected.
swift-reflection-test is now the test that forks a swift executable
and performs remote reflection, making it runnable on other targets,
such as the iOS simulator.
swift-reflection-dump is now a host-side tool that dumps the remote
reflection sections for any platform binary and will continue to
link in LLVM object file support.
This necessitates finally moving lib/Refleciton into stdlib/public,
since we're linking target-specific versions of the test tool and
we would eventually like to adopt some of this functionality in
the runtime anyway.
- Nearly done: TypeRefs and the mangled name decoder.
- Add the swift-reflection-test tool.
The field reflection pipeline is roughly:
- Decode type references
- Substitute generic parameters
- Calculate sizes and offsets
There is currently only one action in the tool, which will test the
*Decode* part of the pipeline: `dump-reflection-section`. This reads
the *swift3_reflect section from an object file and dumps the decoded
type references for all of the stored properties and enum cases in the
file.
- TODO: Write tests with various type arrangements to exercise the
decoder - there are likely some holes in the decoder still since the
AST mangler is quite rich in its kinds.
TODO: The next test mode, `dump-field-types`, will do the following:
1. Launch a swift executable with a canned stopping point
2. Get the address of a heap object instance of interest
3. Dump the fully substituted typerefs of all of the stored properties
or enum case payloads.
That test mode will be more involved since it will attach to another
process and need to read from its address space but will test the
entire out-of-process reflection pipeline in a controlled environment.
We can maybe take this test a step further, with an option or a new
test mode, that prints the entire heap reference graph rooted at that
object of interest, in order to test the ability to detect reference
cycles, for example.
This commit adds a number of compression routines:
1. A dictionary based compression.
2. Huffman based compression.
3. A compression algorithm for swift names that's based on the other two.
This commit also adds two large autogenerated files: CBCTables.h and HuffTables.h.
These files contain the autogenerated string tables and auto-generated code for
fast compression/decompression. The internal tree data structures are lowered
into code that does the variable length encoding/decoding and searching of
fragments in the codebook. The files were generated by processing the symbols
from several large swift applications (stdlib, unittests, simd, ui app, etc).
The list of the programs is listed as part of the output of the tool in the
header file.
I decided to commit the auto-generated files for two reasons. First, we have a
cyclic dependency problem where we need to analyze the output of the compiler
(swift files) in order to generate the tables. And second, these tables will
become a part of the Swift ABI and should remain constant.
It should be possible to split the code that generates the Trie-based data
structure and auto-generate it as part of the Swift build process.
The code goes into its own sub-tree under 'tools' but tests go under 'test',
so that running 'check-swift' will also run all the SourceKit tests.
SourceKit is disabled on non-darwin platforms.
