3c8057e13f
[Parse] Remove unnecessary Lexer fields
2018-03-12 23:35:13 +09:00
1dc3e17f52
[Lexer] Performance improvement for Lexer::kindOfIdentifier ( #15168 )
...
Don't check SIL_KEYWORDs when we're not in SIL mode.
Lexer::kindOfIdentifier is super hot-path because many of
tokens in a swift source file are keywords or identifiers.
2018-03-12 15:03:58 +09:00
8592496b22
[Parse] Remove dead code in lexHash ( #15154 )
2018-03-12 08:47:54 +09:00
5e555868fe
[Parse] Reduce branch by running lexTrivia always ( #15137 )
2018-03-10 21:26:54 +09:00
74ba135008
Merge pull request #15040 from ikesyo/gardening-not-empty
...
[gardening] Use `!empty()` over `size() > 0`
2018-03-08 18:47:12 +09:00
c4c5b7130e
[Parse] Fix error in unsigned char target ( #15068 )
2018-03-08 15:57:43 +09:00
967a48cb77
[Parse] refactor Lexer initialization
2018-03-08 11:17:51 +09:00
cea6c03eb2
[gardening] Use !empty() over size() > 0
2018-03-08 09:21:09 +09:00
25cdc981c7
Merge pull request #14977 from ikesyo/gardening-use-empty
...
[gardening] Use `empty()` over `size() == 0`
2018-03-07 09:34:26 +09:00
22cddbf033
[Syntax] Parse invalid chars as trivia
2018-03-06 08:25:09 -08:00
b4192d80e9
[Parse] EmitDiagnosticsIfToken in lexUnknown
...
It is needed from lexTrivia update future.
2018-03-06 08:25:09 -08:00
e6f42fc63d
[Parse] split lexUnknown function from lexImpl
2018-03-06 08:25:09 -08:00
0a69bd7f8d
[NFC] readable control flow in lexImpl default case
...
There are 3 token flow and 2 trivia flow.
This commit makes it clear and more readable.
And apply clang-format.
2018-03-06 08:25:09 -08:00
3e252c98a8
[NFC] Fix naming style in lexImpl default case
2018-03-06 08:25:09 -08:00
190af6c063
[Syntax] support nul character as garbage trivia
2018-03-05 16:53:24 +09:00
58857fa1cb
[Parse] refactor Lexer by made NulCharacterKind type
2018-03-05 16:53:18 +09:00
26d650292f
[gardening] Use empty() over size() == 0
2018-03-05 14:43:13 +09:00
766774206b
[Lexer] Don't setEscapedIdentifier(true) for tok::eof at ArtificialEOF
...
https://bugs.swift.org/browse/SR-6926
This happens when the Parser re-lexing comment tokens that sets
ArtificialEOF at the end of comment range.
It used to cause an assertion failure:
(!value || Kind == tok::identifier) && "only identifiers can be escaped identifiers"
2018-02-12 14:58:12 +09:00
0780c529c4
[Syntax] Unify RawSyntax and RawTokenSyntax using union and TrailingObjects
...
It better matches with SwiftSyntax model.
Using TrailingObjects reduces the number of heap allocation which
gains 18% performance improvement.
2018-01-18 14:49:46 +09:00
a680768971
SIL: In textual SIL allow global SIL names starting with '$'.
...
For example: @$S1m3fooyyF
It's needed to change the mangling prefix to $S.
The parser change only affects SIL (and not swift).
I didn't add test case because it will be fully tested when changing the mangling prefix.
2018-01-05 11:29:15 -08:00
bc88330740
[Parse] Lexer build backtick trivia around espaced identifier token
2017-12-29 00:22:49 +09:00
ebd5323b42
[Syntax] add UTF-8 BOM support to libSyntax
2017-12-28 01:26:09 +09:00
861ee3a112
[Parse] use pre increment for simple increment ( #13624 )
2017-12-27 15:46:58 +09:00
70986a687f
[Parse] fix lexTrivia LF bug
2017-12-22 14:04:14 +09:00
fbe34e0f6f
[Parse] improve LF handling efficiency.
2017-12-22 01:23:25 +09:00
f86e1c8201
[Parse] add CRLF support in lexTrivia
2017-12-21 14:27:13 +09:00
9daeaf0d06
[Parse] refactor lexTrivia with squash
2017-12-20 14:09:47 +09:00
24509a0bde
[Parse] Change LeadingTrivia type to Trivia
2017-12-20 14:09:47 +09:00
cc72a3b934
[Lexer] Use ContentStart position for hashbang trivia
2017-12-19 09:24:34 +09:00
2c06060165
[Syntax] Add CarriageReturn trivia kind
...
To distinguish '\r' from '\n'.
2017-12-19 09:24:34 +09:00
181333ce0f
[Lexer] Lex conflict marker as a trivia
2017-12-19 09:24:33 +09:00
5de598f34a
[Parse] use skipHashbang in lexTrivia
2017-12-18 18:22:04 +09:00
aeb9ba6f96
[Parse] use skipSlashSlashComment in lexTrivia
2017-12-18 18:22:04 +09:00
f7136ae635
[Parse] delete skipUpToEndOfLine
2017-12-18 18:22:04 +09:00
ed58c152bf
[Parse] Improve Lexer's UTF-8 BOM handling ( #13483 )
...
* Add BOM handling testcases
* Add ContentStart to Lexer for BOM handling
2017-12-18 17:22:11 +09:00
5571e5cc76
[Lexer] Clear trivia at the top of lexImpl()
...
To make sure we only parse trivia for the current token.
2017-12-08 12:08:05 +09:00
9b32c62fbf
[Lexer] Add TODO/FIXMEs for lexTrivia
2017-12-08 12:08:05 +09:00
2b1e316cf6
[Syntax] Add parsing hashbang (shebang) as a trivia.
...
Added GarbageText trivia kind for any skipped text.
2017-12-08 12:07:00 +09:00
e7a393f13f
[Lexer] Lex vertical tab '\v' and form-feed '\t' trivias
2017-12-08 11:36:20 +09:00
d767dc39ba
[Lexer] Improve implementation of lexTrivia
2017-12-08 11:36:20 +09:00
dcc37c3340
[Syntax] Normalize TriviaPiece internal value.
...
Length field of comments are always 1.
Text field of whitespaces are always "".
2017-12-04 10:46:03 -08:00
e01d525621
[Lexer] Remove some special trivia handling in Lexer
...
Even in multiline string mode, we should parse trailing trivia.
Removed special handling for backtick trivias, it's not produced in
Lexer anyway.
2017-12-04 10:46:03 -08:00
d46073dd75
[libSyntax] Backtracking restarts from leading trivia position
...
When reading syntax.
2017-12-04 10:46:03 -08:00
a78fda0720
[Syntax] Always lex Trivia when SF.shouldKeepSyntaxInfo()
...
For backward compatibility, Don't lex comments as trailing trivias.
2017-11-17 14:56:49 +09:00
40b195d98c
[Syntax] Get rid of fullLex
...
Defer (Token, Trivia) -> RawTokenSyntax conversion from Lexer to Parser.
This is a part of effort for consolidating Syntax and AST parsing.
2017-11-17 14:56:49 +09:00
75db3c1db8
Re-apply libSyntax patches after fixing ASAN issue ( #12730 )
...
* Re-apply "libSyntax: Ensure round-trip printing when we build syntax tree from parser incrementally. (#12709 )"
* Re-apply "libSyntax: Root parsing context should hold a reference to the current token in the parser, NFC."
* Re-apply "libSyntax: avoid copying token text when lexing token syntax nodes, NFC. (#12723 )"
* Actually fix the container-overflow issue.
2017-11-03 13:25:33 -07:00
7ebf66ed2d
libSyntax: forward declare libSyntax entities in several header files, NFC. ( #12735 )
2017-11-02 20:55:18 -07:00
4d1249aa82
Revert "libSyntax: Ensure round-trip printing when we build syntax tree from parser incrementally. ( #12709 )"
...
This reverts commit 0d98c4c5df
2017-11-02 14:44:26 -07:00
407db56b8d
Revert "libSyntax: avoid copying token text when lexing token syntax nodes, NFC. ( #12723 )"
...
This reverts commit 7981630ddd
2017-11-02 14:43:42 -07:00
7981630ddd
libSyntax: avoid copying token text when lexing token syntax nodes, NFC. ( #12723 )
...
This is likely the root cause for memory surge when we always turn on
syntax token lexing. Since the underlying buffer outlives the syntax
tree, it's reasonable to refer the text instead of copying and owning it.
2017-11-02 14:04:25 -07:00
omochimetaru
Rintaro Ishizaki
Sho Ikeda
Rintaro Ishizaki
Erik Eckstein
Xi Ge