Argyrios Kyrtzidis
5db368ce7b
[Lexer] Introduce Lexer::getLocForStartOfToken() that returns the location at the start of the token that a given offset points to.
...
Swift SVN r8281
2013-09-16 18:41:16 +00:00
Dmitri Hrybenko
8a8b97985b
Lexer: fix a bug where restoring lexer state could produce duplicate
...
code_complete tokens
Swift SVN r7636
2013-08-27 22:34:29 +00:00
Dmitri Hrybenko
dc439adb59
Lexer: improve recovery for invalid character literals and invalid escape
...
sequences in charater literals
Now we return tok::character_literal with REPLACEMENT CHARACTER U+FFFD instead
of tok::unknown.
Swift SVN r7475
2013-08-22 20:56:47 +00:00
Argyrios Kyrtzidis
4908da8361
[Lexer] Remove the public Lexer constructor that accepts a StringRef.
...
Replace uses of it with the newly introduced constructor that accepts a buffer ID.
The StringRef constructor was rather unsafe since it had the implicit requirement that the StringRef
was null-terminated.
Swift SVN r6942
2013-08-06 14:59:03 +00:00
Dmitri Hrybenko
e1c4ae3174
Wrap llvm::SourceMgr in swift::SourceManager so that we can add new members
...
to the source manager.
Swift SVN r6815
2013-08-01 20:39:22 +00:00
Dmitri Hrybenko
68bba56cec
Lexer tests: simplify helper function: no need to pass down a SourceMgr
...
Swift SVN r6810
2013-08-01 18:39:20 +00:00
Dmitri Hrybenko
72ae1fd842
swift::tokenize: don't include tok::eof, per feedback from Argyrios
...
Swift SVN r6716
2013-07-29 22:23:31 +00:00
Dmitri Hrybenko
464df1cc11
Lexer: make tok::eof length equal to 0
...
It used to be equal to 1, which makes Lexer::getLocForEndOfToken() return
an out-of-bounds location for tok::eof.
Swift SVN r6626
2013-07-26 00:14:09 +00:00
Argyrios Kyrtzidis
6e7d0490f7
Allow optionally to produce comment tokens when lexing and add a tokenize() utility function.
...
Swift SVN r6008
2013-07-05 15:02:42 +00:00
