docs/v2/annotated-source/rewriter.html
browser.coffeecake.coffeecoffeescript.coffeecommand.coffeegrammar.coffeehelpers.coffeeindex.coffeelexer.coffeenodes.coffeeoptparse.coffeeregister.coffeerepl.coffeerewriter.coffeescope.litcoffeesourcemap.litcoffee
The CoffeeScript language has a good deal of optional syntax, implicit syntax, and shorthand syntax. This can greatly complicate a grammar and bloat the resulting parse table. Instead of making the parser handle it all, we take a series of passes over the token stream, using this Rewriter to convert shorthand into the unambiguous long form, add implicit indentation and parentheses, and generally clean things up.
{throwSyntaxError, extractAllCommentTokens} =require'./helpers'
Move attached comments from one token to another.
moveComments = (fromToken, toToken) -\>returnunlessfromToken.commentsiftoToken.commentsandtoToken.comments.lengthisnt0unshiftedComments = []forcommentinfromToken.commentsifcomment.unshift
unshiftedComments.push commentelsetoToken.comments.push comment
toToken.comments = unshiftedComments.concat toToken.commentselsetoToken.comments = fromToken.commentsdeletefromToken.comments
Create a generated token: one that exists due to a use of implicit syntax. Optionally have this new token take the attached comments from another token.
generate = (tag, value, origin, commentsToken) -\>token = [tag, value]
token.generated =yestoken.origin = originiforigin
moveComments commentsToken, tokenifcommentsToken
token
The Rewriter class is used by the Lexer, directly against its internal array of tokens.
exports.Rewriter =class Rewriter
Rewrite the token stream in multiple passes, one logical filter at a time. This could certainly be changed into a single pass through the stream, with a big ol’ efficient switch, but it’s much nicer to work with like this. The order of these passes matters—indentation must be corrected before implicit parentheses can be wrapped around blocks of code.
rewrite:(@tokens) -\>
Set environment variable DEBUG_TOKEN_STREAM to true to output token debugging info. Also set DEBUG_REWRITTEN_TOKEN_STREAM to true to output the token stream after it has been rewritten by this file.
ifprocess?.env?.DEBUG_TOKEN_STREAM
console.log'Initial token stream:'ifprocess.env.DEBUG_REWRITTEN_TOKEN_STREAM
console.log (t[0] +'/'+ t[1] + (ift.commentsthen'\*'else'')fortin@tokens).join' '@removeLeadingNewlines()
@closeOpenCalls()
@closeOpenIndexes()
@normalizeLines()
@tagPostfixConditionals()
@addImplicitBracesAndParens()
@rescueStowawayComments()
@addLocationDataToGeneratedTokens()
@enforceValidJSXAttributes()
@fixIndentationLocationData()
@exposeTokenDataToGrammar()ifprocess?.env?.DEBUG_REWRITTEN_TOKEN_STREAM
console.log'Rewritten token stream:'ifprocess.env.DEBUG_TOKEN_STREAM
console.log (t[0] +'/'+ t[1] + (ift.commentsthen'\*'else'')fortin@tokens).join' '@tokens
Rewrite the token stream, looking one token ahead and behind. Allow the return value of the block to tell us how many tokens to move forwards (or backwards) in the stream, to make sure we don’t miss anything as tokens are inserted and removed, and the stream changes length under our feet.
scanTokens:(block) -\>{tokens} = this
i =0i += block.call this, token, i, tokenswhiletoken = tokens[i]truedetectEnd:(i, condition, action, opts = {}) -\>{tokens} = this
levels =0whiletoken = tokens[i]returnaction.call this, token, iiflevelsis0andcondition.call this, token, iiftoken[0]inEXPRESSION_START
levels +=1elseiftoken[0]inEXPRESSION_END
levels -=1iflevels <0returnifopts.returnOnNegativeLevelreturnaction.call this, token, i
i +=1i -1
Leading newlines would introduce an ambiguity in the grammar, so we dispatch them here.
removeLeadingNewlines:-\>
Find the index of the first non-TERMINATOR token.
breakfor[tag], iin@tokenswhentagisnt'TERMINATOR'returnifiis0
If there are any comments attached to the tokens we’re about to discard, shift them forward to what will become the new first token.
forleadingNewlineTokenin@tokens[0...i]
moveComments leadingNewlineToken, @tokens[i]
Discard all the leading newline tokens.
@tokens.splice0, i
The lexer has tagged the opening parenthesis of a method call. Match it with its paired close.
closeOpenCalls:-\>condition = (token, i) -\>token[0]in[')','CALL\_END'] action = (token, i) -\>token[0] ='CALL\_END'@scanTokens (token, i) ->
@detectEnd i +1, condition, actioniftoken[0]is'CALL\_START'1
The lexer has tagged the opening bracket of an indexing operation call. Match it with its paired close.
closeOpenIndexes:-\>startToken =nullcondition = (token, i) -\>token[0]in[']','INDEX\_END'] action = (token, i) -\>[email protected] >= iand@tokens[i +1][0]is':'startToken[0] ='['token[0] =']'elsetoken[0] ='INDEX\_END'@scanTokens (token, i) ->iftoken[0]is'INDEX\_START'startToken = token
@detectEnd i +1, condition, action1
Match tags in token stream starting at i with pattern. pattern may consist of strings (equality), an array of strings (one of) or null (wildcard). Returns the index of the match or -1 if no match.
indexOfTag:(i, pattern...) -\>fuzz =0forjin[0... pattern.length]continueifnotpattern[j]?
pattern[j] = [pattern[j]]iftypeofpattern[j]is'string'return-1if@tag(i + j + fuzz)notinpattern[j]
i + j + fuzz -1
Returns yes if standing in front of something looking like @<x>:, <x>: or <EXPRESSION_START><x>...<EXPRESSION_END>:.
looksObjectish:(j) -\>returnyesif@indexOfTag(j,'@',null,':')isnt-1or@indexOfTag(j,null,':')isnt-1index = @indexOfTag j, EXPRESSION_STARTifindexisnt-1end =null@detectEnd index +1,((token) -\>token[0]inEXPRESSION_END),((token, i) -\>end = i)returnyesif@tag(end +1)is':'no
Returns yes if current line of tokens contain an element of tags on same expression level. Stop searching at LINEBREAKS or explicit start of containing balanced expression.
findTagsBackwards:(i, tags) -\>backStack = []whilei >=0and(backStack.lengthor@tag(i)notintagsand(@tag(i)notinEXPRESSION_STARTor@tokens[i].generated)and@tag(i)notinLINEBREAKS)
backStack.push @tag(i)if@tag(i)inEXPRESSION_END
backStack.pop()if@tag(i)inEXPRESSION_STARTandbackStack.length
i -=1@tag(i)intags
Look for signs of implicit calls and objects in the token stream and add them.
addImplicitBracesAndParens:-\>
Track current balancing depth (both implicit and explicit) on stack.
stack = []
start =null@scanTokens (token, i, tokens) ->
[tag] = token
[prevTag] = prevToken =ifi >0thentokens[i -1]else[]
[nextTag] = nextToken =ifi < tokens.length -1thentokens[i +1]else[]stackTop = -\>stack[stack.length -1]
startIdx = i
Helper function, used for keeping track of the number of tokens consumed and spliced, when returning for getting a new token.
forward = (n) -\>i - startIdx + n
Helper functions
isImplicit = (stackItem) -\>stackItem?[2]?.oursisImplicitObject = (stackItem) -\>isImplicit(stackItem)andstackItem?[0]is'{'isImplicitCall = (stackItem) -\>isImplicit(stackItem)andstackItem?[0]is'('inImplicit = -\>isImplicit stackTop()inImplicitCall = -\>isImplicitCall stackTop()inImplicitObject = -\>isImplicitObject stackTop()
Unclosed control statement inside implicit parens (like class declaration or if-conditionals).
inImplicitControl = -\>inImplicit()andstackTop()?[0]is'CONTROL' startImplicitCall = (idx) -\>stack.push ['(', idx, ours:yes]
tokens.splice idx,0, generate'CALL\_START','(', ['','implicit function call', token[2]], prevToken endImplicitCall = -\>stack.pop()
tokens.splice i,0, generate'CALL\_END',')', ['','end of input', token[2]], prevToken
i +=1 startImplicitObject = (idx, {startsLine = yes, continuationLineIndent} = {}) -\>stack.push ['{', idx, sameLine:yes, startsLine: startsLine, ours:yes, continuationLineIndent: continuationLineIndent]
val =newString'{'val.generated =yestokens.splice idx,0, generate'{', val, token, prevToken endImplicitObject = (j) -\>j = j ? i
stack.pop()
tokens.splice j,0, generate'}','}', token, prevToken
i +=1 implicitObjectContinues = (j) =\>nextTerminatorIdx =null@detectEnd j,(token) -\>token[0]is'TERMINATOR'(token, i) -> nextTerminatorIdx = i
returnOnNegativeLevel:yesreturnnounlessnextTerminatorIdx?
@looksObjectish nextTerminatorIdx +1
Don’t end an implicit call/object on next indent if any of these are in an argument/value.
if(
(inImplicitCall()orinImplicitObject())andtaginCONTROL_IN_IMPLICITorinImplicitObject()andprevTagis':'andtagis'FOR')
stack.push ['CONTROL', i, ours:yes]returnforward(1)iftagis'INDENT'andinImplicit()
An INDENT closes an implicit call unless
1. We have seen a `CONTROL` argument on the line.
2. The last token before the indent is part of the list below.
ifprevTagnotin['=\>','-\>','[','(',',','{','ELSE','=']whileinImplicitCall()orinImplicitObject()andprevTagisnt':'ifinImplicitCall()
endImplicitCall()elseendImplicitObject()
stack.pop()ifinImplicitControl()
stack.push [tag, i]returnforward(1)
Straightforward start of explicit expression.
iftaginEXPRESSION_START
stack.push [tag, i]returnforward(1)
Close all implicit expressions inside of explicitly closed expressions.
iftaginEXPRESSION_ENDwhileinImplicit()ifinImplicitCall()
endImplicitCall()elseifinImplicitObject()
endImplicitObject()elsestack.pop()
start = stack.pop() inControlFlow = =\>seenFor = @findTagsBackwards(i, ['FOR'])and@findTagsBackwards(i, ['FORIN','FOROF','FORFROM'])
controlFlow = seenForor@findTagsBackwards i, ['WHILE','UNTIL','LOOP','LEADING\_WHEN']returnnounlesscontrolFlow
isFunc =notagCurrentLine = token[2].first_line
@detectEnd i,(token, i) -\>token[0]inLINEBREAKS
(token, i) ->
[prevTag, ,{first_line}] = tokens[i -1] || []
isFunc = tagCurrentLineisfirst_lineandprevTagin['-\>','=\>']
returnOnNegativeLevel:yesisFunc
Recognize standard implicit calls like f a, f() b, f? c, h[0] d etc. Added support for spread dots on the left side: f …a
if(taginIMPLICIT_FUNCandtoken.spacedortagis'?'andi >0andnottokens[i -1].spaced)and(nextTaginIMPLICIT_CALLor(nextTagis'...'and@tag(i +2)inIMPLICIT_CALLandnot@findTagsBackwards(i, ['INDEX\_START','[']))ornextTaginIMPLICIT_UNSPACED_CALLandnotnextToken.spacedandnotnextToken.newLine)andnotinControlFlow()
tag = token[0] ='FUNC\_EXIST'iftagis'?'startImplicitCall i +1returnforward(2)
Implicit call taking an implicit indented object as first argument.
f
a: b
c: d
Don’t accept implicit calls of this type, when on the same line as the control structures below as that may misinterpret constructs like:
if f
a: 1
as
if f(a: 1)
which is probably always unintended. Furthermore don’t allow this in the first line of a literal array or explicit object, as that creates grammatical ambiguities (#5368).
iftaginIMPLICIT_FUNCand@indexOfTag(i +1,'INDENT') >-1and@looksObjectish(i +2)andnot@findTagsBackwards(i, ['CLASS','EXTENDS','IF','CATCH','SWITCH','LEADING\_WHEN','FOR','WHILE','UNTIL'])andnot((s = stackTop()?[0])in['{','[']andnotisImplicit(stackTop())and@findTagsBackwards(i, s))
startImplicitCall i +1stack.push ['INDENT', i +2]returnforward(3)
Implicit objects start here.
iftagis':'
Go back to the (implicit) start of the object.
s =switchwhen@tag(i -1)inEXPRESSION_END
[startTag, startIndex] = startifstartTagis'['andstartIndex >0and@tag(startIndex -1)is'@'andnottokens[startIndex -1].spaced
startIndex -1elsestartIndexwhen@tag(i -2)is'@'theni -2elsei -1startsLine = s <=0or@tag(s -1)inLINEBREAKSortokens[s -1].newLine
Are we just continuing an already declared object? Including the case where we indent on the line after an explicit ‘{‘.
ifstackTop()
[stackTag, stackIdx] = stackTop()
stackNext = stack[stack.length -2]if(stackTagis'{'orstackTagis'INDENT'andstackNext?[0]is'{'andnotisImplicit(stackNext)and@findTagsBackwards(stackIdx-1, ['{']))and(startsLineor@tag(s -1)is','or@tag(s -1)is'{')and@tag(s -1)notinUNFINISHEDreturnforward(1)
preObjectToken =ifi >1thentokens[i -2]else[]
startImplicitObject(s, {startsLine: !!startsLine, continuationLineIndent: preObjectToken.continuationLineIndent})returnforward(2)
End implicit calls when chaining method calls like e.g.:
f ->
a
.g b, ->
c
.h a
and also
f a
.g b
.h a
Mark all enclosing objects as not sameLine
iftaginLINEBREAKSforstackIteminstackby-1breakunlessisImplicit stackItem
stackItem[2].sameLine =noifisImplicitObject stackItem
End indented-continuation-line implicit objects once that indentation is over.
iftagis'TERMINATOR'andtoken.endsContinuationLineIndentation
{preContinuationLineIndent} = token.endsContinuationLineIndentationwhileinImplicitObject()and(implicitObjectIndent = stackTop()[2].continuationLineIndent)?andimplicitObjectIndent > preContinuationLineIndent
endImplicitObject()
newLine = prevTagis'OUTDENT'orprevToken.newLineiftaginIMPLICIT_ENDor(taginCALL_CLOSERSandnewLine)or(tagin['..','...']and@findTagsBackwards(i, ["INDEX\_START"]))whileinImplicit()
[stackTag, stackIdx, {sameLine, startsLine}] = stackTop()
Close implicit calls when reached end of argument list
ifinImplicitCall()andprevTagisnt','or(prevTagis','andtagis'TERMINATOR'andnotnextTag?)
endImplicitCall()
Close implicit objects such as: return a: 1, b: 2 unless true
elseifinImplicitObject()andsameLineandtagisnt'TERMINATOR'andprevTagisnt':'andnot(tagin['POST\_IF','FOR','WHILE','UNTIL']andstartsLineandimplicitObjectContinues(i +1))
endImplicitObject()
Close implicit objects when at end of line, line didn’t end with a comma and the implicit object didn’t start the line or the next line doesn’t look like the continuation of an object.
elseifinImplicitObject()andtagis'TERMINATOR'andprevTagisnt','andnot(startsLineand@looksObjectish(i +1))
endImplicitObject()elseifinImplicitControl()andtokens[stackTop()[1]][0]is'CLASS'andtagis'TERMINATOR'stack.pop()elsebreak
Close implicit object if comma is the last character and what comes after doesn’t look like it belongs. This is used for trailing commas and calls, like:
x =
a: b,
c: d,
e = 2
and
f a, b: c, d: e, f, g: h: i, j
iftagis','andnot@looksObjectish(i +1)andinImplicitObject()andnot(@tag(i +2)in['FOROF','FORIN'])and(nextTagisnt'TERMINATOR'ornot@looksObjectish(i +2))
When nextTag is OUTDENT the comma is insignificant and should just be ignored so embed it in the implicit object.
When it isn’t the comma go on to play a role in a call or array further up the stack, so give it a chance.
offset =ifnextTagis'OUTDENT'then1else0whileinImplicitObject()
endImplicitObject i + offsetreturnforward(1)
Make sure only strings and wrapped expressions are used in JSX attributes.
enforceValidJSXAttributes:-\>@scanTokens (token, i, tokens) ->iftoken.jsxColon
next = tokens[i +1]ifnext[0]notin['STRING\_START','STRING','(']
throwSyntaxError'expected wrapped or quoted JSX attribute', next[2]return1
Not all tokens survive processing by the parser. To avoid comments getting lost into the ether, find comments attached to doomed tokens and move them to a token that will make it to the other side.
rescueStowawayComments:-\>insertPlaceholder = (token, j, tokens, method) -\>tokens[method] generate'TERMINATOR','\n', tokens[j]unlesstokens[j][0]is'TERMINATOR'tokens[method] generate'JS','', tokens[j], token dontShiftForward = (i, tokens) -\>j = i +1whilejisnttokens.lengthandtokens[j][0]inDISCARDEDreturnyesiftokens[j][0]is'INTERPOLATION\_END'j++no shiftCommentsForward = (token, i, tokens) -\>
Find the next surviving token and attach this token’s comments to it, with a flag that we know to output such comments before that token’s own compilation. (Otherwise comments are output following the token they’re attached to.)
j = i
j++whilejisnttokens.lengthandtokens[j][0]inDISCARDEDunlessjistokens.lengthortokens[j][0]inDISCARDED
comment.unshift =yesforcommentintoken.comments
moveComments token, tokens[j]return1else# All following tokens are doomed!j = tokens.length -1insertPlaceholder token, j, tokens,'push'
The generated tokens were added to the end, not inline, so we don’t skip.
return1 shiftCommentsBackward = (token, i, tokens) -\>
Find the last surviving token and attach this token’s comments to it.
j = i
j--whilejisnt-1andtokens[j][0]inDISCARDEDunlessjis-1ortokens[j][0]inDISCARDED
moveComments token, tokens[j]return1else# All previous tokens are doomed!insertPlaceholder token,0, tokens,'unshift'
We added two tokens, so shift forward to account for the insertion.
return3@scanTokens (token, i, tokens) ->return1unlesstoken.comments
ret =1iftoken[0]inDISCARDED
This token won’t survive passage through the parser, so we need to rescue its attached tokens and redistribute them to nearby tokens. Comments that don’t start a new line can shift backwards to the last safe token, while other tokens should shift forward.
dummyToken = comments: []
j = token.comments.length -1untiljis-1iftoken.comments[j].newLineisnoandtoken.comments[j].hereisnodummyToken.comments.unshift token.comments[j]
token.comments.splice j,1j--ifdummyToken.comments.lengthisnt0ret = shiftCommentsBackward dummyToken, i -1, tokensiftoken.comments.lengthisnt0shiftCommentsForward token, i, tokenselseunlessdontShiftForward i, tokens
If any of this token’s comments start a line—there’s only whitespace between the preceding newline and the start of the comment—and this isn’t one of the special JS tokens, then shift this comment forward to precede the next valid token. Block.compileComments also has logic to make sure that “starting new line” comments follow or precede the nearest newline relative to the token that the comment is attached to, but that newline might be inside a } or ) or other generated token that we really want this comment to output after. Therefore we need to shift the comments here, avoiding such generated and discarded tokens.
dummyToken = comments: []
j = token.comments.length -1untiljis-1iftoken.comments[j].newLineandnottoken.comments[j].unshiftandnot(token[0]is'JS'andtoken.generated)
dummyToken.comments.unshift token.comments[j]
token.comments.splice j,1j--ifdummyToken.comments.lengthisnt0ret = shiftCommentsForward dummyToken, i +1, tokensdeletetoken.commentsiftoken.comments?.lengthis0ret
Add location data to all tokens generated by the rewriter.
addLocationDataToGeneratedTokens:-\>@scanTokens (token, i, tokens) ->return1iftoken[2]return1unlesstoken.generatedortoken.explicitiftoken.fromThenandtoken[0]is'INDENT'token[2] = token.origin[2]return1iftoken[0]is'{'andnextLocation=tokens[i +1]?[2]
{first_line: line, first_column: column, range: [rangeIndex]} = nextLocationelseifprevLocation = tokens[i -1]?[2]
{last_line: line, last_column: column, range: [, rangeIndex]} = prevLocation
column +=1elseline = column =0rangeIndex =0token[2] = {
first_line: line
first_column: column
last_line: line
last_column: column
last_line_exclusive: line
last_column_exclusive: column
range: [rangeIndex, rangeIndex]
}return1
OUTDENT tokens should always be positioned at the last character of the previous token, so that AST nodes ending in an OUTDENT token end up with a location corresponding to the last “real” token under the node.
fixIndentationLocationData:-\>@allComments ?= extractAllCommentTokens @tokensfindPrecedingComment = (token, {afterPosition, indentSize, first, indented}) =\>tokenStart = token[2].range[0]matches = (comment) -\>ifcomment.outdentedreturnnounlessindentSize?andcomment.indentSize > indentSizereturnnoifindentedandnotcomment.indentedreturnnounlesscomment.locationData.range[0] < tokenStartreturnnounlesscomment.locationData.range[0] > afterPositionyesiffirst
lastMatching =nullforcommentin@allCommentsby-1ifmatches comment
lastMatching = commentelseiflastMatchingreturnlastMatchingreturnlastMatchingforcommentin@allCommentswhenmatches commentby-1returncommentnull@scanTokens (token, i, tokens) ->return1unlesstoken[0]in['INDENT','OUTDENT']or(token.generatedandtoken[0]is'CALL\_END'andnottoken.data?.closingTagNameToken)or(token.generatedandtoken[0]is'}')
isIndent = token[0]is'INDENT'prevToken = token.prevToken ? tokens[i -1]
prevLocationData = prevToken[2]
addLocationDataToGeneratedTokens() set the outdent’s location data to the preceding token’s, but in order to detect comments inside an empty “block” we want to look for comments preceding the next token.
useNextToken = token.explicitortoken.generatedifuseNextToken
nextToken = token
nextTokenIndex = i
nextToken = tokens[nextTokenIndex++]while(nextToken.explicitornextToken.generated)andnextTokenIndexisnttokens.length -1precedingComment = findPrecedingComment(ifuseNextToken
nextTokenelsetoken
afterPosition: prevLocationData.range[0]
indentSize: token.indentSize
first: isIndent
indented: useNextToken
)ifisIndentreturn1unlessprecedingComment?.newLine
We don’t want e.g. an implicit call at the end of an if condition to include a following indented comment.
return1iftoken.generatedandtoken[0]is'CALL\_END'andprecedingComment?.indented
prevLocationData = precedingComment.locationDataifprecedingComment?
token[2] =
first_line:ifprecedingComment?
prevLocationData.first_lineelseprevLocationData.last_line
first_column:ifprecedingComment?ifisIndent0elseprevLocationData.first_columnelseprevLocationData.last_column
last_line: prevLocationData.last_line
last_column: prevLocationData.last_column
last_line_exclusive: prevLocationData.last_line_exclusive
last_column_exclusive: prevLocationData.last_column_exclusive
range:ifisIndentandprecedingComment?
[
prevLocationData.range[0] - precedingComment.indentSize
prevLocationData.range[1]
]elseprevLocationData.rangereturn1
Because our grammar is LALR(1), it can’t handle some single-line expressions that lack ending delimiters. The Rewriter adds the implicit blocks, so it doesn’t need to. To keep the grammar clean and tidy, trailing newlines within expressions are removed and the indentation tokens of empty blocks are added.
normalizeLines:-\>starter = indent = outdent =nullleading_switch_when =nullleading_if_then =null
Count THEN tags
ifThens = [] condition = (token, i) -\>token[1]isnt';'andtoken[0]inSINGLE_CLOSERSandnot(token[0]is'TERMINATOR'and@tag(i +1)inEXPRESSION_CLOSE)andnot(token[0]is'ELSE'and(starterisnt'THEN'or(leading_if_thenorleading_switch_when)))andnot(token[0]in['CATCH','FINALLY']andstarterin['-\>','=\>'])ortoken[0]inCALL_CLOSERSand(@tokens[i -1].newLineor@tokens[i -1][0]is'OUTDENT') action = (token, i) -\>ifThens.pop()iftoken[0]is'ELSE'andstarteris'THEN'@tokens.splice (if@tag(i -1)is','theni -1elsei),0, outdent closeElseTag = (tokens, i) =\>tlen = ifThens.lengthreturniunlesstlen >0lastThen = ifThens.pop()
[, outdentElse] = @indentation tokens[lastThen]
Insert OUTDENT to close inner IF.
outdentElse[1] = tlen*2tokens.splice(i,0, outdentElse)
Insert OUTDENT to close outer IF.
outdentElse[1] =2tokens.splice(i +1,0, outdentElse)
Remove outdents from the end.
@detectEnd i +2,(token, i) -\>token[0]in['OUTDENT','TERMINATOR']
(token, i) ->if@tag(i)is'OUTDENT'and@tag(i +1)is'OUTDENT'tokens.splice i,2i +2@scanTokens (token, i, tokens) ->
[tag] = token
conditionTag = tagin['-\>','=\>']and@findTagsBackwards(i, ['IF','WHILE','FOR','UNTIL','SWITCH','WHEN','LEADING\_WHEN','[','INDEX\_START'])andnot(@findTagsBackwards i, ['THEN','..','...'])iftagis'TERMINATOR'if@tag(i +1)is'ELSE'and@tag(i -1)isnt'OUTDENT'tokens.splice i,1, @indentation()...return1if@tag(i +1)inEXPRESSION_CLOSEiftoken[1]is';'and@tag(i +1)is'OUTDENT'tokens[i +1].prevToken = token
moveComments token, tokens[i +1]
tokens.splice i,1return0iftagis'CATCH'forjin[1..2]when@tag(i + j)in['OUTDENT','TERMINATOR','FINALLY']
tokens.splice i + j,0, @indentation()...return2+ jiftagin['-\>','=\>']and(@tag(i +1)in[',',']']or@tag(i +1)is'.'andtoken.newLine)
[indent, outdent] = @indentation tokens[i]
tokens.splice i +1,0, indent, outdentreturn1iftaginSINGLE_LINERSand@tag(i +1)isnt'INDENT'andnot(tagis'ELSE'and@tag(i +1)is'IF')andnotconditionTag
starter = tag
[indent, outdent] = @indentation tokens[i]
indent.fromThen =trueifstarteris'THEN'iftagis'THEN'leading_switch_when = @findTagsBackwards(i, ['LEADING\_WHEN'])and@tag(i +1)is'IF'leading_if_then = @findTagsBackwards(i, ['IF'])and@tag(i +1)is'IF'ifThens.push iiftagis'THEN'and@findTagsBackwards(i, ['IF'])
ELSE tag is not closed.
iftagis'ELSE'and@tag(i -1)isnt'OUTDENT'i = closeElseTag tokens, i
tokens.splice i +1,0, indent
@detectEnd i +2, condition, action
tokens.splice i,1iftagis'THEN'return1return1
Tag postfix conditionals as such, so that we can parse them with a different precedence.
tagPostfixConditionals:-\>original =null condition = (token, i) -\>[tag] = token
[prevTag] = @tokens[i -1]
tagis'TERMINATOR'or(tagis'INDENT'andprevTagnotinSINGLE_LINERS) action = (token, i) -\>iftoken[0]isnt'INDENT'or(token.generatedandnottoken.fromThen)
original[0] ='POST\_'+ original[0]
@scanTokens (token, i) ->return1unlesstoken[0]is'IF'original = token
@detectEnd i +1, condition, actionreturn1
For tokens with extra data, we want to make that data visible to the grammar by wrapping the token value as a String() object and setting the data as properties of that object. The grammar should then be responsible for cleaning this up for the node constructor: unwrapping the token value to a primitive string and separately passing any expected token data properties
exposeTokenDataToGrammar:-\>@scanTokens (token, i) ->iftoken.generatedor(token.dataandObject.keys(token.data).lengthisnt0)
token[1] =newStringtoken[1]
token[1][key] = valforown key, valof(token.data ? {})
token[1].generated =yesiftoken.generated1
Generate the indentation tokens, based on another token on the same line.
indentation:(origin) -\>indent = ['INDENT',2]
outdent = ['OUTDENT',2]iforigin
indent.generated = outdent.generated =yesindent.origin = outdent.origin = originelseindent.explicit = outdent.explicit =yes[indent, outdent]
generate: generate
Look up a tag by token index.
tag:(i) -\>@tokens[i]?[0]
List of the token pairs that must be balanced.
BALANCED_PAIRS = [
['(',')']
['[',']']
['{','}']
['INDENT','OUTDENT'],
['CALL\_START','CALL\_END']
['PARAM\_START','PARAM\_END']
['INDEX\_START','INDEX\_END']
['STRING\_START','STRING\_END']
['INTERPOLATION\_START','INTERPOLATION\_END']
['REGEX\_START','REGEX\_END']
]
The inverse mappings of BALANCED_PAIRS we’re trying to fix up, so we can look things up from either end.
exports.INVERSES = INVERSES = {}
The tokens that signal the start/end of a balanced pair.
EXPRESSION_START = []
EXPRESSION_END = []for[left, right]inBALANCED_PAIRS
EXPRESSION_START.push INVERSES[right] = left
EXPRESSION_END .push INVERSES[left] = right
Tokens that indicate the close of a clause of an expression.
EXPRESSION_CLOSE = ['CATCH','THEN','ELSE','FINALLY'].concat EXPRESSION_END
Tokens that, if followed by an IMPLICIT_CALL, indicate a function invocation.
IMPLICIT_FUNC = ['IDENTIFIER','PROPERTY','SUPER',')','CALL\_END',']','INDEX\_END','@','THIS']
If preceded by an IMPLICIT_FUNC, indicates a function invocation.
IMPLICIT_CALL = ['IDENTIFIER','JSX\_TAG','PROPERTY','NUMBER','INFINITY','NAN''STRING','STRING\_START','REGEX','REGEX\_START','JS''NEW','PARAM\_START','CLASS','IF','TRY','SWITCH','THIS''DYNAMIC\_IMPORT','IMPORT\_META','NEW\_TARGET''UNDEFINED','NULL','BOOL''UNARY','DO','DO\_IIFE','YIELD','AWAIT','UNARY\_MATH','SUPER','THROW''@','-\>','=\>','[','(','{','--','++']
IMPLICIT_UNSPACED_CALL = ['+','-']
Tokens that always mark the end of an implicit call for single-liners.
IMPLICIT_END = ['POST\_IF','FOR','WHILE','UNTIL','WHEN','BY','LOOP','TERMINATOR']
Single-line flavors of block expressions that have unclosed endings. The grammar can’t disambiguate them, so we insert the implicit indentation.
SINGLE_LINERS = ['ELSE','-\>','=\>','TRY','FINALLY','THEN']
SINGLE_CLOSERS = ['TERMINATOR','CATCH','FINALLY','ELSE','OUTDENT','LEADING\_WHEN']
Tokens that end a line.
LINEBREAKS = ['TERMINATOR','INDENT','OUTDENT']
Tokens that close open calls when they follow a newline.
CALL_CLOSERS = ['.','?.','::','?::']
Tokens that prevent a subsequent indent from ending implicit calls/objects
CONTROL_IN_IMPLICIT = ['IF','TRY','FINALLY','CATCH','CLASS','SWITCH']
Tokens that are swallowed up by the parser, never leading to code generation. You can spot these in grammar.coffee because the o function second argument doesn’t contain a new call for these tokens. STRING_START isn’t on this list because its locationData matches that of the node that becomes StringWithInterpolations, and therefore addDataToNode attaches STRING_START’s tokens to that node.
DISCARDED = ['(',')','[',']','{','}',':','.','..','...',',','=','++','--','?','AS','AWAIT','CALL\_START','CALL\_END','DEFAULT','DO','DO\_IIFE','ELSE','EXTENDS','EXPORT','FORIN','FOROF','FORFROM','IMPORT','INDENT','INDEX\_SOAK','INTERPOLATION\_START','INTERPOLATION\_END','LEADING\_WHEN','OUTDENT','PARAM\_END','REGEX\_START','REGEX\_END','RETURN','STRING\_END','THROW','UNARY','YIELD'].concat IMPLICIT_UNSPACED_CALL.concat IMPLICIT_END.concat CALL_CLOSERS.concat CONTROL_IN_IMPLICIT
Tokens that, when appearing at the end of a line, suppress a following TERMINATOR/INDENT token
exports.UNFINISHED = UNFINISHED = ['\\','.','?.','?::','UNARY','DO','DO\_IIFE','MATH','UNARY\_MATH','+','-','\*\*','SHIFT','RELATION','COMPARE','&','^','|','&&','||','BIN?','EXTENDS']