Jump To …+

browser.coffee cake.coffee coffee-script.coffee command.coffee grammar.coffee helpers.coffee index.coffee lexer.coffee nodes.coffee optparse.coffee register.coffee repl.coffee rewriter.coffee scope.litcoffee sourcemap.litcoffee

lexer.coffee

The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt matches against the beginning of the source code. When a match is found, a token is produced, we consume the match, and start again. Tokens are in the form:

[tag, value, locationData]

where locationData is {first_line, first_column, last_line, last_column}, which is a format that can be fed directly into Jison. These are read by jison in the parser.lexer function defined in coffee-script.coffee.

{Rewriter, INVERSES} =require'./rewriter'

Import the helpers we need.

{count, starts, compact, repeat, invertLiterate,
locationDataToString, throwSyntaxError} =require'./helpers'

The Lexer Class

The Lexer class reads a stream of CoffeeScript and divvies it up into tagged tokens. Some potential ambiguity in the grammar has been avoided by pushing some extra smarts into the Lexer.

exports.Lexer =class Lexer

tokenize is the Lexer’s main method. Scan by attempting to match tokens one at a time, using a regular expression anchored at the start of the remaining code, or a custom recursive token-matching method (for interpolations). When the next token has been recorded, we move forward within the code past the token, and begin again.

Each tokenizing method is responsible for returning the number of characters it has consumed.

Before returning the token stream, run it through the Rewriter.

tokenize:(code, opts = {}) -\>@literate = opts.literate# Are we lexing literate CoffeeScript?@indent =0# The current indentation level.@baseIndent =0# The overall minimum indentation level@indebt =0# The over-indentation at the current level.@outdebt =0# The under-outdentation at the current level.@indents = []# The stack of all current indentation levels.@ends = []# The stack for pairing up tokens.@tokens = []# Stream of parsed tokens in the form `['TYPE', value, location data]`.@seenFor =no# Used to recognize FORIN, FOROF and FORFROM tokens.@seenImport =no# Used to recognize IMPORT FROM? AS? tokens.@seenExport =no# Used to recognize EXPORT FROM? AS? tokens.@importSpecifierList =no# Used to identify when in an IMPORT {...} FROM? ...@exportSpecifierList =no# Used to identify when in an EXPORT {...} FROM? ...@chunkLine =
      opts.lineor0# The start line for the current @chunk.@chunkColumn =
      opts.columnor0# The start column of the current @chunk.code = @clean code# The stripped, cleaned original source code.

At every position, run through this list of attempted matches, short-circuiting if any of them succeed. Their order determines precedence: @literalToken is the fallback catch-all.

i =0while@chunk = code[i..]
      consumed = \
           @identifierToken()or@commentToken()or@whitespaceToken()or@lineToken()or@stringToken()or@numberToken()or@regexToken()or@jsToken()or@literalToken()

Update position

[@chunkLine, @chunkColumn] = @getLineAndColumnFromChunk consumed

      i += consumedreturn{@tokens, index: i}[email protected]@closeIndentation()
    @error"missing #{end.tag}", end.origin[2]ifend = @ends.pop()[email protected](newRewriter).rewrite @tokens

Preprocess the code to remove leading and trailing whitespace, carriage returns, etc. If we’re lexing literate CoffeeScript, strip external Markdown by removing all lines that aren’t indented by at least four spaces or a tab.

clean:(code) -\>code = code.slice(1)ifcode.charCodeAt(0)isBOM
    code = code.replace(/\r/g,'').replace TRAILING_SPACES,''ifWHITESPACE.test code
      code ="\n#{code}"@chunkLine--
    code = invertLiterate codeif@literate
    code

Tokenizers

Matches identifying literals: variables, keywords, method names, etc. Check to ensure that JavaScript reserved words aren’t being used as identifiers. Because CoffeeScript reserves a handful of keywords that are allowed in JavaScript, we’re careful not to tag them as keywords when referenced as property names here, so you can still do jQuery.is() even though is means === otherwise.

identifierToken:-\>return0unlessmatch = IDENTIFIER.exec @chunk
    [input, id, colon] = match

Preserve length of id for location data

idLength = id.length
    poppedToken =undefinedifidis'own'and@tag()is'FOR'@token'OWN', idreturnid.lengthifidis'from'and@tag()is'YIELD'@token'FROM', idreturnid.lengthifidis'as'and@seenImportif@value()is'\*'@tokens[@tokens.length -1][0] ='IMPORT\_ALL'elseif@value()inCOFFEE_KEYWORDS
        @tokens[@tokens.length -1][0] ='IDENTIFIER'if@tag()in['DEFAULT','IMPORT\_ALL','IDENTIFIER']
        @token'AS', idreturnid.lengthifidis'as'and@seenExportand@tag()in['IDENTIFIER','DEFAULT']
      @token'AS', idreturnid.lengthifidis'default'and@seenExportand@tag()in['EXPORT','AS']
      @token'DEFAULT', idreturnid.length

    [..., prev] = @tokens

    tag =ifcolonorprev?and(prev[0]in['.','?.','::','?::']ornotprev.spacedandprev[0]is'@')'PROPERTY'else'IDENTIFIER'iftagis'IDENTIFIER'and(idinJS_KEYWORDSoridinCOFFEE_KEYWORDS)andnot(@exportSpecifierListandidinCOFFEE_KEYWORDS)
      tag = id.toUpperCase()iftagis'WHEN'and@tag()inLINE_BREAK
        tag ='LEADING\_WHEN'elseiftagis'FOR'@seenFor =yeselseiftagis'UNLESS'tag ='IF'elseiftagis'IMPORT'@seenImport =yeselseiftagis'EXPORT'@seenExport =yeselseiftaginUNARY
        tag ='UNARY'elseiftaginRELATIONiftagisnt'INSTANCEOF'and@seenFor
          tag ='FOR'+ tag
          @seenFor =noelsetag ='RELATION'if@value()is'!'poppedToken = @tokens.pop()
            id ='!'+ idelseiftagis'IDENTIFIER'and@seenForandidis'from'andisForFrom(prev)
      tag ='FORFROM'@seenFor =noiftagis'IDENTIFIER'andidinRESERVED
      @error"reserved word '#{id}'", length: id.lengthunlesstagis'PROPERTY'ifidinCOFFEE_ALIASES
        alias = id
        id = COFFEE_ALIAS_MAP[id]
      tag =switchidwhen'!'then'UNARY'when'==','!='then'COMPARE'when'true','false'then'BOOL'when'break','continue', \'debugger'then'STATEMENT'when'&&','||'thenidelsetag

    tagToken = @token tag, id,0, idLength
    tagToken.origin = [tag, alias, tagToken[2]]ifaliasifpoppedToken
      [tagToken[2].first_line, tagToken[2].first_column] =
        [poppedToken[2].first_line, poppedToken[2].first_column]ifcolon
      colonOffset = input.lastIndexOf':'@token':',':', colonOffset, colon.length

    input.length

Matches numbers, including decimals, hex, and exponential notation. Be careful not to interfere with ranges-in-progress.

numberToken:-\>return0unlessmatch = NUMBER.exec @chunk

    number = match[0]
    lexedLength = number.lengthswitchwhen/^0[BOX]/.test number
        @error"radix prefix in '#{number}' must be lowercase", offset:1when/^(?!0x).\*E/.test number
        @error"exponential notation in '#{number}' must be indicated with a lowercase 'e'",
          offset: number.indexOf('E')when/^0\d\*[89]/.test number
        @error"decimal literal '#{number}' must not be prefixed with '0'", length: lexedLengthwhen/^0\d+/.test number
        @error"octal literal '#{number}' must be prefixed with '0o'", length: lexedLength

    base =switchnumber.charAt1when'b'then2when'o'then8when'x'then16elsenullnumberValue =ifbase?thenparseInt(number[2..], base)elseparseFloat(number)ifnumber.charAt(1)in['b','o']
      number ="0x#{numberValue.toString 16}"tag =ifnumberValueisInfinitythen'INFINITY'else'NUMBER'@token tag, number,0, lexedLength
    lexedLength

Matches strings, including multi-line strings, as well as heredocs, with or without interpolation.

stringToken:-\>[quote] = STRING_START.exec(@chunk) || []return0unlessquote

If the preceding token is from and this is an import or export statement, properly tag the from.

[email protected]@value()is'from'and(@seenImportor@seenExport)
      @tokens[@tokens.length -1][0] ='FROM'regex =switchquotewhen"'"thenSTRING_SINGLEwhen'"'thenSTRING_DOUBLEwhen"'''"thenHEREDOC_SINGLEwhen'"""'thenHEREDOC_DOUBLE
    heredoc = quote.lengthis3{tokens, index: end} = @matchWithInterpolations regex, quote
    $ = tokens.length -1delimiter = quote.charAt(0)ifheredoc

Find the smallest indentation. It will be removed from all lines later.

indent =nulldoc = (token[1]fortoken, iintokenswhentoken[0]is'NEOSTRING').join'#{}'whilematch = HEREDOC_INDENT.exec doc
        attempt = match[1]
        indent = attemptifindentisnullor0< attempt.length < indent.length
      indentRegex =/// \n#{indent} ///gifindent
      @mergeInterpolationTokens tokens, {delimiter},(value, i) =\>value = @formatString value, delimiter: quote
        value = value.replace indentRegex,'\n'ifindentRegex
        value = value.replace LEADING_BLANK_LINE,''ifiis0value = value.replace TRAILING_BLANK_LINE,''ifiis$
        valueelse@mergeInterpolationTokens tokens, {delimiter},(value, i) =\>value = @formatString value, delimiter: quote
        value = value.replace SIMPLE_STRING_OMIT,(match, offset) -\>if(iis0andoffsetis0)or(iis$andoffset + match.lengthisvalue.length)''else' 'value

    end

Matches and consumes comments.

commentToken:-\>return0unlessmatch = @chunk.match COMMENT
    [comment, here] = matchifhereifmatch = HERECOMMENT_ILLEGAL.exec comment
        @error"block comments cannot contain #{match[0]}",
          offset: match.index, length: match[0].lengthifhere.indexOf('\n') >=0here = here.replace/// \n #{repeat ' ', @indent} ///g,'\n'@token'HERECOMMENT', here,0, comment.length
    comment.length

Matches JavaScript interpolated directly into the source via backticks.

jsToken:-\>[email protected](0)is'`'and(match = HERE_JSTOKEN.exec(@chunk)orJSTOKEN.exec(@chunk))

Convert escaped backticks to backticks, and escaped backslashes just before escaped backticks to backslashes

script = match[1].replace/\\+(`|$)/g,(string) -\>

string is always a value like ‘‘, ‘\‘, ‘\‘, etc. By reducing it to its latter half, we turn ‘‘ to ‘', '\\\‘ to ‘`‘, etc.

string[-Math.ceil(string.length /2)..]
    @token'JS', script,0, match[0].length
    match[0].length

Matches regular expression literals, as well as multiline extended ones. Lexing regular expressions is difficult to distinguish from division, so we borrow some basic heuristics from JavaScript and Ruby.

regexToken:-\>switchwhenmatch = REGEX_ILLEGAL.exec @chunk
        @error"regular expressions cannot begin with #{match[2]}",
          offset: match.index + match[1].lengthwhenmatch = @matchWithInterpolations HEREGEX,'///'{tokens, index} = matchwhenmatch = REGEX.exec @chunk
        [regex, body, closed] = match
        @validateEscapes body, isRegex:yes, offsetInChunk:1body = @formatRegex body, delimiter:'/'index = regex.length
        [..., prev] = @tokensifprevifprev.spacedandprev[0]inCALLABLEreturn0ifnotclosedorPOSSIBLY_DIVISION.test regexelseifprev[0]inNOT_REGEXreturn0@error'missing / (unclosed regex)'unlessclosedelsereturn0[flags] = REGEX_FLAGS.exec @chunk[index..]
    end = index + flags.length
    origin = @makeToken'REGEX',null,0, endswitchwhennotVALID_FLAGS.test flags
        @error"invalid regular expression flags #{flags}", offset: index, length: flags.lengthwhenregexortokens.lengthis1body ?= @formatHeregex tokens[0][1]
        @token'REGEX',"#{@makeDelimitedLiteral body, delimiter: '/'}#{flags}",0, end, originelse@token'REGEX\_START','(',0,0, origin
        @token'IDENTIFIER','RegExp',0,0@token'CALL\_START','(',0,0@mergeInterpolationTokens tokens, {delimiter:'"', double:yes}, @formatHeregexifflags
          @token',',',', index -1,0@token'STRING','"'+ flags +'"', index -1, flags.length
        @token')',')', end -1,0@token'REGEX\_END',')', end -1,0end

Matches newlines, indents, and outdents, and determines which is which. If we can detect that the current line is continued onto the next line, then the newline is suppressed:

elements
  .each( ... )
  .map( ... )

Keeps track of the level of indentation, because a single outdent token can close multiple indents, so we need to know how far in we happen to be.

lineToken:-\>return0unlessmatch = MULTI_DENT.exec @chunk
    indent = match[0]

    @seenFor =no@seenImport =nounless@importSpecifierList
    @seenExport =nounless@exportSpecifierList

    size = indent.length -1- indent.lastIndexOf'\n'noNewlines = @unfinished()ifsize - @indebtis@indentifnoNewlinesthen@suppressNewlines()[email protected] > @indentifnoNewlines
        @indebt = size - @indent
        @suppressNewlines()[email protected]
        @baseIndent = @indent = sizereturnindent.length
      diff = size - @indent + @outdebt
      @token'INDENT', diff, indent.length - size, size
      @indents.push diff
      @ends.push {tag:'OUTDENT'}
      @outdebt = @indebt =0@indent = sizeelseifsize < @baseIndent
      @error'missing indentation', offset: indent.lengthelse@indebt =0@outdentToken @indent - size, noNewlines, indent.length
    indent.length

Record an outdent token or multiple tokens, if we happen to be moving back inwards past several recorded indents. Sets new @indent value.

outdentToken:(moveOut, noNewlines, outdentLength) -\>decreasedIndent = @indent - moveOutwhilemoveOut >0lastIndent = @indents[@indents.length -1]ifnotlastIndent
        moveOut =0elseiflastIndentis@outdebt
        moveOut -= @outdebt
        @outdebt =0elseiflastIndent < @outdebt
        @outdebt -= lastIndent
        moveOut -= lastIndentelsedent = @indents.pop() + @outdebtifoutdentLengthand@chunk[outdentLength]inINDENTABLE_CLOSERS
          decreasedIndent -= dent - moveOut
          moveOut = dent
        @outdebt =0

pair might call outdentToken, so preserve decreasedIndent

@pair'OUTDENT'@token'OUTDENT', moveOut,0, outdentLength
        moveOut -= dent
    @outdebt -= moveOutifdent
    @tokens.pop()while@value()is';'@token'TERMINATOR','\n', outdentLength,0unless@tag()is'TERMINATOR'ornoNewlines
    @indent = decreasedIndentthis

Matches and consumes non-meaningful whitespace. Tag the previous token as being “spaced”, because there are some cases where it makes a difference.

whitespaceToken:-\>return0unless(match = WHITESPACE.exec @chunk)or(nline = @chunk.charAt(0)is'\n')
    [..., prev] = @tokens
    prev[ifmatchthen'spaced'else'newLine'] =trueifprevifmatchthenmatch[0].lengthelse0

Generate a newline token. Consecutive newlines get merged together.

newlineToken:(offset) -\>@tokens.pop()while@value()is';'@token'TERMINATOR','\n', offset,0unless@tag()is'TERMINATOR'this

Use a \ at a line-ending to suppress the newline. The slash is removed here once its job is done.

suppressNewlines:-\>@tokens.pop()if@value()is'\\'this

We treat all other single characters as a token. E.g.: ( ) , . ! Multi-character operators are also literal tokens, so that Jison can assign the proper order of operations. There are some symbols that we tag specially here. ; and newlines are both treated as a TERMINATOR, we distinguish parentheses that indicate a method call from regular parentheses, and so on.

literalToken:-\>ifmatch = OPERATOR.exec @chunk
      [value] = match
      @tagParameters()ifCODE.test valueelsevalue = @chunk.charAt0tag = value
    [..., prev] = @tokensifprevandvaluein['=', COMPOUND_ASSIGN...]
      skipToken =falseifvalueis'='andprev[1]in['||','&&']andnotprev.spaced
        prev[0] ='COMPOUND\_ASSIGN'prev[1] +='='prev = @tokens[@tokens.length -2]
        skipToken =trueifprevandprev[0]isnt'PROPERTY'origin = prev.origin ? prev
        message = isUnassignable prev[1], origin[1]
        @error message, origin[2]ifmessagereturnvalue.lengthifskipTokenifvalueis'{'and@seenImport
      @importSpecifierList =yeselseif@importSpecifierListandvalueis'}'@importSpecifierList =noelseifvalueis'{'andprev?[0]is'EXPORT'@exportSpecifierList =yeselseif@exportSpecifierListandvalueis'}'@exportSpecifierList =noifvalueis';'@seenFor = @seenImport = @seenExport =notag ='TERMINATOR'elseifvalueis'\*'andprev[0]is'EXPORT'tag ='EXPORT\_ALL'elseifvalueinMATHthentag ='MATH'elseifvalueinCOMPAREthentag ='COMPARE'elseifvalueinCOMPOUND_ASSIGNthentag ='COMPOUND\_ASSIGN'elseifvalueinUNARYthentag ='UNARY'elseifvalueinUNARY_MATHthentag ='UNARY\_MATH'elseifvalueinSHIFTthentag ='SHIFT'elseifvalueis'?'andprev?.spacedthentag ='BIN?'elseifprevandnotprev.spacedifvalueis'('andprev[0]inCALLABLE
        prev[0] ='FUNC\_EXIST'ifprev[0]is'?'tag ='CALL\_START'elseifvalueis'['andprev[0]inINDEXABLE
        tag ='INDEX\_START'switchprev[0]when'?'thenprev[0] ='INDEX\_SOAK'token = @makeToken tag, valueswitchvaluewhen'(','{','['[email protected] {tag: INVERSES[value], origin: token}when')','}',']'then@pair value
    @tokens.push token
    value.length

Token Manipulators

A source of ambiguity in our grammar used to be parameter lists in function definitions versus argument lists in function calls. Walk backwards, tagging parameters specially in order to make things easier for the parser.

tagParameters:-\>returnthisif@tag()isnt')'stack = []
    {tokens} =thisi = tokens.length
    tokens[--i][0] ='PARAM\_END'whiletok = tokens[--i]switchtok[0]when')'stack.push tokwhen'(','CALL\_START'ifstack.lengththenstack.pop()elseiftok[0]is'('tok[0] ='PARAM\_START'returnthiselsereturnthisthis

Close up all remaining open blocks at the end of the file.

closeIndentation:-\>@outdentToken @indent

Match the contents of a delimited token and expand variables and expressions inside it using Ruby-like notation for substitution of arbitrary expressions.

"Hello #{name.capitalize()}."

If it encounters an interpolation, this method will recursively create a new Lexer and tokenize until the { of #{ is balanced with a }.

- `regex` matches the contents of a token (but not `delimiter`, and not `#{` if interpolations are desired).
- `delimiter` is the delimiter of the token. Examples are `'`, `"`, `'''`, `"""` and `///`.

This method allows us to have strings within interpolations within strings, ad infinitum.

matchWithInterpolations:(regex, delimiter) -\>tokens = []
    offsetInChunk = delimiter.lengthreturnnullunless@chunk[...offsetInChunk]isdelimiter
    str = @chunk[offsetInChunk..]loop[strPart] = regex.exec str

      @validateEscapes strPart, {isRegex: delimiter.charAt(0)is'/', offsetInChunk}

Push a fake ‘NEOSTRING’ token, which will get turned into a real string later.

tokens.push @makeToken'NEOSTRING', strPart, offsetInChunk

      str = str[strPart.length..]
      offsetInChunk += strPart.lengthbreakunlessstr[...2]is'#{'

The 1s are to remove the # in #{.

[line, column] = @getLineAndColumnFromChunk offsetInChunk +1{tokens: nested, index} =newLexer().tokenize str[1..], line: line, column: column, untilBalanced:on

Skip the trailing }.

index +=1

Turn the leading and trailing { and } into parentheses. Unnecessary parentheses will be removed later.

[open, ..., close] = nested
      open[0] = open[1] ='('close[0] = close[1] =')'close.origin = ['','end of interpolation', close[2]]

Remove leading ‘TERMINATOR’ (if any).

nested.splice1,1ifnested[1]?[0]is'TERMINATOR'

Push a fake ‘TOKENS’ token, which will get turned into real tokens later.

tokens.push ['TOKENS', nested]

      str = str[index..]
      offsetInChunk += indexunlessstr[...delimiter.length]isdelimiter
      @error"missing #{delimiter}", length: delimiter.length

    [firstToken, ..., lastToken] = tokens
    firstToken[2].first_column -= delimiter.lengthiflastToken[1].substr(-1)is'\n'lastToken[2].last_line +=1lastToken[2].last_column = delimiter.length -1elselastToken[2].last_column += delimiter.length
    lastToken[2].last_column -=1iflastToken[1].lengthis0{tokens, index: offsetInChunk + delimiter.length}

Merge the array tokens of the fake token types ‘TOKENS’ and ‘NEOSTRING’ (as returned by matchWithInterpolations) into the token stream. The value of ‘NEOSTRING’s are converted using fn and turned into strings using options first.

mergeInterpolationTokens:(tokens, options, fn) -\>iftokens.length >1lparen = @token'STRING\_START','(',0,0firstIndex = @tokens.lengthfortoken, iintokens
      [tag, value] = tokenswitchtagwhen'TOKENS'

Optimize out empty interpolations (an empty pair of parentheses).

continueifvalue.lengthis2

Push all the tokens in the fake ‘TOKENS’ token. These already have sane location data.

locationToken = value[0]
          tokensToPush = valuewhen'NEOSTRING'

Convert ‘NEOSTRING’ into ‘STRING’.

converted = fn.callthis, token[1], i

Optimize out empty strings. We ensure that the tokens stream always starts with a string token, though, to make sure that the result really is a string.

ifconverted.lengthis0ifiis0firstEmptyStringIndex = @tokens.lengthelsecontinue

However, there is one case where we can optimize away a starting empty string.

ifiis2andfirstEmptyStringIndex?
            @tokens.splice firstEmptyStringIndex,2# Remove empty string and the plus.token[0] ='STRING'token[1] = @makeDelimitedLiteral converted, options
          locationToken = token
          tokensToPush = [token][email protected] > firstIndex

Create a 0-length “+” token.

plusToken = @token'+','+'plusToken[2] =
          first_line: locationToken[2].first_line
          first_column: locationToken[2].first_column
          last_line: locationToken[2].first_line
          last_column: locationToken[2].first_column
      @tokens.push tokensToPush...iflparen
      [..., lastToken] = tokens
      lparen.origin = ['STRING',null,
        first_line: lparen[2].first_line
        first_column: lparen[2].first_column
        last_line: lastToken[2].last_line
        last_column: lastToken[2].last_column
      ]
      rparen = @token'STRING\_END',')'rparen[2] =
        first_line: lastToken[2].last_line
        first_column: lastToken[2].last_column
        last_line: lastToken[2].last_line
        last_column: lastToken[2].last_column

Pairs up a closing token, ensuring that all listed pairs of tokens are correctly balanced throughout the course of the token stream.

pair:(tag) -\>[..., prev] = @endsunlesstagiswanted = prev?.tag
      @error"unmatched #{tag}"unless'OUTDENT'iswanted

Auto-close INDENT to support syntax like this:

el.click((event) ->
  el.hide())

[..., lastIndent] = @indents
      @outdentToken lastIndent,truereturn@pair tag
    @ends.pop()

Helpers

Returns the line and column number from an offset into the current chunk.

offset is a number of characters into @chunk.

getLineAndColumnFromChunk:(offset) -\>ifoffsetis0return[@chunkLine, @chunkColumn]ifoffset >= @chunk.length
      string = @chunkelsestring = @chunk[..offset-1]

    lineCount = count string,'\n'column = @chunkColumniflineCount >0[..., lastLine] = string.split'\n'column = lastLine.lengthelsecolumn += string.length

    [@chunkLine + lineCount, column]

Same as “token”, exception this just returns the token without adding it to the results.

makeToken:(tag, value, offsetInChunk = 0, length = value.length) -\>locationData = {}
    [locationData.first_line, locationData.first_column] =
      @getLineAndColumnFromChunk offsetInChunk

Use length - 1 for the final offset - we’re supplying the last_line and the last_column, so if last_column == first_column, then we’re looking at a character of length 1.

lastCharacter =iflength >0then(length -1)else0[locationData.last_line, locationData.last_column] =
      @getLineAndColumnFromChunk offsetInChunk + lastCharacter

    token = [tag, value, locationData]

    token

Add a token to the results. offset is the offset into the current @chunk where the token starts. length is the length of the token in the @chunk, after the offset. If not specified, the length of value will be used.

Returns the new token.

token:(tag, value, offsetInChunk, length, origin) -\>token = @makeToken tag, value, offsetInChunk, length
    token.origin = originiforigin
    @tokens.push token
    token

Peek at the last tag in the token stream.

tag:-\>[..., token] = @tokens
    token?[0]

Peek at the last value in the token stream.

value:-\>[..., token] = @tokens
    token?[1]

Are we in the midst of an unfinished expression?

unfinished:-\>LINE_CONTINUER.test(@chunk)or@tag()inUNFINISHED

  formatString:(str, options) -\>@replaceUnicodeCodePointEscapes str.replace(STRING_OMIT,'$1'), options

  formatHeregex:(str) -\>@formatRegex str.replace(HEREGEX_OMIT,'$1$2'), delimiter:'///'formatRegex:(str, options) -\>@replaceUnicodeCodePointEscapes str, options

  unicodeCodePointToUnicodeEscapes:(codePoint) -\>toUnicodeEscape = (val) -\>str = val.toString16"\\u#{repeat '0', 4 - str.length}#{str}"returntoUnicodeEscape(codePoint)ifcodePoint <0x10000

surrogate pair

high = Math.floor((codePoint -0x10000) /0x400) +0xD800low = (codePoint -0x10000) %0x400+0xDC00"#{toUnicodeEscape(high)}#{toUnicodeEscape(low)}"

Replace \u{…} with \uxxxx[\uxxxx] in strings and regexes

replaceUnicodeCodePointEscapes:(str, options) -\>str.replace UNICODE_CODE_POINT_ESCAPE,(match, escapedBackslash, codePointHex, offset) =\>returnescapedBackslashifescapedBackslash

      codePointDecimal = parseInt codePointHex,16ifcodePointDecimal >0x10ffff@error"unicode code point escapes greater than \\u{10ffff} are not allowed",
          offset: offset + options.delimiter.length
          length: codePointHex.length +4@unicodeCodePointToUnicodeEscapes codePointDecimal

Validates escapes in strings and regexes.

validateEscapes:(str, options = {}) -\>invalidEscapeRegex =ifoptions.isRegex
        REGEX_INVALID_ESCAPEelseSTRING_INVALID_ESCAPE
    match = invalidEscapeRegex.exec strreturnunlessmatch
    [[], before, octal, hex, unicodeCodePoint, unicode] = match
    message =ifoctal"octal escape sequences are not allowed"else"invalid escape sequence"invalidEscape ="\\#{octal or hex or unicodeCodePoint or unicode}"@error"#{message} #{invalidEscape}",
      offset: (options.offsetInChunk ?0) + match.index + before.length
      length: invalidEscape.length

Constructs a string or regex by escaping certain characters.

makeDelimitedLiteral:(body, options = {}) -\>body ='(?:)'ifbodyis''andoptions.delimiteris'/'regex =/// (\\\\) # escaped backslash | (\\0(?=[1-7])) # nul character mistaken as octal escape | \\?(#{options.delimiter}) # (possibly escaped) delimiter | \\?(?: (\n)|(\r)|(\u2028)|(\u2029) ) # (possibly escaped) newlines | (\\.) # other escapes ///g
    body = body.replace regex,(match, backslash, nul, delimiter, lf, cr, ls, ps, other) -\>switch

Ignore escaped backslashes.

whenbackslashthen(ifoptions.doublethenbackslash + backslashelsebackslash)whennulthen'\\x00'whendelimiterthen"\\#{delimiter}"whenlfthen'\\n'whencrthen'\\r'whenlsthen'\\u2028'whenpsthen'\\u2029'whenotherthen(ifoptions.doublethen"\\#{other}"elseother)"#{options.delimiter}#{body}#{options.delimiter}"

Throws an error at either a given offset from the current chunk or at the location of a token (token[2]).

error:(message, options = {}) -\>location =if'first\_line'ofoptions
        optionselse[first_line, first_column] = @getLineAndColumnFromChunk options.offset ?0{first_line, first_column, last_column: first_column + (options.length ?1) -1}
    throwSyntaxError message, location

Helper functions

isUnassignable = (name, displayName = name) -\>switchwhennamein[JS_KEYWORDS..., COFFEE_KEYWORDS...]"keyword '#{displayName}' can't be assigned"whennameinSTRICT_PROSCRIBED"'#{displayName}' can't be assigned"whennameinRESERVED"reserved word '#{displayName}' can't be assigned"elsefalseexports.isUnassignable = isUnassignable

from isn’t a CoffeeScript keyword, but it behaves like one in import and export statements (handled above) and in the declaration line of a for loop. Try to detect when from is a variable identifier and when it is this “sometimes” keyword.

isForFrom = (prev) -\>ifprev[0]is'IDENTIFIER'

for i from from, for from from iterable

ifprev[1]is'from'prev[1][0] ='IDENTIFIER'yes

for i from iterable

yes

for from…

elseifprev[0]is'FOR'no

for {from}…, for [from]…, for {a, from}…, for {a: from}…

elseifprev[1]in['{','[',',',':']noelseyes

Constants

Keywords that CoffeeScript shares in common with JavaScript.

JS_KEYWORDS = ['true','false','null','this''new','delete','typeof','in','instanceof''return','throw','break','continue','debugger','yield''if','else','switch','for','while','do','try','catch','finally''class','extends','super''import','export','default']

CoffeeScript-only keywords.

COFFEE_KEYWORDS = ['undefined','Infinity','NaN''then','unless','until','loop','of','by','when']

COFFEE_ALIAS_MAP =and:'&&'or:'||'is:'=='isnt:'!='not:'!'yes:'true'no:'false'on:'true'off:'false'COFFEE_ALIASES = (keyforkeyofCOFFEE_ALIAS_MAP)
COFFEE_KEYWORDS = COFFEE_KEYWORDS.concat COFFEE_ALIASES

The list of keywords that are reserved by JavaScript, but not used, or are used by CoffeeScript internally. We throw an error when these are encountered, to avoid having a JavaScript error at runtime.

RESERVED = ['case','function','var','void','with','const','let','enum''native','implements','interface','package','private''protected','public','static']

STRICT_PROSCRIBED = ['arguments','eval']

The superset of both JavaScript keywords and reserved words, none of which may be used as identifiers or properties.

exports.JS_FORBIDDEN = JS_KEYWORDS.concat(RESERVED).concat(STRICT_PROSCRIBED)

The character code of the nasty Microsoft madness otherwise known as the BOM.

BOM =65279

Token matching regexes.

IDENTIFIER =/// ^ (?!\d) ( (?: (?!\s)[$\w\x7f-\uffff] )+ ) ( [^\n\S]\* : (?!:) )? # Is this a property name? ///NUMBER =/// ^ 0b[01]+ | # binary ^ 0o[0-7]+ | # octal ^ 0x[\da-f]+ | # hex ^ \d\*\.?\d+ (?:e[+-]?\d+)? # decimal ///i

OPERATOR =/// ^ ( ?: [-=]\> # function | [-+\*/%\<\>&|^!?=]= # compound assign / compare | \>\>\>=? # zero-fill right shift | ([-+:])\1 # doubles | ([&|\<\>\*/%])\2=? # logic / shift / power / floor division / modulo | \?(\.|::) # soak access | \.{2,3} # range or splat ) ///WHITESPACE =/^[^\n\S]+/COMMENT =/^###([^#][\s\S]\*?)(?:###[^\n\S]\*|###$)|^(?:\s\*#(?!##[^#]).\*)+/CODE =/^[-=]\>/MULTI_DENT =/^(?:\n[^\n\S]\*)+/JSTOKEN =///^ `(?!``) ((?: [^`\\] | \\[\s\S] )\*) ` ///HERE_JSTOKEN =///^ ``` ((?: [^`\\] | \\[\s\S] | `(?!``) )*) ``` ///

String-matching-regexes.

STRING_START =/^(?:'''|"""|'|")/STRING_SINGLE =/// ^(?: [^\\'] | \\[\s\S] )\* ///STRING_DOUBLE =/// ^(?: [^\\"#] | \\[\s\S] | \#(?!\{) )\* /// HEREDOC\_SINGLE = ///^(?: [^\\'] | \\[\s\S] | '(?!'') )*/// HEREDOC\_DOUBLE = ///^(?: [^\\"#] | \\[\s\S] | "(?!"") | \#(?!\{) )\* ///STRING_OMIT =/// ((?:\\\\)+) # consume (and preserve) an even number of backslashes | \\[^\S\n]\*\n\s\* # remove escaped newlines ///g
SIMPLE_STRING_OMIT =/\s\*\n\s\*/gHEREDOC_INDENT =/\n+([^\n\S]\*)(?=\S)/g

Regex-matching-regexes.

REGEX =/// ^ / (?!/) (( ?: [^ [/ \n \\] # every other thing | \\[^\n] # anything but newlines escaped | \[# character class (?: \\[^\n] | [^ \] \n \\ ] )\* \] )\*) (/)? ///REGEX_FLAGS =/^\w\*/VALID_FLAGS =/^(?!.\*(.).\*\1)[imguy]\*$/HEREGEX =/// ^(?: [^\\/#] | \\[\s\S] | /(?!//) | \#(?!\{) )\* /// HEREGEX\_OMIT = ///((?:\\\\)+)# consume (and preserve) an even number of backslashes| \\(\s)# preserve escaped whitespace| \s+(?:#.\*)? # remove whitespace and comments///g REGEX\_ILLEGAL = ///^ ( / |/{3}\s\*) (\*) ///POSSIBLY_DIVISION =/// ^ /=?\s ///

Other regexes.

HERECOMMENT_ILLEGAL =/\*\//LINE_CONTINUER =/// ^ \s\* (?: , | \??\.(?![.\d]) | :: ) ///STRING_INVALID_ESCAPE =/// ( (?:^|[^\\]) (?:\\\\)\* ) # make sure the escape isn’t escaped \\ ( ?: (0[0-7]|[1-7]) # octal escape | (x(?![\da-fA-F]{2}).{0,2}) # hex escape | (u\{(?![\da-fA-F]{1,}\})[^}]\*\}?) # unicode code point escape | (u(?!\{|[\da-fA-F]{4}).{0,4}) # unicode escape ) ///REGEX_INVALID_ESCAPE =/// ( (?:^|[^\\]) (?:\\\\)\* ) # make sure the escape isn’t escaped \\ ( ?: (0[0-7]) # octal escape | (x(?![\da-fA-F]{2}).{0,2}) # hex escape | (u\{(?![\da-fA-F]{1,}\})[^}]\*\}?) # unicode code point escape | (u(?!\{|[\da-fA-F]{4}).{0,4}) # unicode escape ) ///UNICODE_CODE_POINT_ESCAPE =/// ( \\\\ ) # make sure the escape isn’t escaped | \\u\{ ( [\da-fA-F]+ ) \} ///g

LEADING_BLANK_LINE =/^[^\n\S]\*\n/TRAILING_BLANK_LINE =/\n[^\n\S]\*$/TRAILING_SPACES =/\s+$/

Compound assignment tokens.

COMPOUND_ASSIGN = ['-=','+=','/=','\*=','%=','||=','&&=','?=','\<\<=','\>\>=','\>\>\>=''&=','^=','|=','\*\*=','//=','%%=']

Unary tokens.

UNARY = ['NEW','TYPEOF','DELETE','DO']

UNARY_MATH = ['!','~']

Bit-shifting tokens.

SHIFT = ['\<\<','\>\>','\>\>\>']

Comparison tokens.

COMPARE = ['==','!=','\<','\>','\<=','\>=']

Mathematical tokens.

MATH = ['\*','/','%','//','%%']

Relational tokens that are negatable with not prefix.

RELATION = ['IN','OF','INSTANCEOF']

Boolean tokens.

BOOL = ['TRUE','FALSE']

Tokens which could legitimately be invoked or indexed. An opening parentheses or bracket following these tokens will be recorded as the start of a function invocation or indexing operation.

CALLABLE = ['IDENTIFIER','PROPERTY',')',']','?','@','THIS','SUPER']
INDEXABLE = CALLABLE.concat ['NUMBER','INFINITY','NAN','STRING','STRING\_END','REGEX','REGEX\_END''BOOL','NULL','UNDEFINED','}','::']

Tokens which a regular expression will never immediately follow (except spaced CALLABLEs in some cases), but which a division operator can.

See: http://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions

NOT_REGEX = INDEXABLE.concat ['++','--']

Tokens that, when immediately preceding a WHEN, indicate that the WHEN occurs at the start of a line. We disambiguate these from trailing whens to avoid an ambiguity in the grammar.

LINE_BREAK = ['INDENT','OUTDENT','TERMINATOR']

Additional indent in front of these is ignored.

INDENTABLE_CLOSERS = [')','}',']']

Tokens that, when appearing at the end of a line, suppress a following TERMINATOR/INDENT token

UNFINISHED = ['\\','.','?.','?::','UNARY','MATH','UNARY\_MATH','+','-','\*\*','SHIFT','RELATION','COMPARE','&','^','|','&&','||','BIN?','THROW','EXTENDS','DEFAULT']