Lex

<!-- Part of the Carbon Language project, under the Apache License v2.0 with LLVM Exceptions. See /LICENSE for license information. SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception --> <!-- toc -->

Table of contents

• Overview
• Bracket matching
• Alternatives considered

<!-- tocstop -->

Overview

Lexing converts input source code into tokenized output. Literals, such as string literals, have their value parsed and form a single token at this stage.

Bracket matching

The lexer handles matching for (), [], and {}. When a bracket lacks a match, it will insert a "recovery" token to produce a match. As a consequence, the lexer's output should always have matched brackets, even with invalid code.

Where the missing bracket belongs is decided by a search over candidate repairs, using indentation, structural facts about the language, and formatting conventions as evidence. See mismatched bracket recovery.

Alternatives considered

• Bracket matching in parser