Chapter 2: Turning text into tokens

This chapter:

Understand why a token is more than its kind: the span it carries is what makes every later error message possible.
Build a hand-rolled scanner over raw source bytes: numbers, strings, identifiers, keywords, operators, comments.
Grow musc its first real subcommand, musc lex, and read a real token dump.

Where we left off: musc <file> reads a .mus file and prints it back verbatim. The project skeleton exists; nothing understands Muspel syntax yet.

2.1 What a token is, and why the span matters from day one

A lexer's job sounds almost too simple to justify a whole chapter: chop source text into a stream of meaningful pieces. var greeting = "hello, muspel" becomes five tokens: a var keyword, an identifier, an =, a string, and (implicitly) the statement's end. The part that's easy to underestimate is that a token is not just what it is, it's also where it came from.

Token :: struct {
	kind:   Token_Kind,
	lexeme: string,
	line:   int,
	column: int,
}

line and column are the token's span: the position its lexeme starts at, with the lexeme's own byte length giving the span its extent. Every error message this book ever prints, from Chapter 4's syntax errors through Chapter 14's runtime stack traces, ultimately traces back to a span captured right here, in the scanner, the only place in the whole pipeline that ever looks at raw source positions directly.

Gotcha: retrofitting spans is painful, so don't defer them. It's tempting to build a lexer that just returns (kind, lexeme) pairs and add position tracking "later, once errors matter." Don't: by the time errors matter, spans need to be threaded through the AST, the resolver, and the bytecode compiler's line table, and adding that after the fact means touching every layer twice. Muspel's Token carries its span from the very first version.

2.2 The token kinds Muspel needs

Token_Kind is Muspel's complete vocabulary of terminal symbols:

Token_Kind :: enum {
	Illegal,
	EOF,

	// Literals and names.
	Number,
	String,
	Ident,

	// Keywords.
	Var, Func, If, Else, While, Return, True, False, Nil,

	// Punctuation.
	Left_Paren, Right_Paren, Left_Brace, Right_Brace, Comma, Semicolon,

	// Operators.
	Plus, Minus, Star, Slash, Bang, Assign,
	Equal, Not_Equal, Less, Less_Equal, Greater, Greater_Equal,
	And_And, Or_Or,
}

A few choices worth calling out:

Illegal is a token, not an early return. When the scanner meets something it doesn't understand, an unterminated string, a stray &, it doesn't stop: it produces an .Illegal token carrying an error message as its lexeme, and keeps going. The parser (Chapter 4) is where malformed input actually gets reported and recovered from; the lexer's job is only ever to describe what it saw.
Keywords are a fixed, small list, checked against an identifier's spelling after it's been scanned as a normal identifier, not matched specially during scanning. var, func, if, else, while, return, true, false, nil, nothing else, for now; more arrive as later chapters need them.
Multi-character operators (==, !=, <=, >=, &&, ||) are each their own kind, not a single-character kind plus a following token: != is one token, not ! followed by =.
Comments produce no token at all. // like this is stripped entirely by the scanner; the parser never sees it.

Why bytes, not runes? The scanner reads src as raw bytes (u8), not decoded Unicode runes. That means Muspel identifiers are ASCII-only: 变量 isn't a legal variable name. This is a real, deliberate scope cut, not an oversight, and it buys a genuinely simpler scanner: no UTF-8 decoding on the hot path, and every position (pos, column) is a plain byte offset. It costs nothing inside string literals, which pass through untouched, so non-ASCII text in a script is fine; only names are restricted. A production language would very likely make the other trade-off; this book states the one it made and moves on.

2.3 The scanner loop

The Lexer is a cursor over the source string plus the position bookkeeping a token's span needs:

Lexer :: struct {
	src:          string,
	pos:          int,
	line:         int,
	column:       int,
	start_line:   int,
	start_column: int,
}

Three small helpers do all the reading: peek looks at the current byte without consuming it, advance consumes one byte and updates line/column (a \n bumps the line and resets the column; anything else just bumps the column), and match conditionally consumes a byte only if it's what the caller expected, the exact primitive multi-character operators need:

match :: proc(l: ^Lexer, want: u8) -> bool {
	if is_at_end(l) || l.src[l.pos] != want do return false
	advance(l)
	return true
}

scan_token ties it together: skip whitespace and comments, record the start position as the token's span, then dispatch on the next byte. Digits and letters fall through to dedicated sub-scanners; everything else is a switch over single characters, with match handling the one-or-two-character operators:

case '!':
	if match(l, '=') do return make_token(l, .Not_Equal, "!=")
	return make_token(l, .Bang, "!")
case '=':
	if match(l, '=') do return make_token(l, .Equal, "==")
	return make_token(l, .Assign, "=")

2.4 Numbers and strings

Numbers are digits, optionally followed by a . and more digits, integers and floats sharing one code path since Muspel's only numeric type is a 64-bit float (Chapter 13 fixes this as the Value design):

scan_number :: proc(l: ^Lexer) -> Token {
	start := l.pos
	for !is_at_end(l) && is_digit(peek(l)) do advance(l)

	if !is_at_end(l) && peek(l) == '.' && is_digit(peek_next(l)) {
		advance(l) // consume '.'
		for !is_at_end(l) && is_digit(peek(l)) do advance(l)
	}

	return make_token(l, .Number, l.src[start:l.pos])
}

The peek_next(l) check before consuming the . matters: without it, 1. followed by a method call syntax later in the language's life would be ambiguous. Requiring a digit after the . sidesteps that entirely, at the cost of Muspel having no bare 1. float literal, a small, acceptable restriction.

Strings track their own start past the opening quote, and handle a single level of backslash escaping by skipping the character after a \ rather than trying to interpret it (Chapter 8's tree-walker, and later the VM's string-constant handling, are where escape sequences actually get decoded; the lexer's job is just to find the string's extent correctly):

scan_string :: proc(l: ^Lexer) -> Token {
	start := l.pos
	for !is_at_end(l) && peek(l) != '"' {
		if peek(l) == '\\' && peek_next(l) != 0 {
			advance(l) // the backslash
		}
		advance(l)
	}

	if is_at_end(l) {
		return make_token(l, .Illegal, "unterminated string")
	}

	lexeme := l.src[start:l.pos]
	advance(l) // the closing quote
	return make_token(l, .String, lexeme)
}

An unterminated string (source ends before a closing ") produces an .Illegal token with a readable message rather than crashing or silently swallowing the rest of the file, exactly the Illegal-as-a-token design from section 2.2 paying off.

2.5 `musc lex`: the first real subcommand

Until now musc took a bare filename and echoed it back. From here it grows subcommands, one per layer, exactly as promised at the end of Chapter 1: musc lex, musc parse, musc check, and more later. musc's main now dispatches on the first argument:

switch cmd {
case "lex":
	cmd_lex(string(src))
case "parse":
	cmd_parse(string(src))
case "check":
	cmd_check(string(src))
case:
	print_usage()
	os.exit(1)
}

cmd_lex scans the whole file with lex_all (a thin loop around scan_token that collects every token, including the trailing .EOF) and prints each one:

./run.sh lex examples/hello.mus

2:1      Var          "var"
2:5      Ident        "greeting"
2:14     Assign       "="
2:16     String       "hello, muspel"
2:29     Semicolon    ""
3:1      Ident        "print"
3:6      Left_Paren   "("
3:7      Ident        "greeting"
3:15     Right_Paren  ")"
3:16     Semicolon    ""
4:1      EOF          ""

Every line is line:column, the token kind, and the exact lexeme, spans and all: proof the scanner is doing its job correctly before a single line of parser exists. Those two Semicolon lines with an empty lexeme are the ones nobody typed; section 2.6 explains where they came from.

Gotcha: fmt's width specifiers pad with '0', even left-aligned. The obvious way to print that columnar table is fmt.printfln("%-8s%-12v", pos, tok.kind). Try it, and numbers come out as 0002 and short strings come out zero-padded on the right (100 for a one-character field), because Odin's fmt package fills width specifiers with '0' regardless of the - flag. musc lex's output above is built with a small pad_right helper using strings.repeat(" ", ...) instead of a width specifier, exactly because of this. It's a minor gotcha, but it's the kind that costs ten confused minutes the first time you hit it.

2.6 Statement ends: automatic semicolon insertion

Look again at that token dump: hello.mus has no ; anywhere in it, yet two Semicolon tokens appear in the stream above, one after the string literal, one right before EOF, both printed with an empty lexeme (""), not ";". Nobody typed them. lex_all inserted them.

This is Muspel's take on a pattern most readers will already know from Go or Kotlin, usually called automatic semicolon insertion (ASI): the lexer, not the parser, decides where a statement ends, using one small rule applied while it builds the token stream. After scanning each token, lex_all checks two things: did the source cross a newline (or hit end of file) since the previous token, and could that previous token plausibly be the last token of a statement?

ends_statement :: proc(kind: Token_Kind) -> bool {
	#partial switch kind {
	case .Ident, .Number, .String, .True, .False, .Nil, .Right_Paren, .Right_Brace, .Return:
		return true
	}
	return false
}

If both are true, lex_all splices a Semicolon token into the stream right there, before moving on. var greeting = "hello, muspel" ends in a String, which is on the list, so the newline after it becomes a semicolon. print(greeting) ends in a Right_Paren, also on the list, and it's the last token before EOF, so it gets one too. Nothing else downstream ever needs to know which Semicolon tokens were typed and which were inserted: the parser's consume(.Semicolon, ...) calls, arriving in Chapter 3, treat them identically. An explicit ; still lexes to exactly the same token kind and still works, it's just almost never necessary.

Why put this in the lexer, not the parser? The alternative, deciding statement boundaries while parsing by comparing a token's line to the previous one, would work, but it would mean every single parsing rule that currently ends with consume(.Semicolon, ...) also needs its own newline logic. Doing it once, in the lexer, keeps the parser's idea of "a statement ends with a Semicolon token" completely unchanged from the version with mandatory semicolons: only the token stream feeding it got smarter.

The trade-off is a handful of rules about which token ends a statement, and readers who know Go's \n insertion rule (or JavaScript's less disciplined version of the same idea) already know Muspel's: end a line with an operator, a comma, or an open bracket if you want the statement to continue onto the next line, since none of those are in ends_statement. End it with a name, a literal, or a closing )/} if you mean it to end there. Chapters 3 and 4 flag the two places this rule bites hardest, once for if/else layout, once for return.

There is a companion rule the lexer plays no part in: the parser lets a statement's terminating semicolon be dropped right before a closing }, which is what makes a one-line block body like if (x < 0) { x = -x } parse even though no newline sits between x = -x and the brace for ASI to act on. That half of the rule lives in the parser (section 3.2), because only the parser knows a given } closes a statement block rather than, say, a struct literal; the lexer, working a token at a time, cannot tell the two apart.

Checkpoint

./run.sh lex examples/hello.mus prints a correct, readable token stream, spans and all, for a real .mus file. Try it against a file with an unterminated string and confirm you get a clean ILLEGAL line instead of a crash.

Exercises

The % (modulo) token now ships in the core lexer (Appendix G §G.2 walks how it was added), so musc lex already reports it on 10 % 3, confirm that first. Then, as the actual exercise, add a single-character operator token Muspel does not have, say ^, the same way, and confirm musc lex reports it on 2 ^ 3.
Feed musc lex a file containing only // a comment\n. Confirm the only token printed is .EOF, proving comments really do disappear rather than becoming an empty token.
Trigger the unterminated-string path deliberately (a " with no closing " before end of file) and read the .Illegal token's message.
is_alpha currently allows _ as a starting character. Trace through scan_ident and explain, in a sentence, why _foo and foo_bar both scan as valid identifiers but 1foo does not (hint: which sub-scanner does scan_token dispatch to based on the first character it sees?).
Run musc lex against a file whose last line is var x = 1 +, an operator with nothing after it before EOF. Confirm no Semicolon is inserted after the +, and explain from ends_statement why not.