Odin and Me · Chapter 4

Chapter 4: Strings, runes and a word-counting tool

This chapter:

  • See how Odin actually stores a string: as bytes, not characters.
  • Meet rune and iterate text correctly, even when it isn't plain ASCII.
  • Slice strings safely, and see what happens when you don't.
  • Build output efficiently with strings.Builder instead of repeated concatenation.
  • Write wordcount.odin, a line/word/character counter for a file or piped input.

Where we left off: Chapter 3 got you reading and parsing a line of input. This chapter is about text itself: what a string really is under the bonnet, and how to work with it properly.

4.1 Strings are bytes

Here's the thing that surprises almost everyone coming from a higher-level language: an Odin string is just an immutable, read-only view over a run of bytes. Nothing more. Those bytes are expected to be UTF-8, but Odin doesn't decode them for you just because you asked for len(s) or s[0].

s := "héllo"
fmt.println(len(s))   // 6
fmt.println(s[0])     // 104
fmt.println(s[1])     // 195

len(s) is 6, not 5, because é takes two bytes to encode in UTF-8, not one. s[0] gives you 104, the byte value for 'h', as a u8, not a character. s[1] gives you 195, which isn't é at all; it's the first of the two bytes that together spell é.

Why does Odin work this way? Because "how long is this string" and "what's the third character" are two genuinely different questions once you leave ASCII behind, and a language that quietly conflates them is lying to you about how much work it's actually doing. Odin makes you ask for what you mean: byte length is len(s), and character-aware operations go through runes, which is exactly what §4.2 covers.

4.2 Runes

A rune is a decoded Unicode codepoint: one actual character, however many bytes it took to encode. Odin's for r in s loop decodes a string's UTF-8 one rune at a time, correctly, so you never have to do that decoding by hand:

s := "héllo"
fmt.println("bytes:", len(s))
fmt.println("runes:", utf8.rune_count_in_string(s))

for r in s {
	fmt.print(r, " ")
}
fmt.println()

bytes: 6
runes: 5
h  é  l  l  o

len(s) still says 6 (bytes); utf8.rune_count_in_string(s), from core:unicode/utf8, says 5 (actual characters), and for r in s visits each of those five characters exactly once, regardless of how many bytes each one took.

Gotcha: len(s) is a byte count, not a character count. For plain ASCII text the two numbers happen to match, which is exactly why this bites people: code that quietly assumes len(s) means "how many characters" works fine in testing and then miscounts the moment a reader's name has an accent in it, or their notes (Chapter 9 onward) contain an emoji. If you want characters, count runes, not bytes.

4.3 Slicing strings

s[i:j] slices a string the same way it slices any other sequence in Odin, but the indices i and j are byte offsets, not character positions. Slice on a rune boundary and you get a clean substring; slice into the middle of a multi-byte rune and you get broken UTF-8:

s := "héllo"
fmt.println(s[0:1])   // "h"
fmt.println(s[3:6])   // "llo"
fmt.println(s[0:2])   // "h" plus half of "é" -- broken

h
llo
h�

s[0:1] and s[3:6] both happen to land on rune boundaries, so they print cleanly. s[0:2] cuts é down the middle: byte 1 is the start of a two-byte rune, and slicing there leaves you with one dangling byte that isn't valid UTF-8 on its own: printed as the replacement character, �.

Odin note: core:unicode/utf8 has your back here. For text that might not be plain ASCII, utf8.rune_size (how many bytes does the rune starting here take?) and utf8.rune_count_in_string (§4.2) let you find safe boundaries instead of guessing. My examples here stay ASCII-safe enough that raw byte slicing is fine, but it's worth knowing the escape hatch exists before you hit real-world text that isn't.

4.4 Building output efficiently

Concatenating strings with + in a loop is a trap in more or less every language: each + allocates a brand-new string and copies everything that came before into it, so a loop that builds up a long string one piece at a time does quadratically more copying than you'd expect. core:strings gives you Builder, which grows a single buffer instead:

b: strings.Builder
strings.builder_init(&b)
defer strings.builder_destroy(&b)

for word, i in ([]string{"one", "two", "three"}) {
	if i > 0 {
		strings.write_string(&b, ", ")
	}
	strings.write_string(&b, word)
}

fmt.println(strings.to_string(b))

one, two, three

Why bother with a Builder for three short words? At this size, honestly, you wouldn't notice the difference either way. The habit is what matters: wordcount.odin in a moment reads a whole file and rebuilds it line by line from standard input, and that's exactly the kind of loop where repeated + concatenation would quietly get slower as the input grows. Reach for Builder whenever you're accumulating text in a loop.

4.5 The program: a word counter

Time to put it together: a program that reports line, word and character counts for either a file named on the command line or whatever's piped into standard input.

// wordcount.odin
package main

import "core:bufio"
import "core:fmt"
import "core:os"
import "core:strings"
import "core:unicode/utf8"

count_text :: proc(text: string) {
	lines := strings.count(text, "\n")
	if len(text) > 0 && text[len(text)-1] != '\n' {
		lines += 1 // a final line with no trailing newline still counts
	}

	words := strings.fields(text)
	defer delete(words)

	chars := utf8.rune_count_in_string(text)

	fmt.printfln("lines: %d", lines)
	fmt.printfln("words: %d", len(words))
	fmt.printfln("chars: %d", chars)
}

read_stdin :: proc() -> string {
	b: strings.Builder
	strings.builder_init(&b)

	scanner: bufio.Scanner
	bufio.scanner_init(&scanner, os.to_reader(os.stdin))
	defer bufio.scanner_destroy(&scanner)

	for bufio.scanner_scan(&scanner) {
		strings.write_string(&b, bufio.scanner_text(&scanner))
		strings.write_byte(&b, '\n')
	}

	return strings.to_string(b)
}

main :: proc() {
	if len(os.args) > 1 {
		data, err := os.read_entire_file_from_path(os.args[1], context.allocator)
		if err != nil {
			fmt.eprintfln("could not read file: %s", os.args[1])
			os.exit(1)
		}
		defer delete(data)
		count_text(string(data))
	} else {
		count_text(read_stdin())
	}
}

Try it both ways against a small file:

Create sample.txt with content:

the quick brown fox
jumps over the lazy dog
the fox runs away

Then

$ odin run wordcount.odin -file -- sample.txt
lines: 3
words: 13
chars: 62

$ cat sample.txt | odin run wordcount.odin -file
lines: 3
words: 13
chars: 62

strings.fields(text) does the actual word-splitting: it breaks text on runs of whitespace (spaces, tabs, newlines) and hands you back a slice of the pieces, which is exactly "word count" for a plain text file. strings.count(text, "\n") counts newlines directly rather than counting lines returned by a scanner, which keeps count_text usable on a whole file's contents or on text you built up yourself from standard input: os.args[1] != "" decides which one you're looking at, but count_text itself doesn't care.

read_stdin reuses §4.4's Builder: it reads line by line with the scanner you met in Chapter 3, and writes each line back out with a newline appended, rebuilding the original text as one string so count_text can treat both input paths identically.

Odin note: os.args. os.args is a []string of the command-line arguments your program was started with; os.args[0] is the program's own name, so a file named on the command line shows up at os.args[1]. len(os.args) > 1 is the check for "was a filename actually given".

4.6 A second pass: longest and most frequent word

strings.fields already gave you a slice of every word in the text: that's most of the work already done for two more useful facts: which word is longest, and which one shows up most often. Add this to wordcount.odin, inside count_text, right after the three fmt.printfln calls:

	if len(words) == 0 {
		return
	}

	longest := words[0]
	tally := make(map[string]int)
	defer delete(tally)

	for word in words {
		if len(word) > len(longest) {
			longest = word
		}
		tally[word] += 1
	}

	most_frequent := words[0]
	for word, count in tally {
		if count > tally[most_frequent] {
			most_frequent = word
		}
	}

	fmt.printfln("longest word: %s", longest)
	fmt.printfln("most frequent word: %s (%d times)", most_frequent, tally[most_frequent])

$ odin run wordcount.odin -file -- sample.txt
lines: 3
words: 13
chars: 62
longest word: quick
most frequent word: the (3 times)

tally is a map[string]int: tally[word] += 1 reads the current count for word (zero, for a map, if it's never been seen before, no separate "does this key exist yet" check needed) and writes back one higher. Finding the longest word and the most frequent word are the same type of loop you've seen since Chapter 2's is_prime: start with a guess (words[0]), then walk everything else looking for something that beats it.

Gotcha: ties go to whichever word the loop happens to see first. sample.txt has three five-letter words (quick, brown, jumps); longest reports quick because it's the first one the loop meets, not because it's special. That's fine for this program, but it's worth noticing: if you needed a stable, well-defined tie-break (alphabetical, say), you'd have to write that rule yourself: Odin's map iteration order in particular isn't something to rely on for breaking ties consistently.

Checkpoint

wordcount.odin reports line, word and character counts for a file or piped input, plus the longest word and the most frequent one. You've handled text as bytes, decoded it into runes where it mattered, and used a map to tally something.

Exercises

  1. Extend wordcount.odin to also report the shortest word. (Solution sketch: the same shape as longest, just flip the comparison: watch out for punctuation-only "words" if your test text has any, since strings.fields splits on whitespace only, not punctuation.)
  2. count_text's line-counting adds 1 for a final line with no trailing newline. Delete that if block and run the program against sample.txt (which does end in a newline) and then against a file you create with printf "one two" (no trailing newline, no \n): what's different, and why does the special case matter for the second file specifically?
  3. Right now, punctuation sticks to words: "fox." and "fox" count as two different words for the tally. Modify the program to strip leading/trailing punctuation from each word before tallying it (a small helper using strings.trim with a set of punctuation characters is enough). (Solution sketch: strings.trim(word, ".,!?;:") handles the common cases; decide for yourself whether that's thorough enough for your test text, and say so in a comment if not.)
  4. What does s[0:2] from §4.3 actually contain, byte for byte? Write a tiny program that prints each byte of s[0:2] with %d, and compare it against the two bytes of é you'd expect from s[1] and a byte you haven't looked at yet. (Solution sketch: s[0:2] is 104, 195: 'h' followed by only the first byte of é's two-byte encoding; the second byte, at s[2], gets left out of the slice entirely, which is why printing it shows the replacement character rather than a real é.)
  5. tally[word] += 1 relies on Odin's maps returning a zero value for a key that isn't there yet. What would you have to write instead if that weren't true, if reading a missing key were an error? Sketch the extra check in words or pseudocode; you don't need to make the map behave that way to answer it. (Solution sketch: something like "if word isn't already a key in tally, first set tally[word] = 0, then increment it": an extra branch for every single word, which is exactly the boilerplate the zero-value default saves you from writing.)

Further reading