Chapter 18: Fault isolation: making a script crash-proof

This chapter:

State the threat model plainly: this defends against bugs, not malicious code.
Add an instruction budget: a soft per-tick limit that pauses a long-running script gracefully, and a hard lifetime cap that catches a genuine infinite loop.
Add a stack-depth limit, catching runaway recursion with a clean error instead of letting it run away.
Make runtime_error (Chapter 14) stop ending the whole process: a script's own runtime fault now tears down that one VM instance and reports .Error, while everything else, other instances, the host process itself, keeps running.
Close a real gap Chapter 14 named and deferred: every arithmetic and comparison opcode now reports a graceful error instead of an Odin type-assertion panic on a type mismatch.

Where we left off: every runtime error this book has produced so far, division by zero, an undefined global, calling a non-callable value, has printed a clean trace and then ended the whole musc process (runtime_error's own os.exit(1), since Chapter 14). That's fine for a one-shot CLI tool running one script. It's fatal for a host running fifty scripted entities: one buggy script would take the other forty-nine down with it, and the host process along with them.

18.1 The threat model, stated plainly

This chapter defends against bugs, not malicious code. Muspel is not a security sandbox against an adversarial script author: nothing here stops a script from spinning the CPU as hard as its own instruction budget allows, and nothing here stops a script from reading or writing any global a host has exposed to it. What this chapter does guarantee is that an honest mistake, an infinite loop where a while condition was meant to eventually go false, a recursive function missing its base case, a value of the wrong type reaching an arithmetic operator, degrades gracefully: that one script instance is torn down, cleanly, with a trace, and every other script and the host process itself keep running exactly as if nothing had happened. A designer or a modder (the audience Chapter 1 opened with) will write bugs like these; this chapter is what makes that a Tuesday, not an incident.

18.2 Instruction budgets

A script that never calls yield and never returns, while (true) {}, would otherwise run vm_call_run's own loop forever, blocking whatever called vm_resume for as long as the host process exists. Two related constants (vm.odin) bound this:

Max_Call_Depth :: 256
Instruction_Tick_Budget :: 100_000
Instruction_Hard_Cap :: 500_000_000

Instruction_Tick_Budget is the soft limit: if a single vm_call_run invocation executes more than 100,000 instructions without the script itself yielding or finishing, the VM is paused exactly as if it had called yield, indistinguishable to a host from an ordinary suspension. This is what lets a legitimately expensive computation, examples/bench_loop.mus's five-million- iteration loop, say, spread itself across many ticks instead of ever tripping anything:

// vm.odin
vm_call_run :: proc(vm: ^VM, floor: int) -> Resume_Result {
    saved := vm.status
    vm.status = .Running
    ticked := 0
    for vm.status == .Running && len(vm.frames) > floor {
        vm.instr_count += 1
        if vm.instr_count > Instruction_Hard_Cap {
            runtime_error(vm, "instruction budget exceeded: over %d instructions total, likely an infinite loop", Instruction_Hard_Cap)
            break
        }
        ticked += 1
        if ticked > Instruction_Tick_Budget {
            vm.status = .Suspended
            break
        }
        vm_step(vm)
    }
    if vm.status == .Error {
        return .Error
    }
    if len(vm.frames) <= floor {
        vm.status = saved
        return .Done
    }
    return .Suspended
}

vm.instr_count (a new int field on VM) counts instructions across the VM's entire lifetime, not reset per tick, and Instruction_Hard_Cap is what turns "keeps getting paused and resumed, tick after tick, forever" into an actual, reported failure: a script that never makes enough real progress to finish, no matter how many ticks it's given, eventually crosses half a billion total instructions and is declared runaway. Nothing about a legitimate script, however long-running, is expected to reach that number over its own reasonable lifetime; musc run examples/infinite_loop.mus (while (true) {}) reaches it in under two seconds and is caught cleanly.

Gotcha: choosing the hard cap. The first version of this chapter set Instruction_Hard_Cap :: 50_000_000, a round, seemingly generous number. bench_loop.mus (Chapter 9's own five-million-iteration benchmark, roughly ten to twelve bytecode instructions per loop iteration) promptly tripped it, reported as a false-positive "infinite loop," a real regression caught only by re-running every existing example after adding the check. The fix was raising the cap tenfold, to 500_000_000; the lesson is that a hard cap chosen without measuring an actual, legitimate heavy workload against it is just a guess, and guesses are exactly the kind of thing this book's own "every design decision gets a why, not just a how" rule exists to catch before it ships.

18.3 Stack-depth limits

Runaway recursion, func recurse(n) { return recurse(n + 1) } with no base case, would otherwise grow vm.frames without bound: a real cost (unbounded memory) and, eventually, a real failure, but not the clean, immediate, source-mapped failure a caught mistake deserves. Call's own ^Func_Proto case (section 13.3) checks the depth before it ever pushes another frame:

// vm.odin, inside vm_step's Call case
case ^Func_Proto:
    if len(vm.frames) >= Max_Call_Depth {
        runtime_error(vm, "stack overflow: call depth exceeded %d", Max_Call_Depth)
    } else {
        new_base := len(vm.stack) - argc
        append(&vm.frames, Frame{chunk = &fn.chunk, ip = 0, base = new_base, name = fn.name})
    }

examples/stack_overflow.mus:

func recurse(n) {
    return recurse(n + 1)
}
print(recurse(0))

./run.sh run examples/stack_overflow.mus

runtime error: stack overflow: call depth exceeded 256
  at recurse (line 2)
  at recurse (line 2)
  ... (253 more identical frames) ...
  at recurse (line 2)
  at <script> (line 4)

Every one of those 256 frames really is on vm.frames when the error fires, Chapter 14's own stack-trace machinery walking every one of them innermost-first, exactly the way it already did for a three-deep call chain in that chapter's own worked example; recursion this deep just makes the same trace long, not different.

18.4 Arena budgets: the failure mode Chapter 10 named and deferred

Chapter 10 named this one honestly and left it open: a script that allocates structs or arrays in an unbounded loop, a plausible bug (building a result array whose size the author miscalculated, say), will eventually exhaust its VM instance's fixed-size arena. Odin's own make/new, given an allocator that's run out of room, don't panic: they return a nil-backed, shorter-than-requested slice (or a nil pointer for new), which is exactly the signal Make_Struct/Make_Array check for before treating either allocation as having actually happened:

// vm.odin, Make_Struct
values := make([]Value, len(layout.field_names), value_arena_allocator(&vm.value_arena))
inst := new(Struct_Instance, value_arena_allocator(&vm.value_arena))
if (len(layout.field_names) > 0 && len(values) != len(layout.field_names)) || inst == nil {
	runtime_error(vm, "arena exhausted: cannot allocate struct '%s'", layout.type_name)
	append(&vm.stack, Value(nil))
} else {
	// ... scatter the popped values into `values` and push `inst`, as normal ...
}

len(values) != len(layout.field_names) is the tell: a successful allocation always returns a slice of exactly the requested length, so any mismatch (Odin's arena allocator falls back to returning a zero-length slice on failure, section 10.5's own core:mem.Arena, never a short-but-nonzero one) means the arena is full. Make_Array checks the identical way. Both push a placeholder nil rather than leaving the stack short, the same "still push something, vm.status being .Error is what actually stops execution" convention every other opcode in this chapter follows.

struct Big { a, b, c, d, e, f, g, h }
var i = 0
while (i < 100000) {
    var x = Big{a:1,b:2,c:3,d:4,e:5,f:6,g:7,h:8}
    i = i + 1
}

./run.sh run examples/arena_exhaust.mus

runtime error: arena exhausted: cannot allocate struct 'Big'
  at <script> (line 7)

(Line 7, not 4: the real file opens with a two-line comment plus a blank line, the same line-number offset section 11.3's own worked example already trimmed away.) A hundred thousand eight-field structs, none of them ever discarded (each loop iteration's x is a fresh top-level-scoped local, still reachable, still live as far as the arena knows, since the arena has no concept of "reachable" at all, only "allocated"), runs the instance's 256 KiB arena dry long before the loop finishes, and the script fails cleanly instead of the allocator silently handing back corrupt memory for the VM to write into. This is the cost Chapter 10 named honestly at the time: a struct or array can never be freed early to reclaim space mid-script, so a script that keeps building without bound will eventually hit this ceiling, no matter how large the arena is sized. Section 17.6 revisits the other half of this story once VM instances have a real teardown moment to reclaim the whole arena at.

18.5 Runtime errors that don't end the process

The mechanism both of the above ultimately lean on, runtime_error no longer ending the whole process, is this chapter's real centre of gravity. The change itself is one field:

// vm.odin, runtime_error's own final line
vm.status = .Error

replacing what had been os.exit(1) since Chapter 14. vm_call_run (section 18.2's own listing) checks for .Error immediately after its loop and returns .Error the moment it sees it, before any further instruction ever executes; vm_resume (Chapter 15) already had an early-return guard for an already-.Error VM, so a second vm_resume call on a VM that's already failed is a harmless no-op, exactly like an already-.Done one. Resume_Result.Error, named back in Chapter 15 for a shape this chapter would eventually fill in, is finally a real, reachable outcome.

Gotcha: every existing runtime_error call site had to be re-audited. os.exit(1) meant no code after a runtime_error call ever ran; removing it meant every single call site's own fall-through behaviour suddenly mattered, and two of them were genuinely broken. native_on (vm.odin) checked len(args) < 2 and called runtime_error, then read args[1] unconditionally on the very next line: calling on("damage") with no handler produced an outright Odin out-of-bounds panic, the exact crash this whole chapter exists to prevent, caught only by deliberately testing the malformed-call path this section describes. native_spawn and native_move_to (tools/musc/host_cmd.odin) had a quieter version of the same bug: after runtime_error reported a bad argument count, execution fell straight through into spawning an entity anyway, with whatever zero-valued arguments happened to be sitting in x/y. Every call site needed an explicit return immediately after runtime_error, unless (as with most of vm_step's own opcodes) whatever came next was already harmless to execute on an about-to-be- discarded stack. The rule this leaves behind: runtime_error reports a failure, it does not stop execution on its own; every caller is responsible for not doing anything unsafe afterward, the same discipline any language without exceptions requires of every fallible call.

18.6 Closing a gap from Chapter 14

Section 14.3's own Design decision named a real, deliberate scope cut: 1 + true still produced a raw Odin type-assertion panic, "Chapter 18's job," that chapter said plainly, "where a failing script tears down that one instance... and every other script, and the host, keeps running." Two small helpers close it:

// vm.odin
as_number :: proc(vm: ^VM, v: Value) -> (f64, bool) {
    n, ok := v.(f64)
    if !ok {
        runtime_error(vm, "expected a number, got %s", value_to_string(v))
    }
    return n, ok
}

as_number_pair :: proc(vm: ^VM, a, b: Value) -> (an, bn: f64, ok: bool) {
    a_n, a_ok := as_number(vm, a)
    if !a_ok do return 0, 0, false
    b_n, b_ok := as_number(vm, b)
    if !b_ok do return 0, 0, false
    return a_n, b_n, true
}

The exact same idea as section 16.4's arg_number/arg_string, extended from "checking a native builtin's own arguments" to "checking every arithmetic and comparison opcode's own operands." Every case that used to reach for an unchecked a.(f64) now goes through one of these instead, skipping the push entirely (harmless: vm.status is already .Error, and the VM is about to stop) when the check fails:

// vm.odin, Chapter 10's Div case, rewritten
case .Div:
    b := pop(&vm.stack); a := pop(&vm.stack)
    if an, bn, ok := as_number_pair(vm, a, b); ok {
        if bn == 0 {
            runtime_error(vm, "division by zero")
        } else {
            append(&vm.stack, Value(an / bn))
        }
    }

./run.sh run examples/type_mismatch.mus   # print(1 + true)

runtime error: expected a number, got true
  at <script> (line 1)

Compare this against Chapter 14's own exercise 3, which asked the reader to trigger exactly this input and read an Odin panic. That panic no longer exists; this is precisely the "vice versa" the exercise asked the reader to notice, made concrete two chapters later rather than left as a permanent gap.

18.7 `musc run`'s own exit code, and fault isolation made visible

vm_resume/vm_run_chunk never call os.exit themselves from this chapter onward: that decision belongs to whoever's actually driving the VM, not the VM itself. musc run, a one-shot CLI command, still wants a non-zero exit code on a script failure (shell scripts and CI both depend on this), so cmd_run (tools/musc/run_cmd.odin) makes that choice explicitly, once, after running:

// tools/musc/run_cmd.odin
muspel.vm_run_chunk(&vm, &chunk)
if vm.status == .Error {
    os.exit(1)
}

musc host does the same. musc run --instances=N (Chapter 17's own stress harness) makes the opposite choice, deliberately: an errored instance is logged and skipped, still_running excludes both .Done and .Error VMs (a real bug caught while testing this chapter: without excluding .Error too, one failing instance kept vm_resume returning .Error forever, which counted as "still running" and hung the whole harness), and every other instance keeps ticking normally:

./run.sh run --instances=10 examples/instances_all_fail.mus   # var x = 5 / 0

runtime error: division by zero
  at <script> (line 1)
[... nine more identical traces ...]
10 instance(s), 1 shared chunk, 1 simulated frame(s), 10 error(s), finished in 194µs

Ten instances, ten failures, reported individually, and the harness itself finishes in microseconds rather than hanging: fault isolation, demonstrated at the exact granularity section 16.1 promised, one script instance at a time, never the process itself.

Checkpoint

musc run examples/infinite_loop.mus is caught by the instruction budget and reports a clean error in under two seconds, rather than hanging musc forever. musc run examples/stack_overflow.mus is caught by the call-depth limit with a full, correctly-ordered trace. musc run examples/arena_exhaust.mus is caught by the arena budget with a clean error naming the struct type that couldn't be allocated. musc run examples/bench_loop.mus (a legitimate five-million-iteration computation) still completes normally, proving the soft instruction budget doesn't false-positive on real work. 1 + true now produces a graceful runtime error, not an Odin panic. musc run --instances=N against a script that fails for every instance reports every failure and still finishes promptly.

Exercises

Instruction_Tick_Budget :: 100_000 is untested against a real host's own frame-time budget (Chapter 21's Embers runs at whatever the browser's own animation-frame rate is). Instrument vm_call_run to report how many soft pauses (not the hard cap) bench_loop.mus takes to finish, and estimate, at 100,000 instructions per soft pause, how many scripted entities doing comparable work a single 16-millisecond host frame could realistically service before the Instruction_Tick_Budget itself becomes the bottleneck.
Section 18.4's Gotcha found two broken runtime_error call sites by testing. Audit tools/musc/host_cmd.odin's native_move_to for the same class of bug: its own trailing runtime_error(vm, "move_to: no entity %d", id) already has a return nil right after it, but confirm, by reading the whole function again, that this was the only unsafe fall-through path, not merely the only one this chapter happened to mention.
as_number_pair calls as_number twice, and runtime_error fires (and vm.status becomes .Error) on the first failing operand without ever checking the second. Write a script that calls 1 + "two" (two type errors are available: string versus number in either position) and confirm the error message names the first mismatched value, not both. Is silently reporting only the first of possibly several problems in one expression a real limitation worth fixing, or is it consistent with how every other error in this book (a syntax error, a resolver error) already only reports what it finds first at that exact spot?
Section 18.7 fixed a real hang in cmd_run_instances's own loop. Deliberately revert that fix (change if vms[i].status == .Done || vms[i].status == .Error do continue back to checking only .Done) and run musc run --instances=5 examples/instances_all_fail.mus again. Confirm it now hangs, and explain, in terms of vm_resume's own early-return guard for an already-.Error VM, exactly why still_running never reaches zero.
Value_Arena_Size is 256 KiB (arena.odin). Compute, on paper, roughly how many Big-shaped (eight-field) struct instances that budget holds before examples/arena_exhaust.mus fails, and compare it against the error's own reported line: does the script get further or less far than your estimate, and what does that say about the arena's own per-allocation overhead beyond the raw field data?