Userspace Developer's Handbook

MontaukOS is a hobbyist 64-bit operating system written in C++20. Userspace programs run in Ring 3, are loaded as static ELF64 binaries, and communicate with the kernel through the x86-64 SYSCALL/SYSRET mechanism.

This document covers everything you need to write, build, and run userspace applications — from a minimal "Hello World" to using every available syscall — as well as how to extend the kernel with new syscalls.

Getting Started

Project Structure

programs/
├── GNUmakefile          # Build system (C++ programs)
├── link.ld              # Linker script (base address 0x400000)
├── include/
│   ├── Api/
│   │   └── Syscall.hpp      # Syscall numbers & data structures
│   ├── montauk/
│   │   ├── syscall.h        # Inline asm wrappers & typed API
│   │   └── heap.h           # Userspace heap allocator (malloc/mfree/realloc)
│   └── libc/                # Minimal C standard library headers
│       ├── stdio.h          # printf, FILE I/O
│       ├── stdlib.h         # malloc, free, atoi, exit
│       ├── string.h         # memcpy, strlen, strcmp, etc.
│       └── ...              # ctype.h, errno.h, assert.h, etc.
├── src/
│   ├── hello/
│   │   └── main.cpp         # Hello world example
│   ├── shell/
│   │   └── main.cpp         # Interactive shell
│   ├── man/
│   │   └── main.cpp         # Manual page viewer
│   └── doom/
│       ├── Makefile             # DOOM build system
│       ├── doomgeneric_montauk.c # MontaukOS platform layer
│       └── libc.c               # C library implementation
├── bin/                     # Compiled .elf binaries
└── obj/                     # Intermediate object files

Each subdirectory under src/ is treated as a separate program. The build system discovers them automatically.

Toolchain

Programs are compiled with a freestanding x86_64-elf cross-compiler. The build system looks for one in toolchain/local/bin/; if not found, it falls back to the host g++.

Key compiler flags:

• -std=gnu++20 — C++20 with GNU extensions
• -ffreestanding -nostdinc -nostdlib — no hosted standard library
• -fno-rtti -fno-exceptions — no RTTI or C++ exceptions
• -fno-PIC -mcmodel=small — static absolute addressing
• -mno-sse -mno-mmx -mno-red-zone — kernel-safe ABI (C++ programs)

Build System

To build all programs:

cd programs/
make          # or: make -j$(nproc)

This produces one ELF binary per program in bin/ (e.g. bin/hello.elf, bin/shell.elf). The binaries are then typically packed into the ramdisk (ramdisk.tar) and loaded by the kernel at boot.

To add a new program, create src/<name>/main.cpp and run make.

Linker Script

programs/link.ld

All userspace programs are linked at virtual address 0x400000. The linker script defines four standard sections:

Debug frames (.eh_frame), notes, and comments are discarded to keep binaries small.

Program Anatomy

Entry Point: _start

Because there is no C runtime, every program must define extern "C" void _start() as its entry point. There is no main(), no argc/argv, and no atexit handlers.

Hello World

programs/src/hello/main.cpp

// Minimal MontaukOS userspace program
#include <montauk/syscall.h>

extern "C" void _start() {
    montauk::print("Hello from userspace!\n");
}

Include <montauk/syscall.h> for the full typed API. That header pulls in <Api/Syscall.hpp> for constants and data structures.

Syscall Architecture

Calling Convention

MontaukOS uses the hardware SYSCALL instruction on x86-64. The kernel sets up the required MSRs (IA32_STAR, IA32_LSTAR, IA32_FMASK) during boot.

The SYSCALL instruction saves RIP in RCX and RFLAGS in R11; both registers are clobbered. The kernel masks IF on entry (via IA32_FMASK) so interrupts are disabled during the transition.

On the kernel side, SyscallEntry (assembly) saves callee-saved registers and all arguments into a SyscallFrame, then calls SyscallDispatch(SyscallFrame*) which dispatches by syscall number.

Raw Syscall Wrappers

programs/include/montauk/syscall.h

Seven inline functions cover 0–6 argument syscalls:

int64_t syscall0(uint64_t nr);
int64_t syscall1(uint64_t nr, uint64_t a1);
int64_t syscall2(uint64_t nr, uint64_t a1, uint64_t a2);
int64_t syscall3(uint64_t nr, uint64_t a1, uint64_t a2, uint64_t a3);
int64_t syscall4(uint64_t nr, uint64_t a1, uint64_t a2, uint64_t a3, uint64_t a4);
int64_t syscall5(uint64_t nr, uint64_t a1, uint64_t a2, uint64_t a3, uint64_t a4, uint64_t a5);
int64_t syscall6(uint64_t nr, uint64_t a1, ..., uint64_t a6);

Each wrapper uses inline assembly with explicit register moves and a full clobber list to ensure correctness. You can use these directly if you need a syscall not yet covered by the typed API.

Typed API Wrappers

The montauk:: namespace provides type-safe wrappers around the raw syscalls. These are the recommended interface for application code. The full reference follows below.

Syscall Reference

MontaukOS v0.1.0 exposes 26 syscalls (numbers 0–25), organized into 10 categories.

Process Process Management

SYS_EXIT (0) — Terminate the current process

[[noreturn]] void montauk::exit(int code = 0);

Terminates the calling process. The exit code is currently unused by the kernel but reserved for future wait/status support. Control never returns to the caller.

SYS_YIELD (1) — Yield the CPU

void montauk::yield();

Voluntarily yields the remainder of the current time slice, allowing the scheduler to run another ready process immediately.

SYS_SLEEP_MS (2) — Sleep for a duration

void montauk::sleep_ms(uint64_t ms);

Suspends the calling process for at least ms milliseconds. Resolution depends on the APIC timer tick rate.

SYS_GETPID (3) — Get process ID

int montauk::getpid();

Returns the PID of the calling process. Returns -1 if called from the idle context (should not happen in userspace).

SYS_SPAWN (20) — Spawn a new process

int montauk::spawn(const char* path, const char* args = nullptr);

Loads the ELF binary at path (a VFS path like "0:/hello.elf") and spawns it as a new process. Returns the new process's PID on success, or -1 on failure (no free slots, invalid ELF, file not found). The optional args string (up to 255 characters) is copied into the new process and can be retrieved with SYS_GETARGS.

SYS_WAITPID (23) — Wait for a process to exit

void montauk::waitpid(int pid);

Blocks the calling process until the process identified by pid has exited. Internally yields the CPU in a loop until the target process is no longer alive. This is used by the shell to wait for foreground processes (e.g. run command) so that keyboard input and the terminal are not shared simultaneously.

SYS_GETARGS (25) — Get process arguments

int montauk::getargs(char* buf, uint64_t maxLen);

Copies the argument string passed to the current process (via spawn()) into buf, writing at most maxLen - 1 characters plus a null terminator. Returns the number of characters copied, or -1 on error. If no arguments were provided at spawn time, the buffer will be empty.

Console Console I/O

SYS_PRINT (4) — Print a string

void montauk::print(const char* text);

Writes a null-terminated string to the kernel terminal. Supports newlines (\n) and standard printable ASCII.

SYS_PUTCHAR (5) — Print a single character

void montauk::putchar(char c);

Writes a single character to the kernel terminal. Useful for building output character by character (e.g. printing integers).

File I/O File I/O

SYS_OPEN (6) — Open a file

int montauk::open(const char* path);

Opens a file on the VFS. Paths use the format "<device>:/<name>" (e.g. "0:/hello.elf" for the ramdisk). Returns a non-negative handle on success, or a negative value on error.

SYS_READ (7) — Read from a file

int montauk::read(int handle, uint8_t* buf, uint64_t offset, uint64_t size);

Reads up to size bytes from the file at the given byte offset into buf. Returns the number of bytes actually read, or a negative value on error. Does not maintain an implicit file position — the offset is explicit on every call.

SYS_GETSIZE (8) — Get file size

uint64_t montauk::getsize(int handle);

Returns the total size (in bytes) of the file associated with handle.

SYS_CLOSE (9) — Close a file

void montauk::close(int handle);

Closes the file handle and releases associated kernel resources.

SYS_READDIR (10) — List directory entries

int montauk::readdir(const char* path, const char** names, int max);

Reads up to max directory entries from path. Entry name pointers are written into the names array. The kernel allocates a user-accessible page for the string data automatically. Returns the number of entries read, or ≤ 0 on error/empty directory. Maximum 64 entries per call.

Memory Memory Management

Memory allocation has two layers: low-level syscalls that map pages from the kernel, and a userspace heap that provides malloc/mfree on top.

Userspace Heap (recommended)

programs/include/montauk/heap.h

Include <montauk/heap.h> for a proper free-list allocator that runs entirely in userspace. It calls SYS_ALLOC internally to obtain pages and manages sub-page allocations with a linked free list — adapted from the kernel's own HeapAllocator.

void* montauk::malloc(uint64_t size);

Allocates size bytes from the userspace free list. Returns a 16-byte-aligned pointer, or nullptr on failure. When the free list is exhausted, it transparently requests more pages from the kernel via SYS_ALLOC (minimum 16 KiB growth).

void montauk::mfree(void* ptr);

Returns the block to the userspace free list. No syscall is made — the memory stays mapped and is immediately available for future malloc calls. Passing nullptr is a safe no-op.

void* montauk::realloc(void* ptr, uint64_t size);

Resizes the allocation at ptr to size bytes. Allocates a new block, copies the smaller of old/new sizes, and frees the old block. If ptr is nullptr, behaves like malloc.

Low-Level Page Syscalls

SYS_ALLOC (11) — Map pages

void* montauk::alloc(uint64_t size);

Maps size bytes of zeroed physical pages into the process's address space (starting at 0x40000000). The size is rounded up to the nearest page boundary (4 KiB). Returns a pointer to the mapped region, or nullptr on failure. This is the backing primitive for montauk::malloc — most programs should use the heap API instead of calling this directly.

SYS_FREE (12) — Unmap pages (no-op)

void montauk::free(void* ptr);

Reserved for future page-level unmapping. Currently a no-op — pages are reclaimed when the process exits. Use montauk::mfree for heap allocations.

Time Timekeeping

SYS_GETTICKS (13) — Get tick count

uint64_t montauk::get_ticks();

Returns the number of APIC timer ticks since boot. The tick rate depends on the hardware and APIC timer calibration.

SYS_GETMILLISECONDS (14) — Get milliseconds since boot

uint64_t montauk::get_milliseconds();

Returns wall-clock milliseconds elapsed since boot. Useful for calculating uptime or measuring durations.

System System Information

SYS_GETINFO (15) — Get OS information

void montauk::get_info(Montauk::SysInfo* info);

Fills in a SysInfo structure with the OS name, version string, API version number, and maximum process count.

Keyboard Keyboard Input

SYS_ISKEYAVAILABLE (16) — Check for pending key

bool montauk::is_key_available();

Returns true if a key event is available in the PS/2 keyboard buffer. Non-blocking.

SYS_GETKEY (17) — Get a key event

void montauk::getkey(Montauk::KeyEvent* out);

Fills in a KeyEvent structure with the next keyboard event (press or release), including scancode, ASCII translation, and modifier state (Shift, Ctrl, Alt).

SYS_GETCHAR (18) — Read a character (blocking)

char montauk::getchar();

Blocks until a printable character key-press is available, then returns the ASCII character. This is the simplest way to read interactive text input.

Network Networking

SYS_PING (19) — Send an ICMP echo request

int32_t montauk::ping(uint32_t ip, uint32_t timeoutMs = 3000);

Sends an ICMP echo request to ip and waits up to timeoutMs milliseconds for a reply. The IP address is in little-endian byte order (e.g. 10.0.2.2 → 0x0202000A). Returns the round-trip time in milliseconds on success, or -1 on timeout.

Framebuffer Framebuffer Access

SYS_FBINFO (21) — Get framebuffer information

void montauk::fb_info(Montauk::FbInfo* info);

Fills in a FbInfo structure with the framebuffer dimensions, pitch (bytes per scanline), and bits per pixel. Call this before SYS_FBMAP to learn the framebuffer geometry.

SYS_FBMAP (22) — Map framebuffer into process memory

void* montauk::fb_map();

Maps the physical framebuffer into the calling process's address space at 0x50000000 and returns the user virtual address. The mapped region covers height × pitch bytes. Each pixel is a 32-bit value in 0xAARRGGBB format (blue in the low byte). Writing to this memory directly updates the screen.

Terminal Terminal

SYS_TERMSIZE (24) — Get terminal dimensions

void montauk::termsize(int* cols, int* rows);

Returns the current terminal dimensions (character grid) via the two output pointers. Columns are packed in the low 32 bits and rows in the high 32 bits of the raw return value; the typed wrapper unpacks them for you. Either pointer may be nullptr if you only need one dimension.

Data Structures

Montauk::SysInfo

programs/include/Api/Syscall.hpp

struct SysInfo {
    char     osName[32];       // e.g. "MontaukOS"
    char     osVersion[32];    // e.g. "0.1.0"
    uint32_t apiVersion;       // Current: 2
    uint32_t maxProcesses;     // Current: 16
};

Montauk::FbInfo

struct FbInfo {
    uint64_t width;       // Framebuffer width in pixels
    uint64_t height;      // Framebuffer height in pixels
    uint64_t pitch;       // Bytes per scanline
    uint64_t bpp;         // Bits per pixel (always 32)
    uint64_t userAddr;    // Reserved (0 until mapped via SYS_FBMAP)
};

Montauk::KeyEvent

struct KeyEvent {
    uint8_t scancode;   // Raw PS/2 scancode
    char    ascii;      // Translated ASCII character (0 if non-printable)
    bool    pressed;    // true = key down, false = key up
    bool    shift;      // Shift modifier active
    bool    ctrl;       // Ctrl modifier active
    bool    alt;        // Alt modifier active
};

Montauk::SyscallFrame (kernel only)

struct SyscallFrame {
    uint64_t r15, r14, r13, r12, rbp, rbx;   // callee-saved
    uint64_t arg6, arg5, arg4, arg3, arg2, arg1;
    uint64_t syscall_nr;
    uint64_t user_rflags, user_rip, user_rsp;
};

This is the stack frame pushed by SyscallEntry.asm and passed to SyscallDispatch. Userspace code never sees this directly.

Shell Application Walkthrough

programs/src/shell/main.cpp

The built-in shell is the best example of a real MontaukOS application. It demonstrates most of the available syscalls.

Initialization

extern "C" void _start() {
    montauk::print("\n  MontaukOS Shell v0.1\n");
    montauk::print("  Type 'help' for available commands.\n\n");

    char line[256];
    int pos = 0;
    prompt();

    while (true) {
        char c = montauk::getchar();       // blocking read
        // ... handle input, echo, backspace ...
    }
}

The shell uses montauk::getchar() in a loop for blocking character-by-character input, manually handling echo and backspace.

Shell Commands & Syscalls Used

Pattern: Reading a File

The shell's cat implementation shows the standard file-reading pattern:

// 1. Build VFS path
char path[128];
// ... copy "0:/" + filename into path ...

// 2. Open
int handle = montauk::open(path);
if (handle < 0) { /* error */ return; }

// 3. Get size
uint64_t size = montauk::getsize(handle);

// 4. Read in chunks
uint8_t buf[512];
uint64_t offset = 0;
while (offset < size) {
    uint64_t chunk = size - offset;
    if (chunk > sizeof(buf) - 1) chunk = sizeof(buf) - 1;
    int bytesRead = montauk::read(handle, buf, offset, chunk);
    if (bytesRead <= 0) break;
    buf[bytesRead] = '\0';
    montauk::print((const char*)buf);
    offset += bytesRead;
}

// 5. Close
montauk::close(handle);

Pattern: IP Networking

// Parse "10.0.2.2" into uint32_t in little-endian order
uint32_t ip;
parse_ip("10.0.2.2", &ip);  // ip = 0x0202000A

// Send 4 pings with 1-second intervals
for (int i = 0; i < 4; i++) {
    int32_t rtt = montauk::ping(ip, 3000);
    if (rtt < 0) { /* timeout */ }
    else { /* reply in rtt ms */ }
    if (i < 3) montauk::sleep_ms(1000);
}

Adding New Syscalls

Adding a syscall requires changes in 3 files (kernel-side) and 2 files (userspace-side). Follow these steps:

Step 1: Assign a Syscall Number

Add a new constant in both copies of the syscall number definitions. The numbers must match exactly.

kernel/src/Api/Syscall.hpp

programs/include/Api/Syscall.hpp

static constexpr uint64_t SYS_MYFUNC = 26;  // next available number

Step 2: Implement the Kernel Handler

kernel/src/Api/Syscall.cpp

Add a static function implementing your syscall's logic:

static int64_t Sys_MyFunc(uint64_t arg1, const char* arg2) {
    // Your kernel-side implementation here.
    // You have full access to kernel subsystems.
    return 0;
}

Step 3: Add the Dispatch Case

kernel/src/Api/Syscall.cpp — SyscallDispatch()

case SYS_MYFUNC:
    return (int64_t)Sys_MyFunc(frame->arg1, (const char*)frame->arg2);

Arguments are accessed through frame->arg1 through frame->arg6, corresponding to RDI, RSI, RDX, R10, R8, R9 respectively.

Step 4: Add a Typed Userspace Wrapper

programs/include/montauk/syscall.h

inline int64_t my_func(uint64_t arg1, const char* arg2) {
    return syscall2(Montauk::SYS_MYFUNC, arg1, (uint64_t)arg2);
}

Choose the appropriate syscallN variant based on the number of arguments. Cast pointer types to uint64_t.

Step 5: Update the Log Message (optional)

kernel/src/Api/Syscall.cpp — InitializeSyscalls()

Update the boot log to reflect the new syscall count:

// Change "26 syscalls" to "27 syscalls"
Kt::KernelLogStream(Kt::OK, "Syscall") << "SYSCALL/SYSRET initialized (LSTAR="
    << kcp::hex << (uint64_t)SyscallEntry << kcp::dec << ", 27 syscalls)";

Complete Checklist

Process Memory Model

Each process gets its own PML4 page table. The kernel half (upper 256 entries) is shared; the lower half is per-process.

The ELF loader maps PT_LOAD segments with user-accessible page flags. BSS is zero-initialized automatically (pages are allocated zeroed).

Current Limitations

• No standard library (C++) — there is no libc++ for C++ programs. String operations, integer formatting, etc. must be written manually (see the shell for examples). A minimal C standard library exists for C programs (used by the DOOM port) in programs/include/libc/ with headers like stdio.h, stdlib.h, and string.h.
• No dynamic linking — all programs are statically linked.
• No argc/argv — _start receives no argc/argv. Use montauk::getargs() to retrieve a single argument string (max 255 chars) passed via montauk::spawn().
• No page-level free() — SYS_FREE is a no-op; pages mapped via SYS_ALLOC are only reclaimed when the process exits. However, the userspace heap (montauk::malloc/montauk::mfree) does reuse memory via a free list.
• 16 process slots — the scheduler supports a maximum of 16 concurrent processes.
• No writable file system — the VFS is read-only (backed by the boot ramdisk).
• Shared console — all processes write to the same terminal; there is no per-process stdout.
• Shared keyboard — keyboard input is global; there is no input focus or multiplexing.
• 10 ms scheduling granularity — preemptive round-robin with a fixed 10 ms time slice.

MontaukOS Documentation — Copyright © 2025-2026 Daniel Hammer