Userspace Developer's Handbook
-
- MontaukOS is a hobbyist 64-bit operating system written in C++20.
- Userspace programs run in Ring 3, are loaded as static ELF64
- binaries, and communicate with the kernel through the x86-64
- SYSCALL/SYSRET mechanism.
-
- This document covers everything you need to write, build, and run - userspace applications — from a minimal "Hello World" to using - every available syscall — as well as how to extend the kernel - with new syscalls. -
- - -Getting Started
- -Project Structure
-programs/
-├── GNUmakefile # Build system (C++ programs)
-├── link.ld # Linker script (base address 0x400000)
-├── include/
-│ ├── Api/
-│ │ └── Syscall.hpp # Syscall numbers & data structures
-│ ├── montauk/
-│ │ ├── syscall.h # Inline asm wrappers & typed API
-│ │ └── heap.h # Userspace heap allocator (malloc/mfree/realloc)
-│ └── libc/ # Minimal C standard library headers
-│ ├── stdio.h # printf, FILE I/O
-│ ├── stdlib.h # malloc, free, atoi, exit
-│ ├── string.h # memcpy, strlen, strcmp, etc.
-│ └── ... # ctype.h, errno.h, assert.h, etc.
-├── src/
-│ ├── hello/
-│ │ └── main.cpp # Hello world example
-│ ├── shell/
-│ │ └── main.cpp # Interactive shell
-│ ├── man/
-│ │ └── main.cpp # Manual page viewer
-│ └── doom/
-│ ├── Makefile # DOOM build system
-│ ├── doomgeneric_montauk.c # MontaukOS platform layer
-│ └── libc.c # C library implementation
-├── bin/ # Compiled .elf binaries
-└── obj/ # Intermediate object files
- Each subdirectory under src/ is treated as a separate
- program. The build system discovers them automatically.
-
Toolchain
-
- Programs are compiled with a freestanding x86_64-elf
- cross-compiler. The build system looks for one in
- toolchain/local/bin/; if not found, it falls back to the
- host g++.
-
Key compiler flags:
--
-
-std=gnu++20— C++20 with GNU extensions
- -ffreestanding -nostdinc -nostdlib— no hosted standard library
- -fno-rtti -fno-exceptions— no RTTI or C++ exceptions
- -fno-PIC -mcmodel=small— static absolute addressing
- -mno-sse -mno-mmx -mno-red-zone— kernel-safe ABI (C++ programs)
-
float/double)
- should compile with -msse -msse2 instead. The kernel enables SSE in
- CR0/CR4 at boot, so SSE instructions are safe in userspace.
-Build System
-To build all programs:
-cd programs/
-make # or: make -j$(nproc)
- This produces one ELF binary per program in bin/
- (e.g. bin/hello.elf, bin/shell.elf).
- The binaries are then typically packed into the ramdisk
- (ramdisk.tar) and loaded by the kernel at boot.
-
To add a new program, create src/<name>/main.cpp and run make.
Linker Script
-programs/link.ld
-
- All userspace programs are linked at virtual address
- 0x400000. The linker script defines four standard
- sections:
-
| Section | Contents | Alignment |
|---|---|---|
.text | Executable code | — (base) |
.rodata | Read-only data, string literals | 4 KiB |
.data | Initialized read/write data | 4 KiB |
.bss | Zero-initialized data | — |
- Debug frames (.eh_frame), notes, and comments are
- discarded to keep binaries small.
-
Program Anatomy
- -Entry Point: _start
-
- Because there is no C runtime, every program must define
- extern "C" void _start() as its entry point. There is no
- main(), no argc/argv, and no
- atexit handlers.
-
_start() returns normally,
- the kernel's exit stub (mapped at 0x3FF000) automatically
- calls SYS_EXIT(0). You can also call
- montauk::exit(code) explicitly at any point.
-Hello World
-programs/src/hello/main.cpp
-// Minimal MontaukOS userspace program
-#include <montauk/syscall.h>
-
-extern "C" void _start() {
- montauk::print("Hello from userspace!\n");
-}
- Include <montauk/syscall.h> for the full typed API.
- That header pulls in <Api/Syscall.hpp> for
- constants and data structures.
-
Syscall Architecture
- -Calling Convention
-
- MontaukOS uses the hardware SYSCALL instruction on
- x86-64. The kernel sets up the required MSRs
- (IA32_STAR, IA32_LSTAR,
- IA32_FMASK) during boot.
-
| Register | Purpose |
|---|---|
RAX | Syscall number (in) / return value (out) |
RDI | Argument 1 |
RSI | Argument 2 |
RDX | Argument 3 |
R10 | Argument 4 (not RCX — SYSCALL clobbers it) |
R8 | Argument 5 |
R9 | Argument 6 |
- The SYSCALL instruction saves RIP in
- RCX and RFLAGS in R11; both
- registers are clobbered. The kernel masks IF on entry
- (via IA32_FMASK) so interrupts are disabled during the
- transition.
-
- On the kernel side, SyscallEntry (assembly) saves
- callee-saved registers and all arguments into a
- SyscallFrame, then calls
- SyscallDispatch(SyscallFrame*) which dispatches by
- syscall number.
-
Raw Syscall Wrappers
-programs/include/montauk/syscall.h
-- Seven inline functions cover 0–6 argument syscalls: -
-int64_t syscall0(uint64_t nr);
-int64_t syscall1(uint64_t nr, uint64_t a1);
-int64_t syscall2(uint64_t nr, uint64_t a1, uint64_t a2);
-int64_t syscall3(uint64_t nr, uint64_t a1, uint64_t a2, uint64_t a3);
-int64_t syscall4(uint64_t nr, uint64_t a1, uint64_t a2, uint64_t a3, uint64_t a4);
-int64_t syscall5(uint64_t nr, uint64_t a1, uint64_t a2, uint64_t a3, uint64_t a4, uint64_t a5);
-int64_t syscall6(uint64_t nr, uint64_t a1, ..., uint64_t a6);- Each wrapper uses inline assembly with explicit register moves and a - full clobber list to ensure correctness. You can use these directly if - you need a syscall not yet covered by the typed API. -
- -Typed API Wrappers
-
- The montauk:: namespace provides type-safe wrappers around
- the raw syscalls. These are the recommended interface for application
- code. The full reference follows below.
-
Syscall Reference
-- MontaukOS v0.1.0 exposes 26 syscalls (numbers - 0–25), organized into 10 categories. -
- - -Process Process Management
- -SYS_EXIT (0) — Terminate the current process
-[[noreturn]] void montauk::exit(int code = 0);- Terminates the calling process. The exit code is currently unused by - the kernel but reserved for future wait/status support. - Control never returns to the caller. -
- -SYS_YIELD (1) — Yield the CPU
-void montauk::yield();- Voluntarily yields the remainder of the current time slice, allowing - the scheduler to run another ready process immediately. -
- -SYS_SLEEP_MS (2) — Sleep for a duration
-void montauk::sleep_ms(uint64_t ms);
- Suspends the calling process for at least ms
- milliseconds. Resolution depends on the APIC timer tick rate.
-
SYS_GETPID (3) — Get process ID
-int montauk::getpid();
- Returns the PID of the calling process. Returns -1 if
- called from the idle context (should not happen in userspace).
-
SYS_SPAWN (20) — Spawn a new process
-int montauk::spawn(const char* path, const char* args = nullptr);
- Loads the ELF binary at path (a VFS path like
- "0:/hello.elf") and spawns it as a new process.
- Returns the new process's PID on success, or -1 on
- failure (no free slots, invalid ELF, file not found).
- The optional args string (up to 255 characters) is
- copied into the new process and can be retrieved with
- SYS_GETARGS.
-
SYS_WAITPID (23) — Wait for a process to exit
-void montauk::waitpid(int pid);
- Blocks the calling process until the process identified by
- pid has exited. Internally yields the CPU in a loop
- until the target process is no longer alive. This is used by the
- shell to wait for foreground processes (e.g. run
- command) so that keyboard input and the terminal are not shared
- simultaneously.
-
SYS_GETARGS (25) — Get process arguments
-int montauk::getargs(char* buf, uint64_t maxLen);
- Copies the argument string passed to the current process (via
- spawn()) into buf, writing at most
- maxLen - 1 characters plus a null terminator.
- Returns the number of characters copied, or -1 on error.
- If no arguments were provided at spawn time, the buffer will be empty.
-
Console Console I/O
- -SYS_PRINT (4) — Print a string
-void montauk::print(const char* text);
- Writes a null-terminated string to the kernel terminal. Supports
- newlines (\n) and standard printable ASCII.
-
SYS_PUTCHAR (5) — Print a single character
-void montauk::putchar(char c);- Writes a single character to the kernel terminal. Useful for building - output character by character (e.g. printing integers). -
- - -File I/O File I/O
- -SYS_OPEN (6) — Open a file
-int montauk::open(const char* path);
- Opens a file on the VFS. Paths use the format
- "<device>:/<name>" (e.g. "0:/hello.elf"
- for the ramdisk). Returns a non-negative handle on success, or a
- negative value on error.
-
SYS_READ (7) — Read from a file
-int montauk::read(int handle, uint8_t* buf, uint64_t offset, uint64_t size);
- Reads up to size bytes from the file at the given byte
- offset into buf. Returns the number of bytes
- actually read, or a negative value on error. Does not maintain an
- implicit file position — the offset is explicit on every call.
-
SYS_GETSIZE (8) — Get file size
-uint64_t montauk::getsize(int handle);Returns the total size (in bytes) of the file associated with handle.
SYS_CLOSE (9) — Close a file
-void montauk::close(int handle);Closes the file handle and releases associated kernel resources.
- -SYS_READDIR (10) — List directory entries
-int montauk::readdir(const char* path, const char** names, int max);
- Reads up to max directory entries from path.
- Entry name pointers are written into the names array.
- The kernel allocates a user-accessible page for the string data
- automatically. Returns the number of entries read, or
- ≤ 0 on error/empty directory. Maximum 64 entries per call.
-
Memory Memory Management
- -
- Memory allocation has two layers: low-level syscalls
- that map pages from the kernel, and a userspace heap
- that provides malloc/mfree on top.
-
Userspace Heap (recommended)
-programs/include/montauk/heap.h
-
- Include <montauk/heap.h> for a proper free-list
- allocator that runs entirely in userspace. It calls
- SYS_ALLOC internally to obtain pages and manages
- sub-page allocations with a linked free list — adapted from
- the kernel's own HeapAllocator.
-
void* montauk::malloc(uint64_t size);
- Allocates size bytes from the userspace free list.
- Returns a 16-byte-aligned pointer, or nullptr on failure.
- When the free list is exhausted, it transparently requests more pages
- from the kernel via SYS_ALLOC (minimum 16 KiB growth).
-
void montauk::mfree(void* ptr);
- Returns the block to the userspace free list.
- No syscall is made — the memory stays mapped
- and is immediately available for future malloc calls.
- Passing nullptr is a safe no-op.
-
void* montauk::realloc(void* ptr, uint64_t size);
- Resizes the allocation at ptr to size bytes.
- Allocates a new block, copies the smaller of old/new sizes, and frees
- the old block. If ptr is nullptr, behaves
- like malloc.
-
Low-Level Page Syscalls
- -SYS_ALLOC (11) — Map pages
-void* montauk::alloc(uint64_t size);
- Maps size bytes of zeroed physical pages into the
- process's address space (starting at 0x40000000).
- The size is rounded up to the nearest page boundary (4 KiB).
- Returns a pointer to the mapped region, or nullptr on
- failure. This is the backing primitive for
- montauk::malloc — most programs should use the heap
- API instead of calling this directly.
-
SYS_FREE (12) — Unmap pages (no-op)
-void montauk::free(void* ptr);
- Reserved for future page-level unmapping. Currently a no-op —
- pages are reclaimed when the process exits. Use
- montauk::mfree for heap allocations.
-
Time Timekeeping
- -SYS_GETTICKS (13) — Get tick count
-uint64_t montauk::get_ticks();- Returns the number of APIC timer ticks since boot. The tick rate - depends on the hardware and APIC timer calibration. -
- -SYS_GETMILLISECONDS (14) — Get milliseconds since boot
-uint64_t montauk::get_milliseconds();- Returns wall-clock milliseconds elapsed since boot. Useful for - calculating uptime or measuring durations. -
- - -System System Information
- -SYS_GETINFO (15) — Get OS information
-void montauk::get_info(Montauk::SysInfo* info);
- Fills in a SysInfo structure
- with the OS name, version string, API version number, and maximum
- process count.
-
Keyboard Keyboard Input
- -SYS_ISKEYAVAILABLE (16) — Check for pending key
-bool montauk::is_key_available();
- Returns true if a key event is available in the PS/2
- keyboard buffer. Non-blocking.
-
SYS_GETKEY (17) — Get a key event
-void montauk::getkey(Montauk::KeyEvent* out);
- Fills in a KeyEvent
- structure with the next keyboard event (press or release), including
- scancode, ASCII translation, and modifier state (Shift, Ctrl, Alt).
-
SYS_GETCHAR (18) — Read a character (blocking)
-char montauk::getchar();- Blocks until a printable character key-press is available, then - returns the ASCII character. This is the simplest way to read - interactive text input. -
- - -Network Networking
- -SYS_PING (19) — Send an ICMP echo request
-int32_t montauk::ping(uint32_t ip, uint32_t timeoutMs = 3000);
- Sends an ICMP echo request to ip and waits up to
- timeoutMs milliseconds for a reply. The IP address is in
- little-endian byte order (e.g. 10.0.2.2 →
- 0x0202000A).
- Returns the round-trip time in milliseconds on success, or
- -1 on timeout.
-
Framebuffer Framebuffer Access
- -SYS_FBINFO (21) — Get framebuffer information
-void montauk::fb_info(Montauk::FbInfo* info);
- Fills in a FbInfo structure
- with the framebuffer dimensions, pitch (bytes per scanline), and
- bits per pixel. Call this before SYS_FBMAP to learn the
- framebuffer geometry.
-
SYS_FBMAP (22) — Map framebuffer into process memory
-void* montauk::fb_map();
- Maps the physical framebuffer into the calling process's address
- space at 0x50000000 and returns the user virtual address.
- The mapped region covers height × pitch bytes. Each
- pixel is a 32-bit value in 0xAARRGGBB format (blue in
- the low byte). Writing to this memory directly updates the screen.
-
Terminal Terminal
- -SYS_TERMSIZE (24) — Get terminal dimensions
-void montauk::termsize(int* cols, int* rows);
- Returns the current terminal dimensions (character grid) via the
- two output pointers. Columns are packed in the low 32 bits and
- rows in the high 32 bits of the raw return value; the typed
- wrapper unpacks them for you. Either pointer may be
- nullptr if you only need one dimension.
-
Data Structures
- -Montauk::SysInfo
-programs/include/Api/Syscall.hpp
-struct SysInfo {
- char osName[32]; // e.g. "MontaukOS"
- char osVersion[32]; // e.g. "0.1.0"
- uint32_t apiVersion; // Current: 2
- uint32_t maxProcesses; // Current: 16
-};Montauk::FbInfo
-struct FbInfo {
- uint64_t width; // Framebuffer width in pixels
- uint64_t height; // Framebuffer height in pixels
- uint64_t pitch; // Bytes per scanline
- uint64_t bpp; // Bits per pixel (always 32)
- uint64_t userAddr; // Reserved (0 until mapped via SYS_FBMAP)
-};Montauk::KeyEvent
-struct KeyEvent {
- uint8_t scancode; // Raw PS/2 scancode
- char ascii; // Translated ASCII character (0 if non-printable)
- bool pressed; // true = key down, false = key up
- bool shift; // Shift modifier active
- bool ctrl; // Ctrl modifier active
- bool alt; // Alt modifier active
-};Montauk::SyscallFrame (kernel only)
-struct SyscallFrame {
- uint64_t r15, r14, r13, r12, rbp, rbx; // callee-saved
- uint64_t arg6, arg5, arg4, arg3, arg2, arg1;
- uint64_t syscall_nr;
- uint64_t user_rflags, user_rip, user_rsp;
-};
- This is the stack frame pushed by SyscallEntry.asm and
- passed to SyscallDispatch. Userspace code never sees this
- directly.
-
Shell Application Walkthrough
-programs/src/shell/main.cpp
-- The built-in shell is the best example of a real MontaukOS application. - It demonstrates most of the available syscalls. -
- -Initialization
-extern "C" void _start() {
- montauk::print("\n MontaukOS Shell v0.1\n");
- montauk::print(" Type 'help' for available commands.\n\n");
-
- char line[256];
- int pos = 0;
- prompt();
-
- while (true) {
- char c = montauk::getchar(); // blocking read
- // ... handle input, echo, backspace ...
- }
-}
- The shell uses montauk::getchar() in a loop for blocking
- character-by-character input, manually handling echo and backspace.
-
Shell Commands & Syscalls Used
-| Command | Description | Syscalls Used |
|---|---|---|
help |
- Print available commands | -SYS_PRINT |
-
info |
- Show OS name, version, API version | -SYS_GETINFO, SYS_PRINT, SYS_PUTCHAR |
-
man <topic> |
- Fullscreen manual page viewer | -SYS_OPEN, SYS_GETSIZE, SYS_READ, SYS_CLOSE, SYS_ALLOC, SYS_TERMSIZE, SYS_GETKEY, SYS_PRINT, SYS_PUTCHAR |
-
ls |
- List ramdisk files | -SYS_READDIR, SYS_PRINT |
-
cat <file> |
- Display file contents in 512-byte chunks | -SYS_OPEN, SYS_GETSIZE, SYS_READ, SYS_CLOSE, SYS_PRINT |
-
run <file> |
- Spawn a new process and wait for it to exit | -SYS_SPAWN, SYS_WAITPID, SYS_PRINT |
-
ping <ip> |
- Send 4 ICMP echo requests | -SYS_PING, SYS_SLEEP_MS, SYS_PRINT, SYS_PUTCHAR |
-
uptime |
- Show uptime in minutes, seconds, ms | -SYS_GETMILLISECONDS, SYS_PRINT, SYS_PUTCHAR |
-
clear |
- Scroll past visible content | -SYS_PUTCHAR |
-
exit |
- Terminate the shell | -SYS_PRINT, SYS_EXIT |
-
Pattern: Reading a File
-The shell's cat implementation shows the standard file-reading pattern:
// 1. Build VFS path
-char path[128];
-// ... copy "0:/" + filename into path ...
-
-// 2. Open
-int handle = montauk::open(path);
-if (handle < 0) { /* error */ return; }
-
-// 3. Get size
-uint64_t size = montauk::getsize(handle);
-
-// 4. Read in chunks
-uint8_t buf[512];
-uint64_t offset = 0;
-while (offset < size) {
- uint64_t chunk = size - offset;
- if (chunk > sizeof(buf) - 1) chunk = sizeof(buf) - 1;
- int bytesRead = montauk::read(handle, buf, offset, chunk);
- if (bytesRead <= 0) break;
- buf[bytesRead] = '\0';
- montauk::print((const char*)buf);
- offset += bytesRead;
-}
-
-// 5. Close
-montauk::close(handle);Pattern: IP Networking
-// Parse "10.0.2.2" into uint32_t in little-endian order
-uint32_t ip;
-parse_ip("10.0.2.2", &ip); // ip = 0x0202000A
-
-// Send 4 pings with 1-second intervals
-for (int i = 0; i < 4; i++) {
- int32_t rtt = montauk::ping(ip, 3000);
- if (rtt < 0) { /* timeout */ }
- else { /* reply in rtt ms */ }
- if (i < 3) montauk::sleep_ms(1000);
-}Adding New Syscalls
-- Adding a syscall requires changes in 3 files - (kernel-side) and 2 files (userspace-side). Follow - these steps: -
- -Step 1: Assign a Syscall Number
-- Add a new constant in both copies of the syscall - number definitions. The numbers must match exactly. -
-kernel/src/Api/Syscall.hpp
-programs/include/Api/Syscall.hpp
-static constexpr uint64_t SYS_MYFUNC = 26; // next available numberStep 2: Implement the Kernel Handler
-kernel/src/Api/Syscall.cpp
-Add a static function implementing your syscall's logic:
-static int64_t Sys_MyFunc(uint64_t arg1, const char* arg2) {
- // Your kernel-side implementation here.
- // You have full access to kernel subsystems.
- return 0;
-}Step 3: Add the Dispatch Case
-kernel/src/Api/Syscall.cpp — SyscallDispatch()
case SYS_MYFUNC:
- return (int64_t)Sys_MyFunc(frame->arg1, (const char*)frame->arg2);
- Arguments are accessed through frame->arg1 through
- frame->arg6, corresponding to RDI, RSI, RDX, R10, R8,
- R9 respectively.
-
Step 4: Add a Typed Userspace Wrapper
-programs/include/montauk/syscall.h
-inline int64_t my_func(uint64_t arg1, const char* arg2) {
- return syscall2(Montauk::SYS_MYFUNC, arg1, (uint64_t)arg2);
-}
- Choose the appropriate syscallN variant based on the
- number of arguments. Cast pointer types to uint64_t.
-
Step 5: Update the Log Message (optional)
-kernel/src/Api/Syscall.cpp — InitializeSyscalls()
- Update the boot log to reflect the new syscall count: -
-// Change "26 syscalls" to "27 syscalls"
-Kt::KernelLogStream(Kt::OK, "Syscall") << "SYSCALL/SYSRET initialized (LSTAR="
- << kcp::hex << (uint64_t)SyscallEntry << kcp::dec << ", 27 syscalls)";Complete Checklist
-| # | File | Change |
|---|---|---|
| 1 | kernel/src/Api/Syscall.hpp | Add SYS_MYFUNC constant (+ any new structs) |
| 2 | kernel/src/Api/Syscall.cpp | Add Sys_MyFunc() implementation + dispatch case |
| 3 | programs/include/Api/Syscall.hpp | Add matching SYS_MYFUNC constant (+ any new structs) |
| 4 | programs/include/montauk/syscall.h | Add typed wrapper in montauk:: namespace |
| 5 | (optional) | Update syscall count in boot log |
Process Memory Model
-- Each process gets its own PML4 page table. The kernel half - (upper 256 entries) is shared; the lower half is per-process. -
-| Region | Virtual Address | Size | Purpose |
|---|---|---|---|
| Exit stub | 0x3FF000 | 4 KiB | Auto-exit trampoline (calls SYS_EXIT(0) if _start returns) |
| Program code | 0x400000+ | Varies | ELF .text, .rodata, .data, .bss |
| User heap | 0x40000000+ | Grows up | Page pool (SYS_ALLOC); managed by userspace free-list heap |
| Framebuffer | 0x50000000+ | height × pitch | Mapped by SYS_FBMAP; direct pixel access (32-bit ARGB) |
| User stack | 0x7FFFFEF000–0x7FFFFFF000 | 16 KiB (4 pages) | Grows down from 0x7FFFFFF000 |
- The ELF loader maps PT_LOAD segments with user-accessible page flags. - BSS is zero-initialized automatically (pages are allocated zeroed). -
- - -Current Limitations
--
-
- No standard library (C++) — there is no libc++ for C++ programs.
- String operations, integer formatting, etc. must be written manually (see the shell for
- examples). A minimal C standard library exists for C programs (used by the DOOM port) in
-
programs/include/libc/with headers likestdio.h, -stdlib.h, andstring.h.
- - No dynamic linking — all programs are statically linked. -
- No argc/argv —
_startreceives no -argc/argv. Usemontauk::getargs()- to retrieve a single argument string (max 255 chars) passed via -montauk::spawn().
- - No page-level
free()— -SYS_FREEis a no-op; pages mapped viaSYS_ALLOC- are only reclaimed when the process exits. However, the userspace heap - (montauk::malloc/montauk::mfree) does reuse - memory via a free list.
- - 16 process slots — the scheduler supports a - maximum of 16 concurrent processes. -
- No writable file system — the VFS is read-only - (backed by the boot ramdisk). -
- Shared console — all processes write to the - same terminal; there is no per-process stdout. -
- Shared keyboard — keyboard input is global; - there is no input focus or multiplexing. -
- 10 ms scheduling granularity — preemptive - round-robin with a fixed 10 ms time slice. -
-
- MontaukOS Documentation — Copyright © 2025-2026 Daniel Hammer -
- -