2024-12-10

<!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang=""> <head> <meta charset="utf-8" /> <meta name="generator" content="pandoc" /> <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" /> <title>File Access for Emulator Services</title> <style>

</style> </head> <body> In order to run useful programs, the VM/OS emulator needs to offer some means of accessing a file system. However, unchecked access to file storage is one of the most frequently abused methods to compromise a user’s account at best, and an entire host computer at worst. Similar to Uxn, VM/OS emulation services elects to sandbox access to files; that is to say, a VM/OS application only has access to the directory or directories it’s been explicitly granted access to. Unlike Uxn, however, multiple sandboxes can be established, each granting a view onto the host’s local filesystem. To facilitate the bootstrapping of my longer-term VM/OS vision, it would be worthwhile to implement as many development tools for VM/OS using VM/OS itself. This allows us to eat our dogfood more regularly, and especially during a time when early design choices are at their most influential. And, to do that, I need the ability to access persistent storage very early in the development of VM/OS – long before we would otherwise have a functional filesystem of its own. <h1 id="naming-conventions-for-files">Naming Conventions for Files</h1> For the sake of implementation simplicity, I choose to implement a multi-rooted filesystem implementation. Instead of having a single root directory under which all other filesystems appear, I instead implement multiple root directories. Each root has its own (unique) name, and is (logically, if not in fact) managed by its own handler. Unlike systems with crippled multi-rooted filesystems, such as MS-DOS or Windows, the naming conventions of a VM/OS path is intended to look like we have a single root. This allows for a future version of VM/OS to migrate to a single-rooted filesystem when it’s convenient. The separation between device or filesystem and the rest of the path is uniform. The syntax of a filename in Windows or DOS requires a drive component to be isolated from its file path via a colon (<code>:</code>), while path elements relative to the root of a given drive are separated by back-slashes (<code>\</code>). Worse, each drive unit has its own idea of what the application’s current directory is. Thus, giving the name of a file like <code>C:FOO.TXT</code> assumes you know what the current directory for C: is. It is not guaranteed to be the same thing as <code>C:\FOO.TXT</code>! From the perspective of the software running under VM/OS, an application locates files by fusing a consistent system of naming files and other useful resources. Such names are always absolute; meaning, VM/OS does not understand the concept of a relative vs. absolute path name. Because all filenames at the application/emulator boundary are absolute, there is no need for a prefix notation indicating an “absolute” path name. More importantly, names always include enough information to allow the emulator, if not the host OS it runs on, to locate on which storage device the file or resource may actually be found on. Thus, a filename generally takes the following form: <pre><code>location / rest-of-pathname</code></pre> Hence, the first element of a filename is generally taken to be a device or filesystem name. The remaining text in a filename is then passed to the unique handler for the indicated device verbatim, without further interpretation by either the application or the VM/OS environment. <h1 id="sandboxes">Sandboxes</h1> One kind of (pseudo-)device is the VM/OS sandbox. Until native DASD devices are supported, all files appear inside sandboxes and must be accessed through emulation services. A sandbox is just a privilege grant from the operator to the VM/OS application to be able to access host-local files at a well-known location in the host’s filesystem hierarchy. This is usually done through configuration settings of the VM/OS emulator or are passed explicitly on the command line used to launch the application. For example, let’s say a programmer is working on a development project and the local filesystem has the following overall shape to it: <ul> <li>home/ <ul> <li>vertigo/ <ul> <li>project_1/ <ul> <li>docs/ <ul> <li>HOWTO.txt</li> <li>README.txt</li> <li>adr/ <ul> <li>adr-0001.md</li> <li>adr-0002.md</li> <li>adr-0003.md</li> </ul></li> </ul></li> <li>src/ <ul> <li>main.asm</li> <li>mylib.inclib</li> <li>yourlib.inclib</li> </ul></li> <li>bin/</li> <li>tools/ <ul> <li>build.sh</li> <li>redo</li> </ul></li> </ul></li> <li>project_2/ <ul> <li>…etc…</li> </ul></li> </ul></li> </ul></li> </ul> We can launch a hypothetical VM/OS text editor to edit the <code>main.asm</code> source listing in a way which only grants the editor access to the <code>src</code> directory. <pre><code>$ vmos editor.bin sandbox "src=~/project_1/src" -- src/main.asm</code></pre> Similarly, if a steering council wants to work on the project’s architecture documentation records (ADRs), they can launch their editor granting exclusive access to just the ADR directory: <pre><code>$ vmos editor.bin sandbox "adr=$HOME/project_1/docs/adr" -- adr/adr-0004.md</code></pre> Someone looking to run their application across multiple directories has several options. One, you can be precise in your permission grants by including multiple sandboxes: <pre><code>$ vmos editor.bin sandbox "adr=project_1/docs/adr" "src=project_1/src" -- etc.</code></pre> Or, if you are more trusting, you can just provide a single grant that covers everything: <pre><code>$ vmos editor.bin sandbox "p1=/home/vertigo/project_1" -- p1/src/main.asm</code></pre> You can also use sandboxes which overlap each other to create convenient shorthand paths. These function in a way similar to symbolic links; although for VMS and AmigaOS users, these are better compared to assignments. For example, we can grant total access to Project_1, but having to type <code>p1/doc/adr/blah.md</code> all the time can be repetitive and error-prone. We can create a sandbox <code>adr</code> to make things more convenient. <pre><code>$ vmos editor.bin sandbox "p1=project_1" "adr=project_1/docs/adr" -- etc.</code></pre> Now, you can access the files in the whole project, but if you’re focusing on ADR documentation, you can directly reference them with, e.g., <code>adr/adr-0004.md</code>. <h1 id="sandbox-security">Sandbox Security</h1> Remember that file name parsing is ultimately the responsibility of individual handlers! The rules described below are intended to document how VM/OS sandboxes work specifically, and are not intended to be more broadly applicable. For example, if a future version of VM/OS creates a 9P bridge to a remote filesystem, these rules will no longer apply, as now you’re going to be playing by 9P’s rules instead. The following rules must be enforced to ensure that nothing can escape the sandbox. <ol type="1"> <li>Leading slashes are always skipped. This ensures that <code>/adr/adr-0004.md</code>, <code>///adr/adr-0004.md</code>, and <code>adr/adr-0004.md</code> all refer to the same file.</li> <li>Embedded multiple slashes are treated as a single slash. This ensures that <code>src///main.asm</code> refers to the same file as <code>src/main.asm</code>.</li> <li>No component of the filename may refer to the parent directory in most host filesystems. E.g., both Unix and Windows use <code>..</code> for this purpose, so that <code>/abc/../def/ghi</code> and <code>/def/ghi</code> are the same filename. However, once the file handler is resolved, it will only see an attempt to access a file named <code>../def/ghi</code>. It’s up to the handler to resolve the remainder of the filename. A path component that steps back a directory must not result in breaking out of the sandbox under any circumstances. One approach is to just forbid <code>..</code> from appearing at all in a filename; another is to try to canonize the filename before translating it to a host pathname.</li> </ol> Be extremely wary of symbolic links! Symbolic links should refer only to directories and files intended to be used by a VM/OS application. Beware of social engineering attempts to get you to create a symbolic link, say, <code>src</code>, which points to data irrelevant to the VM/OS environment (e.g., <code>/etc</code>). <h1 id="vmos-emulator-cli-syntax">VM/OS Emulator CLI Syntax</h1> We’ve already seen how the new emulator CLI syntax looks in the previous section, Sandboxes. However, the formal syntax for options is repeated below for completeness. vmos-cmd := <code>vmos</code> options options := vmos-options | vmos-options <code>--</code> app-options vmos-options := app-name sb-options | <code>FROM</code> app-name sb-options sb-options := <code>SANDBOX</code> sb-list sb-list := sb | sb sb-list sb := sb-name <code>=</code> local-path The use of keyword arguments/options is unconventional for Unix or Windows platforms, but is conventional for Tripos, AmigaDOS, and by extension, my long-term VM/OS vision. Keywords are case insensitive unless specified otherwise. <h1 id="supporting-system-calls-preliminary">Supporting System Calls (PRELIMINARY)</h1> To support file access under emulator control, a number of system calls are added to the emulator services API. As with other contemporary filesystem APIs, a file is treated as an unordered bag of bytes with no system-imposed interpretation. All file access is assumed to be in binary mode. The calling sequence for these API entry points is illustrated in the following code snip: <pre><code>ld a0,arg0(sp) ;load argument from memory addi a1,s0,arg1 ;compute argument from another register addi a2,t3,0 ;transfer argument as-is from another register

addi a7,x0,ecFunction ;Set ECALL service number ecall ;Invoke emulator service bne a0,x0,error_case ;If A0 != 0, an error happened. ;Use value in A1 (if any) in subsequent computation.</code></pre> To more conveniently notate these calling conventions, I use a pseudo-C-like syntax with register annotations, like so: <pre><code>ecFunction(arg0, arg1, arg2) --> error, result

Some functions require passing a string, vector, or a buffer instead of a scalar value. Strings and buffers both require a base address indicating the first element and a length. These are notated like so: <pre><code>ecFunction(filename, buffer, flags) --> error, result

The convention here is that the lower-numbered register (e.g., <code>a0</code> or <code>a2</code>) contains the base address to the buffer, while the higher numbered register (i.e., <code>a1</code> or <code>a3</code>) contains the length of data therein. For buffers, the length is measured in bytes. For vectors, the length is measured in elements (unless otherwise documented). Strings are considered vectors of bytes; thus, it just happens that for strings, the length passed is the byte-length of the string data. <h2 id="ecopen">ecOpen</h2> This opens a file identified by name. If an error occurs, return an error in register A1 (A0 is undefined). Errors can happen for the following reasons: <ul> <li>File is opened for reading or writing, but the file does not exist.</li> <li>File is opened for creation, but there’s no space left or there is a security issue in the host filesystem.</li> </ul> <pre><code>ecOpen(filename, ioflags) --> error, handle

<h2 id="ecclose">ecClose</h2> This closes a previously opened file. <pre><code>ecClose(handle) --> error

<h2 id="ecread">ecRead</h2> This reads a number of bytes from the file into the supplied buffer. At most <code>size</code> bytes will be transferred; however, a smaller number may be transferred if, e.g., VM/OS encounters the end of the file before filling up the buffer. If zero is returned, then the file is at the end; there’s just no further data to read. This is not an error condition. <pre><code>ecRead(handle, buffer) --> error, actual

<h2 id="ecwrite">ecWrite</h2> This writes a number of bytes from the supplied buffer to the file. At most <code>size</code> bytes will be transferred; however, a smaller number may be transferred if, e.g., VM/OS encounters the a maximum size quota before filling up the buffer. If zero is returned, then the file is at capacity; no further data can be written because there’s just no place to put it. This is not an error condition. <pre><code>ecWrite(handle, buffer) --> error, actual

<h2 id="ecseek">ecSeek</h2> This relocates the current read/write position of the file. The file position can be located anywhere within the file’s current extents. You can determine the size of the file by seeking to position 0 relative to the end of the file, then seeking back to its original location. NOTE: In most OSes, you must first flush the file before seeking to avoid filesystem corruption. This seems like a silly oversight to make; if you’re aware enough to document this condition which is obviously a bug (in the sense of maintaining the principle of least surprise), then you should be aware enough to fix this issue. In VM/OS, this system call always flushes before performing a seek operation. Thus, you can also use this call to flush buffers. If all you want to do is flush a file but do not want to relocate the read/write pointer, you can do so simply by seeking to position 0 relative to the current position. <pre><code>ecSeek(handle, position, whence) --> error, old_pos

<h2 id="ecdelete">ecDelete</h2> This function deletes a file by name, if it exists. The operation is idempotent; if the file does not exist, then nothing happens. Attempting to dispose of a directory is currently left unspecified. <pre><code>ecDelete(filename) --> error

<h2 id="ecrename">ecRename</h2> This function renames a file to something new. The new name must reside on the same device as the old name. <pre><code>ecRename(oldname, newname) --> error

<h1 id="conclusion">Conclusion</h1> This interface should provide the nascent VM/OS environment with the facility to process source files and produce its own build artifacts. The interface and file naming convention is believed to be future-proof with a future single-rooted filesystem implementation, while supporting well-confined sandboxes granting localized access to a host filesystem resources. I regret that I could not offer this in an asynchronous manner, thus taking advantage of the multitasking facilities VM/OS already provides; however, as intermediate build tools generally tend to be synchronous anyway, and local storage devices tend to be extremely fast these days, I didn’t think this would be a show-stopping issue. <h1 id="appendix-inspiration">APPENDIX: Inspiration</h1> The inspiration for interpreting filenames the way described above comes from an early home computer which, despite its reputation for being crippled by design, actually came with a system software design which was remarkably ahead of its time. I am speaking of the Texas Instruments TI-99/4 and /4A computer. It came as a shock to me when I learned how the TI-99 platform manages to handle support for installable filesystems. For example, in TI BASIC, you could load a program from an old-fashioned 90KB floppy disk like this: <pre><code>OLD "DSK1.MY-PROGRAM"</code></pre> This would work great if the disk containing MY-PROGRAM resided in unit DSK1. But, what if you don’t know which disk drive the floppy is inserted into, but you do know the name of the volume you assigned when you formatted it? That’s supported too: <pre><code>OLD "DSK.MY-VOL.MY-PROGRAM"</code></pre> Isn’t that neat? We would not see functionality like this in a commercially available product again until the release of the Commodore-Amiga in 1985. <pre><code>1> ; Access My-Program by physical device name 1> DF0:My-program 1> ; Now by volume name, not knowing which physical device it's in 1> My-Volume:My-Program</code></pre> This doesn’t stop with a flat directory structure, either. To be fair, the drivers supporting 90KB, 180KB, and 360KB floppy disks only recognize a single, flat, root directory; but surprisingly, there’s nothing in the system interface which mandates this. This is because the TI software delegates filename parsing to the specific device handler; to the system software itself, a filename is completely opaque beyond the search key used to locate the specific handler. Today, there’s a plurality of 3rd party or even home-made devices which implement hundreds of megabytes worth of storage, complete with support for sub-directories. The <a href="https://www.unige.ch/medecine/nouspikel/ti99/ideal.htm">IDEAL Project</a>, for example, supports not only virtual floppies, but also sub-directories in a manner significantly more natural than how CMD implemented sub-directory support for Commodore 8-bit computers. E.g., <code>.1.THROUGH.THE.FOREST.TO.GRNDMA/TXT</code>. This interface also extends itself to non-block devices as well. For example, if you wanted to use a modem to connect to a BBS, you might configure your terminal to access a device <code>RS232/1.SPEED=9600.HS=HW.FRAME=8N1</code>. Did you want to generate a 16-character random password easily? If you have the appropriate software loaded into RAM, you could use TI BASIC like this to generate it (forgive any syntax errors; I haven’t used a TI since 1981!): <pre class="basic"><code>100 OPEN #1:"RANDOM.ASCII",INTERNAL,INPUT,FIXED 16 110 INPUT #1:P$ 120 CLOSE #1 130 PRINT "PASSWORD IS: "&P$</code></pre> Remember the previously mentioned IDEAL project? There are even virtual files for implementing Blowfish encryption of arbitrary data blocks! Write your data to one virtual file and read back the encrypted result on another. So, how does all this work in practice? <h2 id="ti-99-design-summary">TI-99 Design Summary</h2> There are many web pages which describes all these mechanics better than I can here; however, I’ll briefly summarize so that the reader doesn’t need to flip back and forth between webpages all the time. NOTE: A word on notation before we continue; sometimes, I’ll refer to hexadecimal values using a dollar prefix, which is more or less industry standard notation for anyone not reared on Texas Instruments’ product line. For some reason, TI chose to use the greater-than symbol as its hex prefix. Therefore, <code>>4000</code> and <code>$4000</code> both mean exactly the same thing: <code>0x4000</code> for those of you who understand C notation. The system software divvies up the processor’s address space into 8KB chunks: >0000->1FFF, >2000->3FFF, and so forth. Some of these regions hold special significance in that they’re hard-wired for certain types of peripherals (e.g., HexBus devices, cartridge ROMs, etc.). The figure below is from https://www.unige.ch/medecine/nouspikel/ti99/architec.htm . <pre><code>>0000 ------------------+

When software wants to open a file, it needs to find a handler. The TI convention is to <a href="https://www.unige.ch/medecine/nouspikel/ti99/headers.htm">perform a “scan” of memory</a>, starting at address >0000, then moving to >2000, then moving to >4000, and so forth, looking for a suitable handler for the filename provided. At each 8K boundary, a data structure is checked to see if it is even worth checking that 8K block. For example, it is possible that an expansion device or cartridge is not plugged in; thus, its corresponding region of the address space is missing. Attempting to resolve memory pointers and other attributes in these regions would cause the computer to crash. This presence check structure looks something like this: <pre><code>x000 byte >AA ;Tag indicating a handler header x001 byte 1 ;Version of this data structure (I think) x002 byte 0 ;Number of "programs" x003 byte 0 ;padding x004 word init_list ;procedure list for power-on initialization x006 word 0 ;program list (we're not interested) x008 word dsr_list ;"Device Service Routine" list x00A word isr_list ;list of interrupt service routines</code></pre> NOTE: Although details differ, we wouldn’t see anything like this again until the Commodore-Amiga in 1985, vis-a-vis the <code>RomTag</code> structure described in the <code>Exec: Libraries</code> chapter of the <a href="https://archive.org/details/amiga-rom-kernal-reference-manual-libraries-and-devices/page/229/mode/2up">Amiga ROM Kernel Reference Manual, Libraries and Devices</a>. See also the documentation for the <a href="https://archive.org/details/amiga-rom-kernal-reference-manual-libraries-and-devices/page/525/mode/2up"><code>expansion.library</code></a>, in the same manual. The Device Service Routine (DSR) is what we’re most interested in here, as that provides the linkage between the client software and the device driver we’re looking for. Each node on the dsr_list looks like this: <pre><code>dsr_node0:

So, for example, the handlers for the floppy disk drive might look like this: <pre><code>dsk_node:

Since there are eight 8KB blocks in the processor’s 64KB address space, it follows that there are eight linked lists to check before locating a handler; and, assuming all those are populated, there can be any number of DSRs on each list. If we were to write a simple C pseudo-code to describe the search algorithm that is used, I reckon it would look a lot like the following:

<pre class="sourceCode c"><code class="sourceCode c"><a href="#cb25-1" aria-hidden="true"></a>typedef void HANDLER(void); <a href="#cb25-2" aria-hidden="true"></a>typedef struct node_block NODE; <a href="#cb25-3" aria-hidden="true"></a>typedef struct header_block HEADER; <a href="#cb25-4" aria-hidden="true"></a>typedef union scan_block SCAN; <a href="#cb25-5" aria-hidden="true"></a> <a href="#cb25-6" aria-hidden="true"></a>struct node_block { <a href="#cb25-7" aria-hidden="true"></a> NODE *next; <a href="#cb25-8" aria-hidden="true"></a> HANDLER *proc; <a href="#cb25-9" aria-hidden="true"></a> uint8_t name[8]; //name[0] == length of name <a href="#cb25-10" aria-hidden="true"></a>}; <a href="#cb25-11" aria-hidden="true"></a> <a href="#cb25-12" aria-hidden="true"></a>struct header_block { <a href="#cb25-13" aria-hidden="true"></a> char signature; <a href="#cb25-14" aria-hidden="true"></a> char version; <a href="#cb25-15" aria-hidden="true"></a> char n_progs; <a href="#cb25-16" aria-hidden="true"></a> char padding; <a href="#cb25-17" aria-hidden="true"></a> NODE *init_list; <a href="#cb25-18" aria-hidden="true"></a> NODE *prog_list; <a href="#cb25-19" aria-hidden="true"></a> NODE *dsr_list; <a href="#cb25-20" aria-hidden="true"></a> NODE *isr_list; <a href="#cb25-21" aria-hidden="true"></a>}; <a href="#cb25-22" aria-hidden="true"></a> <a href="#cb25-23" aria-hidden="true"></a>union scan_block { <a href="#cb25-24" aria-hidden="true"></a> HEADER h; <a href="#cb25-25" aria-hidden="true"></a> char gap[8192]; <a href="#cb25-26" aria-hidden="true"></a>}; <a href="#cb25-27" aria-hidden="true"></a> <a href="#cb25-28" aria-hidden="true"></a>HANDLER * <a href="#cb25-29" aria-hidden="true"></a>find_handler_in_block(char *prefix, SCAN *blk) { <a href="#cb25-30" aria-hidden="true"></a> NODE *pn = blk->h.dsr_list, *pnext; <a href="#cb25-31" aria-hidden="true"></a> HANDLER *phnd = NULL; <a href="#cb25-32" aria-hidden="true"></a> <a href="#cb25-33" aria-hidden="true"></a> while(pn != NULL) { <a href="#cb25-34" aria-hidden="true"></a> pnext = pn->next; <a href="#cb25-35" aria-hidden="true"></a> if(name_matches(prefix, pn->name)) { <a href="#cb25-36" aria-hidden="true"></a> phnd = pn->proc; <a href="#cb25-37" aria-hidden="true"></a> break; <a href="#cb25-38" aria-hidden="true"></a> } <a href="#cb25-39" aria-hidden="true"></a> <a href="#cb25-40" aria-hidden="true"></a> pn = pnext; <a href="#cb25-41" aria-hidden="true"></a> } <a href="#cb25-42" aria-hidden="true"></a> <a href="#cb25-43" aria-hidden="true"></a> return phnd; <a href="#cb25-44" aria-hidden="true"></a>} <a href="#cb25-45" aria-hidden="true"></a> <a href="#cb25-46" aria-hidden="true"></a>HANDLER * <a href="#cb25-47" aria-hidden="true"></a>find_handler(char *prefix) { <a href="#cb25-48" aria-hidden="true"></a> SCAN *bp = (SCAN *)0x0000; <a href="#cb25-49" aria-hidden="true"></a> HANDLER *phnd = NULL; <a href="#cb25-50" aria-hidden="true"></a> <a href="#cb25-51" aria-hidden="true"></a> do { <a href="#cb25-52" aria-hidden="true"></a> if((bp->h.signature == 0xAA) && (bp->h.version == 1)) { <a href="#cb25-53" aria-hidden="true"></a> phnd = find_handler_in_block(prefix, bp); <a href="#cb25-54" aria-hidden="true"></a> if(phnd != NULL) <a href="#cb25-55" aria-hidden="true"></a> break; <a href="#cb25-56" aria-hidden="true"></a> } <a href="#cb25-57" aria-hidden="true"></a> ++bp; <a href="#cb25-58" aria-hidden="true"></a> } while(bp != 0x0000); <a href="#cb25-59" aria-hidden="true"></a> <a href="#cb25-60" aria-hidden="true"></a> return phnd; <a href="#cb25-61" aria-hidden="true"></a>}</code></pre>