From: ian@beathoven.com (paramucho) X-Newsgroups: vmsnet.pdp-11 Subject: Re: DOS/BATCH and/or XXDP File Structure info? Organization: www.beathoven.com On Mon, 18 Jun 2001 18:40:41 -0400, Will Kranz wrote: >Hello again: >John Wilson's newest version of PUTR is impressive. >Sadly for me it doesn't seem to solve the problem of creating a >bootable XXDP TU58 tape image (Initialize isn't valid for format above). > >However his documentation suggests the images are a varient of >DOS/BATCH. Can someone point me to documentation on the file >structure and key boot offsets so I can write an initialize routine? >I've figured out a lot of it by inspection, but pieces are missing. > Will, this is an extract from an effort of mine to document XXDP. It's just a draft, so it may have errors. Well, it's sure to have errors. Ian ______________________________________________________________________ 2. Disk Structure NOTE This article forms part of an attempt to document XXDP. The information is based on a preliminary examination of XXDP disks and the scant documentation available on the web. Thus omissions and errors are certain. Any help in extending this document and in correcting errors is be appreciated. Contact me at ian@hammo.com. 2. DISK STRUCTURE This section describes the structure of an XXDP disk volume along with disk initialization, bootstrap and monitor initialization. I suspect that XXDP shares the same philosophy, if not the same data structures, of the DOS-11 disk structure. 2.1 DISK OVERVIEW The basics of the disk structure are illustrated by the layout of the sample XXDPV1 disk: block size area flags area 000000/0 1 BOO FB C Bootstrap 000001/1 1 HOM FB C Home block 000002/2 12. MAP HB L Block allocation bitmap 000030/24. 146. MFD HB L Directory 000252/170. 30. MON HB C Monitor image 000310/200. vary DAT HBDLC File storage area Where: Mapping F = Fixed disk area H = Mapped by home block B = Mapped by bitmap D = Mapped by directory Storage L = Linked file C = Contiguous file There are six major areas identified above: o BOOTSTRAP: A bog-standard PDP-11 bootstrap located in block zero. See below for a description of the operation of the bootstrap. o HOME BLOCK: The XXDP home block contains information about the disk volume. See section 2.3. o BLOCK ALLOCATION BITMAP: This linked file maps each disk block with a single bit flag. See section 2.4. o MONITOR IMAGE: The monitor image is converted and stored in this area as a contiguous file (V1 SAVM, V2 CREATE). The first block of the monitor image is copied to block zero of the device, creating the bootstrap. See sections 2.6 through 2.9. o DIRECTORY: A single-level directory, implemented as a linked file. See section 2.5 o DATA STORAGE: The remainder of the disk is allocated to file storage. There are four forms of mapping: o FIXED DISK AREA: The boot block and home block are fixed at blocks zero and one on all XXDP system disk volumes (the same holds all the PDP-11 systems I know off). o HOME BLOCK MAPPING: the home block contains parameters that map the bitmap, system image, directory and data area. o BITMAP BLOCK MAPPING: indicates which blocks are in use or free. o DIRECTORY MAPPING: the directory maps the linked and contiguous files in the data area. There are two basic kinds of file storage: o LINKED FILES: Linked files reserve the first word of each block for the block number of the succeeding block. See section 2.2. o CONTIGUOUS FILES: Data is stored in a single, continuous disk area. See section 2.2. Two features are notable for their absence: o CLUSTERING: The simplicity of the disk structure would make it easy to add clustering. However, XXDP had neither the design discipline, the available space or the requirement for a disk clusters. o SUB-DIRECTORIES: A single master file directory is present. Again, it would be reasonably easy to add a sub-directory structure since the MFD itself is just a file. ______________________________________________________________________ 2.2 File Storage 2.2 FILE STORAGE o LINKED FILES: The map, directory and most files are linked file structures where the first word of each block contains the block number of the succeeding block. The first word of the last block in a chain is zero. Here is an example of a four block file block link data 350 351 .... First block 351 370 .... 370 353 .... 353 0 .... Last block Linked files must be interpreted, during reading and writing, to remove or insert the block links. This imposes block-at-a-time I/O. Random access I/O requires a sequential search through a file for a given block. Linked files can be extended (and possibly truncated). Note: The linked file structure handled the early papertape-based file structures easily: these were, by their heritage, inherently sequential. Note: The bitmap and directory are both "linked files", but these two files exist outside the directory structure. Note: I have a source MACRO program that converts an "XXDP" loader program to a task or save image. The program "skips" the first two bytes of the file: what it skips is the block linkage. Of coures, the conversion utility would fail with a "contiguous" file. o CONTIGUOUS FILES: These files are stored in a contiguous disk area without a linkage structure. Contiguous files may be read and written without interpretation. They were called "Fast" for that reason: they supported multi-block and random-access I/O. The directory provides all the information that is required to read and write any or all of the file: the start block and the length of the file. Contiguous files were probably not extendable. The last block entry in the directory was probably maintained for compatibility. Two files on XXDPV1 are marked contiguous. XXDPV2 has none. Some utilities refuse to work with contiguous files (indicating that they are deblocking linked files internally!). Note: The block-at-a-time I/O required for linked files, was a real performance killer on early floppy disks. The sudden arrival and departure of contiguous files may have coincided with the floppy- based deployment of XXDP. Note: the bootstrap, homeblock and monitor are stored as contiguous files, but these exist outside the directory system. ______________________________________________________________________ 2.3 Home Block 2.3 HOME BLOCK The XXDP home block contains information about the disk volume. It has little similarity with the later "standard" DEC home block structures. Here is an interpretation of a sample XXDPV1 home block. XXDPV2 values are shown in parentheses where the differ from XXDPV1. address contents usage 001000 000000 0 Link to next block (zero) 001002 000030 24. First block of directory. 001004 000222 146. Directory length, in blocks. 001006 000002 2 First block of bitmap. 001010 000026 22. Bitmap length, in blocks. 001012 000001 1 ?? 001014 000000 0 ?? 001016 047754 20460. Disk size in blocks (10MB = 20480*512) 001020 000310 200. Start block of data area. (000313 203.) Start block of data area on V2. 001022 000001 1 ?? 001024 000000 0 ?? 001026 000252 170. B Start block of monitor system image. 001030 000000 0. ?? 001032 000100 64. No idea what this is, but 1032/1034 (001000 512.) parallels 1036/1040, along with V4 001034 000377 255. changes. (000632 410.) 001036 000100 64. (001000 512.) 001040 000777 511. (001456 814.) The only value referenced by the bootstrap is location 1026: the start block of the monitor system image. Note: An examination of the ZERO and LOAD utilities, which initialize a new volume, might provide more information on the remaining values in the home block. Note: While the location of the bootstrap and homeblock is fixed, the position of the monitor, directory and data areas is defined by the home block and could, theoretically, be ordered in other ways. ______________________________________________________________________ 2.4 Bitmap 2.4 BITMAP Blocks 2 through 25. contain the block allocation bitmap, where a bit value of one indicates that a block is in use. Each bitmap block has the following header: 000 .word successor Block number of next block in bitmap 002 .word predecessor Block number of predecessor block in bitmap 004 .word units Number of words in this map segment 012 .word 2? ? 014 .word flags... First word of bitmap ... 176 .word flags Last word of bitmap Each bitmap block maps 960 blocks on the disk. There are 22 bit map blocks, mapping potentially a total of 21120. blocks. However the disk has only 20480. blocks. In fact location 016 of the home defines the maximum disk block available as 20460. 001016 047754 20460. Disk size in blocks (10MB = 20480*512) 002000 000003 3 Successor block 002002 000001 1 Predecessor block (1 is a bit strange) 002004 000074 60. Bitmap length in words 002006 000002 2 ? 002010 177777 60. bitmap words mapping 960. blocks. 002012 177777 ... 002174 177777 002176 177777 002200 000000 Remaining 75% of block is unused. 002202 000000 The final block in the bitmap does not have a successor link. 027000 000000 027002 000025 027004 000074 etc The bitmap flags for the boot block, home block, bitmap, directory and system image are presumably set during disk initialization. Directory operations that create or delete file blocks must search the bitmap for free blocks. Since only the first 64 words of a block are used, the system could, theoretically, perform bit map operations using a single 64 word buffer. Note: The double link (successor and predecessor), permit a backward walk through the bitmap, which might have been aimed at optimising the use of DECtape. ______________________________________________________________________ 2.5 Master File Directory 2.5 MASTER FILE DIRECTORY The start and length of the directory are specified in the home block. Each directory segment consists of a link word, 29 directory entries and three unused words. The link word specifies the next directory segment (i.e. block). The link of the final segment is zero. The nine word directory entry has the following format: 000 .rad50 /FILNAMTYP/ 6 character filename, 3 character type. 006 .word Date File date and Contiguous flag 010 .word Unknown Unknown 012 .word Start First physical block of file 014 .word Length Total length of file in blocks 010 .word Last Last physical block of file 020 .word Flags Flags Filename: A standard PDP-11 six character name and three character filetype (XXDP uses the outdated term "extension"). Valid characters are A-Z and 0-9. The wildcards are represented by '*' and '?'. Deleted: Deleted files are flagged with a null file name (all three words). Date: The year, less 1970, is multiplied by 1000 and the day-of- year is added. 15-bits are available. The date overflows in year 2003. The display format overflowed in year 2000. Contiguous: The high-order bit of date word is set to mark contiguous files. Contiguous files are marked with a 'C' in listings. Note: the SETUP program does not support contiguous files. Unknown: Radically varying values. Might be a UIC from DOS. Start: The physical block number of the first block of the file. Length: The length of the file, in blocks. Last: The physical block number of the last block of the file. Used when adding blocks to a file. Flags: Unknown. Does not seem to be used before V2. Note: XXDP does not support sub-directories, although they could have been implemented as special files. Note: .SYS version information, displayed by the V2 directory utility, is extracted from file images and not from the directory entries. ______________________________________________________________________ 2.6 Disk Initialization 2.6 DISK INITIALISATION Creating an XXDP disk volume is fairly straight-forward: o Obtain the base initialization parameters: the size of of the disk, bitmap, monitor and directory. These parameters seem to be fixed. o Create the home block. o Create the block bitmap. o Create the initial empty directory segment(s). o Optionally, create a dummy bootstrap block The following V1 UPD1/UPD2 commands initialize DL1: ZERO DL1: ; initialize basic structures LOAD DL0:RLDP.BIN ; load the monitor SAVM DL1: ; save monitor to block 26. DUMP DL1:RLDP.BIN ; copy monitor LOAD DL0:UPD1.BIN ; copy upd1 DUMP DL1:UPD1.BIN ; LOAD DL0:UPD2.BIN ; copy upd2 DUMP DL1:UPD2.BIN ; The V2 CREATE command combines some of the commands above. 2.7 SYSTEM DISK INITIALISATION A separate operation copies a target monitor to the system. HMDLD0.SYS is used as an example. o HMDLD0.SYS contains the DL: monitor executable stored in absolute loader format. This is converted to a memory image. o The monitor is stored as a contiguous image starting at the block specified in location 026 of the Home Block. o The first block of the memory image is copied to block zero of the device, where it becomes the bootstrap. o Required system files are then manually copied to the volume. ______________________________________________________________________ 2.7 Bootstrap Operation 2.8 BOOTSTRAP OPERATION The XXDP+ boot sequence is as follows: o Read the home block into a buffer at 1000. o Extract required system parameters from home block. o Read blocks 1 to 15 of the XXDP monitor (starting with block 0253) into memory starting at location 1000. Block zero of the monitor is the boot block which has already been loaded (and modified). o Pass control to the XXDP monitor. Here is an edited trace of the bootstrap disk activity: Read Unit=0 Block=1|1 Buffer=1000 Count=512. Read Unit=0 Block=253|171 Buffer=1000 Count=512. Read Unit=0 Block=254|172 Buffer=2000 Count=512. ... more of the same ... Read Unit=0 Block=270|184 Buffer=16000 Count=512. Read Unit=0 Block=271|185 Buffer=17000 Count=512. The XXDP V2 bootstrap is considerably more ornate. ______________________________________________________________________ 2.9 Monitor Startup 2.9 MONITOR STARTUP (V1) The monitor loads into low memory at 01000. The initiation code of the monitor relocates the monitor to high memory, leaving it in the 15xxxx (26kw) region. +----------+ | i/o page | +----------+ 160000 PDP-11 4kw I/O Page | monitor | +----------+ 15xxxx XXDP Monitor | hooks | Relocated components | | +----------+ | program | +----------+ 000200 Application Programs | vectors | +----------+ 000000 Interrupt Vectors The V1 Monitor then prompts for the date: CHMDLD0 XXDP+ DL MONITOR BOOTED VIA UNIT 0 28K NON-UNIBUS SYSTEM ENTER DATE (DD-MMM-YY): The V2 startup is more elaborate. It has not been studied in detail. ______________________________________________________________________ ---------- To unsubscribe (or subscribe) from (to) this list, send a message to info-pdp11-request@village.org, with the first line of the message body being "unsubscribe" or "subscribe", respectively (without the quotes). -------------------- In a subseqent note emailed directly to me on Date: Thu, 2 Aug 2001 Ian added the following information: -------------------- 3. File Structures 3. FILE STRUCTURES This section briefly outlines the major file XXDP structures, many of which were adopted from DOS/BATCH which in turn were based on PDP-11 papertape formats, using ASCII for text files and Formatted Binary for most other structures. More investigation is required to decode many of the file structures listed below. 3.1 TEXT FILES (.txt) Text files are stored in 8-bit Ascii. Newline is represented by . The unused area of the final block is zero padded. Some files are terminated with Ascii EOF . 3.2 FORMATTED BINARY The formatted binary structure is used to wrap binary data in a simple byte stream. Nulls may precede and separate records. It has a two word header and a concluding checksum byte: .word 1 ; signature .word BC+6 ; data byte count + 6 .blkb BC ; data .byte CHK ; checksum The checksum, CHK, is the negated sum of the preceding bytes in the record, including the four bytes of the header. This structure is the fundamental vehicle of binary record structures. The loader and object formats are built on top of it as are other less well known or private structures (such as XXDP configuration files). 3.3 OBJECT FILES (.obj) Similar to standard PDP-11 object files, object record structures are layered on top of Formatted Binary records. TBS: Details of differences with the standard PDP-11 format. 3.4 LIBARY (.lib) DEC/X11 Object Libraries consisting of a number of headers followed by zero or more object files. The headers include an ascii name index, a Rad50 name index and a set of mapping records. TBS: Details of the library structure. 3.5 MAP (.map) TBS: Details of the map structure. 3.6 ABSOLUTE LOADER (.lda) Standard PDP-11 formatted binary loader format. Nulls may precede and separate records. The record header has three words: .word 1 ; signature .word bc+6 ; data byte count + 6 .word address ; load or transfer address .blkb bc ; load data .byte cs ; There are two record types. If the data count is greater than six, the record contains one or more bytes to be stored beginning with the load address. If the data count is exactly six, the address specifies the transfer address and the record concludes the file. If the transfer address is even, then control is passed immediately to that address. If the address is odd, the program is not executed. In a stand-alone environment, a [continue] from the halt causes the program to start execution at location 0200. 3.7 XXDP EXECUTABLE (.bin,.bic,.sys) An XXDP executable is stored in either Absolute Loader format or as a a contiguous file. Information in the system communication area (040- 060) is used by the monitor (e.g. mapped/unmapped image). TBS: The content of the system communication area and other aspects of images stored as contiguous files (e.g. start address). 3.8 XXDP DRIVERS TBS: Details of the driver structure. 3.9 CONFIGURATION FILES (.cnf) TBS: Details of the configuration file structure. 3.10 P-TABLES (.bin,.bic) Some .bin/.bic files support an internal parameter table structure used by SETUP to prepare programs for DRS. I haven't decoded this structure, but it would be quite useful for listing because it would supply the standard CSR addresses associated with the test programs. TBS: Details of the P-Table structure.