43. Dumping

43.1 Dumping Justification

The C code of XEmacs is just a Lisp engine with a lot of built-in primitives useful for writing an editor. The editor itself is written mostly in Lisp, and represents around 100K lines of code. Loading and executing the initialization of all this code takes a bit a time (five to ten times the usual startup time of current xemacs) and requires having all the lisp source files around. Having to reload them each time the editor is started would not be acceptable.

The solution to this problem is called dumping: the build process first creates the lisp engine under the name ‘temacs’, then runs it until it has finished loading and initializing all the lisp code, and eventually creates a new executable called ‘xemacs’ including both the object code in ‘temacs’, and, traditionally all the contents of the memory after the initialization. This last feature was termed “unexec”.

Unexec, while it worked, had a huge problem: the creation of the new executable from the actual contents of memory was an extremely system-specific process, quite error-prone, and interfered with a lot of system libraries (like malloc). This got worse over time with libraries using constructors which are automatically called when the program is started (even before main()); those constructors tend to crash when they were called multiple times, once before dumping and once after (IRIX 6.x ‘libz.so’ pulls in some C++ image libraries through dependencies which have this problem).

Writing the dumper was also one of the most difficult parts of porting XEmacs to a new operating system.

Oliver Galibert replaced this approach by adding explicit descrptions of objects in memory and implementing a “portable dumper”, which removes the need for unexec. See the rest of this section.

43.2 Overview

The portable dumping system has to:

1. At dump time, write all initialized, non-quickly-rebuildable data to a file named ‘xemacs.dmp’, along with all information needed for the reloading.
2. When starting xemacs, reload the dump file, reinitialize all pointers to this data, and adjust pointers within the dump file to reflect their new addresses. Also, rebuild all the quickly rebuildable data.

Note: As of 21.5.18, the dump file has been moved inside of the executable. There are occasional problems with this on some, rare, systems. The configure script will pick up these problems and default to an external dump file there. The build process is slower with the dump file inside the executable, so using an external dump file can be of value in development.

43.3 Data descriptions

The more complex task of the dumper is to be able to write memory blocks on the heap (lisp objects, i.e. lrecords, and C-allocated memory, such as structs and arrays) to disk and reload them at a different address, updating all the pointers they include in the process. This is done by using external data descriptions that give information about the layout of the blocks in memory.

The specification of these descriptions is in lrecord.h. A description of an lrecord is an array of struct memory_description. Each of these structs include a type, an offset in the block and some optional parameters depending on the type. For instance, here is the string description:

static const struct memory_description string_description[] = {
  { XD_BYTECOUNT,         offsetof (Lisp_String, size) },
  { XD_OPAQUE_DATA_PTR,   offsetof (Lisp_String, data), XD_INDIRECT(0, 1) },
  { XD_LISP_OBJECT,       offsetof (Lisp_String, plist) },
  { XD_END }
};

The first line indicates a member of type Bytecount, which is used by the next, indirect directive. The second means "there is a pointer to some opaque data in the field data". The length of said data is given by the expression XD_INDIRECT(0, 1), which means "the value in the 0th line of the description (welcome to C) plus one". The third line means "there is a Lisp_Object member plist in the Lisp_String structure". XD_END then ends the description.

This gives us all the information we need to move around what is pointed to by a memory block (C or lrecord) and, by transitivity, everything that it points to. The only missing information for dumping is the size of the block. For lrecords, this is part of the lrecord_implementation, so we don’t need to duplicate it. For C blocks we use a struct sized_memory_description, which includes a size field and a pointer to an associated array of memory_description.

43.4 Dumping phase

Dumping is done by calling the function pdump() (in ‘dumper.c’) which is invoked from Fdump_emacs (in ‘emacs.c’). This function performs a number of tasks.

43.4.1 Object inventory

The first task is to build the list of the objects to dump. This includes:

• lisp objects
• other memory blocks (C structures, arrays. etc)

We end up with one pdump_block_list_elt per object group (arrays of C structs are kept together) which includes a pointer to the first object of the group, the per-object size and the count of objects in the group, along with some other information which is initialized later.

These entries are linked together in pdump_block_list structures and can be enumerated through either:

1. the pdump_object_table, an array of pdump_block_list, one per lrecord type, indexed by type number.
2. the pdump_opaque_data_list, used for the opaque data which does not include pointers, and hence does not need descriptions.
3. the pdump_desc_table, which is a vector of memory_description/pdump_block_list pairs, used for non-opaque C memory blocks.

This uses a marking strategy similar to the garbage collector. Some differences though:

1. We do not use the mark bit (which does not exist for generic memory blocks anyway); we use a big hash table instead.
2. We use the external descriptions to follow pointers to generic memory blocks and opaque data in addition to Lisp_Object members, above and beyond what the garbage collector needs to do.

This is done by pdump_register_object(), which handles Lisp_Object variables, and pdump_register_block() which handles generic memory blocks (C structures, arrays, etc.), which both delegate the description management to pdump_register_sub(). This latter function also checks if the address of a Lisp_Object encountered is in staticpros, and if so, it moves the corresponding entry from staticpros to staticpros_nodump, since staticpros is relocated as an array of pointers in the data segment and needs to be handled differently to heap pointers.

The hash table doubles as a map from an object to pdump_block_list_elt (i.e. allows us to look up a pdump_block_list_elt with the object it points to). Entries are added with pdump_add_block() and looked up with pdump_get_block(). The Lisp hash table implementation is used; objects are converted to Lisp_Objects using STORE_VOID_IN_LISP() or make_opaque_ptr() depending on whether the low-order bit is set.

The roots for the marking are:

1. the Lisp_Object variables registered via dump_add_root_lisp_object
2. the data-segment memory blocks registered via dump_add_root_block (for blocks with relocatable pointers), or dump_add_opaque (for "opaque" blocks with no relocatable pointers; this is just a shortcut for calling dump_add_root_block with a NULL description).
3. the pointer variables registered via dump_add_root_block_ptr, each of which points to a block of heap memory (generally a C structure or array). Note that dump_add_root_block_ptr is not technically necessary, as a pointer variable can be seen as a special case of a data-segment memory block and registered using dump_add_root_block. Doing it this way, however, would require another level of static structures declared. Since pointer variables are quite common, dump_add_root_block_ptr is provided for convenience. Note also that internally we have to treat it separately from dump_add_root_block rather than writing the former as a call to the latter, since we don’t have support for creating and using memory descriptions on the fly – they must all be statically declared in the data-segment.
4. the staticpro’ed variables, a special case of dump_add_root_block_ptr.

This does not include the GCPRO’ed variables, the specbinds, the catchtags, the backlist, the redisplay or the profiling info, since we do not want to rebuild the actual chain of lisp calls which end up to the dump-emacs call, only the global variables.

Weak lists and weak hash tables are dumped as if they were their non-weak equivalent (without changing their type, of course). This has not yet been a problem, but may lead to small leaks if objects that would not otherwise be dumped are in those structures.

43.4.2 Address allocation

The next step is to allocate the offsets of each of the objects in the final dump file. This is done by pdump_allocate_offset() which is called indirectly by pdump_scan_by_alignment().

The strategy to deal with alignment problems uses these facts:

1. real world alignment requirements are powers of two.
2. the C compiler is required to adjust the size of a struct so that you can have an array of them next to each other. This means you can have an upper bound of the alignment requirements of a given structure by looking at which power of two its size is a multiple.
3. the non-variant part of variable size lrecords has an alignment requirement of 4.

Hence, for each lrecord type, C struct type or opaque data block the alignment requirement is computed as a power of two, with a minimum of 2^2 for lrecords. pdump_scan_by_alignment() then scans all the pdump_block_list_elt’s, the ones with the highest requirements first. This ensures the best packing.

The maximum alignment requirement we take into account is 2^8.

pdump_allocate_offset() only has to do a linear allocation, starting at an aligned address after the header.

43.4.3 The header

The next step creates the file and writes a header with a signature and some important information in it. For alignment reasons to keep us portable to Microsoft Windows, there are actually two headers; pdump_signature_header, which is UINT_32_BIT aligned, contains a signature specific to this dump file, and, following it, pdump_real_header which is 64-bit aligned, which contains the following more important information needed to restore the dump file:

The block_ptrs_offset field gives the offset within the dump file of the block pointers that need to be restored; the nb_root_block_ptrs gives a count for them.

The reloc_address field, which indicates at which address the file should be loaded if we want to avoid post-reload relocation, is set to 0 (if the dump file is external) or the known address of the dump data (if dumping is into the executable file).

The Fcons_address field gives the dump-time address of the Fcons(), allowing a delta to be worked out for C functions.

The lisp_object_description_address field gives the dump time address of lisp_object_description, guaranteed to be in the data segment, and allowing a delta to be worked out for C data segment addresses.

There are also counts for the root blocks and the serialized objects. These are stored immediately after the root blocks so there is no need for a specific offset to be saved for them.

43.4.4 Data dumping

The data is dumped in the same order as the addresses were allocated by pdump_dump_data(), called from pdump_scan_by_alignment(). This function copies the data to a temporary buffer, relocates all pointers in the object to the addresses allocated in step Address Allocation, and writes it to the file. Using the same order means that, if we are careful with lrecords whose size is not a multiple of 4, we are ensured that the object is always written at the offset in the file allocated in step Address Allocation.

There is special handling for hash tables; since hashing routinely depends on pointer values, and pointer values differ from dump time to load time, pdump_dump_data() offers the hash table code the opportunity to reorganize a hash table’s entries before they are written, after the offsets are determined. This is not always possible but as of 2026-05 there is normally no need for any load-time reorganization for the existing dumped hash tables.

43.4.5 Pointers dumping

A bunch of tables needed to reassign properly the global pointers are then written. They are:

1. a vector of all the offsets to the objects in the file that include a description (to allow relocation at reload time, and, for dumped Lisp_Objects, to allow them to be traced for GC at run time).
2. details of serialized objects (fields described by XD_SERIALIZABLE_DATA, XD_SERIALIZABLE_PTR), the addresses to which they should be restored, the functions needed to call to restore them, and the serialized data.
3. the pdump_root_block_ptrs dynarr
4. the pdump_root_blocks dynarr

For each of the dynarrs we write both the pointer to the variables and the relocated offset of the object they point to. Since these variables are global, the pointers are still valid when restarting the program and are used to regenerate the global pointers.

Some very important information like lrecord_implementations_table is handled indirectly using dump_add_root_block_ptr.

This is the end of the dumping part.

43.5 Reloading phase

43.5.1 File loading

The dump file contains C function pointers and Lisp code and should be regarded as executable C code. As a result, the preferred place for the dump file is within the XEmacs executable, since this avoids security worries at startup (see LD_LIBRARY_PATH and its setuid restrictions for motivation; ld.so(8) on Linux); this also has the benefit that no path searching is needed very early before we have any significant natural language processing available.

This is easily achieved on MS Windows by storing it in a “resource” at compile time and loading that at runtime. This only guarantees DWORD (UINT_32_BIT) alignment. Unfortunately there is no better way to do this with Visual Studio, and so the platform-independent dump code has been adjusted to deal with this worst-case alignment scenario.

On other platforms, if the current compiler supports C23’s #embed directive, that is used. That directive is not yet universal, and if it is unavailable, we use “INCBIN”, a reasonably portable assembler macro from Dale Weiler that is equivalent. Both of these require re-linking the executable after the dump file (of the correct size!) is generated. We only request four byte alignment with #embed and “INCBIN”, with a view to eliciting likely problems on native Windows builds when building on non-Windows platforms.

In these days of reproducible builds these approaches work well. The previous approach, from Olivier Galibert, binary-patched the generated executable, which interacts poorly with signed executables, as is standard with macOS and the ARM architecture.

If DUMP_IN_EXEC is turned off, the file is found either in the same directory in which the XEmacs executable was found (for in-place execution) or in “exec-directory”, usually under ../lib under the same prefix. It is then mmap’ed in memory (which ensures a PAGESIZE alignment, at least 4096), or if mmap is unavailable or fails, a max_align_t-aligned malloc is done and the file is loaded.

The difference between the actual loading address and the reloc_address is computed and will be used for relocating the Lisp data. The difference between the address of Fcons at dump time and at load time is computed and is used for relocating C function pointers, and the difference between the address of lisp_object_description at dump time and at load time is computed and used for relocating C data pointers, necessary for ASLR support.

43.5.2 Putting back the pdump_root_block_ptrs

The variables pointed to by pdump_root_block_ptrs in the dump phase are reset to the right relocated object addresses.

43.5.3 Object relocation

All the Lisp_Objects are relocated using their description and their offset by pdump_reloc_one. Once this is done, all non-Lisp_Object blocks are relocated using their descriptions and offsets by pdump_reloc_one in the same way. Neither step is necessary if the dump file is in the executable and ASLR is not present, so they are not done in that case.

43.5.4 Putting back the pdump_root_blocks

The pdump_root_blocks have their contents restored from the dump file (to the data segment address), and the relocation is performed on the data segment copy (if appropriate).

43.5.5 Putting back the XD_SERIALIZABLE_DATA, XD_SERIALIZABLE_PTR objects

The metadata for these objects is read, the address of the deconversion function is relocated appropriately, and then that function is called with the (relocated) addresses to which the XD_SERIALIZABLE_DATA objects should be restored, together with the address of the serialized data and its size. The deconversion function is then called with the XD_SERIALIZABLE_PTR serialized data, with NULL as the address to deconvert to, and the various addresses appropriate for that object are set to the result.

43.6 Dumping and restoring hash tables

Many, but not all, of the hash tables dumped by XEmacs use the pointer values of the Lisp_Object keys to hash those keys. This makes it difficult to dump them in a way that will give reliable look-up after load (since the pointer values change), and a previous version of this code just reorganized every hash table at load time to reflect the new hashes.

Currently the dumper (in close co-operation with the hash table code) can usually avoid the need to reorganize hash tables at load time. This relies on several observations:

1. Empty hash tables have no keys that might depend on pointer values, and so they do not need reorganization.
2. #'eq hash tables where the keys comprise exclusively fixnums and characters do not depend on pointer values, so they do not need reorganization.
3. #'eql hash tables where the keys comprise exclusively numbers and characters do not depend on pointer values, so they do not need reorganization.
4. #'equal hash tables where the keys comprise exclusively strings do not depend on pointer values, so they do not need reorganization. This is most relevant for obarray, mapping strings to symbols, the largest hash table currently dumped.
5. The low-order bits of the dump file are the same from run to run, since mmap() gives a page-aligned address, and the XEmacs executable (when dumping into the executable) is similarly page-aligned.

In the first four cases above, it suffices to just not reorganize the relevant hash tables. For other cases the fifth observation becomes relevant. The disksave method for a hash table, if it determines that the first four cases (and some very similar, related cases) do not apply, and that the “size” of the hash table is less than the (byte count) page size, can often resize that hash table to be a power of two in size, change its “golden ratio” to be one (meaning that the hash just depends on the key pointer value modulo the hash table size), and mark that the table’s entries should be reorganized deep in the bowels of the dumper once the load-time addresses (or file offsets) are available.

The hash table disksave method stores a key for sorting for each hash table, such that the largest hash table that may need reorganization will come first, and those that do not need reorganization come last. The dumper sorts the list of hash tables to be dumped using those keys, and stores the count of hash tables that may need reorganization in pdump_hash_table_reorganize_count. This sorted order is the same order that the pointers to the hash tables appear in the relocation table (and, indeed, the same order that the hash tables themselves appear in the dump file).

At run time, pdump_load_finish() examines the current page size and if it is greater than or equal to the dump time value, it sets pdump_hash_table_reorganize_count to zero. main_1() in emacs.c checks this variable and if it is non-zero it calls calls pdump_reorganize_hash_tables(). This will do any reorganization needed, but it will never reorganize e.g. empty hash tables or obarray.

This is complicated slightly by the rare need to allocate the dump data using xnew_array() (ultimately malloc() rather than mmap()), which only guarantees max_align_t alignment and by the more likely, but currently (2026) unnecessary, need to reorganize hash tables of a size larger than the dump time page size.

This document was generated on September 25, 2026 using texi2html 1.82.