Introduction

This appendix describes the format in which resources are stored in object files. This doesn't apply to PECOFF file format, where a format already exists and is well described in documentation from Microsoft.

On Microsoft Windows, resources are natively supported by the operating system. On other systems, Free Pascal RTL provides access to resources, which are embedded in the executable file in the format that is here described.

This appendix doesn't describe a particular object file format implementation (e.g. ELF or Mach-O), but the layout of the sections involved in supporting resources: the way these sections are actually laid out on a file are subject to the rules of the hosting object file format.

For external resource file details, see Description of external resource file format

Conventions

In this document, data sizes are specified with pascal-style data types. They are the following:

Byte order is the one of the target machine.

All data structures in the sections must be aligned on machine boundaries (4 bytes for 32 bit machines, 8 bytes for 64 bit machines).

Note that all pointers must be valid at runtime. This means that relocations must be written to the object file so that the linker can relocate pointers to the addresses they will have at runtime. Note also that pointers to strings are really pointers, and not offsets to the beginning of the string table.

Resource sections

Free Pascal uses two sections to store resource information:

• fpc.resources. This is a data section that contains all required structures. It must be writable.
• fpc.rehandles. This is a bss section whose size must be equal to (total number of resources)*(size of pointer). It must be writable.

The rest of this chapter will describe the contents of fpc.resources section.

Resource section layout

The fpc.resources section consists of these parts:

• The initial header
• The resource tree, in the form of nodes
• The string table, which can be optional
• The resource data

The initial header

The fpc.resources section starts with this header:

The resource tree

Immediately following the initial header, the resource tree comes. It is made up by nodes that represent resource types, names and language ids.

Data is organized so that resource information (type, name and language id) is represented by a tree: root node contains resource types, that in turn contain resource names, which contain language ids, which describe resource data.

Given a node, its sub-nodes are ordered as follows:

• First the "named" nodes (nodes that use a string as identifier) come, then the id ones (nodes that use an integer as identifier).
• Named nodes are alphabetically sorted, in ascending order.
• Id nodes are sorted in ascending order.

In the file, all sub-nodes of a node are written in the order described above. Then, all sub-nodes of the first sub-node are written, and so on.

Example:

There are three resources:

1. a BITMAP resource with name MYBITMAP and language id $0409
2. a BITMAP resource with name 1 and language id 0
3. a resource with type MYTYPE and name 1 and language id 0

Nodes are laid out this way (note that BITMAP resources have type 2):

	
	root | MYTYPE 2 | 1 | 0 | MYBITMAP 1 | $0409 | 0

That is, types (MYTYPE is a string, so it comes before 2 which is BITMAP), then names for MYTYPE (1), then language id for resource 3 (0), then names for BITMAP (MYBITMAP and 1), then language id for resource 1 ($0409), then language id for resource 2 (0).

Node format

If the node is identified by a string, nameid is a pointer to the null-terminated string holding the name. If it is identified by an id, nameid is that id. Language id nodes are always identified by and ID.

ncount is the number of named sub-nodes of this node (nodes that are identified by a string).

idcountsize is the number of id sub-nodes of this node (nodes that are identified by an integer id). For language id nodes, this field holds the size of the resource data.

subptr is an pointer to the first subnode of this node. Note that it allows to access every subnode of this node, since subnodes of a node always come one after the other. For language id nodes, subptr is the pointer to the resource data.

The string table

The string table is used to store strings used for resource types and names. If all resources use integer ids for name and types, it may not be present in the file.

The string table simply contains null-terminated strings, one after the other.

If present, the string table always contains a 0 (zero) at the beginning. This way, the empty string is located at the first byte of the string table.

The resource data

This part of the file contains raw resource data. As written before, all data structures must be aligned on machine boundaries, so if a resource data size is not a multiple of 4 (for 32 bit machines) or 8 (for 64 bit machines), bytes of padding must be inserted after that resource data.

Exported symbols

Object files containing resources must export a pointer to the first byte of fpc.resources section. The name of this symbol is FPC_RESSYMBOL.

Mach-O specific notes

fpc.resources and fpc.reshandles sections are to be contained in a __DATA segment.

We'll see how to implement a writer for a new resource file format. A resource writer is a descendant of TAbstractResourceWriter, and it's usually implemented in a unit named namewriter, where name is file format name.

Suppose we must write a writer for file format foo; we could start with a unit like this:

Note that in the initialization section, TFooResourceWriter is registered for extension .foo.

We must implement abstract methods of TAbstractResourceWriter. Let's start with the simpler ones, GetExtensions and GetDescription.

Now let's see Write. This method must write all resources in the TResources object to the stream.

Suppose that our foo format is very simple:

• the header is made by a 4-byte signature (FOO*), a longword holding the number of resources in the file, and other 8 bytes of padding.

• each resource has a 16-byte header containing:

  • a longword for the resource type (only IDs are allowed for types)
  • a longword for the resource name (only IDs are allowed for names)
  • a longword for the language ID
  • a longword for the size of the resource data
• after the resource header immediately comes the resource data, possibly padded so that it ends on a 4 byte boundary.
• file format is little-endian

Our Write method could be something like this:

So we must implement WriteFileHeader, which writes the 16-byte file header, and WriteResource, which writes a single resource with its header.

Let's start from the first one:

This code simply writes the file header as defined in foo format. Note that if we are running on a big endian system we should swap bytes before writing, e.g. calling SwapEndian function, but for simplicity this is omitted.

Now WriteResource comes. This method could be like this:

We write the 16-byte resource header, and then resource data. Note that if resources have been loaded from a stream and the user didn't modify resource data, we are copying data directly from the original stream.

Align4Bytes is a private method (not shown for simplicity) that writes some padding bytes to the stream if needed, since FOO file format specifies that resource data must be padded to end on a 4 byte boundary. Again, we didn't consider endianess for simplicity. And finally, note that foo format only supports IDs for types and names, so if one of them is a name (that is, a string) this code might cause exceptions to be raised.

Note: More complex file formats store resources in a tree hierarchy; since TResources internally stores resources in this way too, a writer can choose to acquire a reference to the internal tree used by the TResources object (see TAbstractResourceWriter.GetTree) and use it directly. For these file formats resources can be written faster, since there is no overhead involved in building a separate resource tree in the writer.

That's all. Now you should be able to create a real resource writer.

We'll see how to implement a reader for a new resource file format. A resource reader is a descendant of TAbstractResourceReader, and it's usually implemented in a unit named namereader, where name is file format name.

Suppose we must write a reader for file format foo; we could start with a unit like this:

Note that in the initialization section, TFooResourceReader is registered for extension .foo.

We must implement abstract methods of TAbstractResourceReader. Let's start with the simpler ones, GetExtensions and GetDescription.

Now let's see CheckMagic method. This method is called with a stream as a parameter, and the reader must return true if it recognizes the stream as a valid one. Usually this means checking some magic number or header.

Suppose our foo files start with a 4-byte signature 'FOO*'. This method checks the signature and returns true if it is verified. Note that it catches EReadError exception raised by TStream: this way, if the stream is too short it returns false (as it should, since magic is not valid) instead of letting the exception to propagate.

Now let's see Load. This method must read the stream and create resources in the TResources object, with information about their name, type, position and size of their raw data, and so on.

First of all, this method checks file magic number, calling CheckMagic method we already implemented. This is necessary since CheckMagic is not called before Load: CheckMagic is invoked by TResources when probing a stream, while Load is invoked when loading resources (so if the user passed a reader object to a TResources object, CheckMagic is never called). Note also that the stream is always at its starting position when these methods are called.

If magic number is ok, our method invokes another method to do the actual loading. If during this process the stream can't be read, an EResourceReaderUnexpectedEndOfStreamException exception is raised.

So, let's implement the private method which will load resources.

Suppose that our foo format is very simple:

• the header is made by the 4-byte signature we already saw, a longword holding the number of resources in the file, and other 8 bytes of padding.

• each resource has a 16-byte header containing:

  • a longword for the resource type (only IDs are allowed for types)
  • a longword for the resource name (only IDs are allowed for names)
  • a longword for the language ID
  • a longword for the size of the resource data
• after the resource header immediately comes the resource data, possibly padded so that it ends on a 4 byte boundary.
• file format is little-endian

To start with, our method will be:

Since in Load we called CheckMagic, which read the first 4 bytes of the header, we must read the remaining 12: we read the number of resources, and we skip the other 8 bytes of padding.

Then, for each resource, we read the resource header. Note that if we are running on a big endian system we should swap the bytes we read, e.g. calling SwapEndian function, but for simplicity this is omitted.

Now, we should create a resource. Of which class? Well, we must use unit. In fact it contains TResourceFactory class, which is an expert in creating resources of the right class: when the user adds a unit containing a resource class to the uses clause of its program, the resource class registers itself with TResourceFactory. This way it knows how to map resource types to resource classes.

We need to have type and name of the resource to create as TResourceDesc objects: instead of creating and destroying these objects for each resource, we'll create a couple in the creator of our reader and we'll destroy them in the destructor, so that they will live for the whole life of our reader. Let's name them workType and workName.

Our code becomes:

Note that after the resource has been created we set its data size and data offset. Data offset is the current position in the stream, since in our FOO file format resource data immediately follows resource header.

What else do we need to do? Of course we must create RawData stream for our resource, so that raw data can be accessed with the caching mechanism. We will create a TResourceDataStream object, telling it which resource and stream it is associated to, which its size will be and which class its underlying cached stream must be created from.

So we add to the uses clause, declare another local variable

and add these lines in the for loop

That is, aRawData will create its underlying stream as a TCachedResourceDataStream over the portion of aStream that starts at current position and ends after aRes.DataSize bytes.

We almost finished: now we must add the newly created resource to the TResources object and move stream position to the next resource header. Complete code for ReadResources method is:

Align4Bytes is a private method (not shown for simplicity) that sets stream position to the next multiple of 4 if needed, since FOO file format specifies that resource data must be padded to end on a 4 byte boundary.

Note: We have used Add method to populate the TResources object. More complex file formats store resources in a tree hierarchy; since TResources internally stores resources in this way too, a reader can choose to acquire a reference to the internal tree used by the TResources object (see TAbstractResourceReader.GetTree), populate it and notify the TResources object about the added resources (see TAbstractResourceReader.AddNoTree). For these file formats resources can be loaded faster, since there is no overhead involved in keeping a separate resource tree in the reader.

That's all. Now you should be able to create a real resource reader.

Some considerations

Usually, a specific resource class is needed when resource data is encoded in a specific binary format that makes working with RawData uncomfortable.

However, there aren't many reasons to design a new binary format requiring a specific resource class: the classes provided with this package exist for compatibility with Microsoft Windows, but in the general case one should avoid such approach.

In Microsoft Windows, some resource types have a specific format, and the operating system supports them at runtime making it easy to access that kind of data: e.g. icons and bitmaps are stored in resources in a format that is slightly different from the one found in regular files, but the operating system allows the user to easily load them using LoadImage function, without having to deal with their internal format. Other resource types are used to obtain information about the executable: RT_VERSION resource type and RT_GROUP_ICON contain version information and program icon that can be displayed in Windows Explorer, respectively.

Using this kind of resources in a cross-platform perspective doesn't make much sense however, since there is no support by other operating systems for these types of resources (and for resources in general), and this means that it's up to you to provide support at runtime for these binary formats. So if you need to store images as resources it's better to use TGenericResource and store an image in PNG format (for instance), which can be read by existing libraries at runtime, instead of creating a RT_BITMAP resource with TBitmapResource, since libraries that read BMP files can't handle that resource contents.

New resource classes thus make sense when you want to add support for existing Windows-specific resources, e.g. because you are writing a resource compiler or editor, but in the general case they should be avoided.

Now that you've been warned, let's start with the topic of this chapter.

How to implement a new resource class

A resource class is a descendant of TAbstractResource, and it's usually implemented in a unit named typeresource, where type is resource type.

If you are implementing a new resource class, you are doing it to provide additional methods or properties to utilize resource data. You resource class must thus be able to read its RawData stream format and write data back to it when it is requested to do so.

Generally, your class shouldn't create private objects, records or memory buffers to provide these specific means of accessing data until it's requested to do so (lazy loading), and it should free these things when it is requested to write back data to the RawData stream.

We'll see these points in more detail, using TAcceleratorsResource as an example.

TAcceleratorsResource holds a collection of accelerators. An accelerator is a record defined as follows:

The resource simply contains accelerators, one immediately following the other. Last accelerator must have $80 in its flags.

To provide easy access to the elements it contains, our accelerator resource class exposes these methods and properties in its public section:

We must also implement abstract methods (and an abstract constructor) of TAbstractResource:

The protected methods are very easy to implement, so let's start from them. For GetType and GetName, we need to add two private fields, fType and fName, of type TResourceDesc. We create them in the constructor and destroy them in the destructor. Type will always be RT_ACCELERATOR. We make the parameterless constructor of TAbstractResource public, using 1 as the resource name, while in the other constructor we use the name passed as parameter, ignoring the type (since it must always be RT_ACCELERATOR).

So, GetType, GetName, the constructors and the destructor are implemented as follows:

Note that we used SetDescOwner to let type and name know the resource that owns them.

Now ChangeDescTypeAllowed and ChangeDescValueAllowed come. As we said, resource type must be RT_ACCELERATOR, always. Thus, we only allow name to change value or type:

NotifyResourcesLoaded is called by TResources when it finishes loading all resources. Since we are not interested in this fact, we simply leave this method empty. This method is useful for resources that "own" other resources, like TGroupIconResource and TGroupCursorResource (note: you should not implement resource types that depend on other resources: it complicates things a lot and gives you a lot of headaches).

Now, let's see the more interesting - and more difficult - part: providing an easy way to deal with accelerators.

As we said earlier, we must implement some methods and properties to do so. Surely we'll need a list to hold pointers to accelerator records, but we must think a little bit about how this list is created, populated, written to RawData and so on.

When the object is created, we don't have to create (yet) single accelerator records, as said before; but if the user tries to access them we must do it. If the object is created and its RawData contains data (e.g. because a reader has created the resource when loading a resource file) and the user tries to access an accelerator, we must create accelerators from RawData contents. So, until a user tries to access accelerators our class doesn't do anything, while when the user does so it "lazy-loads" data, or creates empty structures if RawData is empty.

When data is loaded, RawData contents aren't considered anymore. When, however, our resource is requested to update RawData (that is, when UpdateRawData method is invoked), our "lazy-loaded" data must be freed. In fact, a user could ask our resource to update raw data, then he/she could modify it directly and then could use our resource's specialized methods and properties to do further processing: for this reason, when RawData is written, other buffered things must be freed, so that they'll created again from RawData if needed.

Note that other resource classes could behave differently: e.g. TBitmapResource uses a copy-on-write mechanism on top of RawData to insert a BMP file header at the beginning of the stream, but it doesn't copy RawData contents whenever possible.

Coming back to our TAcceleratorsResource example, let's see how to implement this behaviour.

Let's add a fList field in the private section of our class:

In the constructor, we set this field to nil: we use it to check if data has been loaded from RawData or not. Consequently in the destructor we'll free the list only if it's not nil:

(we did not implement Clear method yet: we'll see it later).

We said that our resource must load data only when needed; to do this we add a private method, CheckDataLoaded that ensures that data is loaded. This method is called by whatever method needs to operate on the list: if data has already been loaded it will quickly return, otherwise it will load data.

If fList is not nil data is already loaded, so exit. Otherwise, create the list. If RawData is empty we don't need to load anything, so exit. Otherwise allocate room for accelerators, read them from the stream, and add them to the list.

Note that if we are running on a big endian system we should swap the bytes we read, e.g. calling SwapEndian function, but for simplicity this is omitted.

The counterpart of CheckDataLoaded is UpdateRawData. When it is called, data from the list must be written back to RawData, and the list and its contents must be freed:

If fList is nil data hasn't been loaded, so RawData is up to date, so exit. Otherwise, write each accelerator (ensuring that only last one has $80 flag set), clear the list, free it and set it to nil. Again, if we are on a big endian system we should swap each accelerator contents before writing it, but for simplicity this is omitted.

Other methods we named earlier (Add, Delete, Clear) are trivial to implement: remember only to call CheckDataLoaded before doing anything. The same is true for accessor methods of relevant properties (Count, Items).

If you are curious, you can check the full implementation of TAcceleratorsResource looking at source code.

Resource files and TResources class

One of the most important classes is TResources class, contained in unit, which represents a format-independent view of a resource file. In fact, while single resources are important, they are of little use alone, since they can't be read or written to file directly: they need to be contained in a TResources object.

TResources provides methods to read itself from a file or stream, using specific objects that are able to read resource data from such a stream: these are the so called resource readers, that descend from TAbstractResourceReader.

There are also resource writers that do the opposite, and that descend from TAbstractResourceWriter.

Usually readers and writers register themselves with TResources in the initialization section of the unit they are implemented in, so you only need to add a certain unit to your program uses clause to let TResources "know" about a particular file format.

Let's see a very simple example: a program that converts a .res file to an object file in COFF format (the object file format used by Microsoft Windows).

As you can see, the code is trivial. Note that resreader and coffwriter units were added to the uses clause of the program: this way, the resource reader for .res files and the resource writer for COFF files have been registered, letting the resources object know how to handle these file types.

There are cases where one doesn't want to let the TResources object to choose readers and writers by itself. In fact, while generally it is a good idea to let TResources probe all readers it knows to find one able to read the input file, this isn't true when it comes to write files: writers are selected based on the file extension, so if you are trying to write a file with .o extension you can't be sure about which writer will be selected: it could be the COFF or the ELF writer (it depends on which writer gets registered first). Moreover, writers generally make an object file for the host architecture, so if you are running the program on a i386 machine it will produce a COFF or ELF file for i386.

The solution is to provide TResources with a specific writer. In the following example the reader is automatically chosen among various readers, and we use a specific writer to produce an ELF file for SPARC.

Note that the file to convert is taken from the command line. Its format is automatically detected among res (), coff (), elf (), PE (, e.g. a Windows exe or dll), and is written as an ELF file for SPARC. Note that we had to use unit since we used emtsparc constant to specify the machine type of the object file to generate.

With a small change to the above program we can let the user know which reader was selected to read the input file: we can use TResources.FindReader class method to obtain the appropriate reader for a given stream.

Output example:

user@localhost:~$ ./res3 myresource.res
Selected reader: .res resource reader
user@localhost:~$

Single resources

You can do more with resources than simply converting between file formats.

TResources.Items property provides a simple way to access all resources contained in the TResources object.

In the following example we read a resource file and then dump each resource data in a file whose name is built from type and name of the dumped resource.

This code simply copies the content of each resource's RawData stream to a file stream, whose name is resourcetype_resourcename.

Resource raw data isn't always what one expected, however. While some resource types simply contain a copy of a file in their raw data, other types do some processing, so that dumping raw data doesn't result in a file in the format one expected.

E.g. a resource of type RT_MANIFEST is of the former type: its raw data is like an XML manifest file. On the other hand, in a resource of type RT_BITMAP the RawData stream isn't like a BMP file.

For this reason, several classes (descendants of TAbstractResource) are provided to handle the peculiarities of this or that resource type. Much like it's done with readers and writers, resource classes can be registered: adding the unit that contains a resource class to the uses clause of your program registers that class. This way, when resources are read from a file, they are created with the class that is registered for their type (the class responsible to do this is TResourceFactory, but probably you won't need to use it unless you're implementing a new resource reader or resource class).

In the following example, we read a resource file and then dump data of each resource of type RT_BITMAP as a BMP file.

Note that we included in the uses clause of our program. This way, resources of type RT_BITMAP are created from TBitmapResource class. This class provides a stream, BitmapData that allows resource raw data to be accessed as if it was a bmp file.

We can of course do the opposite. In the following code we are creating a manifest resource from manifest.xml file.

Note that resources of type RT_MANIFEST contain a straight copy of a xml file, so TGenericResource class fits our needs. TGenericResource is a basic implementation of TAbstractResource. It is the default class used by TResourceFactory when it must create a resource whose type wasn't registered with any resource class.

Please note that instead of copying inFile contents to RawData we used SetCustomRawDataStream method: it sets a stream as the underlying stream for RawData, so that when final resource file is written, data is read directly from the original file.

Let's see a similar example, in which we use a specific class instead of TGenericResource. In the following code we are creating a resource containing the main program icon, which is read from mainicon.ico file.

In this program we created a new TGroupIconResource with 'MAINICON' as name, and we loaded its contents from file 'mainicon.ico'. Please note that RT_GROUP_ICON resource raw data doesn't contain a .ico file, so we have to write to ItemData which is a ico-like stream. As we did for res6 program, we tell the resource to use our stream as the underlying stream for resource data: the only difference is that we are using TGroupResource.SetCustomItemDataStream instead of TAbstractResource.SetCustomRawDataStream method, for obvious reasons.

Other resource types

There are other resource types that allow to easily deal with resource data. E.g. TVersionResource makes it easy to create and read version information for Windows executables and dlls, TStringTableResource deals with string tables, and so on.

Data caching

Whenever possible, fcl-res tries to avoid to duplicate data. Generally a reader doesn't copy resource data from the original stream to RawData stream: instead, it only informs the resource about where its raw data is in the original stream. RawData uses a caching system so that it appears as a stream while it only redirects operations to its underlying stream, with a copy-on-write mechanism. Of course this behaviour can be controlled by the user. For further information, see TAbstractResource and TAbstractResource.RawData.

This package contains a library to easily work with Microsoft Windows resources in a cross-platform way.

Classes are provided to create, load and write resources from/to different file formats in a transparent way, and to handle most common resource types without having to deal with their internal format.

Whenever possible data caching is performed, helped by a copy-on-write mechanism. This improves performance especially when converting big resources from a file format to another.

Since fcl-res architecture is extensible, it's always possible to extend the library with custom resource types or new file readers/writers.

Please note that resources aren't limited to Windows platform: Free Pascal can use them also on ELF and Mach-O targets. Moreover, this library can be useful for cross-compilation purposes even on other targets.

It is highly recommended to read topic if you are approaching this library for the first time.