`Shared Memory` for a fast Queue mechanism (shmq)

0.0.3

Context

In some prototype experimentation on the construction of an event based phrase dictionary there arose a need to provide a minimal impact refactoring a semantic enhancement service. The implementation placed some uncomfortable restrictions on the size of an ontology that could be used. This was part of an attempt to break that process into multiple processes.

In general there seems to be a need to provide some buffering between processes, much in the fashion of the command line pipeline so useful in linux/UNIX shells. Some times it is not convenient to have a mechanism that allows buffering between processes that are not strictly synchronously invoked or executed. Namely, if one process constructs a shared memory area and populates it, whereby another process, some time later, can connect to that memory area and retrieve the information. Since one can construct queue operations that are properly synchronized, the calling processes can use the fast queues without special consideration for timing issues. As part of the implementation, two utilities should be provided, a producer and consumer, which can read from standard input and output streams and redirect those stream to and from the queue respectively. One approach to constructing fast queues would be to construct a mechanism for using appropriate shared memory containers.

There is a web page, SharedMemory , that addresses a relevantly similar use case and can was used as an example for consideration. Though Josh provided a rough cut design and implementation fragments, he can hardly be held responsible for the current mayhem checked into this shmq project.

This design was originally implemented in Solaris, however at the time could not be ported to linux due to the partial support of the POSIX standard, particularly condition variables. Since that time, linux support for POSIX interprocess communication mechanisms has improved and the Solaris operating system is being open sourced, the hope is that _now_ we can run in a free environment.

A project to host this effort has been created on SourceForge [0]. The shmq "Web Site" is generated from the source code using doxygen [1] and the summary contains links to the discussion forums, mailing lists, source code repository and more.

Highlights

There are three main components of this work, the shared segment of memory, an allocator of memory chunks and the queue itself.

• Segment : the shared memory class is a thin object implemented on the shmget() and shmat() calls [2]. Since an additional call to the function shmctl() is required to remove the segment (using the option IPC_RMID), queue persistence is obtained with lack of an action in the segment destructor.
• Allocator : the shared allocator uses a sorted circular doubly linked list for free blocks and should try to merge contiguous blocks on free.
• Queue : the queue is a template and can be used for basic FIFO operation.

There is some amount of work required to build out the facilities to be a "well behaved" package. Some effort has been put towards the automake and autoconf definitions to support cross platform builds. A limited number of architecture and operating system combinations have been attempted. Additional effort will be required to construct the package definitions for particulat targets, for example Redhat Package Manager (RPM).

Memory

An alternative to using shared memory segments is memory mapped files. Some work would be required to provide a simple method for switching between shared memory flavors. In either case, a convenient way is needed to work with pointers in memory. This is done as a template class [3], so that any sort of data type can be used in this fashion.

References

Project shmq hosted by

Documentation generated on Sat Sep 2 10:07:40 2006 for shmq by   1.4.6