`Shared Memory` for a fast Queue mechanism (shmq)



In some prototype experimentation on the construction of an event based phrase dictionary there arose a need to provide a minimal impact refactoring a semantic enhancement service. The implementation placed some uncomfortable restrictions on the size of an ontology that could be used. This was part of an attempt to break that process into multiple processes.

In general there seems to be a need to provide some buffering between processes, much in the fashion of the command line pipeline so useful in linux/UNIX shells. Some times it is not convenient to have a mechanism that allows buffering between processes that are not strictly synchronously invoked or executed. Namely, if one process constructs a shared memory area and populates it, whereby another process, some time later, can connect to that memory area and retrieve the information. Since one can construct queue operations that are properly synchronized, the calling processes can use the fast queues without special consideration for timing issues. As part of the implementation, two utilities should be provided, a producer and consumer, which can read from standard input and output streams and redirect those stream to and from the queue respectively. One approach to constructing fast queues would be to construct a mechanism for using appropriate shared memory containers.

There is a web page, SharedMemory , that addresses a relevantly similar use case and can was used as an example for consideration. Though Josh provided a rough cut design and implementation fragments, he can hardly be held responsible for the current mayhem checked into this shmq project.

This design was originally implemented in Solaris, however at the time could not be ported to linux due to the partial support of the POSIX standard, particularly condition variables. Since that time, linux support for POSIX interprocess communication mechanisms has improved and the Solaris operating system is being open sourced, the hope is that _now_ we can run in a free environment.

A project to host this effort has been created on SourceForge [0]. The shmq "Web Site" is generated from the source code using doxygen [1] and the summary contains links to the discussion forums, mailing lists, source code repository and more.


There are three main components of this work, the shared segment of memory, an allocator of memory chunks and the queue itself.

There is some amount of work required to build out the facilities to be a "well behaved" package. Some effort has been put towards the automake and autoconf definitions to support cross platform builds. A limited number of architecture and operating system combinations have been attempted. Additional effort will be required to construct the package definitions for particulat targets, for example Redhat Package Manager (RPM).


An alternative to using shared memory segments is memory mapped files. Some work would be required to provide a simple method for switching between shared memory flavors. In either case, a convenient way is needed to work with pointers in memory. This is done as a template class [3], so that any sort of data type can be used in this fashion.


[0] SourceForge.net is the world\'s largest Open Source software development web site. SourceForge.net provides free hosting to Open Source software development projects with a centralized resource for managing projects, issues, communications, and code.
[1] Doxygen is a documentation system for C++, C, Java, Objective-C, Python, IDL (Corba and Microsoft flavors), Fortran, VHDL, PHP, C#, and to some extent D.
[2] W. Richard Stevens: UNIX Network Programming, Volume 2: Interprocess Communications, Prentice Hall, 1998, ISBN 0-130-81081-9.
[3] Andrei Alexandrescu: Modern C++ Design: Generic Programming and Design Patterns Applied, Addison-Wesley, 2000, ISBN 0-201-70431-5.
[4] A C++ library of designs, containing flexible implementations of common design patterns and idioms.
Project shmq hosted by   SourceForge.net
Documentation generated on Sat Sep 2 10:07:40 2006 for shmq by   doxygen.org 1.4.6