SDBM(3)

NAME
sdbm, sdbm_open, sdbm_prep, sdbm_close, sdbm_unlink, sdbm_fetch, sdbm_store, sdbm_replace, sdbm_delete, sdbm_exists, sdbm_firstkey, sdbm_nextkey, sdbm_hash, sdbm_rdonly, sdbm_error, sdbm_clearerr, sdbm_dirfno, sdbm_pagfno, sdbm_datfno - data base subroutines

SYNOPSIS
#include <sdbm.h>

typedef struct {
	char *dptr;
	size_t dsize;
} datum;

datum nullitem = { NULL, 0 };

DBM *sdbm_open(char *file, int flags, int mode)
DBM *sdbm_prep(char *dirname, char *pagname, char *datname,
        int flags, int mode)
void sdbm_close(DBM *db)
void sdbm_unlink(DBM *db)
int sdbm_rebuild(DBM *db)

datum sdbm_fetch(DBM *db, key)
int sdbm_store(DBM *db, datum key, datum val, int flags)
int sdbm_replace(DBM *db, datum key, datum val, bool *existed)
int sdbm_delete(DBM *db, datum key)
int sdbm_exists(DBM *db, datum key)

datum sdbm_firstkey(DBM *db)
datum sdbm_firstkey_safe(DBM *db)
datum sdbm_nextkey(DBM *db)
datum sdbm_value(DBM *db)
int sdbm_deletekey(DBM *db)
void sdbm_endkey(DBM *db)

long sdbm_hash(char *string, size_t len)

ssize_t sdbm_sync(DBM *db)
ssize_t sdbm_count(const DBM *db)
ssize_t sdbm_delta(const DBM *db)
void sdbm_delta_reset(DBM *db)

int sdbm_set_cache(DBM *db, long pages)
int sdbm_set_wdelay(DBM *db, bool on)
int sdbm_set_volatile(DBM *db, bool yes)

long sdbm_get_cache(const DBM *db)
bool sdbm_get_wdelay(const DBM *db)
bool sdbm_is_volatile(const DBM *db)

void sdbm_set_name(DBM *db, const char *string)
const char *sdbm_name(const DBM *db)

int sdbm_rename(DBM *db, char *base)
int sdbm_rename_files(DBM *db,
	char *dirname, char *pagname, char *datname)

bool sdbm_rdonly(const DBM *db)
bool sdbm_error(const DBM *db)
bool sdbm_shrink(DBM *db)
bool sdbm_clear(DBM *db)
void sdbm_clearerr(DBM *db)
int sdbm_dirfno(DBM *db)
int sdbm_pagfno(DBM *db)
int sdbm_datfno(DBM *db)

bool sdbm_is_storable(size_t key, size_t value)

void sdbm_thread_safe(DBM *db)
void sdbm_lock(DBM *db)
void sdbm_unlock(DBM *db)
bool sdbm_is_thread_safe(const DBM *db)
bool sdbm_is_locked(const DBM *db)
DBM *sdbm_ref(const DBM *db)
void sdbm_unref(const DBM **db_ptr)
int sdbm_refcnt(const DBM *db)
int sdbm_rebuild_async(DBM *db)

size_t sdbm_foreach(DBM *db, int flags, sdbm_cb_t cb, void *arg);
size_t sdbm_foreach_remove(DBM *db, int flags, sdbm_cbr_t cb, void *arg);

size_t sdbm_loose_foreach(DBM *db, int flags, sdbm_cb_t cb, void *arg);
size_t sdbm_loose_foreach_remove(DBM *db,
        int flags, sdbm_cbr_t cb, void *arg);

DESCRIPTION

This package allows an application to maintain a mapping of <key,value> pairs in disk files. This is not to be considered a real database system, but is still useful in many simple applications built around fast retrieval of a data value from a key. This implementation uses an external hashing scheme, called Dynamic Hashing, as described by Per-Aake Larson in BIT 18 (1978) pp. 184-201. Retrieval of any item usually requires a single disk access, unless some keys or values are large enough to be stored via indirections, in which case more disk accesses are required. The application interface is compatible with the ndbm(3) library.

An sdbm database is kept in two files usually given the extensions .dir and .pag. The .dir file contains a bitmap representing a forest of binary hash trees, the leaves of which indicate data pages in the .pag file. When facing large keys or values that would not otherwise fit in one page, a third file (carrying the .dat extension) is created to store the large data, with indirection pointers left in the .pag file to be able to access the underlying data.

A .tmp file can also be created to store all the temporary file extensions used by the database during a rebuild operation. If the process is killed in the middle of a rebuild, the left-over files can then be properly reclaimed when the application calls sdbm_prep() or sdbm_open() in a later invocation, from another process.

The application interface uses the datum structure to describe both keys and values. A datum specifies a byte sequence of dsize size pointed to by dptr. If you use ASCII strings as keys or values, then you must decide whether or not to include the terminating NUL byte which sometimes defines strings. Including it will require larger database files, but it will be possible to get sensible output from a strings(1) command applied to the data file.

In order to allow a process using this package to manipulate multiple databases, the applications interface always requires a handle, a DBM *, to identify the database to be manipulated. Such a handle can be obtained from the only routines that do not require it, namely sdbm_open() or sdbm_prep(). Either of these will open or create the two or three necessary files. The difference is that the latter allows explicitly naming the bitmap and data files whereas sdbm_open() will take a base file name and call sdbm_prep() with the default extensions. Furthermore, giving a NULL pointer for datname in sdbm_prep() will disable support for large keys and values. The flags and mode parameters are the same as for open(2).

To free the resources occupied while a database handle is active, call sdbm_close(). The sdbm_unlink() call is similar to sdbm_close() but will additionally force the deletion of the database files.

Given a handle, one can retrieve data associated with a key by using the sdbm_fetch() routine, and associate data with a key by using the sdbm_store() routine. sdbm_exists() will say whether a given key exists in the database.

The values of the flags parameter for sdbm_store() can be either DBM_INSERT, which will not change an existing entry with the same key, or DBM_REPLACE, which will replace an existing entry with the same key. Keys are unique within the database.

To replace a possibly existing key and know whether the key existed or was actually created, use sdbm_replace() and the parameter existed will be set upon return to indicate whether the key existed already, provided the call is not returning -1.

To delete a key and its associated value use the sdbm_delete() routine.

To retrieve every key/value pairss in the database, use a loop like:

for (key = sdbm_firstkey(db); key.dptr != NULL; key = sdbm_nextkey(db)) {
    datum value = sdbm_value(db);
}

The order of retrieval is unspecified. During traversal, no insertion or deletion of any other key should occur. However, the current key may be deleted through the sdbm_deletekey() routine. Usage of sdbm_value() or sdbm_deletekey() outside of the outlined iterator loop is forbidden. Should that happen, an error is returned (either nullitem or -1) and errno is set to ENOENT.

The iteration may begin with sdbm_firstkey_safe() instead, which will activate extended checks of each page: keys not belonging to the page are deleted on the fly. This is useful when iterating over a database which could have been corrupted due to I/O errors, and the .dir and .pag files are not consistent with eachother.

If for some reason the iteration must be aborted before sdbm_nextkey() or sdbm_firstkey() return nullitem, then the sdbm_endkey() routine must be called to clear the iteration context and mark the end of the iteration loop.

If you determine that the performance of the database is inadequate or you notice clustering or other effects that may be due to the hashing algorithm used by this package, you can override it by supplying your own sdbm_hash() routine. Doing so will make the database unintelligable to any other applications that do not use your specialized hash function.

The following functions are also defined in the header file:

    sdbm_rdonly() returns true if the database has been opened read-only.

    sdbm_error() returns true if an I/O error has occurred. After a write I/O error, the database is put in read-only mode until sdbm_clearerr() has been called, to prevent cascading errors.

    sdbm_clearerr() allows you to clear the error flag if you think you know what the error was and insist on ignoring it.

    sdbm_dirfno() returns the file descriptor associated with the bitmap file.

    sdbm_pagfno() returns the file descriptor associated with the page file.

    sdbm_datfno() returns the file descriptor associated with the data file (large keys and values), or -1 if the file was not opened yet because it was not needed so far.

    sdbm_name() returns the database name previously set using sdbm_set_name() or a string starting with "file " and followed by the pathname of the .pag file if no name was set yet.

    sdbm_is_storable() indicates whether a combination of key and value sizes can fit within the size constraints we have.

    sdbm_shrink() attempts to shrink the allocated files on disk to reclaim unused trailing space, after many value deletions for instance. Returns FALSE on errors.

    sdbm_count() quickly counts the amount of entries (key/value pairs) in the database by scanning the whole database file. Returns -1 on error.

    sdbm_delta() returns the algebraic count of entries added (if positive) or removed (if negative) from the database since the last reset of the counter. This is simply the result of added minus deleted pairs. It is initially set to 0 after the database has been opened.

    sdbm_delta_reset() resets the algebraic count of added minus deleted pairs to 0.

    sdbm_clear() erases all the data held in the database but keeps it opened. Returns -1 on error. It also issues a sdbm_delta_reset() call.

    sdbm_rename() renames the database files using the specified name base (without a .pag or other .dir extension) and transparently re-opens the database using the new files. Returns -1 on error.

    sdbm_rename_files() works like sdbm_rename() but with the new filenames explicitly given.

    sdbm_rebuild() rebuilds the database, hopefully leading to a more compact on-disk representation. It returns -1 on failure. It is also possible to rebuild the database asynchronously, from a separate thread by using sdbm_rebuild_async() instead: concurrent usage from other threads is possible during that asynchronous rebuild.

ITERATING
It is possible to use high-level iterators on the database to process all the items (key / value pairs) via a common routine. That processing callback of type sdbm_cb_t has the following signature:

void callback(const datum key, const datum value, void *arg);

The key and value arguments are the current item key and value, and arg is an opaque argument given as context to the routine. To invoke the callback on all the items, one uses:

sdbm_foreach(db, flags, callback, arg);

The arg here will be passed as-is to the callback routine for every item. The flags govern the way the traversal is made. It can be a combination of the following flags, or 0 to use the default processing: DBM_F_SAFE will activate key checking during the iteration whilst DBM_F_SKIP will skip unreadable values, especially for big values that cannot be correctly fetched from the data file.

The sdbm_foreach() routine returns the number of items that were successfully iterated over, in other words, the amount of entries in the databse. It is possible to use a NULL argument for the callback parameter to have sdbm_foreach() simply iterate over the database to count the items.

Using sdbm_foreach_remove(), one can also iterate over all the items and use a callback of type sdbm_cbr_t whose signature is:

bool callback(const datum key, const datum value, void *arg);

It returns a boolean value, with TRUE indicating that the item being iterated over should be removed and FALSE indicating that the item is to be kept.

The sdbm_foreach_remove() routine returns the number of items that were successfully iterated over, and for which the callback returned FALSE, in other words, it returns the amount of remaining entries in the database.

THREAD SAFETY
By default, the database handles can only be used by the thread that created them. However, invoking sdbm_thread_safe() on a newly created database will allow safe concurrent accesses to the database: this locks the database handle for the duration of the call and routines which return a datum will make sure that the data returned are thread-private, making the necessary copy to allow concurrent updates to the database after the value was returned.

For multiple operations that need to be performed consistently over the database without interruptions by other threads, one may call sdbm_lock() to begin the critical section and sdbm_unlock() to end the critical section. Calls can nest as long as they are paired correctly. It is forbidden to use these routines on a database that was not marked as being concurrent via sdbm_thread_safe() beforehand.

The sdbm_is_thread_safe() and sdbm_is_locked() predicates can be used in assertions to check respectively whether the database was flagged as being thread-safe and whether it is currently locked by the calling thread. The sdbm_is_locked() routine will always return FALSE for databases that have not been marked thread-safe.

Concurrent iterations on the database are forbidden, and this is enforced by having the iterators return nullitem and set errno to EPERM.

When iterating, it is required to take the DB lock during the whole operation. Iterating routines such as sdbm_firstkey() or sdbm_value() do not return a thread-private datum and enforce the requirement that the database be locked.

When passing the database descriptor to other threads, be sure to use sdbm_ref() on the receiving side to properly account for the extra reference. The sdbm_close() call will panic when attempting to close a descriptor that has more than one reference still. Thus, when sharing descriptors, it is more suitable to call sdbm_unref() which will only close when the last reference is gone and which nullifies the pointer to the descriptor in the thread, preventing further access through that variable since the reference was dropped.

Iterating over a large database can take a very long time, and locks the database for the whole duration of the iteration. However, sometimes the application does not require such a strict iterating policy, but can very well cope with a loose iterating policy. The sdbm_loose_foreach() and sdbm_loose_foreach_remove() routines do just that: they do not grab the lock for the entire duration of the iteration, allowing concurrent updates to be done by other threads. However, they do not guarantee that all the items will be traversed, that no item will not be traversed more than once, that the traversed items hold the latest value (in case of concurrent update), or even that the items still exist (in case of concurrent deletion).

However, the loose traversal guarantees that the keys and values passed to the callbacks are consistent, and especially that values are not containing parts of several concurrent updates. Furthermore, the delete operation in sdbm_loose_foreach_remove() only takes place if no changes occurred on the page where the item lies since it was given to the callback, to avoid deleting a record for the wrong reasons.

By passing the DBM_F_ALLKEYS flag to the loosely iterating routines, the database is locked as pages are processed, meaning all the keys are guaranteed to be traversed and all deletions will succeed. However, callbacks are invoked with the database locked and that reduces opportunities for more concurrency. The lock is yield back after each processed page.

PAGE CACHING
This sdbm library is equipped with an LRU (Least Recently Used) page cache whose aim is to reduce the amount of read() system calls performed when accessing data. By default, it is set to cache 64 pages, but this may be changed at any time by calling sdbm_set_cache() with the amount of pages desired for caching. Use 1 to disable LRU caching altogether and only keep the last loaded page in memory.

It is also possible to enhance the performance of sdbm by turning write delay on via sdbm_set_wdelay() at any time. This will also greatly reduce the amount of write() system calls performed when changing data, but it could make the database inconsistent in the advent of a crash before sdbm_close() can be called. To mitigate that risk, the application can call sdbm_sync() on a regular basis (say every 5 seconds). That call returns the amount of pages flushed if everything was OK, and -1 if an I/O error occurred during flushing.

Even with deferred writes, there are important operations that are nonetheless flushed immediately to disk, when splitting a page for instance. Otherwise, in the advent of a crash, the disk data could contain twice the same key / value pair. However, if the database is recreated from scratch each time it is opened, there is no need for this precaution, since disk consistency does not matter then. The database is said to be volatile and this behaviour can be turned on by calling sdbm_set_volatile() with a TRUE argument. In which case it also automatically enables delayed writes and the database files will be deleted when sdbm_close() is called.

To know how a database descriptor has been configured, one can call sdbm_get_cache() to get the amount of pages configured for LRU caching, use sdbm_get_wdelay() to know whether deferred writes have been enabled, and check volatility by calling sdbm_is_volatile().

SEE ALSO
open(2).

DIAGNOSTICS
Functions that return a DBM * handle will use NULL to indicate an error. Functions that return an int will use -1 to indicate an error. The normal returned value in that case is 0. Functions that return a datum will return nullitem to indicate an error.

As a special case, sdbm_store(), when called with the DBM_INSERT flag and the key happens to already exist in the database, will return 1 and errno is set to EEXIST.

In general, if a function parameter is invalid, errno will be set to EINVAL. If a write operation is requested on a read-only database, errno will be set to ENOPERM. If a memory allocation (using malloc(3)) failed, errno will be set to ENOMEM. For I/O operation failures errno will contain the value set by the relevant failed system call, either read(2), write(2), or lseek(2).

When a sdbm_rename() or sdbm_rebuild() operation fails, the database handle may be left in a broken state, causing any further access to the database to fail with errno set to ESTALE. At this point, the database handle should be closed.

If the sdbm_rebuild() routine is called from within an iteration loop, it will fail and set errno to EBUSY. That same error is also returned when sdbm_rebuild_async() is called whilst another asynchronous rebuilding is in progress.

Conversely, if sdbm_nextkey(), sdbm_value() or sdbm_deletekey() are called outside an iteration loop, they will fail with errno set to ENOENT and they will loudly complain to indicate misuse.

When dealing with large keys or values, an inconsistency in the .pag or .dat files (such as bad block number list, requesting access of unallocated data blocks, etc...) is reported by setting errno to EFAULT.

AUTHORS
"Ozan S. Yigit" <oz@nexus.yorku.ca>

Raphael Manfredi <Raphael_Manfredi@pobox.com> (for gtk-gnutella extensions)

BUGS
The sum of key and value data sizes must not exceed PAIRMAX (1008 bytes) if large key/value support was disabled by calling sdbm_prep() with a NULL value for datname. Otherwise, with large key/value support enabled, the sum of key and value data sizes cannot exceed 250 KiB currently, but the real limit can be lower depending on the respective sizes of keys and values. Use sdbm_is_storable() to be sure.

The sum of the key and value data sizes where several keys hash to the same value must fit within one bitmap page.

The .pag file will contain holes, so its apparent size is larger than its contents. When copied through the filesystem the holes will be filled.

The contents of datum values returned are in volatile storage. If you want to retain the values pointed to, you must copy them immediately before another call to this package.

The only safe way for multiple processes to (read and) update a database at the same time, is to implement a private locking scheme outside this package and open and close the database between lock acquisitions. It is safe for multiple processes to concurrently access a database read-only.

APPLICATIONS PORTABILITY
For complete source code compatibility with the Berkeley Unix ndbm(3) library, the sdbm.h header file should be installed in /usr/include/ndbm.h.

The nullitem data item, and the sdbm_prep(), sdbm_hash(), sdbm_rdonly(), sdbm_dirfno(), and sdbm_pagfno() functions are unique to this package.

The following functions:

sdbm_datfno()
sdbm_replace()
sdbm_firstkey_safe()
sdbm_deletekey()
sdbm_endkey()
sdbm_value()
sdbm_is_storable()
sdbm_count()
sdbm_delta()
sdbm_delta_reset()
sdbm_sync()
sdbm_shrink()
sdbm_clear()
sdbm_unlink()
sdbm_rename()
sdbm_rename_files()
sdbm_rebuild()
sdbm_rebuild_async()
sdbm_get_cache()
sdbm_get_wdelay()
sdbm_is_volatile()
sdbm_set_cache()
sdbm_set_wdelay()
sdbm_set_volatile()
sdbm_set_name()
sdbm_name()
sdbm_thread_safe()
sdbm_is_thread_safe()
sdbm_lock()
sdbm_unlock()
sdbm_is_locked()
sdbm_ref()
sdbm_unref()
sdbm_refcnt()
sdbm_foreach()
sdbm_foreach_remove()
sdbm_loose_foreach()
sdbm_loose_foreach_remove()

are additions made by Raphael Manfredi for gtk-gnutella but anyone is welcome reusing these extensions.

Support for large keys and values was added by Raphael Manfredi in a way that is backward compatible (i.e. databases not requiring this feature should see no difference). When activated (as needed), a third file is created to store the larger keys and values, ending with the extension .dat.

The API was also made thread-safe by Raphael Manfredi.