Under the hood KF 4.6 has moved from using ZDBase for its backing store, instead using a tuplebase. This makes it possible to split HTML page generation into separate processes, potentially running on multiple front end machines. It also restores support for a rich client application, now written in Java, using ZTSoup to efficiently communicate changes in the tuplebase, whether made by other clients or by the web interface.
The tuplebase API is well-suited to data processing needs. But it's clumsy as the mechanism by which data in a tuplebase is to be presented and maintained in a live UI, rather than web pages or generated reports.
ZTSoup gets its name and inspiration from the Apple Newton concept. A ZTSoup is backed by the same data as a tuplebase, but that data is accessed by instantiating a ZTCrouton for each tuple that code is interested in, and a ZTSieve for the result set of any query of interest. Why the funny names? A soup has croutons floating in it, interesting ones are sieved out of it.
Initially I provided Java access to a tuplebase instance by implementing Java classes whose most interesting methods were marked as native, and thus invoked via JNI. This was very powerful because Java could use any tuplebase implementation simply by calling the appropriate factory function and I could expose any existing C++ functionality simply by implementing the appropriate JNI glue.
ZooLib's tuplebase is derived from the tuplespace concept initially explored in the Linda coordination language, another well known derivation of which is Sun's JavaSpaces system. Whereas JavaSpaces is Java-only and relies on many of that languages's features, ZooLib's tuplebase works today with C++ and Java, and is well suited to work with other languages.
IFF is a venerable data meta-format, introduced in 1985 by Electronic Arts as a standard framing mechanism for multimedia data. In short it defines a nested chunked format, where the file as a whole is considered to be a chunk. A chunk has as its first four bytes a tag that indicates the type of data in the payload, a 32 bit (four bytes again) count of the number of bytes in the payload, and then that number bytes being the payload itself. Some chunk types are known to contain zero or more other chunks in their payload, so an arbitrarily complex hiearchy can be established. There are other details, but chunk types and sizes are IFF's essentials. QuickTime's file format is almost the same, except that the size comes first, and includes the eight bytes of the chunk header.
IFF/QT are simpler than most file formats, but can still be fiddly to work with. To make it easier for me to parse and generate QT files manually (on platforms which don't have QT libraries) I put together a suite of ZooLib streams that take care of all the bookkeeping.
Most of my work till this point had not required 'interesting' operations with file systems, being restricted to creating and opening files in externally determined locations, then reading and writing the files' contents. When it became necessary to ennumerate the contents of directories, and to deal with permissions and locking I took the opportunity to define an API that would be consistent across Windows, Mac and UNIX whilst cleanly accomodating their differences.
Applications often have to satisfy two conflicting requirements. On the one hand the data created by a user has a structure whose parts should be managed independently of one another, ideally with each piece placed in its own file. On the other hand users like to think of their data as a single entity which can be copied, emailed and backed up in its entirety. Although Mac OS X has the notion of a bundle, a specially marked directory which behaves like a single file when manipulated by the Finder, other operating systems do not.
I picked the name 'asset' as an alternative to 'resource', a term that already has too many disparate meanings on different platforms. That said, assets are used in the same situations that MacOS/Win32/BeOS resources would be, although the mechanism is more flexible.
The data making up an asset tree is directly usable by any processor, big or little-endian. It can be kept in a file, loaded into RAM, memory-mapped from disk or accessed from a stream.
ZDBase is a relatively simple database storage engine. It's more of an alternative to Berkeley DB (as it was) than a replacement for MySQL. It supports an arbitrary number of tables, records and typed/named fields within those records. Each table maintains as many indices as desired, and the schema can be updated at any time. All data is kept in a blockstore, so everything lives in a single file, and the code and data are portable across platforms.
Marrakech took the hypermedia concepts I explored in WorkSpace and applied them to the problem of managing workflow and assets for multimedia development.
