February 2014

WireOver - Fast, Free, Secure File Sending

WireOver is a Mac, Windows and Linux application for sending and receiving files. It's easy to use and very low friction, requiring only entry of an email address and click on a received web link to set up. Receiving an unsolicited send is even easier, as acting on the receipt of the notifying email naturally associates that email with the recipient.

Transfers (by subscribers) use perfect forward secrecy, and can be entirely asynchronous, so sender and receiver need never be online at the same time. Of course if they are, then transmission goes as fast as the physical network allows.

We built the backend and client mainly in Python, supporting Mac and Windows back to 10.5 and XP respectively. For Linux we support 32 and 64 bit Ubuntu 12 and later.

Much of my focus has been on keeping OS variations from impacting the main body of the code. To that end, we're using wxWidgets to abstract the GUI, and I've implemented such native code as needed to interface with unwrapped aspects of the OS and third-party libraries. In some cases the fluent Python bindings to OS facilities just aren't fast enough, and native code is used there also.

June 2010

Natter

Multiplexing comms over a single bidirectional channel is a useful ability. ZStreamMUX has proven itself, but to support fully bidirectional substreams it uses a pair of threads for each MUX instance, and another thread for each pending listener. For RPC-ish situations this is overkill.
Natter is a lovely English word that well describes a lightweight back and forth between participants. A ZNatter is constructed with a ZStreamerR and a ZStreamerW instance (often two aspects of a ZStreamerRWCon instance).
From instantiation the two sides of a ZNatter are symmetrical, each sending and blocking till a response is received. Of course the interesting situations are the beginning and the end of such an exchange. On the server-ish side we call ZNatter::Receive. The result of such a call is a ZQ<ZData_Any>, with a ZRef<ZNatter::Exhange> as an optional output. On the client-ish side we can call ZNatter::Send and disregard any response, or we can instantiate a ZRef<ZNatter::Exchange> passing the ZNatter with which we wish to participate, and then call its SendReceive method.
It's certainly a constrained communication mechanism, but one that maps cleanly to common patterns with no extraneous resource implications.

April 2010

Jabber (XMPP) and AIM (OSCAR)

To integrate with existing social networks, the team at Zorap defined a mechanism that passed structured information via instant messages. My role was to implement Jabber (XMPP) and OSCAR, the AOL instant messenger protocol, in a portable and robust fashion.

Zorap ran as a browser extension, so could be instantiated and destroyed many times within the life of a process. Any code it used had to be robust, and could not depend on operating system facilities to dispose of resources like network sockets.

March 2009

ZooLib BBDaemon

The ZooLib BBDaemon lets multiple Mac applications talk to USB-connected BlackBerrys concurrently. Obviously this is something that Research in Motion could make possible, but as they haven't we're posting a pre-built installer that takes the pain out of getting things working.

If you're using Missing Sync for BlackBerry 2.0.1 and iMobimac Modem then this is what you need.

BBDaemon runs as a user-space process, not a kernel extension. The installer simply puts the executable in /Library/PrivilegedHelperTools and a launchd control file in /Library/LaunchDaemons.

RIM's own Mac applications (PocketMac for BlackBerry and BlackBerry Media Sync) take a different approach. They each install a kernel extension which grabs any USB-connected BlackBerry, and prevent any other application (including each other) from being able to talk to it.

The installer we've posted was built with revision 533 of ZooLib. You can check out the source and build it yourself:

svn export -r 533 https://zoolib.svn.sourceforge.net/svnroot/zoolib/trunk/zoolib

svn export -r 533 https://zoolib.svn.sourceforge.net/svnroot/zoolib/trunk/zoolib_samples

cd zoolib_samples/BlackBerry/BBDaemon/osxpackage

sh build.sh

The executable and launchd file must be owned by root/wheel, so the build.sh script uses sudo to change the ownership – thus you must be logged in with an administrator account and you may be prompted to enter your password.

February 2009

The Missing Sync for BlackBerry

Mark/Space has released The Missing Sync for BlackBerry 2.0. This latest version provides a rich suite of synchronization options, made possible by a device-side Java application developed by Electric Magic. The Mac application uses our ZBlackBerry Mac/Black SDK for its USB communications.

March 2008

Mac/BlackBerry SDK

The BlackBerry is a very popular mobile communications device. Official Mac support from Research in Motion is limited to providing the PocketMac utility as a free download. With no official SDK the Mac/BlackBerry ecosystem has seen very little activity.

ZBlackBerry is a suite of code that implements the BlackBerry USB communications protocol in a generic fashion. A few hundred lines of code let Macs use that protocol. A few hundred more allow multiple Mac applications to talk to a single BlackBerry simultaneously, something that has not been possible till now.

ZBlackBerry can also make a BlackBerry connected to a PC accessible via the same API as is used when it's connected directly to a Mac. This lets Mac application developers run their BlackBerry application under the Windows-based debugger and still have it communicate with their Mac application. This is a crucial capability when writing anything more than the most trivial application.

ZBlackBerry is part of ZooLib, our open source C++ library.

January 2008

iMobimac Modem

The Research In Motion BlackBerry is famous for its connectivity. iMobimac Modem runs on a BlackBerry and the Mac to which it is connected, and lets Mac applications access the Internet using the BlackBerry's connection.

The BlackBerry application is written in Java. It maintains a single communications channel with the Mac application, over which it receives requests to make, use and break network connections. The channel is multiplexed with ZooLib's Java ZStreamMUX on the BlackBerry side, and its C++ ZStreamMUX on the Mac.

Much of the Mac application's functionality is a C++ HTTP and port-forwarding proxy server, most of which is in ZHTTP. Its UI is implemented in Objective C using Cocoa Bindings.

The remainder of the Mac implementation effort was in reverse-engineering the BlackBerry/Host USB communications protocol.

May 2006

Knowledge Forum 4.6

Knowledge Forum 4.6's minor version number change belies just how different from its predecessor it really is.

Under the hood KF 4.6 has moved from using ZDBase for its backing store, instead using a tuplebase. This makes it possible to split HTML page generation into separate processes, potentially running on multiple front end machines. It also restores support for a rich client application, now written in Java, using ZTSoup to efficiently communicate changes in the tuplebase, whether made by other clients or by the web interface.

January 2006

WebDAV

WebDAV is an extension to the HTTP protocol. It is the basis for Apple's iDisk and Windows' Web Folders, standard features of Mac OS X and Windows XP. It is thus the easiest way for a server to make data available to a client machine without requiring that client software be installed first.

ZooLib provides a generic WebDAV server. Your application need only implement subclasses of ZNodeRep to represent nodes in your desired hierarchy. Standard subclasses of ZNodeRep let you lay one tree over another, or expose part of the server's file system to clients.

Frankly, WebDAV is an inelegant protocol. It puts some information in the URL, some in header lines, and some is encoded as XML in the body of requests and responses. But it has three sterling characteristics:

  • It exists
  • It works
  • It's usable out of the box on Mac, Windows and many flavors of UNIX

August 2003

Knowledge Forum 4.5

Knowledge Forum 4.5 was a significant refinement of Knowledge Forum 4.0, with a much richer web interface, although still constrained by the need to support Netscape 4.x-era web browsers.

New features included support for upload and download of files as attachments to views and notes, headings within views, full text and attribute-based searches, and notification by email. It also supported graphs of useful metrics, implemented across platforms using ZDC_ZooLib.

On the back end, a significant change was moving from the original blockstore implementation to one based on B+Trees that uses a neat upward propagting copy-on-write of nodes to make it virtually bullet-proof - only a physical failure of the backing store can cause data loss.

September 2002

Knowledge Forum 4.0

Knowledge Forum 4.0 took the radical direction of being web-only. We'd had some support for web access by virtue of a perl program that used the client's communications protocol to talk directly to the server, but perl wasn't pre-installed on Mac OS (Classic) or Windows, and it had some performance problems. So we ported the perl software to C++ and incorporated it into the server directly.

For a time we had native client and web access running simultaneously, but never in a released version. The 3.x client remains buildable as of May 2006, but still talks only to a 3.x server.

January 2002

ZStreamMUX

There's a maxim, a citation for which I can't locate now, which pours scorn on the idea of multiplexing streams over TCP. It kinda made sense when I first saw it, but in these days of ornery system adminstrators and their firewalls that sometimes allow connections seemingly on the roll of a die, there's a lot to be said for the all or nothing of getting a connection to a server, and then sharing it locally.

I wrote ZStreamMUX after coming across an early draft of WebMUX. ZStreamMUX takes a read stream and a write stream, and runs a lightweight protocol over them to support multiple independent sessions. The overall interface is similar to that provided by sockets. The protocol is asynchronous, and uses buffer credits to prevent deadlock at the protocol level.

December 2000

ZooLib

ZooLib is an Open Source (MIT License) C++ library that makes it easy to write one set of source and build an application for Windows, Mac and UNIX. It provides a foundation suite of facilities that in essence form a virtual operating system API, and wide range of higher-level facilities that build on that foundation.

ZooLib abstracts threads, files, networking, graphics and the user interface. And there are escape hatches whereby OS-specific features can be accessed without compromising the platform-agnostic nature of the rest of the code.

ZooLib has been under continual development since 1992, and incorporates fixes for every OS bug or anomaly I've encountered, and generally provides cleaner mechanisms for most tasks than OSes traditionally provide. It's been the basis of Measurement in Motion, NetPhone, and every version of the Knowledge Forum server and client applications.

January 2000

Knowledge Forum 3.0

Knowledge Forum 3.0 took advantage of major enhancements in ZooLib that let the client be released for Windows as well as Mac OS. Much of Knowledge Forum 2.0 had been built around the Mac-only Zoom framework, and so had to be reimplemented. The parts of Knowledge Forum 2.0 built with ZooLib were simply carried forward, with refinements and enhancements.

By this time ZooLib abstracted Mac OS, Windows and UNIX, so the server could be hosted on all three OSes.
The text editing in Knowledge Forum 3.0's note windows used the PAIGE text engine, which was a fairly crusty 60,000 lines by this time, but was portable across Mac and Windows. So although the rest of the client could be built for UNIX, without the text editor it wasn't particularly useful and we never released a UNIX client, although that wasn't a big deal in 2000.
Unfortunately we also had to abandon Kevin Parichan's excellent painting implementation, as it was also Mac-specific. Instead we used ZooLib's NPainter, which used only ZooLib features and was thus completely portable.

August 1997

Knowledge Forum 2.0

Knowledge Forum 2.0 is a computer supported collaboration environment designed to foster the growth of knowledge building communities. It's based on the CSILE project, developed at the Ontario Institute for Studies in Education and is published by published by Learning in Motion.

CSILE was pioneering and innovative, but in May 1994 limitations in its feature set and a desire to see it in wider use led to a collaboration with Learning in Motion to redesign it. The scope of the necessary changes neccessitated a complete reimplementation and the decision was taken that it would be released as a commercial product.

Cyrus Ghalambor started work on the KF 2.0 client application in July 1995. I was busy with NetPhone till December 1995, when I started working full time on the client's data API and the server that would satisfy its requests. We looked at several existing database libraries, but they were either too limiting or could not run on Mac OS (Classic, of course). So I implemented ZDBase, a portable database engine that was built on ZooLib. Client and server communications and the server's administration UI were also built using ZooLib.

The client application's event loop, menus and windows were handled by a refinement of Berkeley Systems' Zoom framework. It had two main types of windows. The note window used Zoom for its controls, and the WASTE text engine for its text editing panel. The view window provided a two-dimensional icon-based representation of collections of content, its operation was closely modeled on that of Apple's Finder, and it was implemented using ZooLib. The note and view windows also contained pictures, in-line with the text in the note window and in a layer behind the icons in the view window. Pictures were edited in place using a nice painting implementation by Kevin Parichan.

Other dialogs and property inspectors were based on Zoom or ZooLib, depending on whether they were written by Cyrus or by me.

April 1996

DigiPhone

The publishers of Digiphone, a VOIP application for Windows, took a shortcut to getting Mac compatibility - they (indirectly) bought NetPhone from Electric Magic, and had me add support for DigiPhone's communications protocol and audio codec.

January 1995

NetPhone

In 1994 the Internet was just starting to be available in people's homes and offices. NetPhone was the first application to support what's now known as VOIP (Voice Over IP) without requiring a high-speed connection.

Project Archives