hughpyle : Internals

GWS Security

hpyle — Mon, 15 Aug 2005 16:25:00 GMT

Access to a Groove workstation is controlled by the account passphrase; once the account is unlocked, you can create workspaces, and read and modify data within all your workspaces (according to the permissions you have in those workspaces). Similarly, the web services interface doesn't introduce any additional access control restrictions. The Groove Web Services APIs allow essentially unrestricted access to everything within an account.

The secret key to unlock GWS access, however, is not the account passphrase. (Your account passphrase is never stored on disk, and we don't want to send it over a TCP port either; it's not just secret, it's personal too). So the GWS designers needed to create something else for authentication -- authentication of an application (a process running on the local machine, usually) which wants access to your Groove data. (The account must also be logged in, by the user typing their passphrase, because the passphrase really does key the cryptographic stuff of the account. If a GWS application requests access to a workspace when the account is not logged in, Groove will pop up the login dialog).

The solution is implemented using a registry key in the HKCU hive. When Groove starts, it creates a strong random secret, and writes that to a "current user" registry location (HKCU\Software\Groove Networks, Inc.\Groove\WebServices\LocalRequestKey). Local web services requests must provide that key value, which they can obtain by reading the registry (ie. by having a certain level of access to the local system). The key value may change at any time.

To avoid man-in-the-middle attacks on web services accesses, Groove also provides a LocalResponseKey in its SOAP responses, which again can be validated against the local registry if you want. Groove also writes its process ID to HKCU\Software\Groove Networks, Inc.\Groove\GrooveLocalHTTPServerPID.

Remote access -- usually only deployed for EDB servers where an integration application is running on a separate server in the data center -- uses a slightly different method; there's a different registry key (HKCU\...\RemoteRequestKey), but this one can be set to any value and remain constant, if appropriate. Confidentiality of the remote key and the network traffic is not done by Groove, so for remote applications you probably want a dedicated LAN segment and IPSEC, or some other mechanism, to secure the SOAP access. (Remote access is also disabled by default, and you need policy settings to switch it on. WS-Security: not yet.)

The other missing piece from my first code was the GWS HTTP port (which defaults to 9080). Again, there's a local registry key (HKCU\Software\Groove Networks, Inc.\Groove\GrooveLocalHTTPPort) updated by Groove when it starts listening.

To write an application calling local Groove web services, then, we need a few helper routines to provide access to these values. Each SOAP request sent to Groove must include the request key in the SOAP header, and the call must be targeted at the appropriate TCP port.

Here's my (static) util class which implements this. (Note: there's no dependency on the Groove WSDL web references):

namespace GrooveContactsAddin

{

static class GWSUtil

{

static public string GrooveURL

{

get

{

string url;

Microsoft.Win32.RegistryKey HKCU_GrooveRegKey = Microsoft.Win32.Registry.CurrentUser.OpenSubKey(@"Software\Groove Networks, Inc.\Groove");

object DesiredGroovePort = HKCU_GrooveRegKey.GetValue("GrooveHTTPDesiredPort", null);

if (DesiredGroovePort != null)

{

url = "http://localhost:" + DesiredGroovePort;

}

else

{

object GroovePort = HKCU_GrooveRegKey.GetValue("GrooveLocalHTTPPort", "9080");

url = "http://localhost:" + GroovePort;

}

return url;

}

static public string GrooveLocalRequestKey

{

get

{

Microsoft.Win32.RegistryKey HKCU_GrooveWebServicesRegKey = Microsoft.Win32.Registry.CurrentUser.OpenSubKey(@"Software\Groove Networks, Inc.\Groove\WebServices");

string key = (string)HKCU_GrooveWebServicesRegKey.GetValue("LocalRequestKey");

return key;

}

static public string GrooveLocalResponseKey

{

get

{

Microsoft.Win32.RegistryKey HKCU_GrooveWebServicesRegKey = Microsoft.Win32.Registry.CurrentUser.OpenSubKey(@"Software\Groove Networks, Inc.\Groove\WebServices");

string key = (string)HKCU_GrooveWebServicesRegKey.GetValue("LocalResponseKey");

return key;

}

Deep dive: how Groove Forms synchronizes data

hpyle — Fri, 22 Jul 2005 18:36:00 GMT

As I mentioned earlier, Groove's essential magic is that all members of a workspace see the same information in the space. That's maintained even though members can work offline (each member has a copy of each of their workspaces, so all the information is available when you're disconnected, and you can work in the space regardless of network connectivity), or be in separate companies behind firewalls. And there's no central server; none of your workspace data is permanently stored anywhere else than on the machines of the workspace members. The synchronization happens automatically, immediately, and is extremely resilient; it's designed to be "sufficiently advanced".

There's lots of depth in the mechanics, and I don't plan to dive into all of it. (If you want reading material, try the product docs, or even look up the patent numbers on Groove's About box: 6446113, 6640241, 6859821). But it'll be helpful to understand the overall structure of Groove and how it works. (Standard "how-internals-work" disclaimer goes here. I'm just writing down what I believe to be the case about how things work; it may not all be correct, and of course might change in the future too, so don't rely on this the way you'd use official documentation.)

Physically, each workspace is a database file; they're files with .XSS extension (and corresponding transaction logs), usually to be found under <user>\Local Settings\Application Data\Groove Networks\Groove\accounts\<id>\telespaces. These databases are encrypted (with your account key). The file format is Groove-developed; essentially each XSS file is a collection of binary and/or XML documents. There's a Groove subsystem called Storage Manager which is responsible for reading and writing the databases on disk, managing the transaction integrity, and so on.

Your Groove account is also stored in an XSS database file (id.xss). Actually, the "account telespace" shares lots of characteristics with regular Groove workspaces; it's a distributed database in exactly the same way, so you can have your whole Groove account installed on multiple devices, which will all keep in sync.

Synchronization is performed by a subsystem called Dynamics Manager, reading and writing to the network via a subsystem called Communications Manager. But to understand Dynamics, we should first take apart a workspace and peek inside.

The workspace is a container for Tools (there's a ToolContainer COM component which implements tool-containing-stuff). We've already seen some of the user-visible tools; and there are several invisible tools too; one to manage the member list, one for managing the tabs which display UI (the RootDisplay tool), and a few others.

Tools themselves are quite complex. There's a tool template: an XML document which defines all the components and their connections (the components, at this level, are really COM components which implement parts of the tool's functionality). Tools mostly all have the same sorts of components, in a Model-View-Controller structure: there's a data-management piece (one or more Engines, and a DataDelegate providing programming interfaces to the outside world), a user interface (with its widgets usually being COM components arranged according to the spec in the tool template, and a ViewContainer managing the layout), and glue between them (UIDelegate code of some sort, usually written in C++, but historically also plenty of JavaScript). A tool also has a ToolDescriptor, which is a small XML document describing the tool: its name, version, the location of the template, and so on. (If you invite someone to a workspace, usually they're sent each tool descriptor and the engine data but not the tool template; the descriptor tells the client where to get the template).

Internally, every component -- every component -- is addressable by URL. In the case of an Engine, this URL is really important; it's used as the name of a message queue.

Engines are where work happens on a tool's data. Let's take the example of a Forms tool; the tool has an engine called RDB (for "record database"), and RDB knows how to maintain reasonably large data sets with multiple schemas and indexes. (RDB uses a btree-type structure; maybe Ed can expand on the details sometime, but honestly the details aren't very important. One change in RDB over its predecessor RSE (recordset engine) is that RSE stored XML, where RDB is binary - Jack Ozzie and Jon Udell discussed this a while back).

The outside world talks to an Engine by sending it transactions, or commands. So in the Forms tool and the RDB underneath it, there are a few basic transaction commands: "CreateRecord", "DeleteRecord", "SetField". When a user (at the Groove user interface) saves a Forms record, the View code tells the DataDelegate to create a record with certain field values, and the Engine receives a "CreateRecord" command with those field values inside. Or, if the user edits an existing record to change a field value, the engine sees one or more "SetField" commands (possibly wrapped inside an outer transaction, so the field changes all apply atomically).

Synchronization between workspace members happens at this level: the transaction (a distributed transaction in Groove is called a Delta). Dynamics Manager sees the command to the local engine, and creates a copy of that command for each other member of the workspace, wraps them inside a routing packet (XML), and puts those commands into a queue. The queue is "targeted" at the same engine URL, but for each other device in the workspace. Communications Manager can then route data in the queues (encrypted) across the network to every other member; depending on your communications state, the transactions can be sent directly to the receiving device (peer to peer), or be sent to the recipient's relay server (to sit in a queue -- if you have access to a relay server's management UI you can see the queues -- the relay admin guide has lots more detail, but unfortunately no screenshots). Or you might be completely offline, in which case the transactions sit in queue on your own machine until they can be offloaded.

Meanwhile, Communications Manager is also listening to connections from other devices, and dequeueing transactions from your own relay server, and sending those transactions to Dynamics Manager to tell your tools' Engines that their state must change. Dynamics Manager is the middleman, and for good reason: it takes care of transaction sequencing.

Since you can work offline, there's no guarantee that your machine will receive transactions in anything like the same order as other members of the workspace. You might create some deltas yourself offline, which then need to be "later" than work done by other members while you were disconnected. In an extreme (but common) case, for example, you could be working offline for a long period, then drive by a WiFi connection and manage to send a fraction of your outgoing queue data, then disconnect again, do some more work, and reconnect. So it's entirely possible that other workspace members see a different set of deltas than you.

Each delta is assigned a sequence number, and Dynamics Manager maintains a table of which deltas are known to you, and which of those you know are also known to all other members of the space (so they can be purged from the log), and which might be fetchable from other online members, and so on. If you have delta A and C, then receive delta B from another member, then Dynamics will decide (based on the dependency nesting of the transactions) whether it can just insert B right away, or whether it should tell the engine to undo transaction C, then apply B, then reapply C.

The magic of this is twofold. First, the Dynamics Manager sequencing doesn't care about the actual data, or the implementation of the particular Engine; it's only concerned with the transaction sequencing and dependencies, and filling in any gaps in your transaction history. Second, everything happens completely transparently to the user (even if there's a fairly large rollback and rollforward -- you might notice a performance glitch there, but big rollbacks are very rare), and the result is that all members of the workspace will always see a consistent transaction sequence, and they'll all see the same sequence (eventually).

So, back to the Forms tool. The RDB engine underneath Forms manages distributed transactions (insert, update, delete) against records, and other things such as schema changes, within your tool. If a record changes because another user changed it, the engine will assimilate that transaction, then notify the View to update the display -- immediately, if the user's working in that tool at the time.

Now, not all data in Groove is always disseminated. Files in a Files tool are one example; the "stub" of the file is always disseminated, but the file contents can be fetched on demand if appropriate. The Forms Designer Sandbox is another example: if you open the forms designer to make changes in the tool (adding or modifying fields, forms, views), those changes are only stored in your local device until you hit the "Save to Groove" button.

The end result is that users can work with your forms, creating, reading and updating rich documents, and Groove takes care of all the synchronization stuff to ensure that everyone has a consistent view of the world. (Should I talk about conflicts at this point? Naah, never happens. Or rather, with most Forms applications conflicts are rare enough that I can leave that for another day. Remind me, later).

Digression: the types of fields

hpyle — Fri, 22 Jul 2005 15:26:00 GMT

Groove Forms forms are constructed by laying out fields on a page. Actually, fields are just about the only things on the page (apart from field-groups and tab-groups, which we'll get to some time later). So a "field", in the form designer, encompasses several things which you might not usually consider as data entry or data definition -- buttons, for example.

That's why the list of field types in the "Create New Field" dialog is quite so long.

Let's break down that list and look at each type of field in turn. First the usual suspects: fields for entering data in a form.

Data entry fields

Text: a single-line text box. There's no real limit on the size of text you can enter in one of these (but it's always entered on a single line); optionally, you can specify a maximum length, which restricts users from entering text beyond that limit, as well as other types of input validation. When displayed on a form, the field actually has two parts: the input box, and a label beside it (to the left, or to the right, or above it).

Multi-line Text: a multiple-line text box. You can specify the width and height. Like all text fields, the amount of text Groove can store in a multi-line field is effectively unlimited. You can also specify validation in the same way as a text field: to ensure that the text contains a particular string, or doesn't, or is a valid email address or ZIP, and so on -- quite a long list of options. (It's also possible to write code for extra validation).

Number: a single-line entry to hold a number. It looks just like a text field, but doesn't allow non-numeric values. Number fields can define a precision (the number of digits after the decimal point), and can restrict input to numbers in a particular range. Now, numbers are stored differently from text in the Groove records' data; for example, you can build a view which adds up the total of a number field across several records -- can't do that with text. Internally, numbers are double-precision floating point values.

Unformatted Number: like a number field, but doesn't format the number with the user's locale's numeric formatting -- separating thousands with commas, or whatever. (You can't change the type of a field after you create it, so it's unfortunate that display formatting isn't just a property on a regular numeric field. Oh well.)

Currency: like a number field, but displays using a currency format.

Password: a text field where the input only displays as dots to hide your secret. (The data you enter in a password field is NOT encrypted in any special way. Which is to say, it's encrypted multiple times with astounding confidentiality strength before being stored in Groove or transmitted to other members of the workspace, just like all data in a Groove workspace; and when the data arrives on another member's machine, after they handshake to verify that it wasn't spoofed or manipulated in transit and that the information originated from exactly who you claim to be, it's decrypted with a stunningly deep reverse of the original encryption algorithm, so that the other members can look at it if they want to).

Date: a date (with no time portion; just the date). In the form, date fields show a calendar button beside the date input box; pressing this button opens a calendar (defaulting to today's date if the field is empty). You can design the field to have an initial value (a particular date, or "today", "tomorrow"), choose the date format, and validate that the date lies within a particular range. (Again, it's possible to add code for more particular validation).

Date Time: a date and time. The calendar picker popup for a date/time field includes a time picker too.

Option Buttons, aka Radio Buttons. The field includes a list of options; for each option, you can specify the "display value" (which appears beside the selection button in the list) and also a "stored value" (which is saved into the record when that option is chosen).

The options needn't be hard-coded; they can come from a lookup (ie. pulling data from a set of records in Groove). Since we haven't yet created any records in the contacts-form example I'm building, we'll cover lookups another time.

Check Box: just one checkbox.

Drop-down List: what it says. Options are defined the same way as for Option Buttons fields and List Box fields. Optionally, you can have the drop-down list include a blank value; otherwise, it'll default to the first option. And that "+" button is another option, allowing users to add their own values to the list (pressing it opens a little custom entry dialog).

List Box: a list box. Specify the options, and the size of the listbox, and whether to allow multiple selections.

Other types of data

All the standard field types above should be fairly familiar. If you're looking closely, you might notice that they're implemented with HTML; the form, it turns out, is actually a web page inside of the Groove forms tool (which is the basis for the custom scripting we'll get into later). But there are some types of fields which go way beyond HTML's usual capabilities, allowing the form to become more dynamic. These fields are:

Rich Text. A proper rich-text entry field, with text and paragraph formatting. In the example pictured here, I set some default contents for the richtext field, and also turned the background blue. The toolbar is optional; right-clicking pops up a formatting menu.

The richtext field, like the other fields, has an optional field label beside it; and you can specify the width and height. One thing you can't easily do is specify validation of the contents, though.

It even has a built-in spell-checker; press F7.

Attachments: a place to attach one or more files of any type.

Contact: a field which contains a real, "live", groovey contact. This is very special, actually; maybe we'll use one of these in the contact-form example. If there's a Groove contact in the field it will display online awareness, and you can right-click to send instant message, add to your personal contact list, invite to a workspace, and so on.

Non-Data Fields

Finally, the oddments and assortments. These fields appear on a form, but there's no data stored in them. (When I first started using Forms, I immediately wanted to add a button onto a form, and have some JavaScript run when the button was clicked. Turns out this is really easy, but it took me a while to realize how: the secret is that buttons are fields. And newlines are fields too. Now isn't that cute?).

I'm all out of screenshots for this lot. Their look-and-feel is defined mostly by the stylesheet used on the form anyway; stylesheets are another important thing on the "later" list.

Form Heading: a heading for the form. You usually never have more than one of these per form.

Section Heading: another piece of heading text, usually big and bold.

Static Text: just a random piece of text; for an extra label, for example. The default formatting is different from a section heading or a form heading.

Horizontal Line: a rule across the form (or across part of the form).

New Line: a line-break on the form.

Image: a picture (not editable by the user). Use an image field to put your logo on a form, for example.

Script Button: a button, with some script attached; when you press the button it runs the script. The script has access to everything in the current state of the form, and the record underneath the form; forms scripting is quite powerful (and really not difficult, but we'll get to the details soon),