GPII Architecture Overview

From wiki.gpii
Revision as of 01:44, 6 August 2015 by Colinbdclark (talk | contribs) (Matchmaker Implementations)
Jump to: navigation, search


Executive Summary

This document provides an overview of the Cloud4all architecture and realtime framework, which supports the scalability, modularity, and interoperability of the Cloud4all project’s technology development efforts. This architecture is intended to reach across a variety of platforms and devices, helping to deliver on Cloud4all’s promise of ubiquitous personalization using cloud technologies.

The essential goals of this architecture are to support the long-term growth, sustainability, and viability of the technologies developed across the Cloud4all project. These goals are realized by providing:

  • a set of interoperable components, service APIs, and data exchange formats
  • a framework for developing and connecting components together in a modular and loosely-coupled manner
  • development practices and techniques to ensure scalability, reliability, security, performance, and sustainability
  • a working, cross-platform reference implementation

The architectural approach taken is “web oriented,” emphasizing the use of standard web technologies and RESTful service APIs. This helps to ensure that the Cloud4all technology can be easily ported to new platforms and devices, as well as providing a cloud-ready flexible deployment model.

The core components of the system include the Preferences Server, Flow Manager, Device Reporter, Solutions Registry, Transformer, and Lifecycle Manager. These components are built as independent, loosely coupled components that are “wired” together using a declarative, event-based Inversion of Control container. This approach helps to ensure that the Cloud4all system can be extended, adapted, and modified to suit future features, requirements, and technological changes. The Transformer and “data lensing” approach within the Cloud4all architecture ensures greater interoperability by enabling different schemas, ontologies, and data structures to be bi-directionally transformed between standard and implementation-specific formats.

The Cloud4all architecture also provides a robust framework for preferences management, including the cloud-based preferences server and the preferences framework that supports the development of a variety of preferences editing user interfaces such as the preferences management tool and the personal control panel. These architectural features help to promote a “one size fits one” approach to accessible and user interface design, where different preferences editing user interfaces can be provided for diverse user needs and contexts.

The Cloud4all architecture is being designed and developed in an open source manner that will help foster a growing and sustainable development community for the long term, in tandem with the larger GPII effort. This development is happening on the popular Github social coding site, which helps us disseminate our activities and recruit new contributors.

Overview of the Cloud4All Architecture

The Cloud4all architecture encompasses an overall technical approach, a development framework, and a working reference implementation that serves as the basis for developing an ecosystem of tools to support user interface personalization. This architecture is designed to work across platforms and devices, including desktop computers, mobile devices, and web applications. It includes the specification and implementation of the data interchange formats, component APIs, and extension points that comprise Cloud4ll's personalization workflow.

This architecture is designed to promote the long term growth, sustainability, and viability of the Cloud4all technology. The primary concerns of the architecture are to ensure that all critical components within the system are built in a performant, modular, reusable, secure, and well-tested manner. These goals are primarily accomplished by providing a core framework that manages the personalization workflow and all relationships between components within the system, ensuring that it can be extended and reconfigured to support new functionality and new implementation strategies over time.

The Cloud4all architecture reference implementation (often referred to as "the realtime framework") builds on the familiar cross-platform and standards-driven approach of the web. In particular, it makes extensive use of both client and server-side JavaScript (using Node.js as its primary cross-platform runtime) as well as HTML and CSS for user interfaces. This enables the core of Cloud4all to be easily ported to a wide variety of devices and operating systems without requiring costly code rewrites and duplication. JSON, a lightweight interchange format, is used for all data in the system. The Fluid Project’s accessible Infusion Framework is used extensively to provide Cloud4all's context-aware, model-driven, and highly decoupled development infrastructure.

Design Goals

Cloud4all is positioned in the midst of a complex and rapidly-changing technological environment. Over the past several years there has been an unprecedented proliferation of different devices, platforms, and form factors available for users. The web, mobile, and hybrid apps have driven a massive shift towards synchronized, distributed computing environments and multiscreen experiences. In the midst of an environment that is highly fragmented, Cloud4all offers the possibility to weave together our digital lives seamlessly across these diverse technologies.

In this context, the goal of the Cloud4all architecture is to provide the APIs, coding practices, and frameworks that will enable an ecosystem for personalized accessibility to emerge, grow, and be sustained for the long run. The Cloud4all architecture has been designed to provide the software development tools and techniques that will support:

  • Security and privacy
  • Deployment across multiple platforms and devices
  • Cloud-level scalability
  • Extensibility and long-term growth
  • Community-based sustainability

Each of these goals and non-functional requirements, and how the Cloud4all architecture addresses them, are discussed in detail in Section 7, Architectural Considerations and Non-Functional Requirements.

As part of the larger Global Public Inclusive Infrastructure effort, the code developed in Cloud4all is intended to be usable by real people, in the real world. As a result, our code—especially the core kernel that supports the configuration and extension of the system with new functionality over time— must be robust, stable, and forward-looking. To achieve these goals, Cloud4all's architectural approach emphasizes the importance of separation of concerns, data-oriented service APIs, Inversion of Control (IoC), unit testing, and community code review. These approaches work together to ensure that the Cloud4all system can be easily extended and reconfigured to support new use cases, features, and components in the future without requiring major code changes.


Web-Oriented Architecture

Web-based technologies offer a nearly ubiquitous development platform for building cross-platform applications. For both desktop and mobile development, the trio of JavaScript, HTML, and CSS provide a highly flexible and increasingly performance-oriented technology stack that is available everywhere. Notably, the web has succeeded where other attempts at creating a cross-platform runtime failed; Java Swing and Flash, despite their "write once, run anywhere" promise, both struggled to deliver in the consumer application and mobile markets due their single-vendor control and inflexibility when faced with new device capabilities and form factors. Over the past decade, significant effort has been invested to ensure that web technologies are accessible. The declarative approach of the web, particularly the built-in separation of structure from presentation inherent in the HTML and CSS model, makes web technologies particularly amenable to realizing the goals of the Cloud4all architecture.

Web technologies, JavaScript in particular, are no longer confined to the browser. JavaScript has emerged as a popular general-purpose programming language that can run on both the client and server using tools such as Node.js. JavaScript-based code can also run on mobile devices. This web technology stack has become an integral part of many commercial and open source desktop and mobile development toolkits, including those for Windows 8 and Windows Phone, Adobe PhoneGap, Google Chrome OS, Firefox OS, and more. JavaScript performance has, over the past several years, made significant performance improvements, leapfrogging other familiar dynamically-interpreted languages such as PHP, Python, and Ruby. In many cases, it is now comparable in performance to traditional statically typed enterprise programming languages such as Java and C++. There are also millions of skilled web developers and designers around the world; this is a significant factor when considering the long-term sustainability of Cloud4all in terms of being able to continue to attracted skilled and enthusiastic developers to our community.

JSON, REST, and Repartitioning

Components in the Cloud4all architecture take input data and produce output in a simple JSON (JavaScript Object Notation) format. JSON is the lowest common denominator of interchange formats: it's widely supported and can be trivially parsed by any programming language and toolkit available today. It has a simple and lightweight syntax and maps directly to standard data structures. It is ideal for representing user preferences and application settings, and enables them to be easily transformed into other formats for interoperability.

Components in the system communicate with each other using simple Representational State Transfer (REST) conventions over the HTTP protocol. Each component is represented as a "resource" available at a unique URL. This approach specifically avoids a dependence on language or toolkit-specific messaging APIs that would prevent cross-platform compatibility. The system can easily be extended with new components and features by adding and defining new URLs, and the dependency between components is configured, not hard-coded.

Most importantly, it is trivial to move components between the local system and the web, ensuring that the system works as well in highly connected, cloud-hosted environments as it does without an Internet connection.

In the Cloud4all reference implementation, local components are hosted within a single application instance that responds only to internal HTTP requests. If required, Implementers and system administrators can redeploy a local component to a remote web server and update a plain text configuration file in order to re-partition the system according to their needs.

More information about this repartition ability of the Cloud4all architecture is described in the Deployment section.

Components of the Architecture

Overview of Components

Below, figure 1 illustrates the primary components of the Cloud4all realtime framework as well as the sequence of events that occur when a user requests that a device be configured to suit their personal needs and preferences. This sequence is described in detail below. Solid arrows represent the flow of data between components during the personalization process. Dashed arrows show non-realtime dependencies between components.


The Auto-Personalization Flow

The purpose of Cloud4all/GPII architecture is to enable the personalization of desktop, mobile, and web applications and content based on the user's needs and preferences. The system is able to automatically configure and launch the user's preferred applications (for example, a screen reader) and settings (e.g. their preferred speech rate).

Typically, the user initially gets started with Cloud4All by describing their needs and preferences to the system using an editor, wizard, or discovery aid such as the Preferences Management Tool.

Once they have created their preference set and are ready for a system to be configured for them, they can key-in to the system and the Flow Manager component will orchestrate the autopersonalization process. This involves the following steps:

  1. When the user approaches a Cloud4all-enabled device such as a computer at a library or their new smartphone, they offer a key (a unique string) to the device. The User Listener component, typically running on the local device, will detect this key and send it to the Flow Manager. User Listeners employ a variety of approaches for keying in, such as USB drives, RFID tags, QR codes, and more.
  2. Using this key, the user’s preferences are fetched from the Preferences Server. If desired, the needs and preferences set can be transformed into a particular format, structure, or ontology by the Ontology Server.
  3. With permission from the user, information about their device’s characteristics are retrieved from the Device Reporter.
  4. With the user’s permission, a connection is made to the Environment Reporter running on the local device and the Context Aware Server running remotely. These components provide live, periodic updates about the user’s physical environment using local and remote sensors.
  5. A Matchmaker is given the user’s needs and preferences set, the information about their device, and the environmental data from steps 3-5. It then fetches information about each of the applications installed on the device and available on the web from the Solutions Registry. The Matchmaker is responsible for making decisions about how to match the user’s needs with the available solutions, taking into account the user’s context and environment-specific conditions. The Matchmaker will assemble a complete list of all the features and applications that need to be configured on the device.
  6. The list of solutions and settings is passed to the Lifecycle Manager, which is responsible for invoking applications and configuring settings. The Lifecycle Manager also takes a snapshot of the system’s current settings so it can restore them later. The Lifecycle Manager, in turn, invokes a set of Lifecycle Actions and Settings Handlers that do the actual launching and setting up of solutions
  7. Once the system has been configured, the user can use the Personal Control Panel to adjust and refine their settings at any time.
GPII Personalization Activity Diagram.png

Figure 2. A UML activity diagram illustrating the basic auto-personalization procedure

A Dynamic Process

The Cloud4all system is designed as a dynamic, user-controlled process that supports realtime adaptations of a user's computing environment. The Flow Manager's event-driven architecture (described below) allows the system to respond to on-the-fly changes to the user's preference set as well as to changes in the environment and context around them.

For example, a user can use the Personal Control Panel to refine their preferences and see the results immediately. When context and environmental changes occur, the system invokes a realtime-friendly Matchmaker implementation (often referred to informally as a "mini" or "spinal", e.g. the Flat or Canopy Matchmakers, described in the Matchmakers section) that enable on-the-fly adjustments of assistive technologies and access features running both on the user's device and in the cloud.

The Matchmakers are particularly dynamic components, providing device and context-specific personalization decisions based on the particular capabilities, platform, and installed solutions available to the user. Multiple matchmaker implementations are supported within the system. If a user's preferred solution isn't available, each Matchmaker takes a unique approach to choosing alternatives. As an example, the Canopy Matchmaker dynamically determines the appropriate solutions to a user's needs by employing a fast and simple algorithm that measures fitness of a match based on weighting the proximity of solutions within a pluggable ontology (Basman, "The Matchmaker Ecology").

Cloud4all’s Adaptations are not hard-coded for a particular set of applications. The Solutions Registry enables new assistive technologies to be linked into the Cloud4all ecosystem at any time, and for new and refined transformation rules to be specified by both vendors and the larger community.

Flow Manager

The Flow Manager is the event-driven, asynchronous orchestrator of the personalization workflow. It is implemented as a very small server that runs on the local device and is responsible for bringing together the various components in the system and invoking them at the appropriate points in the process. An instance of the Flow Manager is also available in the cloud (running as a web service) for use by web applications and resource-constrained devices such as Java Mobile Edition (JME)-based feature phones.

The key responsibility of the Flow Manager is to manage the workflow of the personalization process, providing the means to register components with the system and bind them to specific events and lifecycle points within this workflow. The Flow Manager uses an Inversion of Control approach (Fowler, 2004) (provided by Fluid Infusion) to instantiate and wire up dependencies between components. This means that components rarely have a direct coupling with other components in the Cloud4all system. Instead, they are implemented as small, self-contained (and independently deployable) objects that are responsible for a single concern. Components are registered and bound to the Flow Manager via JSON-based configuration. This means that the system is trivially unit testable (since default implementations can be replaced easily with testing mocks) and, more importantly, that it can be easily extended and evolved over time. New features and components can be added to the Flow Manager without requiring code changes to the other components or the Flow Manager itself.

The Flow Manager thus provides the critical infrastructure for many of the functional and non-functional requirements of Cloud4all, particularly the requirement for modularity and extensibility by ensuring that components within the system have minimal code-level dependencies on each other and that they can be substituted for alternative implementations as features and requirements change with time.

Figure 3 presents a UML diagram illustrating the realtime framework’s sequence of method invocations during the personalization workflow described above. These operations are represented in the Flow Manager by a collection of events that fire at each point in the personalization workflow, and for which new listeners can be bound.


Figure 3: UML sequence diagram showing the order of operations in the realtime framework.

User Listeners

The User Listener is the first step in the process of customizing a device or application to fit the user's needs and preferences. The user shares their key with the listener, which is then responsible for starting the process of personalization. Since users may want to stay as anonymous as possible, this key may be a random, unique, opaque string—not necessarily a personal ID. The architecture is designed to support many different types of interchangeable User Listeners, and the rest of the system is decoupled from the particular listener employed. Listener implementations may include RFID tag readers, USB key readers, 2D barcode (QR code) readers, username/password-based login, face recognition, voice recognition, etc. It is straightforward to add new listeners to the system, since they can be integrated into the architecture with a single HTTP request to the Flow Manager. Currently, there are two implemented User Listeners:

  • RFID Reader, which reads the user token from a low-cost Near Field Communication (NFC) tag that can be embedded in almost any object (cards, pens, rings, and even many Android mobile devices are available with NFC built-in). The RFID Reader works on all currently supported Cloud4all platforms including Windows, Linux, and Android.
  • USB Drive Reader, which reads a user token from a file on any USB storage media such as a thumb drive or SD card reader. The USB Reader is available on all supported desktop platforms including Windows and Linux.

Context and Environmental Awareness

The Environment Reporter and the Context Aware Server provide two different and complementary strategies for providing realtime data about the state of the physical environment around the user. Both represent, to the Flow Manager and other components, sources of periodically-changing data.

The environment reporter and the context aware server share a common set of responsibilities:

  • Aggregating multiple sensors or sources of environmental data
  • Representing this sensor data as a normalized set of changes to the rest of the architecture (via events)

The two components each accomplish these goals in different ways: the Environment Reporter is responsible for aggregating data for sensors that are installed on the local device (such as the user’s webcam, microphone, ambient light sensor, and GPS radio), while the Context Aware Server collects together diverse signals from motes—small sensors installed in space that work independently from the device that is currently being used. When the personalization process occurs, the Flow Manager will initiate two connections in order to establish sources of environmental data:

  1. A local method call to the Environment Reporter, telling it to start providing periodic on-device sensor data if available
  2. a remote REST call to the Context Aware Server, which will open up a persistent socket connection that the CAS can use to stream periodic remote sensor data to

Whenever new data is sent from either of these sources of context, the Flow Manager will fire an onEnvironmentChanged event, passing along a normalized payload representing the change.


Figure 4: A UML sequence diagram illustrating the relationship between the Flow Manager, Context Aware Server, and Matchmaker.

The "mini-matchmaker" described in D103.1 has been implemented as a series of model transformations using the Transformer’s “data lenses.” This provides a low-cost way to handle regularly changing data without requiring the invocation of a full matchmaker. These transformations will provide a form of mediation and synchronization between the two different views of the world: the CAS and Environment Reporter's model of "the state of the external environment" and the Flow Manager's view of "the user’s current needs and preferences." An advantage to this approach is that the transformation infrastructure is already used throughout the Cloud4All architecture and is represented in a declarative format that can be produced by a Matchmaker without complex code generation techniques or inflexible hardcoding of rules.


The Matchmakers are responsible for deciding which solutions and settings will best meet the user's needs. To do this, they take into account:

  • the user's preferences
  • the applications, features, and assistive technologies available on their current device and out on the cloud
  • information about the user's device, environment, and context, supplied by the Device Reporter, the Environment Reporter, and the Context Aware Server

Invoked by the Flow Manager, the Matchmaker will return a comprehensive list of solutions and settings that have been selected specifically for the user’s needs, which is in turn passed on to the Launch Manager. This information will then be used to invoke the relevant Settings Handlers and Launch Handlers.

The Matchmaker Workflow

The Cloud4all architecture defines a common, high-level workflow for all Matchmakers. This process is managed by the Cloud4all framework. Like the rest of the architecture, it is event-based and configuration-driven, enabling each Matchmaker to easily customize the default workflow and inject custom behaviour at any phase in the process without requiring any code to be cut and pasted or duplicated. Currently, the matchmaking workflow is as follows:

  1. Fetch solutions from the Solutions Registry
  2. Filter the solutions based on:
    1. Platform
    2. Platform version
    3. Device
    4. Context and environment
  3. Match
    1. Modify, synthesize, or infer user preferences
    2. Modify the solutions list
    3. Run a custom matchmaking strategy (using preferences set and solutions list)
    4. Produce and return the appropriate list of solutions and settings for the Lifecycle Manager

Matchmaker Implementations

The Cloud4all/GPII architecture currently supports a variety of matchmaker implementations, each of which take a unique approach to matching a user’s needs and preferences with the appropriate solutions. These implementations include:

  • The Flat Matchmaker, which makes a direct match between user preferences and the declared capabilities of available solutions
  • The Canopy Matchmaker, which uses a CSS-style fitness measurement algorithm and pluggable ontology to determine the alternative solutions that will most closely match the user's needs.
  • The Statistical Matchmaker, which directly infers application-specific settings using a complex statistical algorithm outlined in depth in D204.1, Profile Matching Technical Concept
  • The Rule-Based Matchmaker, which evaluates a set of expert-defined rules for resolving conflict scenarios (such as when a preferred solution is unavailable on the current device)

The motivation for developing the technical concept of matchmaking is described in (Loitsch, et al., 2012)

Lifecycle Manager and Lifecycle Actions

Located on the user's device, the Lifecycle Manager receives the list of appropriate solutions and settings from the Matchmaker and manages the process of setting them up. This configurable process generally includes:

  1. Taking a snapshot of the device's current settings so they can be restored later
  2. Invoking the appropriate Settings Handler for each solution
  3. Launching each solution
  4. Stopping each solution when the user is done
  5. Restoring the previous settings (if required) once the user is finished

Each of these activities is accomplished by Lifecycle Actions, which are functions that can be registered with the architecture and are responsible for performing a single, specific task of the configuration workflow.

The Cloud4All real-time framework has a set of built-in Lifecycle Actions, including:

  • The System Call Handler, which executes system commands using "exec" or similar calls.
  • The GSettings handler, which can set GSettings values used in GNOME applications (GSettings is GNOME's high-level API for application settings.)
  • "setSettings," a special lifecycle handler that tells the settings handlers to set the settings
  • "restoreSettings," a special lifecycle handler that tells the settings handlers to set the settings

These actions can be sequenced arbitrarily, allowing an assistive technology developer to specify exactly the order in which an application and its dependencies should be set up and launched.

Settings Handlers

Settings Handlers are a type of Lifecycle Action that are responsible for configuring the settings of an application, access feature, or assistive technology. They are often, but not always, specific to a particular platform. Platform-specific Settings Handlers typically interface with the operating system's built in APIs for storing settings (such as the Registry on Windows and GSettings on Linux). Others are capable of configuring standard formats such as XML and JSON files.

Settings Handlers are configured declaratively (within a JSON-based Solutions Registry entry), and assistive technology developers will typically use the ones that are built in to the system. In cases where the default Settings Handlers aren't sufficient, developers can choose to write their own Settings Handlers and hook them into the GPII Personalization Framework.

Out of the box, the Cloud4all/GPII architecture ships with the following Settings Handlers:

Preferences Server

The Preferences Server stores user preferences. These preferences include general preferences ("I prefer high contrast"), context-specific preferences ("In the evening, I want the text to be 1.5x larger"), and application-specific settings ("I want X screenreader to read at Y speed").

The Preferences Server is a web-based service that provides a REST API for accessing preferences in a JSON format. Preferences are represented according to a new version of the AccessForAll ISO 24751 standard. More information about this evolving standard is available in the GPII wiki on the AccessForAll working group's discussion page and the Common Terms Registry page, and a summary is also provided below.

In the case of the GPII Preferences Server, the primary resource (in the RESTful sense) is the user and their preferences. These are identified by URLs such as:<anonymous user token>

The Preferences Server supports multiple representations of a user's preferences, corresponding to particular formats and structures that are optimized for distinct use cases and implementations. While the server does indeed offer a standardized JSON representation, consisting of the preferences mapped as flat key-value pairs that correspond to the AccessForAll preferences registry, other formats can also be provided and used as needed via the Transformer. For example, a more terse and hierarchical JSON structure suitable for the high-performance processing and interchange requirements of the GPII Personalization Framework. One can even imagine HTML representations of the user's preferences, enabling them to review the preferences they've set and edit them accordingly.


The Preferences Server is designed to store each revision of a user's preference set. This provides a foundation for implementing a number of important long-term features, including:

  • Undo support, allowing a user to roll back to a previous version of their preferences set
  • Audit trails, ensuring that users, their helpers, and the system's administrators can see a history of all the changes made to a preference set (e.g. for security and privacy reasons)
  • Comparison and "diffs" of preference set, allowing a user to easily see and compare the difference between their sets
  • Metrics gathering, enabling privacy-respecting information to be gathered about the success of the system, how often users update their settings, etc.

In order to support such versioning, each revision of a preferences JSON document is tagged with a unique identifier and stored in a document-oriented database, ensuring it can be retrieved in the future. The Preference Server also specifies a REST-based service for tracking the changes to a document over time based on the SLEEP open protocol. The real-time framework reference implementation uses Apache CouchDB to provide versioning support out of the box.

Example Preference Sets

Flat, URL-keyed Preferences Profile in JSON Format (this is the default format)

    "": [
            "value": "fr-ca"
    "": [
            "value": 100
    "": [
            "value": 0.5
    "": [
            "value": true

Hierarchical Preferences Profile in JSON Format Conforming to the Current ISO 24751 Standard

    user: {
        language: "fr-ca"

    display: {
        screenReader: {
            speechRate: 100

    control: {
        alternativeKeyboard: {
            repeatKeys: {
                repeatRate: 0.5
                automaticDelay: true

In each case where the Preferences Server exposes a particular data representation (in JSON format), there will also be an accompanying "schema" or map, describing how a data structure corresponds to the underlying AccessForAll preferences registry. This will ensure that multiple formats can coexist harmoniously; each representation will be self-describing, ensuring that implementations can easily understand the format and its meaning. Here's an example of how this schema might look:

    "user.language": "",
    "display.screenReader.speechRate": "",
    "control.alternativeKeyboard.repeatKeys.repeatRate": ""
    "control.alternativeKeyboard.repeatKeys.automaticDelay": ""

In this example, keys within the hierarchical structure (separated by dots) are mapped to their corresponding AccessForAll Registry entries via their unique URI. Any application built in a Registry-aware way would be able to understand the meaning of this preferences profile, regardless of its particular structure.

In this way, the Preferences Server takes advantage of RESTful principles and self-describing Resources to ensure that we can widely interoperate, supporting the diverse needs and requirements of implementers.

Solutions Registry

The Solutions Registry is a comprehensive source for information about applications, access features, and assistive technologies (referred to collectively as Solutions). It can be queried to determine which solutions are available to the user on their particular platform. The Solutions Reporter consists of declarative information about a solution, how it is configured, and what features it provides. This information can then be used to make a successful match with the user's needs and preferences. The Solutions Registry is where new implementers of Cloud4all will add entries that describe their software.

The Solutions Registry is held in a document-oriented database. Like all other documents in the Cloud4All reference implementation, the format is JSON. This list of solutions will include both those locally available (AT installed on computers, features built into the OS) and available via the cloud (services that the user is subscribed to or free services).

In order to register their software with the Solutions Reporter, developers need simply to create a standard JSON file called a solution description, which will contain:

  • a name, ID, and other basic information about the assistive technology
  • the name of a built-in Settings Handler that can set and retrieve the application's settings
  • a settings transformation map, which describes how to map standard AccessForAll user preferences into the assistive technology's native settings format

In the future, graphical tools can be provided so that solutions developers (or even the broader AT user community) can create new Solutions Registry entries without having to hand-author JSON files directly.

This solution metadata is used by several components of the GPII:

  • The Matchmaker, in order to know what needs and preferences are matched by solution.
  • The Lifecycle Handler, in order to know how to start and stop the application and detect whether it's running, and similar information used by other lifecycle actions.
  • The Settings Handler, for information about how to find and set the preferences
  • The Device Reporter via the Solution Registry, in order to find out what solutions are installed on the system.
  • The Transformer, for information about transformation between common terms and application specific settings

The Format of a Solution Entry

Each application will have at least one solution entry. If it behaves differently on different operating systems, versions, etc., it may have more than one solution entry.

The solution entry is a JSON object with at least the following mandatory fields:

  • name (string): The name of the application/solution in a human readable format.
  • id (string): A unique ID for the application. The general format used is the Reverse-DNS notation (e.g. "")
  • contexts (object): Describes the context required for this solution to be applicable
  • lifecycleManager (object): Instructions for the Lifecycle Manager describing what should happen when a user logs in and logs out
  • settingsHandlers (array): An array of settings handlers (and their settings) that should used to configure this solution

Device Characteristics Reporter

The Device Characteristics Reporter provides information about the user's device. This includes operating system, versions, and the list of applications currently installed. It will also provide information about the device's capabilities, such as screen resolution and size, support for features such as HTML5, and the availability of devices such as microphones and cameras, etc.

Over time, the Device Characteristics Reporter will be harmonized with evolving standards such as W3C Media Queries and the HTML5 device APIs.

Preferences Editors

Given Cloud4all's broad and diverse audience, it is clear that no single preferences editor user interface design will meet the needs of all users. There is an incredible diversity of experience levels, comfort with technology, ability, age, and other factors that influence the design of user interfaces for preference editing. We need to speak the language of our users, giving them an environment in which they feel comfortable discovering what they need and experimenting with new settings. The Cloud4all architecture needs to take into account this diversity, making it easier for designers and developers to create different user interfaces for different users.

To this end, the project's architectural approach emphasizes modularity and adaptability. Rather than creating the infrastructure for a single, hard-coded preferences editor, the goal is to provide the technical infrastructure that will allow for maximum diversity of user experiences while avoiding the proliferation of duplicated and mostly unshared code typical of ad-hoc UI development. The Cloud4all architecture follows Fluid Infusion's declarative Inversion of Control (IoC) philosophy, making it easier for third-party developers and designers to refine and extend Cloud4all's preferences editing environment over time. With Infusion IoC, instead of writing code with tightly-coupled dependencies, developers express their application structure as a declarative "component tree" that is instantiated and managed by a context-aware framework (Basman et. al. 1). This makes it possible to change the application later, either statically by third-party developers or at run time by evaluating contextual information such as device capabilities and physical sensors.

In keeping with the overall AccessForAll approach, in which we understand disability and accessibility to be usability issues which we all experience, preferences are always represented functionally. They never require the user to identify medical conditions or limitations; instead, they express functional preferences that may be useful to a wide variety of users.

The PMT and PCP Editors

The two main preferences editors that are part of the Cloud4all deliverables are the Preferences Management Tool (PMT) and the Personal Control Panel (PCP). Both will be developed with a common core framework (the UI Options Preferences Framework) and will share a consistent set of user interface controls and interaction idioms. They will, however, provide distinct user experiences that are optimized for different use cases. The PMT provides a more in-depth means to edit an entire preference set (including contexts and conditions where appropriate). The PCP, on the other hand, is optimized for quick, on-the-fly adjustment of frequently-used settings.

From an architectural perspective, both the PMT and PCP will be built using web technologies (HTML, CSS, and JavaScript) to ensure that they:

  1. can themselves be automatically personalized and transformed to suit a users preferences
  2. will be available across the myriad of platforms, operating systems, and devices we anticipate supporting in Cloud4all without costly rewrites
  3. will share a common design approach and technical foundation to many other aspects of the Cloud4all architecture, including the Flow Manager and the Preferences Server

The PMT will be accessed as a familiar in-browser web application, optimized for larger screens and longer time spans of usage. The PCP, on the other hand, will be always available on the particular device or system the user is interacting with. As a result, the PCP will be wrapped in a "native app" runtime so that, despite being cross-platform and built with web technologies, it will still work in offline contexts and idiomatically like any other app on the platform. This native wrapping will be accomplished on desktop using a Node.js-based runtime such as the Node Webkit project and on mobile with Apache Cordova (also known as "PhoneGap").

The UI Options Preference Framework

User Interface Options (UI Options) provides a programming framework with which new user interfaces for editing preferences can be more easily built. UI Options will provide all the lifecycle events, configuration hooks, and persistence infrastructure required to support a variety of different editing experiences, ensuring that there is a robust and well-tested framework in which to incrementally move the Cloud4all designs from prototype to production.

The UI Options framework is intended to give developers the ability to supply custom appearances, behaviours, and schemas for any preference, including new preferences that aren't part of the default UI Options package. This is achieved architecturally by modeling each preference as three discrete objects: a View (presentation) component, an Enactor (action) component, and a Persistence component. This three-part model ensures that developers and designers who either are using UI Options as a general framework for building a preferences editor (such as the Preferences Management Tool and Personal Control Panel) have complete control over how a preference 1) appears to the user, 2) is enacted, and 3) is stored.

Preference Components


Diagram showing the primary architectural components of Preferences for Global Access, including the UI Options framework, the GPII Preferences Server, and server side transformation components.

  1. The View component is responsible for specifying the markup, styling, and template required to render the panel.
  2. The Enactor does the actual work of accommodating the user's preference. In the case of web application, Enactors are typically registered with the UI Options framework's UI Enhancer component, which is responsible aggregating multiple Enactors together and managing the interaction between these Enactors and the page's Document Object Model (DOM).
  3. The Persistence component provides schematic information about the preference---its value types, ranges and default values. Developers can also plug in custom Data Store components, which enable the preferences editor to connect to different data stores. The primary DataStore in Cloud4all saves preferences directly into the GPII cloud-based Preferences Server via its REST API.

The UI Options framework provides a declarative JSON data structure for registering custom components, enabling easy customization of the PMT, PCP, and other editors by third-party developers and integrators without having to modify any built-in code. As a result, the job of the developer is to compose preferences together into a tree of JSON configuration that is passed to the UI Options framework at instantiation time.

By separating the responsibilities of view and action, this architectural approach will enable developers to more rapidly create diverse preference editing experiences. For example, a designer may want to provide both a highly simplified user interface that is more fun and approachable for children as well as an in-depth experience for more technical users. In this case, the same Enactor code can be reused while different Views can be written and swapped in without having to change any core application code.

The UI Options framework also provides support for pluggable previews, enabling a user to see the effect that a preference change will have before committing to it. Developers can provide their own custom preview strategies. The default "live preview" strategy shows the changes applied to a live view of the application, allowing users to see their preferences in context. Another built-in strategy in include a "sample window" preview showing a small selection of typical content. We also envision specialized previews where the user can try out the preferences in practice in a safe environment before committing to them globally.

The Transformer and Data Lensing

The Cloud4all architecture takes a powerful and forward-looking approach to data modelling and interoperability. Our goal is to enable developers to represent data (such as user preferences) in whatever formats and structures best fit their needs and implementation constraints. Rather than trying to impose yet another competing, overlapping, or conflicting "grand unifying standard ontology" that further fragments the development ecosystem, our goal is to embrace the "one size fits one" philosophy in our architecture, too. The Transformer represents a system for smoothly transferring information back and forth between different perspectives on data formats, structures, and ontologies.

This "minimising the costs of disagreement" approach emphasizes the inherent mutability of simple data structures (such as those represented in JSON), allowing the Cloud4all real-time framework to easily adapt and transform data structures as needed in order to interoperate with others. Using as its basis the Model Transformation library that was developed for Fluid's Infusion application framework implementers will have tools available to them that enable the use their preferred formats and structures while still retaining interoperability and compatibility with others.

This approach has been inspired by the work of Benjamin Pierce and his Boomerang language, from which we have borrowed the metaphor of lensing to describe the bi-directional process of transforming data structures back and forth between different formats.

The Transformer is an interoperability service provided by the Cloud4All architecture to enable such transformations. For example, the Transformer is currently used by most Matchmakers to translate preferences between common terms and application specific settings. It is stateless, relatively easy to extend, and has an easy-to-use API. The transformation rules are structured in a declarative, transparent format. Transformation rules are themselves JSON documents. This approach makes it easier to:

  • understand programmatically how a transformation can be reversed
  • produce tools for authoring and visualizing transformation rules
  • store, retrieve and parse in database technologies
  • “lift” a transformation of documents into the corresponding transformation of schemata

When transforming between common terms and application-specific settings, rules are defined by a developer (or user) in the description of a solution in the Solution Registry. They describe how an application-specific setting should be understood and transformed into a common term and vice versa.

Examples of Transformation

As mentioned above, transformation rules for solution settings are declared along with the solutions in the settingsHandler section of the Solution Registry.

<syntaxhighlight lang="javascript">

                   "Magnification": {
                       "transform": {
                           "type": "gpii.transformer.scaleValue",
                           "inputPath": "display.screenEnhancement.magnification",
                           "outputPath": "value",
                           "factor": 100


This example rule describes a transformation between an AccessForAll common term and a Windows Magnifier-specific setting. Here, a proper transformation will take place; the common term represents the range of the "magnification" setting differently than the Windows Magnifier does. As a result, the value of the common term (listed here as "display.screenEnhancement.magnification") must be scaled by 100 before being assigned to the "Magnification" application setting.

Ontologies and the Ontology Server

The ease of producing JSON schematic structures represented in JSON itself makes it economic and desirable to produce a dedicated piece of architecture for managing this process. Each of the JSON formats in which user preferences documents can be served can be thought of as being derived from an ontology - a schema describing the layout and meaning of the terms appearing in the format. Use of the model transformation and data lensing features referred to above makes it easy to provide an open infrastructure for ontologies - users can define and register new ontologies without writing programming language code.

The two formats described above in the Preferences Server section—the standard "Flat, URL-keyed Preferences Profile" as well as the "Format Conforming to the Current ISO 24751 Standard"—both constitute ontologies in our terminology, even though the first format conveys no ontological information beyond the names of keys. Further ontologies may easily be added to the system by integrators, AT vendors and other stakeholders simply by defining a transformation document in the format accepted by the model transformation system, which translates their ontology into one of the two standard ones. As long as there exists a path linked by transformations between any one ontology and another, the system can broker between preferences stored in the two formats automatically, ultimately for writing as well as for reading. Here is a sample document defining a transformation between the standard registry format and a part of the ISO 24751-2 schema (the hierarchy under "display.screenEnhancement") described above:

    "transformations": {
        "display.screenEnhancement.fontSize": "{}.0.value",
        "display.screenEnhancement.foregroundColor": "{}.0.value",
        "display.screenEnhancement.backgroundColor": "{}.0.value",
        "display.screenEnhancement.fontFaceFontName": "{}.0.value",
        "display.screenEnhancement.fontFaceGenericFontFace": "{}.0.value",
        "display.screenEnhancement.magnification": "{}.0.value",
        "display.screenEnhancement.tracking": "{}.0.value",
        "display.screenEnhancement.invertImages": "{}.0.value",
        "": {
            "transform": {
                "type": "gpii.ontologyServer.transform.application",
                "inputPath": "",
                "outputPath": "applications"

Note that this transformation format is not final and will evolve over the next few releases of the Cloud4all real-time framework, in particular to eliminate the indirection ".0.value" appearing on the right hand side of rules.

The Cloud4all Architecture Reference Implementation


Each component in the GPII Personalization Framework reference implementation is developed using a standard cross-platform technology stack. Components are written in JavaScript and hosted within the Node.js runtime environment. Node.js is a highly scalable, event-driven system that can run on all major desktop platforms (Mac, Windows, and Linux) as well as mobile platforms such as Android. It is commonly used as a server-side web development toolkit, but can be used to create any kind of local or connected application. It features prominently within the development tools offered on desktop and mobile platforms such as Windows 8.

Due to this design, most components within the real-time framework have the ability to run across platforms without recoding.

We've designed the real-time framework specifically to be production-ready, cross-platform, and reliable. It has extensive unit tests written for it already; automated integration tests being developed, too. These will have a significant impact on lowering the costs associated with maintaining the Cloud4all technology. The Node.js-based real-time framework represents one code base that the GPII community can invest in, support, and sustain for the long run across a variety of the platforms we support.

Integration with Native Technologies

Although the Cloud4all real-time framework extensively uses JavaScript as the primary development language, it is also possible to integrate with native technologies such as operating system settings and device APIs. For example, Settings Handlers (described above), which are responsible for configuring the settings of assistive technologies, are often written in C, C++, or Java in order to bind to platform APIs such as the Windows Registry or Linux GNOME's GSettings. In cases where native bindings are required, the architecture provides several means to integrate them:

  • On Windows, Linux, and Android, native code can be called by creating Node.js modules in either C++ or Java that bridge between platform code and JavaScript
  • The Cloud4all real-time framework also ships with node-ffi, a Node.js module for calling "foreign" (i.e. C-based) functions based on the widely-used libffi library. The use of node-ffi enables Cloud4all developers to bind JavaScript code to C-based APIs without having to write any actual C code.
  • Non-JavaScript code can, in many cases, pass data to and from the Flow Manager by invoking its RESTful service endpoints and exchanging payloads in JSON. The USB User Listener, for example, which is responsible for invoking the Flow Manager whenever a user plugs in their GPII-enabled USB drive, communicates with the Flow Manager via REST.

The end goal is to ensure as much code as possible is rewritten in a cross-platform and highly reusable manner. However, whenever platform-specific native code is required, the architecture enables this while ensuring that it is carefully localized to avoid difficulties when unit testing or porting the code to new platforms.


With the exception of the components that are designed to bind with platform-specific APIs, any component in the system can be located either locally on the device or out on the web. For example, some Settings Handlers and User Listeners are bound to the local device, though others are cross-platform. The Flow Manager, Preferences Server, Matchmaker, and Solutions Registry can easily move between locations. It's simply a matter of updating a configuration file for the Flow Manager.

Default Deployment Layout


This is the default deployment layout of the Cloud4all/GPII architecture, but does not represent all of the possible places that any component could reside. For example:

  • Preferences can be on the server, or a USB drive, or in a phone, or in an NFC ring, or elsewhere
  • Matchmaking can be done locally, on the server, or distributed across both locations
  • Content transformation and transcoding can be done on the server, on the local device, or distributed


  • Preferences Server
  • Solutions Registry
  • Matchmaker-specific server-side components (e.g. the Statistical Matchmaker's stats processing engine)

Cross-platform capable, but installed locally

  • Flow Manager
  • Device Reporter
  • Matchmaker
  • Transformer
  • Lifecycle Manager
  • XML, INI and JSON File Settings Handlers

Platform-specific and installed locally

  • USB Drive User Listener (Windows and Linux implementations)
  • Registry SettingsHandler (Windows)
  • SystemParametersInfo SettingsHandler (Windows)
  • GSettings Settings Handler (Linux GNOME)
  • GSettings Lifecycle Handler (Linux GNOME)
  • D-Bus Settings Handler (Linux)

It is important to recognize that each component in the system doesn't have be its own server. By default, we have two server instances:

  1. one that runs on the local device and is fully firewalled and secure
  2. a second server that contains both the cloud-based Preferences Server the Solutions Registry

When components are co-located in the same application, they are able to communicate via direct function calls (without the overhead of REST). This, too, is a configurable aspect of the Flow Manager.

We expect mobile devices can be supported on most platforms using the same approach. iOS is always a more complex issue due to Apple's very restrictive terms, but a comparable implementation using Apple's own JavaScript runtimes should be possible to implement.

Platform-Specific Support


As mentioned in the Integration with Native Technologies section above, the Cloud4all architecture takes a hybrid approach to delivering cross-platform applications. The core "kernel" of the real-time framework (the Flow Manager, Matchmaker framework, Solutions Registry, Transformer, and Lifecycle Manager in particular) are written in JavaScript and Node.js, thus enabling them to be deployed on nearly any operating system or device. However, many "leaf" components of the system (the Setting Handlers and Lifecycle Actions, for example) are written in the programming language and toolkits that are idiomatic for a particular platform. This hybrid approach substantially lowers the cost of supporting multiple platforms while ensuring that we can take advantage of the unique features of each platform.

On modern smartphone platforms such as Android, the real-time framework will be deployed directly on the device using ANode, a customized distribution of Node.js. ANode provides a bridge between JavaScript-based Node.js applications and native Android code written in Java, including the built-in Android frameworks. Native Settings Handlers and Lifecycle Actions will be implemented to connect with Android's system-level settings and launch APIs as well as to third-party assistive technologies.

On older-style "feature phones" that run Java, a native application will be built that is responsible for the Settings Handler-related functionality in the architecture. It will delegate to a remote Flow Manager and Matchmaker via REST to do the work of determining which settings to adjust on the local device, comparable to the approach for web applications described below.

Web Applications

Web applications will be supported by a cloud-based instance of the Flow Manager and Matchmakers. This means that web applications can be integrated into the Cloud4all architecture in an analogous way to a native application. Web app developers will create a Solutions Registry entry that describes the personalization capabilities of their software, and the rules by which a user's preferences can be transformed into site-specific settings.

However, unlike locally installed Solutions, web applications will also be granted a "consumer" or "application" key (via OAuth 2.0) that uniquely identifies an application whenever it requests settings from the web-based Flow Manager. This helps ensure that only vetted and user-sanctioned web sites will be able to access their preferences, and that they will always receive only the portions of a preference set that they can actually act upon.

In addition, developers will be able to use a set pre-built components and Enactors (described above in the Preferences Editors section), which they can plug into their application to reduce the cost of adding autopersonalization features.


Figure x.y, showing the partitioning of responsibility in a web application scenario. Here, many of the same components from the local personalization workflow are reused but made available via a cloud-based server.

Bridging the Web and Local Devices

With the Web-based Flow Manager, the Cloud4all system is able to configure purely web-based applications. However, there are several cases where communication between a web application and the natively-installed Cloud4all application may be necessary, such as:

  • Preferences editors and discovery aids that are developed as standard web applications (e.g. the PMT) may need to be able to notify a local instance of the Flow Manager when a user has updated their preferences
  • The Flow Manager may need to coordinate between locally installed assistive technology and equivalent cloud- and web-based assistive technology (e.g. a web-based screen reader such as WebAnywhere) to ensure that there aren't conflicts
  • A user may want to remain anonymous, and thus not want to make an association between their online identity (e.g. their Facebook account) and their GPII user token; as a result, the local Flow Manager may need to pass a portion of the preference set or a temporary token into the web browser.

In order to address these cases, we are exploring several potential avenues for orchestrating web-based and local personalization, including:

  • Platform push notifications, enabling periodic, large-scale messaging between instances of the Flow Manager running both in the cloud and on the device
  • A browser extension (e.g. a Firefox Jetpack) that will allow minimal, secure communication between web applications and the local Flow Manager

Architectural Considerations & Non-Functional Requirements

Security and Privacy

A user's preferences may reveal personal information about their lives and needs, either directly or indirectly. Privacy and security are thus of critical importance to the design of the Cloud4all architecture. While we are still in the process of designing the Cloud4all security and privacy infrastructure as part of Work Package 104.2, the intention is to use tools and standards that are broadly used on the web and in industry, such as the OAuth 2.0 framework. OAuth will help ensure that the Security Gateway and core architecture will protect access to a user's preferences by third party applications such as the web sites, content delivery tools, and other appilcations (Hardt 2012). The use of OAuth will let users approve or deny access to their information on a per-site basis, ensuring they control who gets to see and act on their needs and preferences.

After the basic OAuth 2.0-based authentication infrastructure is established, an attribute release layer such as the Kantara Initiative's User Managed Access will be layered on top (Machulak 2010). This will give users the further ability to specify that a site or application is able to see only portions of their preferences set, ensuring that the risk of "privacy leakage" is reduced by only sharing the minimum information required by a service to meet the user's needs (Krishnamurthy 2009).

The architecture's security and privacy infrastructure will be developed as a set of low-level libraries that can be integrated directly into the real-time framework's Node.js-based web services, ensuring that all components within the system can be protected appropriately and in a performance-optimized manner.

A key goal of this effort is to ensure that this architecture can support diverse privacy and security requirements internationally.

For other information about Cloud4all's security and privacy infrastructure goals, see available in Cloud4all Deliverable 104.2, Security, privacy and ethical policy assurance system requirements and design.

Performance, Scalability, and Cloud Technology

Performance and scalability are critical concerns of the Cloud4all architecture. The Cloud4all architecture and real-time framework have been designed specifically to be deployed on cloud-based infrastructure such as OpenStack, Amazon EC2, and more.

In particular, key components of the Cloud4all architecture have been designed to avoid heavy server-side session state, which is a common barrier to cloud-scale horizontal load balancing. As a result of this statelessness, many instances of the Cloud4all services can be "spun up" and run cooperatively with a load balancer such as HAProxy in order to spread high load more evenly across multiple servers and cores. Traditional state-heavy models such as Enterprise Java Beans-based architectures have struggled to scale in this manner, whereas Cloud4all's real-time framework makes use of non-blocking, event-based JavaScript tools such as Node.js that are specifically designed for this kind of scaling.

The use of document-oriented databases and map/reduce querying techniques, originally popularized by Google for their large-scale search infrastructure, also contributes to Cloud4all's potential scalability (Dean and Ghemawat). The real-time framework extensively uses Apache's CouchDB database, which enables master-to-master replication over a large cluster of database instances and provides several options for cloud-based sharding of the database. CouchDB has been used in a number of cloud-scale production deployments in industry.

In order to ensure that the real-time framework is "cloud native," we have continually deployed and tested it on a large OpenStack-based cloud cluster (currently comprising 35 compute nodes powered by three Nebula cloud controllers running at the IDRC in Toronto). Work is underway to create automated benchmarking and load testing scripts to ensure that scalability is an integral metric to our architectural and development process. Automated deployment recipes and virtual machine profiles are also being created in order to streamline the process of spinning up new Cloud4all virtual machine instances. These recipes and tools will ultimately ship with the Cloud4all real-time framework as a toolkit for administrators and production engineers who will deploy Cloud4all internationally.

Modularity and Long-Term Growth

This architecture particularly emphasizes the importance of modularity in ensuring the long-term growth and extensibility of Cloud4all's technical platform. While it might be possible to implement similar features to Cloud4all using a "simpler" or more tightly coupled architectural approach, such a tradeoff would significantly inhibit the ability for developers to continue to productively extend and maintain the system over the long run. Change is inevitable in the cross-platform and ubiquitously connected world that typifies cloud computing; in this context, a system's long-term viability and resilience is largely determined by its ability to adapt to change in features, technology, and user needs without requiring costly code rewites over time. This characteristic needs to be actively designed into a system from the start.

In order to promote long-term sustainability, the Cloud4all real-time framework ensures that a separation of concerns is rigorously maintained across each component, making it viable to swap out existing implementations for improved versions as technology changes. Similarly, the Flow Manager's open architecture, Inversion of Control container, and event-driven API ensures that the Cloud4all system will be resilient to long-term change by allowing new, as-yet-unimagined features, components, and logic to be added without modifying existing ones. Cloud4all's data-oriented API design philosophy helps promote long-term viability in the face of technological change by emphasizing transparent, declarative data structures that can be parsed and used across a wide variety of environments. Unlike imperative, code-oriented techniques, Cloud4all's declarative approach can more easily be adapted and optimized to suit the radically different requirements of the massively parallel, multicore, and GPU-driven future.


Open source practices are an integral part of the design and development of Cloud4all's architecture and real-time framework. Fostering a community is a significant strategy to support the long-term growth, viability, and sustainability of Cloud4all's technology throughout and beyond the initial grant funding period. By promoting open governance practices and an inclusive meritocracy, contributors and users will be able to influence and support the direction and ongoing progress of Cloud4all's technical infrastructure.


This document has provided an overview of the goals and approach of the Cloud4all architecture, as well as in-depth coverage of the components, APIs, and data formats that it offers to developers. This architecture represents a scalable, modular, and forward-looking technical foundation for the Cloud4all project as well as the greater Global Public Inclusive Infrastructure. It is an open source and community-driven effort infused with production-quality software development techniques at its core, helping to ensure that it will remain sustainable and viable for the long term. Design and development of the architecture is an ongoing activity, performed in close coordination with technical and functional teams spanning the whole project, and culminating in regular releases of the real-time framework. Progress on the Cloud4all architecture can be followed in these open community forums: