Preferences Framework Overview
- 1 Summary
- 2 Approach
- 3 The Landscape for Developing Preference Tools
- 4 How Developers Build Preferences Editors Using the Framework
- 5 Native vs. Web Based
- 6 Platform Support
- 7 Privacy and security
Based on our experience researching and designing preference editing tools over the past decade, it is clear that no single design will meet the needs of all users. There is an incredible diversity of experience levels, comfort with technology, ability, age, and other factors that influence the design of user interfaces for preference discovery. We need to speak the language of users, giving them an environment in which they feel comfortable discovering what they need and experimenting with new settings. The GPII software architecture takes into account this diversity, making it easier for designers and developers to create different user interfaces for different users.
This architecture is designed to deliver personalized user interfaces, adapted web content, and native accessibility solutions that meet an individual user’s needs in a reliable and predictable way across all their devices and operating contexts. It also helps support a variety of privacy and security techniques required to protect personal information. And finally, it offers a rich set of technical “building blocks” and tools in the form of a framework that enables the rapid development and testing of different types of preference gathering and editing tools.
Given this diversity of preference editing tools, a primary concern of the architecture is to ensure that there is technical consistency and reusability across all of them. There is no reason to incur the costs of redundant or duplicative efforts—as well as the potential confusion for users—that typically result from ad-hoc software development. In particular, developers of preferences editing tools need to be able to pool their efforts and share a collection of:
- flexible user interface components for editing and exploring preferences (Views and Adjusters)
- pluggable personalization services that do the work of transformation and delivery (Enactors)
- tools that can configure web-based and native platform accessibility features and third-party assistive technologies (Lifecycle Actions and Settings Handlers)
- user interface components that allow a user to manage, activate, and share their preference sets for different contexts, devices, and environment (Context and Management Views)
- robust and scalable programming idioms and techniques (the Framework)
In addition to simplifying the process of developing personalizable applications, this approach will also help provide users with a stable, comprehensible, and consistent user experience across tools and applications.
To this end, the Preferences for Global Access Project, in tandem with the international efforts of the Cloud4all and Floe projects, have contributed to the specification of the GPII Preferences Framework. This framework offers a reusable set of schemas, programming APIs, and UI building blocks that are being employed in the development of several preference tools in the ecosystem, including the PGA Exploration Tool prototype and the Cloud4all Preferences Management Tool (PMT) and Personal Control Panel (PCP).
The Global Public Inclusive Infrastructure initiative is establishing a robust and comprehensive architecture designed to support the personalization of desktop, mobile, and web applications. The preferences framework architecture builds on this foundation, providing tools specific to the creation, persistence, and integration of preference editors into a variety of web-based applications, content management systems, and delivery environments. It does not prescribe a single means by which content should delivered to users. Instead, the Preferences Framework architecture provides an event-driven API that enables different, pluggable personalizations strategies (called Enactors) to listen for changes in a user's preferences and respond accordingly.
A key goal of the GPII architectural approach is to help developers create applications that are designed from the beginning to be personalizable, rather than trying to retrofit or transcode inaccessible applications and content after the fact.
The Problem of Transcoding
Many members of the GPII community have extensive experience in delivering adaptive user interfaces designed to meet the needs of individual users. We have found that a reliance on transcoding, where arbitrary content is converted to match a user’s needs, is error-prone and complex. Many factors contribute to the complexity of after-the-fact transcoding, including:
- lack of semantics at the presentation layer
- poorly-formed content
- transcoding over secure connections
- Digital Rights Management (DRM) restrictions
For example, the IBM Web Accessibility Gateway was developed in the early 2000s to provide alternative interfaces for seniors. It relied heavily on proxy-based transcoding techniques. The most serious problem with this approach was that it had to perform HTML markup correction on poorly-formed content before a user’s adaptations could be applied. This correction process became a significant performance bottleneck when the server supported many simultaneous users. This overhead is predicated by the fact in the early browser wars, browsers like Internet Explorer, would perform extensive, non-standard, and undocumented error correction. This resulted in developers getting careless and it forced browsers to have to reproduce other browsers error correction from scratch.
Building Personalization in at the Beginning
In contrast to the pitfalls of after-the-fact transcoding, the GPII architecture recognizes the importance of building applications from the beginning to be personalization-aware. This requires a different approach to software development. Instead of hardcoding a single user interface, personalizable applications need to support flexible connections between modular user interfaces and underlying services.
When applications are composed from modular pieces and the connections between these modules are flexible, it is significantly easier to:
- deliver a transformed or augmented version of the user interface
- connect up an entirely different user interface
- reuse portions of an application in different contexts, including those that are situational (e.g. detection of context changes such as ambient noise and lighting, etc.)
- extend or alter the features of an application
Reusable Components and Services
In addition to the overall framework for supporting flexible connections between user interfaces and services, developers also need a pool of reusable, modular components and services that help to simplify the process of building personalizable applications. These components, called Enactors, do the work of providing alternative presentations based on a user’s needs and preferences set. Several web-based Enactor components have been developed as part of the PGA prototype development effort and complementary efforts from the Fluid and Cloud4all projects, including:
- text to speech
- high contrast
- larger text, line spacing, and inputs
- synchronized captions and transcripts
- table of contents
- layout simplification
Based on the valuable research gleaned from this project, we have identified the need for many other Enactors that haven’t yet been developed, a few of which include:
- highlight and speak
- dictionary and thesaurus
- word simplification
- grammar check
- spell check
- text simplification
- alarms, reminders, and notifications
- math rendering, navigation, and verbalization
- group formation and communication on a task
In this model, each developer doesn’t have to write their own adaptable interfaces and services from scratch. Companies, libraries, schools, and third-party application developers can reuse and customize the components available in the Preferences Framework as well as create their own novel tools.
This pool of reusable Enactors components, along with the Preference Framework API with which others can create new Enactors, serves several interrelated goals. First, it reduces the cost, complexity, and specialized knowledge required to build personalizable applications; and second, it helps foster an open ecosystem of alternative user interfaces and personalization delivery strategies that are tailored to different contexts and applications.
The Landscape for Developing Preference Tools
The Preferences Framework is intended to provide the critical infrastructure needed by developers of preferences editing and discovery tools. Its architecture has been informed by the practical, day-to-day development issues that have arisen during this project’s prototyping efforts as well as those of Cloud4all’s in-progress Personal Control Panel and Preferences Management Tool. For example, developers of these tools found that they were writing overlapping and redundant code for managing preference sets, dynamically presenting user interfaces tailored to each user’s preferences, and handling common tasks like internationalization and skinning.
Rather than wasting time and effort developing such features over again in each new tool, the Preferences Framework offers robust abstractions that assist in the creation of flexible editing user interfaces, personalized delivery, and interaction with the larger GPII infrastructure.
The open source code for the Preferences Framework will be released in beta form this fall, and will continue to grow and evolve to support the more advanced functionality envisioned by this project’s research and design activities. Development of the PGA Exploration Tool is proceeding alongside the Cloud4all preference editors in a shared GPII code repository on Github. This collaboration is an excellent illustration of how PGA’s research insights and prototyping activities have helped to foster collaboration on personalization tools across communities and projects.
What the Preferences Framework Offers
Today, the Preferences Framework offers components and APIs that address several key development activities common to all preference editors, including:
- saving, retrieving, removing, and versioning a user’s preferences with the GPII Preferences server and to the local device
- dynamically generating user interface controls that allow users to edit and refine individual and clustered preferences
- enacting the user’s needs and preferences by transforming web content and user interfaces, injecting enhancements and scaffolds into an application, and configuring and launching web, cloud, and native access features and services
Upcoming versions of the framework will add support for several other key features such as:
- managing multiple preference sets and contexts
- support for undoing changes to preference sets
- sharing preferences with individuals as well as group collaboration
- inferring preferences and settings based on activities and metrics
- deeper integration with native platform services and applications
- many new enactors, adjustors, and supporting services
The design of each of these aspects of the architecture is discussed briefly below.
Figure 1: The components of the Preferences Framework. This diagram illustrates the components that are offered within the Preferences Framework, as well as some of the services and tools they interact within in the broader GPII ecosystem. The Preferences Framework offers: 1) Views and Adjusters, which provide reusable widgets and controls with which users can edit their preferences; 2) Enactors, which do the work of transforming the user interfaces based on these preferences; 3) Data Stores, which provide persistence for the user’s preferences; and 4) Recognizers, which provide activities to help infer new preferences and settings. These components typically interact with both web content and a suite of native/server-side GPII transformation services.
All preference editing tools need to be able to store, retrieve, and keep track of revisions to a user’s preference sets. The Data Store abstraction has been designed to provide a single programming interface for different persistence strategies, including saving to the GPII Preferences Server and to local data storage on the device.
Views and Adjusters
Users often need to refine and adjust their preferences. Views provide a generalized abstraction for rendering editable interfaces that can be bound to particular needs and preferences. The Preferences Framework provides support for both individual and clustered preferences, ensuring that designers can tailor the user interface to the context and user.
Adjusters provide fine-grained control over individual preferences. An effort is currently underway in the Cloud4all project to create a common pool of Adjusters that can be used in both the PCP and the PMT.
Enactors, as discussed above, do the work of transforming the user interface and content to suit a user’s needs and preferences. They typically interact with web, cloud, or native services such as the GPII Flow Manager and Text to Speech Server. Enactors typically require a deep integration with a web application or delivery system, and as a result there will be many different types of these components. Indeed, it is possible to consider the entire GPII native Flow Manager and its Settings Handlers as a collection of Enactors tailored to Linux, Windows, and Android devices.
Recognizers and the Proposed Inference Architecture
A significant theme of the PGA research has been the role of for inferring and suggesting user preferences within the discovery and exploration process. As mentioned above, inference is often best done in the context of activities and games that the user participates in. This approach has the potential to streamline and enhance the user’s experience, though it offering its own set of complexities and limitations.
Given this, there is significant value in providing an extensible and reusable architecture for supporting inference. Such an architecture has been recently designed, though this design has not yet been implemented in the Preferences Framework. Further design and development of this inference architecture is a significant opportunity for future research and innovation.
Recognizers are an architectural abstraction that encapsulate several different aspects of the inference process, including:
- the sources of input and activities that a user engages in
- different metrics gathering and analysis strategies
- inference strategies, which translate metrics into a suggested preferences set
It is envisioned that developers will be able to create and share new types of Recognizers by composing together different Input Sources (also referred to as Activities), Metrics Analyzers, and Inferrers.
Figure 2: The anatomy of a Recognizer. This diagram illustrates the event and model-driven relationships between parts of a Recognizer. Activities are a type of View component that provide a source of user input events. MetricAnalyzers use these events to produce a JSON-based model of the behaviour of the user during the activity, and Inferrers use this raw data to produce suggested preferences for the user. This architecture enables different approaches to metrics gathering and inference to be plugged into reusable activity UIs.
In order to clarify how these components interact, we’ll consider one of the inference examples described in Section 4:
"Given a demonstration of an individual’s typing, their keystroke timing and errors can suggest changes to the keyboard response settings that would improve accuracy. If an individual presses the same key twice in quick succession, the timing of these key presses gives a strong indication of whether the second press was deliberate or accidental (Trewin, 2004). From this it is possible to infer whether the BounceKeys feature would help to eliminate unwanted extra characters."
Activities and Input Sources
An Activity some kind of interaction presented to the user as a source for inference. In practice, this is often an exercise or game. In this example, it consists of a typing exercise designed to measure the user’s keyboard accuracy. Activities are typically composed of a user interface (a View, in architectural terminology) and one or more Input Sources, which connect with an input device of some kind (e.g. keyboard, mouse, etc) . The result of the user’s interaction is exposed to the system in real time as a stream of events. These events usually represent higher-level concepts such as typing errors, goals or achievements, etc. They are connected to low-level input events from the browser or native hardware.
Metrics Analyzers do the work of listening for one or more user events and producing a data model consisting of metrics about the user’s interaction with an Activity. These metrics are stored in a JSON object (a Model, in architectural terminology) that can be later processed and transformed by the rest of the inference pipeline.
Typically, a Recognizer will employ multiple metrics gathering strategies. In our example, the “successive keystroke” analyzer will listen for keyboard event and keep track of the time between key presses. If keyboard events occur within quick succession and close proximity, the analyzer identifies this as a likely error and updates the model accordingly. Metrics Analyzers are also responsible for tracking and drawing metrics from temporal characteristics of the user’s activity. As they listen for events they may choose to record data about how long it takes for the user to accomplish a particular activity or, as mentioned in the example above, the timing between particular events.
Inferrers are responsible for translating an activity model into a list of suggested preferences. In our example, the typing inferrer will suggest values for “bounce keys” and “acceptance delay” based on metrics such as the number of successive keystrokes and their timing. These preferences represent an “inferred set,” which can then be shown the user, who can confirm whether or not they want to include these inferred preferences in one of their saved sets. Inferrers typically operate on multiple metrics.
How Developers Build Preferences Editors Using the Framework
Rather than having to write a preferences editor from scratch, the Preferences Framework provides a means to generate the user interface and persistence layers from a set of declarative (i.e. JSON-based) schemas that are assembled by a Builder into an editor that can be instantiated and presented to the user. The advantage of this declarative approach is that it is extremely easy for developers to borrow and adapt portions of an existing preferences editor and extend them according to their own requirements, increasing the opportunity for sharing and code reuse.
Preferences editors are constructed by combining three main parts:
- Panels, which contain Views and Adjusters for users to edit their preferences with
- a Data Store that provides persistence for the user's selections
- Enactors that respond to a user's settings
The Preferences Framework provides all the lifecycle events, configuration hooks, and persistence infrastructure required to support the aggregation of these pieces, as well as providing some pre-existing panels and enactors that can be reused if desired.
The starting point for working with the Preferences Framework is the Builder, a tool that creates all the necessary components (the settings store, panels and enactors) based on a set of schema files. Schemas define:
- the settings available to be edited
- the panels used to configure the settings
- the enactors that respond to the settings
The Primary Schema
The Primary Schema defines each of the settings that a preference editor will allow the user to adjust. This includes:
- their type (e.g. boolean, string, number)
- their default values
- ranges, enumerations, and any other information necessary to define them, depending on their type
In future versions of the Preferences Framework, primary schemas will typically be generated by exporting information from the ISO 24751 AccessForAll Common Terms Registry, which provides an open forum for defining standardized preferences and settings.
Panels and the Auxiliary Schema
Along with the panel definitions, an auxiliary schema binds together the panels and enactors to be used for each of the preferences defined in the primary schema. It’s how the all of the component pieces of a preferences get woven together to deliver a seamless user experience.
The Preferences Framework also includes a number of CSS stylesheets that are used to control the layout and appearance of the interface. These stylesheets can be used as the basis for flexible styling, and can also be overridden and extended as necessary.
A tutorial that teaches developers how to create a preferences editor is currently being written.
Native vs. Web Based
The GPII architecture takes a “web-first” approach, using standard web technologies wherever possible. This helps to ensure the broadest possible compatibility with different platforms and devices. It also lowers the cost of developing user interfaces and infrastructure by reducing the amount of platform-specific code that must be written. The motivations for this approach are described in further detail in the GPII wiki.
Over the past several years, the capabilities of the web platform have expanded substantially. Features that were previously only possible in native applications are now available to web applications, such as advanced graphics, audio, camera support, touch and device orientation input, text to speech, and more. Indeed, new operating systems such as Firefox OS, Windows 8, and Chrome OS support full-featured application development entirely with web technologies. As a result, the web platform is a potentially significant source of new assistive technology development in the coming years. Traditional “monolithic” AT such as screen readers (e.g. WebAnywhere and ChromeVox) are already available. More importantly, the web is enabling the creation of more modular assistive technology “bits” (such as the Enactors that accompany the Preferences Framework) that can be assembled into novel applications and configurations.
Nonetheless, not all user preferences can be supported in a purely web-based environment, including some of those in the Preferences Matrix documented in PGA deliverable 1. This is due to the fact that web browsers implement a strict “sandbox” security policy. A web application cannot modify the settings stored on a user’s computer. This policy helps to protect users from having their system changed by malicious or poorly-coded web sites. As a result, some features need to be configured and activated at the operating system level. For example:
- enabling system-wide high contrast mode
- launching a dedicated desktop-based screen reader
- manipulating the keyboard response rate
- interfacing directly with hardware such as switches or eye trackers
The GPII Flow Manager, which is the component responsible for configuring the settings and assistive technology on a local device, includes a collection of pluggable, operating system-specific settings handlers and lifecycle actions that are able to configure and activate settings such as the ones listed above. To do so, however, the Flow Manager must be installed directly on the user’s desktop computer or mobile device so that it can interact with the operating system and native applications.
This lack of a built-in way for a web application to communicate directly with the Flow Manager presents a challenge for preferences editing and discovery tools. These web-based tools typically need to provide users with the ability to try out and preview the effects of a setting before saving them. In order to accomplish this for all preferences, a communications channel is required between the web browser and the locally-installed Flow Manager.
In the past, browser plugins have been used to bridge the gap between the web platform and the native operating system. In recent years, however, plugin usage has decreased, perhaps due to a combination of growth in the capabilities of the web platform along with a tendency for poorly-written plugins to cause incompatibilities and security issues. Another disadvantage of the plugin-based approach is that each browser vendor has a different plugin architecture and API; consequently, separate plugins must be developed for each browser. As a result, our goal is to avoid browser plugin technology wherever possible.
A recent feature of modern web browsers is HTML5 Web Sockets. Web Sockets are designed to allow a web application to communicate with a server using sockets. Notably, Web Sockets allow for “messages to be passed back and forth while keeping the connection open. In this way a two-way (bi-directional) ongoing conversation can take place between a browser and the server” (Wikipedia). This API is now included in today’s browsers.
In keeping with the GPII’s web-oriented approach, the Flow Manager has already been implemented as a web server, even when it is installed locally on the user’s system. For security reasons, the Flow Manager server is typically kept behind a firewall so that it only accepts local requests. This nonetheless allows a preference tool running in the user’s browser to use Web Sockets to communicate with the Flow Manager. Tools can thus instruct the Flow Manager to initiate all or part of the personalization workflow so that the user can actually try out new settings in practice. This solution will likely work in most modern web browsers.
Although the core infrastructure is already available in the Flow Manager today, some further development work will be required to:
- provide a reusable Web Sockets client library in the Preferences Framework
- extend the Flow Manager’s available RESTful APIs, exposing preference set change events that can be called by a web application via Web Sockets
With the aforementioned web-first architectural approach, we are well positioned to support a very broad range of platforms and devices. Preferences editors and discovery tools, for example, will run on any platform with a modern browser. Supporting native access features and assistive technologies, however, requires deeper integration on low-level platform services that can be complex and highly variable across platforms.
Increasingly, many users are employing mobile devices and tablets. iOS has become particularly popular due to its excellent suite of built-in access features. The GPII community is keen to support this platform, but it presents a number of fundamental and troubling limitations. Apple places intense legal restrictions and a rigorous app review process on developers. On iOS, apps are completely sandboxed—they are not allowed to access system settings or the settings of other applications. Any app that attempts to do so will be refused by Apple. Mac OS X, with its new GateKeeper feature, is increasingly taking this troublesome direction as well.
As a result, given limited resources and a desire to make an impact on users today, iOS is not currently supported for native application personalization. All web-based tools and access features are supported, however. We are optimistic that future support for Apple-based platforms can be added, but pressure needs to be put on Apple to expose approved APIs for personalization into their platforms.
Privacy and security
Privacy and security must be principal tenets of a preferences acquisition system. Users want to have control over who can access their personal needs and preferences; our systems must protect this potentially sensitive information and give users easy-to-understand ways to identify their trust networks. When personal information is provided, an effort must be made to ensure those preferences don’t reveal their medical information.
Security support must also extend beyond the protection of a user needs to address security walls laid down between the web browser and the local system. Security considerations must also be addressed between the native operating system assistive technologies themselves.
Security and Privacy of User Preferences
A user’s preferences may reveal personal information about their lives and needs, either directly or indirectly; privacy and security is thus of critical importance to the design of the GPII architecture. While this security and privacy infrastructure work is still in its early stages, the intention is to use the OAuth 2.0 framework to protect access to a user’s preferences by third party applications.
The use of OAuth will let users approve or deny access to their information on a per-site basis, ensuring they control who gets to see and act on their needs and preferences. In designing this system, the GPII architecture will strive for compatibility with efforts such as the National Strategy for Trusted Identities in Cyberspace (NSTIC), a White House initiative focused on the establishment of an identity ecosystem that will allow individuals to validate their identities securely to facilitate exchange of personally identifiable information such as accessibility preferences. This layer of protection is particularly critical for the exchange of data indicating medical needs of users (such as the need to use a screen reader) in contexts where that information may be exposed to third party applications unbeknownst to the user.
In the future, an attribute release layer such as the Kantara Initiative’s User Managed Access will be layered on top of the basic OAuth 2.0 authorization system (Machulak 2010). This will give users the further ability to specify that a site or application is able to see only portions of their preferences set, ensuring that the risk of “privacy leakage” is reduced by only sharing the minimum information required by a service to meet users’ needs (Krishnamurthy 2009).
The goal is to ensure that the architecture can support diverse privacy and security requirements internationally.
Security Between the Browser and the Native Platform
As mentioned previously, web browsers protect the user from malware by “sandboxing”—not allowing direct access to features such as system settings or local files. Over time, standardized browser services are being added to provide access to some of these features securely. In the interim, developers sometimes make use of browser plug-ins to access features in the native platform or they embed the browser in containers like Cordova to gain access to these extended features on mobile devices as plug-ins themselves are not supported on most mobile browsers. Future work on preference acquisition must work within these restrictions as direct access to the local features will be necessary. However, new trusted API services are being added to web browsers such as WebSockets and IndieUI User Context that allow access to local information either directly or indirectly. Future preference acquisition and manipulation solutions will make effort to use these services to reduce management costs due to increased the portability of our solution.
Assistive Technologies and Security
At times activating native assistive technologies may be required in order to acquire a user’s preferences. Many assistive technologies can be mistaken by the operating system as malware, in that they must access native operating system services that would not normally be accessed by native platform applications. For example, screen readers and screen magnifiers still hook into graphics engine drawing calls and ‘steal’ the keyboard away from applications in order to facilitate control of their assistive technology by the user. Consequently, on Windows these applications must follow security requirements for Windows such as being signed. On other platforms, such as iOS, assistive technologies with this level of system access are not allowed and we must be dependent on what the native operating system platform provides.
Additionally, we must also consider that during preference acquisition a user may be using a shared system. Our preference gathering tool must be able to return the preferences back to the the original state that it was prior to a user’s use of the preference acquisition tool. In the GPII infrastructure, this process of “snapshotting” a system’s state prior to personalizing it and then restoring it afterwards is accomplished by the Lifecycle Manager component. The Lifecycle Manager is the part of the Flow Manager that is responsible for coordinating all aspects of the workflow of configuring, starting, and stopping assistive technologies and solutions on a local device. All settings that are set by the Lifecycle Manager are automatically reset to their previous values when the user’s session is finished. Further information about the Lifecycle Manager and its role in the GPII architecture can be found in the GPII wiki.
We must ensure that when native platform assistive technologies are involved in the preferences gathering that platform conventions and policies will be adhered to ensure the security of the user and the system on which they operate.