- 1 Summary
- 2 Requirements
- 3 Ongoing work
- 4 People involved
Nowadays, web browsers are one of the most important pieces of the set of software we use as a daily basis. Usually, web browsers don’t follow the settings applied at the OS level such as the font size or the high contrast themes, and for this reason, we need a mechanism to enable such settings.
Right now, web browsers don’t allow 3rd party applications to access/change their settings but have support for extensions that can enhance or add new features to the browser when installed. These extensions are clearly the best option for the GPII to auto-personalize the web browsers to fit the users’ needs and preferences.
Also, there is another layer of adaptation that the GPII would like to cover, the auto-personalization of web pages that allow the customization of the interface to fit the users’ needs and preferences. By covering such layer, the GPII can go deeper and perform a more fine grained auto-personalization through the different layers.
Normally, web browsers include APIs for creating extensions that can interact with the browser in many different ways, and these are clearly enough for us to achieve these goals. Initially, we would like to cover the most relevant web browsers: Mozilla Firefox, Google Chrome and MS Edge. Since both Chrome and Firefox can run in different platforms we will be able to use the same extensions across different Operating Systems with no additional work.
During the Cloud4all project, several extensions were created in order to demonstrate the APfP capabilities of the GPII. These extensions have support for adapting the browser and the web content. Initially, these extension were “pull based”, this means that the extensions interact themselves with the GPII by making requests to a Cloud Based Flow Manager. Since the GPII tends to use a “push based” approach to perform the auto-personalization of the OS and the applications, these extensions had to be converted to follow this approach. In order to address that problem, the WebSockets Settings Handler and the Browser Channel were created in order to fulfill this requirement so the extensions could be refactored to follow this approach (see Bridging the Browser-OS Personalization Gap). By the end of the Cloud4all project, we ended up with these extensions:
- Chrome4Cloud: An extension that can configure the web content (font size, high contrast themes, etc) and enable/disable ChromeVox in Google Chrome. It uses the Browser Channel and can also be used standalone without the GPII.
- Cloud4Firefox: An extension that uses the "pull based" approach and that can configure settings such as font size, make Firefox follow the OS font styles, etc.
- GWC-Chrome: This extension uses the Browser Channel and adds support for web pages to receive settings from a locally running Flow Manager.
- SST: This extension uses the Browser Channel too and performs the adaptation of web content through the Cloud4all's Service Synthetizer Tool.
Given that all these extensions follow similar approaches and/or use similar (or even literally copy/pasted) source code, there is no reason to keep them as they are. Ideally, we should build a single one that has all the features we want and works cross-platform, or at least, to create them by using common components and avoiding duplication of code.
These are the general features that we want to build into our browser extensions.
- Enable/Disable built-in features
- Enable/Disable 3rd party extensions
- Perform adaptations of the web content
- Provide support for GPII compatible web pages
- Be able to apply the settings selectively, i.e.: if a web page is GPII compatible and supports font size, take this into account and do not apply the font size at browser level (although this is something that has to be filtered out at some point in the matchmaking process, the extension needs to be able to deal with these situations)
Also, besides the features that the extensions must include, we want to:
- Establish a coherent API between the browser extensions and the Browser Channel - Although the registration process starts in the browser side, the rest of the communication is “push based” (The Browser Channel pushes settings to the extension whenever they change)
- Ensure that the communication between the Browser Channel and the extension is secure and that all the security concerns have been taken into account
Currently we’re working on the extension for Google Chrome, which will help us to sketch up and consolidate the way the rest of the extensions must be implemented. Once this first implementation is done, we will go for Firefox. Potentially, some of the components that are part of the extension for Google Chrome will be moved into a library that can be used by the extensions for the other browsers.
Right now, the implementation will use the GPII Flow Manager’s Browser Channel, but this may change in the future once the Nexus has support for this kind of implementations.
UIO+ is a browser extension that enacts settings in the browser of a GPII enabled machine. These will only be settings that could not be applied by the OS or other AT running on the machine ( see the 5 layers ). Which settings to apply via UIO+ will be determined by the “match maker”.
(See: uio-f2f-toronto GPII Pad)
(in order of priority)
- Text Size - can we use the browsers zoom feature - ( see: "display.screenEnhancement.screenScale" term)
- Line spacing
- Colour contrast
- Character spacing
- Larger links and form controls
- TTS read and highlight all text on the page (with play/pause button and auto-read mode)
- TTS read and highlight selected text
- Table of contents - talk to the UX team about where on the page it should be put.
- Syllabification - splits words into syllables - see onenote learning tools for an example, but there may be webservices we can use
These settings are coming from this document
Using browser extensions
- can only change settings of other extensions that you own (ie. have developed and can make custom channels)
- could also work if the extension integrated with GPII
- extensions can enable/disable other extensions, so if an extension applies its feature on "enable" we can use it for enacting the setting
- the user needs to initiate an extension installation, could possibly be done as part of the GPII app install.
- should be able to install encacting extensions on demand, but this will be lower priority.
- The installation part of the extensions could be responsability of IoD.
Using in page enacting
- Will inject CSS and JS into the page to enact settings in fashion similar to the prefs framework's UI Enhancer
Other Use Cases
Another potential case is when the extension is run on a machine that isn't GPII enabled. In this case it could be like the standard UIO, including the adjuster panels and store locally or connect to the cloud based flow manager. (this isn't priority for the summer 2017).
Timelines - Due Dates
Pre-pilots: 3rd week of August 2017 at the latest (should be done by mid to end of July 2017 so that it can go through testing and etc. first ).
- Look into Javi and Cindys extension
- Write up iteration plans for the tasks
- Start by using other browser extensions for enacting on Chrome
- Add any additional enactors through the browser ( similar to how standard UIO works )
- Port to MS Edge ( target latest MS Edge? )
While a feature may be marked as "DONE", it may still require bug fixes and/or improvements in the future. Features marked as "IMPROVE" are currently working but have a requested improvement.
|Captions||DONE||Completed, however it only works a limited number of videos. Ones that use the YouTube iframe API and that provide captions (not autogenerated).|
|Dictionary||REMOVED||Removed as the requirement, from the Google Dictionary extension, to reload existing tabs, was causing too much confusion.|
|Table of Contents||DONE|
|Text-to-Speech||DONE||Complete however there are some JIRAs for improving the interactions, and due to a Chrome bug related to pausing, the page level tts has been temporarily removed.|
|Apply preferences from GPII||DONE|
|Display a set of adjuster for modifying preferences locally||DONE|
|Re-oraganize / hide/show Preference Adjusters||Needs more information
In looking at the related Google Doc, it seemed that the conversation regarding this is still open.
|Reset||Requested feature for use when modifying through the UIO+ panel.
The UX team will discuss on the week of Feb 5, 2018
|Right Click UIO+ Gear Icon||Pops up menu of items that can be turned on and off (ONLY) - For example Reading on highlight , Captions|
|Google Translate||Need more information
I believe this is intended to be an adjuster. Is that correct? Are there designs?
Publishing UIO+ to the Chrome Web Store
Although this work is part of the whole GPII core team, these are the most active members working on this project.
- Javier Hernández
- Colin Clark
- Cindy Li
- Justin Obara
Right now we're having sporadic meetings to settle down the concept, requirements, design, etc.