Browser Extensions

From wiki.gpii
Jump to: navigation, search

Summary

Motivation

Nowadays, web browsers are one of the most important pieces of the set of software we use as a daily basis. Usually, web browsers don’t follow the settings applied at the OS level such as the font size or the high contrast themes, and for this reason, we need a mechanism to enable such settings.

Right now, web browsers don’t allow 3rd party applications to access/change their settings but have support for extensions that can enhance or add new features to the browser when installed. These extensions are clearly the best option for the GPII to auto-personalize the web browsers to fit the users’ needs and preferences.

Also, there is another layer of adaptation that the GPII would like to cover, the auto-personalization of web pages that allow the customization of the interface to fit the users’ needs and preferences. By covering such layer, the GPII can go deeper and perform a more fine grained auto-personalization through the different layers.

Normally, web browsers include APIs for creating extensions that can interact with the browser in many different ways, and these are clearly enough for us to achieve these goals. Initially, we would like to cover the most relevant web browsers: Mozilla Firefox, Google Chrome and MS Edge. Since both Chrome and Firefox can run in different platforms we will be able to use the same extensions across different Operating Systems with no additional work.

Background

During the Cloud4all project, several extensions were created in order to demonstrate the APfP capabilities of the GPII. These extensions have support for adapting the browser and the web content. Initially, these extension were “pull based”, this means that the extensions interact themselves with the GPII by making requests to a Cloud Based Flow Manager. Since the GPII tends to use a “push based” approach to perform the auto-personalization of the OS and the applications, these extensions had to be converted to follow this approach. In order to address that problem, the WebSockets Settings Handler and the Browser Channel were created in order to fulfill this requirement so the extensions could be refactored to follow this approach (see Bridging the Browser-OS Personalization Gap). By the end of the Cloud4all project, we ended up with these extensions:

  • Chrome4Cloud: An extension that can configure the web content (font size, high contrast themes, etc) and enable/disable ChromeVox in Google Chrome. It uses the Browser Channel and can also be used standalone without the GPII.
  • Cloud4Firefox: An extension that uses the "pull based" approach and that can configure settings such as font size, make Firefox follow the OS font styles, etc.
  • GWC-Chrome: This extension uses the Browser Channel and adds support for web pages to receive settings from a locally running Flow Manager.
  • SST: This extension uses the Browser Channel too and performs the adaptation of web content through the Cloud4all's Service Synthetizer Tool.

Given that all these extensions follow similar approaches and/or use similar (or even literally copy/pasted) source code, there is no reason to keep them as they are. Ideally, we should build a single one that has all the features we want and works cross-platform, or at least, to create them by using common components and avoiding duplication of code.

Requirements

These are the general features that we want to build into our browser extensions.

  • Enable/Disable built-in features
  • Enable/Disable 3rd party extensions
  • Perform adaptations of the web content
  • Provide support for GPII compatible web pages
  • Be able to apply the settings selectively, i.e.: if a web page is GPII compatible and supports font size, take this into account and do not apply the font size at browser level (although this is something that has to be filtered out at some point in the matchmaking process, the extension needs to be able to deal with these situations)

Also, besides the features that the extensions must include, we want to:

  • Establish a coherent API between the browser extensions and the Browser Channel - Although the registration process starts in the browser side, the rest of the communication is “push based” (The Browser Channel pushes settings to the extension whenever they change)
  • Ensure that the communication between the Browser Channel and the extension is secure and that all the security concerns have been taken into account

Ongoing work

Currently we’re working on the extension for Google Chrome, which will help us to sketch up and consolidate the way the rest of the extensions must be implemented. Once this first implementation is done, we will go for Firefox. Potentially, some of the components that are part of the extension for Google Chrome will be moved into a library that can be used by the extensions for the other browsers.

Right now, the implementation will use the GPII Flow Manager’s Browser Channel, but this may change in the future once the Nexus has support for this kind of implementations.

Consult the project’s README.md file to check about the details and status of the implementation. There's also a TODO list that you can consult to have an idea of what's going on.

UIO+

UIO+ is a browser extension that enacts settings in the browser of a GPII enabled machine. These will only be settings that could not be applied by the OS or other AT running on the machine ( see the 5 layers ). Which settings to apply via UIO+ will be determined by the “match maker”.

(See: uio-f2f-toronto GPII Pad)

Relavent JIRAs

https://issues.gpii.net/issues/?jql=project%20%3D%20GPII%20AND%20component%20%3D%20%22Web%20Personalization%20Browser%20Extension%22

Settings

(in order of priority)

  1. Text Size - can we use the browsers zoom feature - ( see: "display.screenEnhancement.screenScale" term)
  2. Line spacing
  3. Colour contrast
  4. Character spacing
  5. Larger links and form controls
  6. TTS read and highlight all text on the page (with play/pause button and auto-read mode)
  7. TTS read and highlight selected text
  8. Table of contents - talk to the UX team about where on the page it should be put.
  9. Dictionary
  10. Simplification
  11. Syllabification - splits words into syllables - see onenote learning tools for an example, but there may be webservices we can use

These settings are coming from this document

Enacting

Using browser extensions
  • can only change settings of other extensions that you own (ie. have developed and can make custom channels)
  • could also work if the extension integrated with GPII
  • extensions can enable/disable other extensions, so if an extension applies its feature on "enable" we can use it for enacting the setting 
  • the user needs to initiate an extension installation, could possibly be done as part of the GPII app install.
  • should be able to install encacting extensions on demand, but this will be lower priority.
  • The installation part of the extensions could be responsability of IoD.
Using in page enacting
  • Will inject CSS and JS into the page to enact settings in fashion similar to the prefs framework's UI Enhancer

Other Use Cases

Another potential case is when the extension is run on a machine that isn't GPII enabled. In this case it could be like the standard UIO, including the adjuster panels and store locally or connect to the cloud based flow manager. (this isn't priority for the summer 2017).

Design Research

Browser Adaptation Design Research

Timelines - Due Dates

Pre-pilots: 3rd week of August 2017 at the latest (should be done by mid to end of July 2017 so that it can go through testing and etc. first ).

Iteration Plan

Next steps

  • Look into Javi and Cindys extension
  • Write up iteration plans for the tasks
  • Start by using other browser extensions for enacting on Chrome
  • Add any additional enactors through the browser ( similar to how standard UIO works )
  • Port to MS Edge ( target latest MS Edge? )

Current Status

While a feature may be marked as "DONE", it may still require bug fixes and/or improvements in the future. Features marked as "IMPROVE" are currently working but have a requested improvement.

Preferences Status Notes
Captions DONE Completed, however it only works a limited number of videos. Ones that use the YouTube iframe API and that provide captions (not autogenerated).
Character Spacing DONE
Contrast DONE
Dictionary REMOVED Removed as the requirement, from the Google Dictionary extension, to reload existing tabs, was causing too much confusion.
Emphasize Inputs DONE
Line Spacing DONE
Right-Click Select DONE
Selection Highlight DONE
Simplification DONE
Syllabification DONE
Table of Contents DONE
Text Size DONE
Text-to-Speech DONE Complete however there are some JIRAs for improving the interactions, and due to a Chrome bug related to pausing, the page level tts has been temporarily removed.
Word Spacing DONE
Capabilities Status Notes
Apply preferences from GPII DONE
Display a set of adjuster for modifying preferences locally DONE
Re-oraganize / hide/show Preference Adjusters Needs more information

In looking at the related Google Doc, it seemed that the conversation regarding this is still open.

Reset Requested feature for use when modifying through the UIO+ panel.

https://issues.gpii.net/browse/GPII-2840

The UX team will discuss on the week of Feb 5, 2018

Right Click UIO+ Gear Icon Pops up menu of items that can be turned on and off (ONLY) - For example   Reading on highlight , Captions
Google Translate Need more information

I believe this is intended to be an adjuster. Is that correct? Are there designs?

Publishing UIO+ to the Chrome Web Store

See the documentation about publishing UIO+ to the Chrome Web Store.

People involved

Although this work is part of the whole GPII core team, these are the most active members working on this project.

  • Javier Hernández
  • Colin Clark
  • Cindy Li
  • Justin Obara

Right now we're having sporadic meetings to settle down the concept, requirements, design, etc.