Cloud4All Testing

Cloud4all uses a UCD methodology, so prospective users will be involved in all stages of the conceptualizacion, design and evaluation of the products developed under it. The organization and management of the testing falls into the sub-project SP4 Pilot testing and thus this is focused on testing the ideas, concepts, and methods identified or developed by this project in real world settings. This means that they must work across platforms and with a sufficient variety of solutions and contexts to test the robustness of the approaches. It is understood that much more extensive testing will be needed than can be done through this IP alone. However, already in the context of this project, there are 3 iterative testing phases scheduled, upon predefined UCD methodologies and UI style guides, that will accompany project prototypes and tools from proof of project concepts (1st iterative phase) to Lo/Me-Fi prototypes assessment of SP3 applications and SP2 tools and algorithms (2nd iterative phase) and, finally, to Hi-Fi prototypes assessment in the context of 3rd iteration phase, before final demonstration events of A403.3. All types of users will participate in all three phases, including beneficiaries of various groups (i.e. older people and people with disabilities), developers and key stakeholders.

The activities in SP4 are centred on the overall project guidance, guidance for user testing and prototype development in relation to user testing and involvement, and it will prepare documentation for a Style guide,Testing plan, Pilot evaluation, User Testing and Prototype Development.

Links to relevant pages

• Getting SP3 applications ready for pilot testing - round 1
• http://wiki.gpii.net/index.php/Cloud4all_Pilot_Scenario_Tests

Three planned iterations

The three planned iterations are detailed in the following table:

Iteration Cycles	Dates	Prototypes to be tested	Pilot sites / Number of users
			Greece (CERTH)	Spain (FONCE)	Germany (SDC)
1st Iteration	M15-M18	WP102: Interfaces (ID 102.1, ID 102.2, ID 102.3). Input to MS4. WP104: Security gateway mockup  (ID.104.3). Input to MS7. Early version of Cloud semantic infrastructure (D.201.1). Input to MS10. Semantic alignment mechanism (ID201.2) WP204: Comparative testing of rule-based and statistics-based matchmakers, plus expert-provided adaptations. Input to MS17. Success criterion: All algorithms score at least 50% of the expert-provided solution. First round of SP3 Applications. Input to MS12.	30 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)	30 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)	30 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)
2nd Iteration	M25-M30	WP103: Context related profile tests (D103.2). Input for MS9. Security gateway prototype (D104.3) Business rules inference and trust ontology model (D201.2) Metadata tool boxes and utility (D202.1, D202.2, D202.4). Input for MS13. Repository federation algorithms (D203.1) Input for MS15. WP204: Comparative testing of rule-based, statistics-based and hybrid matchmakers, plus expert-provided adaptations. Input for MS17. Success criterion: All algorithms score at least 65%, and one algorithm at least 75% of the expert-provided solution. 2nd group of SP3 application. Input for MS23	40 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)	40 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)	40 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)
3rd Iteration	M37-M44	WP204: Comparative testing of rule-based, statistics-based and hybrid matchmakers, plus expert-provided adaptations. Success criterion: All algorithms score at least 75%, and one algorithm at least 80% of the expert-provided solution. Matching tools of WP205 (D205.1, D205.2). Remaining SP3 applications plus some repeats. Input for MS24.	50 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)	50 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)	50 beneficiaries of various groups 10 developers 5 key stakeholders (in expert walkthroughs)

TESTING DISCUSSION

What we would like to tested at these pilots

1. A Rule-based and a Statistical matchmaker that will be able to infer settings between diferent

• devices (desktop and mobile)
• platforms (Windows, Linux, Android, Symbian)
• applications (SP3 apps)

2. Multiple implementations of Cloud4all compatible implementations (SP3 apps) in all pilot sites languages ready and connected to the architecture.
3. A functional PMT with all the basic and some extended functions that will be connected to the architecture and it will be used from the user to create and save his/her initial set of preferences (create an account).

• A tool that will allow the automatic creation of the user's token from the user

4. A first Low-Fi PCP ready, working on desktop and mobile, that will be used by the user to fine-tune and save this fine-tuned settings.

• A way to automatically save the optimum settings of the user as these have been finetuned by the user, using the PCP or manully done by the application settings.

5. A first Low-Fi functional implementation of the MMs function to propose a new solution/AT to the users
6. An updated version of the SAT with federated repositories integrated, ready to be tested.
7. A first Me-Fi prototype of the context related profile adaptations.
8. Mock-up prototypes of the final PMT design, with the sharing among peers function.
9. Mock-up prototypes of the final PCP design and work flow.

Auto-configuration scenario

The auto-configuration scenario details, as well as, details for the whole proceedure are available here [1]

https://lh3.googleusercontent.com/EkyTnbH6V_on64rR0Zr6-KR9Br7G6V8nBds-HAZ0Bw-Racno8k8ERaEPayDVjqkrNoKsOYadhlMD_i3IlsFx83q6EA0ANls1IDUJenrRYfzW2v7SdtqzxdbJGw

We should create a training session for the user that will explain the SP3 applications. The training curriculum will include screenshots and videos.

1. To create an account. (sub-scenario 1.1)

This means that the user is asked to create an initial set of N&P using the PMT, in a desktop environment.  

• The user inserts specific values in the settings (which are presented in common terms entities) presented at the PMT.  
• When the user fill in all the settings required, he/she saves the account.
• The user creates his/her token.
• The preferences are automatically aplied to the device -"apply my changes button"
• The users sees an infered interface on Windows or Linux
• According to the disability of the user, the facilitator will launch a specific SP3 application that will be used as an interface for the user to see the inferred settings.

2. To edit inferred settings. (sub-scenario 1.2)

This means that the user is asked to edit/optimise the settings presented to him/her, using the PCP.

• The facilitator launches to the user the following (according to his/her disability profile)
  • Visual impaired users
    • Platform A: Desktop (Windows or Linux)
      • will do a simple task on Window
      • or will do a simple task on Linux
    • Blind users
      • Platform A: Desktop (Windows or Linux)
        • will do a simple task on Windows
        • or will do a simple task on Linux
        • will do a simple task on webanywhere???
    • Cognitive impaired users
      • Platform A: Desktop (Windows or Linux)
        • will do a simple task onMaavis (Maavispeople should tell us which tasks will work 3-4 moves) - Pilot facilitators will have demonstrated the specific task to the users beforehand.
    • Elderly users
      • Platform A: Desktop (Windows or Linux)
        • will do a simple task on Windows
        • or will do a simple task on Linux
    • Hearing impaired users
      • Platform A: Desktop (Windows or Linux)
        • will do a simple task on Windows
        • or will do a simple task on Linux
    • Deaf users
      • Platform A: Desktop (Windows or Linux)
        • will do a simple task on Windows
        • or will do a simple task on Linux
    • Deaf and blind users (users with visual and hearing impairements)
      • Platform A: Desktop (Windows or Linux)
        • will do a simple task on Windows
        • or will do a simple task on Linux

• PCP is launched to the user -->HOW????????????  Boyan: The PCP would most probably be launched together with user login (e.g. after clicking "apply my changes" button)
• The user is asked to modigy (fine-tune) the interface he/she sees using PCP as he/she does a specific task depending on the app that is launched.
• When the user is finished using the PCP he is asked if there are additional settings he/she wants to change
• If yes, the facilitator has to assist the user on making these changes manually, diving into the settings of the OS/Device (following the protocol of the 1st pilots iteration)
• When the user reaches the optimum interface according to his/her needs, the settings are saved -->HOW????????????? Boyan: Option 1: Save them from the preferences server. Option 2: Log them locally. In both cases it's easy to detect the settings changed from the PCP. It's harder to detect the settings the facilitator changes manually.
• The user Keys-out (logs -out).

3. To change platform (sub-scenario 1.3)

The user will be asked to change platform. This means that the user will be asked to use one of the following:

• Switch device. Since device A will always be desktop, device B can be one of the following:

  • tablet

  • mobile

    • Symbian 

    • Android

• Switch OS. This means that we are staying at the same device (desktop) and we change only OS. If platform A was Windows, platform B will be Linux, and vice versa.

• Switch application. This means that we are staying at the same device (desktop) and the same platform (Windows, or Linux) and we change only application.

• The facilitator launches to the user the following (according to his/her disability profile)

4. Edit inferred settings. (sub-scenario 1.4)

• The facilitator launches to the user the following (according to his/her disability

  • Visual impaired users
    • Platfrom B:Desktop (Windows or Linux) or mobile (Android or Symbian)
      • will do a simple task on Google chrome in desktop
      • or will do a simple task onAndroid
      • orwill do a simple task onSymbian  (Android/Symbian  people should tell us which tasks will work) - Will Android and Symbian support settings for visually impaired users like high contrast??? IF NOT WE WILL DO THE Chrome SCENARIO
  • Blind users
    • Platform B: Mobile (Android)
      • The user will do a simple task on Mobile accessibility
  • Cognitive impaired users
    • Platform B: Mobile (Android or Symbian)
      • will do a simple task on Android
      • orwill do a simple task onSymbian  (Android/Symbian  people should tell us which tasks will work)
  • Elderly users
    • Platform B: Tablet (Windows)
      • will see will do a simple task on Sociable (Sociable people should tell us which tasks will work)
  • Hearing impaired users
    • Platform B: Tablet (Windows or Linux))
      • will do a simple task on Windows Media Player
      • or will do a simple task on Linux Player app
  • Deaf users
    • Platform B: Mobile (Android) or Tablet (????)
      • The user will do a simple task on EC touch
      • The user will do a simple task on EC mobile
  • Deaf and blind users (users with visual and hearing impairements)
    • Platfrom B:Desktop (Windows or Linux) or mobile (Android or Symbian)
      • will do a simple task on Google chrome in desktop
      • or will do a simple task onAndroid
      • orwill do a simple task onSymbian  (Android/Symbian  people should tell us which tasks will work) - Will Android and Symbian support settings for visually impaired users like high contrast??? IF NOT WE WILL DO THE Chrome SCENARIO

• PCP is launched to the user -->HOW?????????????? (See above)
• The user is asked to modigy (fine-tune) the interface he/she sees using PCP as he/she does a specific task depending on the app that is launched.
• When the user is finished using the PCP he is asked if there are additional settings he/she wants to change
• If yes, the facilitator has to assist the user on making these changes manually, diving into the settings of the OS/Device (following the protocol of the 1st pilots iteration)
• When the user reaches the optimum interface according to his/her needs, the settings are saved -->HOW?????????????? (See above)
• The user Keys-out (logs -out).

Questions we have

Question	Team	Answer
Which authentication methods will be available?	Architecture
How are we going to create the token?	Architecture	We will have both USB and NFC tags. The architercture team will discuss this. If there is no solution, the facilitators of the pilots will have to do it manually.
Which of the extended functions will be ready to be tested?	PMT
How is the PCP going to be launched?	PCP	?? Probably together with user login (i.e. the settings will be applied and the PCP will be launched) ??
How is the PMT going to be launched?	PMT	We will have an open URL with the web page.
Which settings (in common terms) will be used?	PMT and PCP	?? See http://wiki.gpii.net/index.php/Cloud4all_Testing:_Essential_Registry_Terms ??
Is it possible to have a functional PCP for mobile, tablet and desktop? If yes, by when?	Architecture and PCP	?? Depends on whether we are able to implement WebSockets communication - in this case we will have the PCP in a browser for mobile devices ??
When we switch devices we don’t evaluate the MMs. What do we evaluate? The transformer? Is there a point in doing this by the means of development? Is it reasonable to test the transformer? Does anyone need this evaluation data?	MMs and Architecture
What do we want to capture from the user when we demonstrate the context scenario?	Context
We need to define the research questions per tool/scenario. What do the developers what to get from users at this phase? Check https://docs.google.com/document/d/1rwNaRYfagOY8IWl0Xa4UNS0K2qQWwSgGgb0OSAkqo10/edit# -page 12 and add your thoughts.	Architecture MMs PMT PCP SAT Context
How are we going to tackle security issues?	Architecture Security	We are going to have a SP3 application (Easit4all) integrated with the Security Gateway and the Preferences Server in the cloud (http://preferences.gpii.net/user/)
What does the architecture team need to know from the MMs teams for developing the logging tool?	Architecture MMs
Which SP3 will be available for the pilots and in which language?	SP3 developers and Ignacio (template already sent)
How are the setting gathered from the PMT applied to platform? Is this an automatic procedure or the user has to key-out and then key-in?	Architecture
Since we will have inverse transformations why can’t we have the “capturing my device settings” of the PMT ready for the pilots?	Architecture PMT

Auto-configuration Features for 2nd Pilot Phase

Cloud4all Solution	Auto-configuration features	Tested in the 2nd phase (Yes/ No)
APfP of Linux/GNOME and key AT applications
APfP of Windows OSs and built-in features
APfP of Web browsers (Internet Explorer and Mozilla Firefox)
APfP of Simple (Java) phones
APfP of Android smartphones
APfP of Mobile Accessibility for Android
APfP of Total conversation Allan eC
APfP of Maavis
APfP of Read&Write GOLD
APfP of Texthelp WebApps
APfP of SuperNova Access Suite
APfP of Web applications and components
APfP of ASIT - Adapted Social Interaction Tool
APfP of Mobile Accessibility (Android App)
APfP of WebAnywhere (Cloud screenreader)
APfP of SAToGo (Web-based screenreader)
APfP of Microsoft PixelSense/Sociable
APfP of Smart Houses (home appliances)
APfP of Online Banking Application
APfP of Infokiosks

Pilots roadmap

This roadmap has been integrated in the Cloud4all roadmap for test - Iteration II

We from SP4 need to do the following actions:

• Talk with architercture team and discuss the plan we have with them. --> Eleni, Ignacio on 9/10/2013
• Define concrete pilot scenarios for the functional tools (autoconfiguration - PMT, PCP, MMs- and the SAT) --> Eleni, by 11/10/2013
• Ask the SP3 developers to procide us with specific scenarios, comrised by 3-4 tasks for each application --> Eleni, by 18/10/2013
• Create a plan for the tech validation of the end-to-end system --> Eleni and Ignacio, by 18/10/2013
• Define the research topics/objectives/hypothesis/questions for each pilot sub-scenario --> Eleni, by 18/10/2013
• Identify how we will answer each research question --> TECH, UPMM CERTH????, by 25/10/2013
• Create the tools we will use for the evaluation --> TECH, UPMM CERTH????, by 1/11/2013
• Define how many users will test what --> TECH, by 1/11/2013

Cloud4all roadmap for test - Iteration II

• 2-Oct: Ignacio will pass the planning document on Google Doc - with the SP3 apps and their proposed settings to Kostas - Including the common terms.
• 4-Oct: Kostas, Christophe and Claudia will insert all the info gathered until now to the SAT (http://160.40.50.183:8080)
• 4-Oct: On Friday Gianna will send an email to SP3 devs who have not filled in Ignacio’s doc yet and ask them to fill in the SAT with the assistance of Kostas (within a week - 7/10-11/10) - scheduled appointments. It should be done by 9-Oct
  • Update 10 October: preference terms (including data types, value ranges etc) are being collected in a new spreadsheet on Google Docs.
• 9-Oct. Talk with the architecture team and discuss our plan with them (Eleni, Ignacio)
• 10-Oct. Kostas adds 2 new fields (string) in the SAT to enter the transformation rules from Pilot Common Terms to Specific Terms and viceversa. Kostas meets Gianna to explain how to introduce the transformation rules. Gianna will pursue SP3 developers to introduce this information in the SAT.
• 11-Oct. Define concrete pilot scenarios for the functional tools (autoconfiguration - PMT, PCP, MMs- and the SAT) --> Eleni
• 11-Oct. Define the final list of solutions/platforms/components that are part of the test. Eleni, Gianna and Kasper
• 14-Oct. SP3 developers to provide SP4 with specific scenarios, comprised by 3-4 tasks for each application. SP3 developers (coordinated by Gianna) to Eleni
• 14-Oct. Decide which Pilot common terms we will use at the 2nd pilots. Kostas, Christophe and Claudia. Dependences of PMT/PCT design force us to come up with the common terms before 14-Oct. This can be a first iteration to help PMT team do their job. Publish it the wiki at Pilots iteration 2. Christophe and Claudia do the matching of the Pilot common terms with the specific terms entered by SP3 developers. Output: list of Pilot Common terms.
• 14-Oct. Draft of the technical/functional pre-test plan. Kasper, Colin, Ignacio, Christophe
• 14-Oct. Check of the Pilot Common terms by SP3 and SP4. Output: approval/rejection/suggestion of Pilot Common Terms. Eleni
• 15-Oct. Final version of the common terms. Christophe, Claudia and Kasper
• 15-Oct. PMT/PCP teams get all the information about the pilot scenarios and the common terms and start working in some wireframes for the Pilots PCP – PMT (PCP, PMT, Pilot teams)
• 15-Oct. Final version of the technical/functional pre-test plan. Kasper, Colin, Ignacio, Christophe
• 18-Oct. efine the research topics/objectives/hypothesis/questions for each pilot sub-scenario --> Eleni
• 18-Oct. Architecture-SP3 meeting to help SP3 create the transformers (json files).
• 21-Oct. All solution developers have added the transformations in pseudocode in the SAT.
• 21-Oct. The SAT is completely finished. Kostas exports to json files + pseudocode transformations.
• 22-Oct. PCP/PMT teams to discuss the Pilot-PCP/PMT wireframes and estimate development dates.
• 22-Oct. Architecture-SP3 meeting to solve last question SP3 produce the final version of the transformers (json files).
• 24-Oct. Identify how we will answer each research question --> TECH, UPMM CERTH????
• 24-Oct. Make the transformation of the common terms to app specific terms (Claudia, Christophe, Ignacio, Colin - by end of October)
• 28-Oct. Coding integration tests (SILO) Example: (https://github.com/GPII/windows/blob/v0.2/tests/AcceptanceTests.js#L475-L511)
• 28-Oct. Start of technical/functional validation. MMs, PMT/PCP and SP3 solutions must be ready at this moment.
• 01-Nov. Create the tools we will use for the evaluation --> TECH, UPMM CERTH????
• 01-Nov. Define how many users will test what --> TECH
• 19-20-Nov. Plenary board and Consortium workshop
• 21-Nov. Scientific Advisory Board.
• 25-Nov. Start of Pre-test with first users and support from all development teams.
• 13-Dic. End of Pre-tests.

What MMs and PCP/PMT will (not) do in Iteration II

• The RB-MM will not work for Android, it will only be deployed locally.
• The RB-MM is available in desktop environments, and it enhances the transformation terms of constrainsts, etc.
• The ST-MM will run wherever the real-time framework can run -> it should run on Android
• The ST-MM will work on Symbian -> Eleni to ask Kostas
• The hybrid MM development starts in M23
• For the main test scenario, we will not use any recommendations -> only pre-defined conditions.
• PCP will show you the current value of your preference set
• The PMT will only save settings that the user has actually adjusted
• The goal of the Settings Snapshotter for this pilot would be to provide a kind of "logging" or "research facility," unconnected to any user interfaces. The user will tweak their settings in the UI of all their native applications (e.g. NVDA's settings dialog, Windows control panels, etc) and then the pilot facilitator will take a "snapshot" (by running a script or something), which will cause these settings to be saved to a log.
• Next week, after we get the common terms and the pilot scenarios, the design team will work on wireframes of the PMT and PCT. (Oct 18) (The PCP/PMT will estimate how much time they will need to implement these prototypes.

Critical Next Steps

These two steps are blockers for designing the necessary "pilot wireframes" for the PCP & PMT

• Pilot scenarios need to be clearly defined by 11-Nov
• All the common terms need to be listed by 14-Nov

Apps so far and people in charge

• Sociable: Thanos with Kasper
• Google Chrome: Ignacio with Colin
• SAToGo: Kasper
  • (still needs to be done)
• WebAnywhere: Kasper
  • (still needs to be done, need to determine what settings it has). This isn't integration with the web-based flow manager, but rather just via query parameters in the URL
• Windows: Kasper.
  • There are new settings we need to support, probably
• Linux: Kasper and Javi.
  • There are new settings we need to support, probably
• Android: Javi.
  • There are new settings we need to support, most definitely
• Mobile Accessibility: Ferran
  • (might need help--Javi)
• Symbian: Kostas K
  • (Kasper will check in)
• Maavis: Steve Lee
  • (might need help). Check with him today at the archi meeting
• Smarthouse: Boyan and crew
  • check with him today at the archi meeting

Risks, bugs and stuff to panic about

• Something wrong with font sizes on Windows (Flow Manager seems to freeze)
• RFID reading doesn't work on Windows
• Kasper has a 3-persons workload