[meta] Presentation API Testing

[anssiko]

When doing a larger change spanning across components it is often helpful to have a single issue for tracking the bigger change. This is such a "meta issue" for tracking the development of the test suite for the Presentation API.

Resources:

• w3c/web-platform-tests/presentation-api/ - GitHub repo for tests
• w3c-test.org/presentation-api/ - published tests
• Second_Screen/Work_Mode/#Testing - how to contribute

Tasks and related sub-issues:

• Land initial web-platform-tests/presentation-api PR: Presentation API test web-platform-tests/wpt#2432
• Split interface Presentation IDL into partials: Presentation API IDL Tests for Presentation Interface #230
• Update web-platform-tests/presentation-api to match the ED spec
• Annotate IDL blocks with idl-controlling-ua and/or idl-receiving-ua: Annotate IDL blocks with idl-controlling-ua and/or idl-receiving-ua classes #268
• Improve test coverage
• Publish test results (to be updated as test coverage increases)
  • https://w3c.github.io/test-results/presentation-api/controlling-ua/all.html
  • https://w3c.github.io/test-results/presentation-api/receiving-ua/all.html
• [presentation-api] Test coverage for start() event order web-platform-tests/wpt#4168

All - @louaybassbouss is your test facilitator for the Presentation API spec and will be responsible for coordinating the test suite development. Please work with @louaybassbouss by contributing new test cases and reviewing existing tests. See the Work Mode > Testing wiki for more information on how to setup your own test environment.

Happy testing!

[louaybassbouss]

thx @anssiko. I will take the responsibility to keep the IDL-harness up-to-date with the spec. I will try to extract the IDL for each conformance class (controlling and receiving UA) programatically.
I'd like also to ask Presentation API implementors to share with the group more information about their implementations e.g. which version of controlling UA, which presentation devices, limitations, etc. These information may help testers while writing or reviewing tests.
For testing activities, I'd like to start with basic tests around the following features:

• monitor screen availability.
• launch new presentation.
• reconnect to a running presentation.
• join a running presentation.
• presentation connection lifecycle (and transition between connection states as stated in the spec).
• messaging (send and receive messages in the formats stated in the spec).

If you like to open an new issue please use the label testing.
I will let you know when the initial web-platform-tests PR is merged.

[anssiko]

Thanks @louaybassbouss. I updated the initial comment with the suggested tasks for basic tests, and will link to related issues when they appear.

[anssiko]

The initial PR has landed to establish the test structure for the Presentation API, so you're now able to run the work-in-progress tests in-browser at:

http://w3c-test.org/presentation-api/

The Presentation API tests are located in the w3c/web-platform-tests repo at:

https://github.com/w3c/web-platform-tests/tree/master/presentation-api

(I added these links to the head of the spec and updated the first comment in this issue accordingly.)

w3c-test.org mirrors the master branch of the web-platform-tests repo. Whenever a new Pull Request is merged to master, the mirror should be updated.

[louaybassbouss]

thx @anssiko. I just submitted a new PR to the web-platform-tests repo containing an update of idlharness.html for controlling and receiving UAs according to the lastet ED from today.
Volunteers a welcome to review this PR. Please add a comment to the PR after review (if everything is fine just put LGTM).

[anssiko]

@zqzhang Could you look into creating some of the basic tests enumerated in #266 (comment)?

[zqzhang]

@yhe39 will jump in this after the International Labor Day holidays.

[anssiko]

Thanks @zqzhang and @yhe39 -- please coordinate your test contributions with the Test Facilitator @louaybassbouss. He indicated he'd have new contributors to help with testing too.

@louaybassbouss feel free to share your plans and expected schedule. It'd be good if this meta issue would be kept updated as the testing progresses as to allow more people jump in and help.

[anssiko]

Initial test results published (thanks @louaybassbouss and @zqzhang):

https://w3c.github.io/test-results/presentation-api/controlling-ua/all.html
https://w3c.github.io/test-results/presentation-api/receiving-ua/all.html

These test results contain idlharness.js tests only at the moment. For increased coverage, @louaybassbouss, @zqzhang, and @yhe39 will be adding basic tests (see the first comment) beyond the existing tests for WebIDL interfaces. Any ambiguities or issues in the spec that may be discovered while writing further test cases should be reported to the group so we can improve the spec accordingly.

[louaybassbouss]

I am happy to announce that the first test report of Presentation API is published. It includes separate test results for Controlling 1 and Receiving 2 UAs. It considers for now only WebIDL tests. Many thanks to @zqzhang for the support. Also welcome @yhe39 from Intel, @mariuswsl and @taff-franck from Fraunhofer FOKUS who volunteered to support this activity. @mariuswsl and @taff-franck are working on the first 4 basic tests from the list in the first comment. Other volunteers are welcome to write, review or run tests. We plan to publish a second test report before the next F2F Meeting and a third one by the end of June. The second report will considers a subset of the basic tests and third report all basic tests and hopefully more Browsers. To Presentation API implementers: can you please share information with the group about current status, limitations, setup of presentation devices, etc.. This will help us a lot in writing and running tests.