Engineering Notes
Created by admin on Sat, 07/01/2012 - 11:21
Sub Topic:
Writing Good Test Cases
Department:
Chapter Name:
User Interface Design
Description:
General tips for writing a good Test Case for a effective User Interface Design
Content:
<h1>
Writing good test cases</h1>
<h2>
General tips</h2>
<p>A good test case should have the following properties:</p>
<ul>
<li>
portable</li>
<li>
fast</li>
<li>
understandable & clear</li>
<li>
deterministic</li>
</ul>
<h3>
Portable</h3>
<p>A test should work in DumpRenderTree (of course!) <strong>and in a WebKit-base browser</strong>. This means that anything that is specific to DumpRenderTree should not make the test fail if it is not available.</p>
<p>Bad test:</p>
<div style="margin-left:21.0pt;">
<pre>
<script></pre>
<pre>
layoutTestController.dumpAsText();</pre>
<pre>
</script></pre>
</div>
<p>This will throw an exception in any WebKit-based browser.</p>
<p>Good test:</p>
<div style="margin-left:21.0pt;">
<pre>
<script></pre>
<pre>
if (window.layoutTestController)</pre>
<pre>
layoutTestController.dumpAsText()</pre>
<pre>
</script></pre>
</div>
<p><strong>Note:</strong> You get more points if your test works in ANY browsers.</p>
<h3>
Fast</h3>
<p>WebKit has several thousands layout tests that are run in parallel. If yourlayout test takes a lot time, it will slow down everyone's testing. Also DumpRenderTree has a default timer of several seconds before it considers that a test timed out. <strong>This is true even if you are using waitUntilDone / notifyDone.</strong></p>
<p>Here are some advices to avoid having a slow test case:</p>
<ul>
<li>
Do not make unnecessary use of <tt>window.setTimeout</tt> and similar functions which create a lower bound on the time it takes the test to complete. For separating events in time, you can use<tt>eventSender.leapForward</tt>; for waiting on sub-resources, you can use load events (e.g. iframe.onload = doNextStep;)</li>
</ul>
<ul>
<li>
Do not make unnecessary use of external resources: use inline JavaScript and <tt>style</tt> elements instead of <tt>link</tt>s to external stylesheets. You can also use <tt>data:</tt> URLs sometimes for things like frames' <tt>src</tt> attribute. The exceptions are stylesheets and JavaScript libraries that are shared by multiple tests, and cases that test the loading of external resources. (There are various data: url generators on the net.)</li>
</ul>
<h3>
Understandable & clear</h3>
<p>A good reading on how to write awesome test cases is the <a href="http://www.w3.org/Style/CSS/Test/guidelines.html"> CSS2.1 Test Case Authoring Guidelines</a>. Especially read the section: <em>4. Writing ideal tests</em></p>
<p>Now in the context of WebKit, a test should at least indicate:</p>
<ul>
<li>
which bug it belongs to. That means the bug number or the bugzilla URL (the latter is better as you can paste it in a browser) but also the bug title to see what is tested.</li>
<li>
what is the condition for success or failure.</li>
</ul>
<p>The rule to understand what you need to include is to position yourself as someone who has to investigate why your test is failing. He need to kickly understand if the new baseline is a progression, a regression or just a small change unrelated to what is tested.</p>
<p>Bad test:</p>
<div style="margin-left:21.0pt;">
<pre>
<style></pre>
<pre>
.block { color: red; }</pre>
<pre>
</style></pre>
<pre>
<p class="block"></p></pre>
</div>
<p>This is bad because it misses all the WebKit information but also because it uses <strong>red</strong> when it means passing. Red should be reserved for failures.</p>
<p><strong>Note:</strong> This test is badly formatted and if this is not something your test needs, it is better to avoid it.</p>
<p>Better test:</p>
<div style="margin-left:21.0pt;">
<pre>
<!DOCTYPE html></pre>
<pre>
<html></pre>
<pre>
<head></pre>
<pre>
<style></pre>
<pre>
.block { color: green; }</pre>
<pre>
</style></pre>
<pre>
</head></pre>
<pre>
<body></pre>
<pre>
<p> Bug <a href="http://webkit.org/b/XXXX">XXXX</a>: WebKit does not apply class-selector</p></pre>
<pre>
<p> For this test to pass, you should see a green rectangle below. </p></pre>
<pre>
<p class="block"></p></pre>
<pre>
</body></pre>
<pre>
</html></pre>
</div>
<p>This test is a huge improvement over the previous one but we can actually go one step further! (see the next section about writing pixel tests on that)</p>
<p>One last comment (FIXME: move it to a better section):</p>
<ul>
<li>
Tests should not access the Internet. Avoid <tt>http:</tt> URLs in <tt>src</tt> and <tt>href</tt> attributes, in CSS properties and in XMLHttpRequest. Testing WebKit's network layer should be done using the HTTP test facility, to be described below.</li>
</ul>
<h2>
Pixel test tips</h2>
<h3>
Do you really need a pixel test?</h3>
<p>That should be your first question. Pixel tests are a burden on every ports as any small change in the engine could lead to your test needing a new pixel result. That does not mean that pixel tests are bad, they are invaluable for catching visual regressions but there may be a way to avoid dumping the pixel while checking the behavior you need!</p>
<p>Let's take the example from the previous section and make it better.</p>
<p>Even better test:</p>
<div style="margin-left:21.0pt;">
<pre>
<!DOCTYPE html></pre>
<pre>
<html></pre>
<pre>
<head></pre>
<pre>
<style></pre>
<pre>
.block { color: green; }</pre>
<pre>
</style></pre>
<pre>
<script></pre>
<pre>
if (window.layoutTestController)</pre>
<pre>
layoutTestController.dumpAsText();</pre>
<pre>
</pre>
<pre>
function log(message)</pre>
<pre>
{</pre>
<pre>
var console = document.getElementById('console');</pre>
<pre>
console.appendChild(document.createTextNode(message);</pre>
<pre>
console.appendChild(document.createElement('br'));</pre>
<pre>
}</pre>
<pre>
</pre>
<pre>
function checkBlockColor()</pre>
<pre>
{</pre>
<pre>
var block = document.getElementsByClassName('block')[0];</pre>
<pre>
var color = document.defaultView.getComputedStyle(block, null).getPropertyValue('color');</pre>
<pre>
if (color === 'rgb(0, 255, 0)')</pre>
<pre>
log('PASSED');</pre>
<pre>
else</pre>
<pre>
log('FAILED: color was not green but ' + color);</pre>
<pre>
}</pre>
<pre>
window.addEventListener('load', checkBlockColor, true);</pre>
<pre>
</script></pre>
<pre>
</head></pre>
<pre>
<body></pre>
<pre>
<p> Bug <a href="http://webkit.org/b/XXXX">XXXX</a>: WebKit does not apply class-selector</p></pre>
<pre>
<p> For this test to pass, you should see a green rectangle below and / or you should see PASS below. </p></pre>
<pre>
<p id="console"></p></pre>
<pre>
<p class="block"></p></pre>
<pre>
</html></pre>
</div>
<p>As you can see, we did not need the pixels to get the information. The idea is that in this case, we were testing CSS and not the painting part of WebKit so querying the property directly would give us the result we need without having to dump the pixels.</p>
<h3>
How to write portable pixel tests</h3>
<p>The problem with a pixel dump is that it <strong>can be</strong> highly depend on the platform on which you run it. To be avoided to have more portable tests cases are the following:</p>
<ul>
<li>
fonts</li>
<li>
native controls</li>
</ul>
<h4>
Fonts</h4>
<p>Fonts are very dependent on the platform and the source of a lot of differences in our tests' output. That's because not all platform share the same fonts and the fallback mechanism is pretty much defined per machine as it depends on the OS and the available fonts!</p>
<p>The golden rule for writing pixel tests is that <strong>if you can avoid text, you should</strong>.</p>
<p>This seems against the previous section about writing understandable test cases and it is. However there are ways to put the information you need while keeping the portability of the no-text mantra:</p>
<ul>
<li>
put your information as comments in the HTML code (preferably somewhere prominent like the beginning)</li>
<li>
put your text outside of the dumped area: by default DumpRenderTree dumps a viewport of 600 * 800 pixels (FIXME: not 100% sure about the exact size). This means that anything outside this range would not be dumped in DRT but will appear in a browser opening the test if the resolution is sufficient.</li>
<li>
add a 0 opacity to your text.</li>
</ul>
<p><strong>Important note:</strong> For the 2 last points, the text will still be in the text render tree dump! You can however disable the text dump if you don't need it to achieve maximum portability, see below.</p>
<p>If you test <strong>really needs to use some fonts</strong>, make sure you use the <em>Ahem</em> font as it is a very portable font. Make sure to specify the font-height too not to depend on the platform for that (FIXME: not sure if the font-size is really needed but better safe than sorry).</p>
<h4>
Native controls</h4>
<p>Native controls cover anything that is painted by the OS. In WebKit, this means buttons, scroll bars, media players, ...</p>
<p>You can use the previous tips for text here to hide your button if you don't need to show it.</p>
<p>For scroll bars, you just have to use the following rule in your CSS to disable them:</p>
<div style="margin-left:21.0pt;">
<pre>
overflow: hidden;</pre>
</div>
<h3>
How to disable render tree dumps but keep the pixels</h3>
<p>If you want to just dump the pixels but don't care about DRT's text dump. You can use the following trick:</p>
<div style="margin-left:21.0pt;">
<pre>
if (window.layoutTestController)</pre>
<pre>
layoutTestController.dumpAsText(true);</pre>
</div>
<p>This is <strong>not</strong> a typo. It is an extremely confusing syntax but this disables the text dump (you basically dump an empty text file).</p>
<h3>
How to disable the pixel dumps but keep the render tree dump</h3>
<p>There is currently no way for you to do that.</p>
<h1>
Writing JavaScript-based DOM-only Test Cases</h1>
<p>When writing test cases that only test the DOM it is still preferred to use an .html file since it only requires one test file instead of two.</p>
<p>When writing these tests it is often useful to include <a href="http://trac.webkit.org/browser/trunk/LayoutTests/fast/js/resources/js-test-pre.js"> LayoutTests/fast/js/resources/js-test-pre.js</a> and <a href="http://trac.webkit.org/browser/trunk/LayoutTests/fast/js/resources/js-test-post.js"> LayoutTests/fast/js/resources/js-test-post.js</a>.</p>
<p>To see what sort of special js functions are exposed to js-only tests, see <a href="http://trac.webkit.org/browser/trunk/LayoutTests/fast/js/resources/js-test-pre.js"> LayoutTests/fast/js/resources/js-test-pre.js</a></p>
<h1>
DumpRenderTree JavaScript Environment</h1>
<p>DumpRenderTree exposes a number of additional JavaScript objects in the testing environment.</p>
<p>These can be used to perform additional debugging-related tasks.</p>
<h2>
<tt>window.</tt>layoutTest<tt>Controller</tt></h2>
<h3>
<tt>dumpAsText()</tt></h3>
<p>Call this method to make your test output plain text instead of a render tree. This is useful if your test prints messages rather than testing fancylayout. For an example of how to print to a console in a test, check out <a href="http://trac.webkit.org/browser/trunk/LayoutTests/fast/dom/Element/attribute-uppercase.html"> LayoutTests/fast/dom/Element/attribute-uppercase.html</a>.</p>
<h3>
<tt>waitUntilDone()</tt> and <tt>notifyDone()</tt></h3>
<p>By default, DumpRenderTree dumps each test file immediately after the document has loaded and the load event handlers have executed. If yourtest needs to do further processing after loading -- for example, waiting for a timer to fire -- call layout<tt>TestController.waitUntilDone()</tt> to tell DumpRenderTree to delay its dump, and then call <tt>notifyDone</tt> when your results are ready.</p>
<h3>
<tt>overridePreference(key, value)</tt></h3>
<p>Changes a preference with name <em>key</em> (for example <tt>"WebKitEnableCaretBrowsing"</tt>) with <em>value</em> (for example <tt>"1"</tt>) for the duration of the test. The preference is reset when the test ends.</p>
<h3>
<tt>setCanOpen</tt>Windows<tt>()</tt></h3>
<p>If your layout test needs to open pop-up windows, call this method before it does.</p>
<h3>
<tt>clearBackForwardList()</tt></h3>
<p>Clears the back/forward list (i.e. history).</p>
<h2>
<tt>window.eventSender</tt></h2>
<h3>
<tt>mouseMoveTo(x, y)</tt></h3>
<p>Used to change the current mouse position.</p>
<h3>
<tt>leapForward(ms)</tt></h3>
<p>Jumps the current event time forward by a specified number of miliseconds.</p>
<h3>
<tt>mouseDown([buttonNumber [, modifiers]])</tt></h3>
<p>Sends a mouseDown event to the WebView at the current mouse position.</p>
<p>buttonNumber; 0:left button, 1:middle button, 2:right button.</p>
<p>modifiers; See keyDown().</p>
<h3>
<tt>mouseUp([buttonNumber [, modifiers]])</tt></h3>
<p>Sends a mouseUp event to the WebView at the current mouse position.</p>
<h3>
<tt>keyDown(character [, modifiers]])</tt></h3>
<p>Sends a keyDown event to the WebView.</p>
<p>modifiers: An array of strings. A string should be a modifier key name in the followings:</p>
<ul>
<li>
<tt>"ctrlKey"</tt></li>
<li>
<tt>"shiftKey"</tt></li>
<li>
<tt>"altKey"</tt></li>
<li>
<tt>"metaKey"</tt> (Command key in Mac)</li>
<li>
<tt>"addSelectionKey"</tt> (equivalent to metaKey in Mac, ctrlKey in Windows)</li>
<li>
<tt>"rangeSelectionKey"</tt> (equivalent to shiftKey in Mac and Windows)</li>
</ul>
<h3>
<tt>enableDOMUIEventLogging</tt></h3>
<h3>
<tt>fireKeyboardEventsToElement</tt></h3>
<h3>
<tt>setDragMode</tt></h3>
<h2>
<tt>window.GCController</tt></h2>
<h3>
<tt>collect()</tt></h3>
<p>Performs JavaScript garbage collection.</p>
<h3>
<tt>collectOnAlternateThread(wait)</tt></h3>
<p>Performs JavaScript garbage collection on an alternate thread. The <tt>wait</tt> argument specifies whether script execution waits for garbage collection to finish.</p>
<h2>
<tt>window.textInputController</tt></h2>
<p><em>Needs to be filled in.</em></p>
<h3>
<tt>insertText</tt></h3>
<h3>
<tt>doCommand</tt></h3>
<h3>
<tt>setMarkedText</tt></h3>
<h3>
<tt>substringFromRange</tt></h3>
<h3>
<tt>attributedSubstringFromRange</tt></h3>
<h3>
<tt>firstRectForCharacterRange</tt></h3>
<h3>
<tt>characterIndexForPoint</tt></h3>
<h3>
<tt>makeAttributedString</tt></h3>
<h2>
<tt>window.appleScriptController</tt></h2>
<h3>
<tt>doJavaScript()</tt></h3>
<p><em>Needs to be filled in.</em></p>
<h2>
<tt>window.navigationController</tt></h2>
<p><strong>The navigation controller is currently broken.</strong> <a href="http://bugs.webkit.org/show_bug.cgi?id=11042"> http://bugs.webkit.org/show_bug.cgi?id=11042</a></p>
<h3>
<tt>evalAfterBackForwardNavigation(script [, destination])</tt></h3>
<p>To test a bug having to do with the loader or the back/forward cache, call this method to run a script after executing a back/forward navigation. The first argument is the script to run, and the second argument is the page to load during the navigation. The second argument is optional. It defaults to <tt>about:blank</tt>.</p>
<h2>
<tt>window.internals</tt></h2>
<h3>
<tt>createShadowContentElement(Document)</tt></h3>
<p>Creates a <tt><content></tt> element for use in shadow DOM. These elements can not be created from JavaScript, hence this constructor in the testharness.</p>
<h3>
<tt>elementRenderTreeAsText(Element)</tt></h3>
<p>Gets and returns the element’s renderer’s description of its tree as a String.</p>
<h3>
<tt>ensureShadowRoot(Element)</tt></h3>
<p>Given a host element, returns its shadow root node. If the element doesn’t have a shadow root one is created and attached to the element. If you want to just retrieve a shadow root without creating one, use <em>shadowRoot</em>. <em>ensureShadowRoot</em> only inspects DOM shadows; not SVG shadows.</p>
<h3>
<tt>isPreloaded(Document, String url)</tt></h3>
<p>Gets whether the specified document’s cached resource loader has the specified URL preloaded.</p>
<h3>
<tt>removeShadowRoot(Element)</tt></h3>
<p>Given a host element, removes its shadow root if it has one. <em>removeShadowRoot</em> only inspects DOM shadows; not SVG shadows.</p>
<h3>
<tt>shadowPseudoId(Element)</tt></h3>
<p>Gets the specified element’s CSS pseudo-id for styling when in shadow DOM.</p>
<h3>
<tt>shadowRoot(Element)</tt></h3>
<p>Given a host element gets its shadow root, if it has one; otherwise <em>shadowRoot</em> returns <tt>null</tt>.</p>
<h1>
Writing tests which require network access</h1>
<p><tt>run-webkit-</tt>test<tt>s</tt> (the script which runs DumpRenderTree) also launches a local Apache daemon (<tt>httpd</tt>) to allow real local-only network basedtesting (for incremental loads, XMLHttpRequest, etc.) ap needs to document how best to use this.</p>
<p> </p>
Add new comment