The Web Testing Companion: The Insider's Guide to Efficient and Effective Tests

Lydia Ash

Appendix H - Questions and Checklists

Taking information from all the previous chapters, this appendix compiles very detailed and encompassing lists of questions for verifying your site or application. These questions need to be asked of the specifications before the code is written, but they need to be asked again when you are testing. Asking these questions before coding has begun helps identify problems while they are still very cheap to fix, but it also helps to identify where information is missing and more details need to be determined.

Note: Not all of these will apply, and your application will raise many not listed here, but these questions are a starting point. Always ask yourself the question, "What's missing?"

General

• How will the application be used?
• What is the purpose of the application? Information, entertainment, business, and so on.
• What is the target audience? Who is the current audience? Who is the potential audience?
• What is the user scenario for using it? There may be several representing various classes of users.
• Will this be used from an Internet kiosk or other machine that the user has no control over? Should the application be verified in kiosk mode?
• Are alternative devices going to use this? (Web TV, PDAs, iMode phones, and so on.)
• What is the estimated user base for the first 6 months? First year?
• Are users paying to access your site or use your application? If so, their expectations may be very different than if it were free.
• Are there partners you are dependent on? Are partners dependent on you? Is there a plan if either your service is down or a partner you rely on is down?
• Is there a new user or first-time user experience that differs from subsequent visits?
• What dependencies does your site/application have? It may require that WinAmp, AOL Instant Messenger, or some other piece of software be installed, or it may make use of system- or browser-provided DLLs. If it does not rely on any system DLLs, then there may be no need to test on various versions of each platform family.
• Does your application install any components or controls? What if the user cannot/does not install it? What if you release a new version-is there an upgrade path for the users?
• If your application installs components or controls, where are they installed? Is there an assumed or hard-coded location? Will that break on a nonstandard configuration (for example, a Windows machine where the active drive is the F:\ drive instead of the more common C:\ drive)
• If there is a component or control installed, is there an uninstall path for it?
• Does the component or control that is installed set any registry keys? When it uninstalls, all of these need to be cleaned up appropriately.
• What platforms is this supported on? What are the various Windows, Macintosh, and Unix versions, as well as various devices?
• Which browsers are supported? Microsoft Internet Explorer, Microsoft MSN Explorer, Netscape Navigator, Microsoft Pocket IE and Generic IE, Opera, Omni Group Omni Web, Amaya?
• Are there varying levels of support?
• What does the test matrix look like?
• What does the support matrix look like?
• What happens if you access the application from an unsupported platform? Consider a notification mechanism.
• What other requirements does the application have-security settings, script settings, cookie settings, resolution, line speed, and so on?
• Is there an internal coding standard?
• Is this feature necessary?
• Is the role of the feature in the application clearly understood?
• How will a user access this piece of functionality? List all ways to get into this feature.
• How will the user put data into this piece of functionality?
• What are those areas of user-defined input? What are the sizes of those areas (in bytes or characters)? What type of input do they accept?
• Where is the data output displayed?
• What boundaries exist for this data?
• What error states can come of this data or action?
• What states is the data available in? Map the creation, edit, save, display, and delete states.
• How is the user expected to navigate? Are Back and Forward controls provided in the application interface or is the user expected to use the browser buttons? Make this clear, but know what the behavior is when using the browser buttons.
• If the application makes use of frames, what should the behavior be when the user clicks the browser Back or Forward buttons? What about Refresh or Stop?
• Do errors occur when the page loads?
• Do all graphics appear properly?
• Are all similar icons of the same size (visually), giving a clean appearance?
• Is the download time perceptibly slow?
• Do all graphics have width and height tags to correctly place them into a formatted page while they are downloading?
• Are all links active and working? Are they accurate and/or descriptive?
• Are links relative or absolute?
• Should all links or active (hot) areas have the finger pointer (hand) displayed when the cursor is over them?
• Do the mailto links also display the mailing address?
• Can the user print various pages?

Usability

• Does the user interaction for this feature make sense? Will the user easily figure out how to use it?
• Are Help files provided?
• Are error messages to the user helpful?
• Will server errors (such as 404s) be given a more usable interface when delivered to the user?
• Can the user easily find the necessary navigational controls, or are the application controls easily confused with the browser's controls? (For example, if users are looking for the Help pages for your application, do they instead click the Help for the browser?)
• What is the result of each click? Where does it navigate to?
• What happens if the user right-clicks? Is this disabled or is there a special menu that appears? Is it supported?
• Is the information architecture clearly mapped out with content topics mapped to it?
• Would the user try to bookmark the page? Can the user return to a bookmarked page of yours? What if the page has been removed? There should at least be a polite general statement and links to the main navigation page of your site/application.
• Will there be usability studies performed with typical users prior to the release?

Accessibility

• After reading Section 508 (the Rehabilitation Act Amendment [RAA]) of the Workforce Investment Act of 1988, do you notice particulars that apply to your application?
• Will your application be made Section 508 compliant?
• Can the application be used without a mouse? This means keyboard-only navigation and the use of the Tab key to navigate. Try placing the mouse behind the computer to force you to try this.
• Is the tab order through the application logical (TabIndex attribute)?
• Does the focus default to the most logical place when a user enters a page?
• Are there any keyboard shortcuts in the product (AccessKey attributes)?
• Are these shortcuts unique on each page compared to any other access key, browser hotkey, or system command? You cannot have duplicates.
• Do all pages, tables, columns, frames, forms, graphics, links, animations, applets, and controls have a descriptive real-language label (ALT text and TITLE text)?
• Do all tables have header attributes? (<TH> with a real-language description.)
• Do all links, buttons, and graphics have tooltips?
• Do all images used as anchors for links have a text-based alternative?
• Is the application able to function with all graphics turned off? With the sound turned off?
• Can the application function when the user has turned off style sheets in the browser?
• Is color or sound the only method of communicating information to the user in any place?
• Are the user's specified colors of fonts, links, and the background applied?
• Are the user's specified font face and size applied?
• Can the application be used while in High Contrast mode?
• Does the application or site provide the appropriate properties for the controls in MSAA?
• If any text edit boxes are used, can you change the value in the box and have MSAA recognize the new input?
• What level of MSAA is the application standardized to (v1.3 or v2.0)? The standardization is a good marketing point and worth at least a mention on the Web page. More information on Microsoft Active Accessibility can be found at http:/www.microsoft.com/enable/.
• Do included video or animations have closed captioning, narration, or text equivalents?

Localization

• What languages will the application be localized into?
• When you view each localization, are all the strings localized? Are all the strings supposed to be localized? (Some strings, such as the company name, may not be translated.)
• If you provide more than one language, how do you determine which one to show to the user? Read the browser's accept language? A user preference when they register?
• Will the localizations from a previous release be reused?
• Will server errors (such as response codes-404) be localized?
• Is there country-dependent formatting?
• Sorting
• Number formats (thousand place separator, decimal, and negative)
• Numerals-ordinal and cardinal numbers
• Units of measurement (U.S. versus metric)
• Currency (symbol and placement and the new Euro-some contain more than one symbol; place the symbol before the amount and after as well as with leading and trailing 0s and spaces)
• Calendar (Gregorian, Lunar, Hebrew, Thai, Korean, Japanese, Chinese)
• First week of year
• First day of week (Sunday or Monday)
• Date format and separator (mm-dd-yyyy or dd-mm-yyyy or mm-dd-yy, and using a period, dash, or slash to separate the fields)
• Time format (12-hour clock versus 24-hour clock and a.m./p.m. formats)
• Punctuation (quotes, exclamation marks, question marks, list separators)
• Address formats
• Name formats (last name last versus last name first)
• Phone number formats
• Is there language-dependent formatting such as spell check or sort order? (for example, in Finnish or Swedish, the word waffle should be sorted ahead of the word vegetable).
• Will complex scripts be supported? Are there different behaviors that should be observed here?
• Does the interface display correctly with RTL localizations?
• Does the interface properly display the extended ASCII range for each language you localize into?
• Does the interface properly display double-byte characters?
• Does the interface properly display Unicode-only characters?
• What encodings are supported? (particularly for Russian [Cyrillic versus KOI-8] and Japanese [JIS versus SJIS])
• Does your application specify fonts to be used? If so, make sure that the fonts you hard-code are able to display the language you are localizing into.
• Will this application be code page dependent?
• Will the application be pseudolocalized prior to release?
• Will all localizable strings be placed in a central location?
• Do all graphics and symbols make sense for the language? Many symbols, graphics, and images do not carry the same meaning in different cultures, for example, flags should not represent a language, and icons such as a mailbox may not exist in all regions.
• Are there other internationally sensitive issues-maps, country names, and so on?

Globalization

• What languages will users be allowed to input information in?
• Will complex script input be supported? Are there different behaviors that should be observed here?
• What encodings are supported? Not just code-page encodings, but Unicode UTF format and conversions as well.
• What code pages are supported? Microsoft? Mac? OEM?
• Will this application be code-page-dependent?
• Does your application need to do any escapings of the input?
• Does the application accept extended ASCII, double-byte, complex script, RTL, or Unicode input?

Performance

• What are the performance criteria that the application needs to meet? (Page load time over various line speeds, bytes over the wire, server-side performance issues, client response times, and so on.)
• What are the baseline numbers for these, as determined by testing?
• What is the typical customer bandwidth? Make sure that metrics are taken with this.
• Will there be automation to quantify the performance numbers?
• Will this automation require developers to provide performance markers in the code? Are these documented?
• Can the server configuration be changed to set the caching on images higher, allowing users revisiting the site from the same machine the ability to use the images from cache instead of downloading them from the server?
• Have all the graphics been optimized for the Web?
• What operations are the most expensive?
• What do the most expensive operations cost?
• What do the cheapest operations cost?
• Is there a base cost that cannot be escaped; any fixed overhead costs?
• What is the cost (in megahertz or other value) of a typical user scenario?
• Is there any instrumentation being used to calculate performance?
• Are other tools being used to calculate performance?
• What are the metrics for quantifiable entities such as:
• Performance versus users
• Performance versus time
• Transactions per second
• Errors per user interaction
• Errors per unit of time
• Kilobytes of data per second
• Average amount of data downloaded per user visit
• Round-trip time
• Round-trip count
• Client response time
• Bytes per request
• Bytes over the wire per transaction
• Has the application been profiled when performing at peak capacity-75 or 80 percent CPU utilization?

Security

• What security problems were present in the previous version? Examine these and understand how they could have been prevented. Make sure that these and any new ones are all documented through the development process.
• Could a malicious user get the server to respond or send an error that gives away information about the server (name, IP address, and so on)?
• When connected to the application/site with IE, can a user go to File --> Edit with Microsoft FrontPage or File --> Edit with Microsoft Word or File --> Edit with Notepad to alter and save the files back to your servers?
• Are cookies used to store user state or other user information? What features read or set a cookie?
• For each cookie, is it persisted or session only? Is it marked as HTTP only (IE6 feature)?
• What if the user does not accept the cookie? What if the user has cookies set to Prompt and then accepts the cookie?
• What information is stored in the cookie? What happens if the user edits any of this information?
• Is any user information passed in a URL (parameter, path, and so on)?
• Are any forms being submitted to the server using the GET method?
• What information do we have of the users? Is any of it sensitive (credit card, SSN, address, and so on)? How could it be used to their detriment? Make sure that this information is not passed in cookies or in the parameters in the URL.
• Do we send any of the sensitive information in the body of the request? If so, could a malicious user grab it with a packet sniffer?
• How could any of these pieces of information be put together to be used maliciously?
• What information is the user expecting to be kept secure? (A user ID may not be something that the user has an opinion about, but in answer to the previous question, having the user's ID plus a time stamp may allow a malicious user to guess at a user's email locations, as the Hotmail bug did.)
• Can a malicious user deny a valid user access to his own data?
• Could a malicious user alter or corrupt data?
• What software do you depend upon? (Windows NT, IIS, Sun Solaris, Apache, SQL, and so on.) What vulnerabilities do these pieces of software have? Make sure you are up to date on all patches and security releases. Monitor BugTraq at http://www.securityfocus.com to keep up with recent exploitations.
• What security problems do your competitors' or other similar software have? How can you avoid them? Again, some of this might be available at http://www.securityfocus.com, but also on other sites.
• How are your servers configured? What permissions are set on those directories and files?
• Are users required to log into the application? What if they are behind a firewall or proxy server?
• Does your application perform any encryption? Does it use SSL?
• Does your application generate passwords for the user? Are they easily guessed?
• Does your application allow users to select their own password? Are there guidelines that they must follow?
• Are there any points where data could be shared or accessed by users other than the creator?
• Is this written in managed code? Are there any unsafe functions? Are there legacy pieces or other components that are unmanaged? Where are the transitions from managed to native code?
• Could a malicious user set up scripts to guess at a password 10,000 times until it is guessed correctly? What safeguards could be put in place to safeguard against that?
• Could a malicious user flood your application with requests? Do you want your application to try to respond to them all, or are safeguards in place for that?
• Could a malicious user send a malformed request? What does your application do with it?
• Could a malicious user send a NULL request? What would your application do with it?
• Could a malicious user send a very large request? What would your application do with it?
• Could a malicious user send a small request and tell the server it was a large request? Does the server hang?
• Could a malicious user open many connections to your server? What would your application do about that?
• What ports are open? How could these be exploited?
• What is the response to a denial of service attack?
• What buffer overruns could exist in the software? You do not have to go through the work of identifying how they can be exploited as long as you fix them when you find them.
• Is there a data conversion in this feature or between two features?
• Is data parsed in this feature or between two features?
• Does your application try to write to the server? Could a malicious user exploit this by convincing the server to write more data than it has space for?
• What kind of logging does your application do on the server side? Does it make note of attacks on the servers, and are they descriptive enough to help identify attacks? Are these logs too granular or too general?
• Is there a user-side cache? Does the application rely on it?
• Always assume that the worst happens. How will the server recover? Have a plan in place for each scenario.
• What gets cached on the client side?
• Do cookies contain sensitive data or data that can be combined to be sensitive?
• Should the code stipulate Cache-control: no-cache headers?
• How can you programmatically access the data in the SQL database?
• Is there a way to do this from the outside?
• Can the administrators or product support personnel read the information in the database? HTML
• Which HTML version will you create your application with and standardize to (v 3.2 or v 4.0)? What about CHTML?
• What file extension will be used-.htm or .html?
• Is there a default file set up for your folders? This default file will allow a user who tries to navigate to a folder directly to have something to navigate with. What is the name of this file (default.htm, index.htm, home.htm, or something else)?
• Are there tags supported in one browser and not another that you want to take advantage of?
• Are there attributes supported in one browser and not another that you want to take advantage of?
• Will you provide a separate code path for various clients based on their levels of support for HTML tags or attributes?
• Does your application make use of any other technologies such as CSS? What happens if no styles are applied or custom styles are applied?
• Can the client request the page and have something appear?
• Do the graphics show up in their proper place?
• Are the graphic dimensions too large, forcing users to scroll horizontally or vertically?
• Are table dimensions too large, forcing users to scroll horizontally or vertically?
• Can the image's file size be reduced to improve the download time, either by saving it in a different file format or reducing the number of colors in the graphic?
• Are there extra images that are unnecessary?
• Are there images that can be reused throughout the site?
• Are WIDTH and HEIGHT attributes associated with each graphic?
• Is an ALT text supplied for each graphic? Is it useful?
• If the client system is reduced to 256 colors, does the application render too poorly? What about in High Resolution mode (for flat screens)?
• Are the plaintext files of the HTML code bloated due to extra tags? Should these be removed for cleanliness and performance?

Script

• Is script used? Is server-side script implemented? Client-side script? Both?
• Is JavaScript, Jscript, or VBScript used? If JavaScript is used, is it ECMA-262/ISO/IEC 16262 compliant?
• Does your script check to make sure it is fully loaded before any references are made to the document object model? (This can be particularly bad if used in conjunction with frames because there is a larger opportunity for them to be out of sync.)
• Is all script commented out so browsers without scripting engines will ignore it? Can these users still access some amount of helpful information?
• What happens when the application is run maximized? Less than maximized? Minimized?
• If you resize the browser so that the window, frames, or controls will not have enough room to load and be displayed, are there script errors when they are navigated to?
• If there is client-side data validation, what happens if the data is input in a different order than the application is expecting? (For example, zip code then state.)

Automation

• Will there be automation written to test this product? This feature?
• Do the developers need to provide any hooks to allow for automation? (For example, element IDs for tags to identify objects.)
• Are the most important test cases automated?
• Is there API-level automation that should be done calling directly into the functions that provide no UI?
• Are the typical user scenarios automated?
• Does some part of the automation rely on random data?
• Is the wait time (latency) between user interactions also randomized?

ASP

• Are Active Server Pages (ASP) used in this application?
• What is the performance of these pages?
• Where is the performance bottleneck in your pages?
• Should your pages check if machines are still connected when expensive pages are requested?
• Can the number of requests or the data transferred be minimized further?

Other Technologies

• Are ActiveX controls used? If they are planned in the product, is this product intended to be used anywhere except for Microsoft Internet Explorer on Windows?
• Do the ActiveX controls allow code to be executed? Do they read or write files? Could they be made to?
• Do the ActiveX controls create or delete any persisted data?
• Do the ActiveX controls touch any system files or registry settings? (For example, reading or modifying.)
• Are any other objects created?
• Does the object ShellExecute?
• Does the control expose any personal information? (Think about file names, user login name, paths, and so on.)
• Are all of your ActiveX controls signed?
• Are all of your ActiveX controls virus checked?
• Are Java applets implemented?
• Is CGI used? If not, can it be disabled on the server?
• Are XML requests being sent?
• Can a malformed XML query come from the client?
• What happens if an XML request is sent to the server without the closing tag </...>? Does the server hang waiting for the end of the request?

Server-Side Issues

• What is the required stability and uptime for this application?
• What type of servers are being planned for deployment? Microsoft Internet Server (IIS)? Netscape Web server? Apache Web server?
• What server-side technologies are being planned? ISAPI? CGI? ASP? Scripts? Make sure these are compatible with the server platform planned.
• What is the server architecture?
• What is the server technology?
• Are there other servers involved? (Hardware load balancers, routers, hardware SSL, and so on.)
• What software dependencies does your application have? (OS, components, third-party components, and so on.)
• Is there a test lab for you to test the application in? Does it accurately mirror the intended deployment topology?
• What ports are left open? Are these necessary?
• Is a content expiration set on items? Should some items be cached and others not?
• Is the content expiration set in the response header or in the META data for the page?
• How many users total are expected? How many simultaneously?
• How does the application scale? Up or out? What is the scaling unit, and how many users are supported there?
• What is the anticipated traffic?
• How are backups, restores, failovers, and disasters handled?
• Is there any user management? How are they managed?
• How are the live boxes administered?
• Where is the bottleneck? Is it acceptable? (Disk I/O, network, memory paging, database, and so on.)
• What memory leaks exist? Have you looked for them?
• Is server-side caching implemented?
• Is the architecture redundant and distributed?
• Are server logs analyzed while testing?
• Is there a plan to dogfood this application or have a progressive roll-out to users?
• What happens if the server hard drive space on servers fills up?
• Could a malicious user constantly fire errors into the log to attempt to fill up the hard drive space?
• How are requests received from the client? ASP, ISAPI filters, CGI?
• What user context does the receiving code run under? (Administrator, sa, local system, local user, or so forth.)
• Is the communication between the client and the server (or the server and the other servers) in plaintext or encrypted?
• How does the server know the user is who he says he is?
• How does the user know the server is who it says it is?
• Is data stored securely on the server?

Client-Side Issues

• Can users behind proxies and firewalls use the application effectively? Certain requests can be refused by some older firewalls.
• Is client-side caching relied upon?
• Are there any temp files created? This question could also apply to the server side.
• Do any files created rely on a particular name? What if another file of that name was already created? What if a folder of that name has been created there? I have seen programs handle assumed file and path names well and increment a number on the end. Try creating a file and adding those numbers, or the DBCS number, on the end.
• What are the permissions on these files?
• Does your feature allow scripts to run?
• Can anything be embedded in input areas? (Images dropped in, files, controls with code behind them, and so on)
• Is data stored securely locally?