Web Testing

Has anyone put any thought into a RegressionTesting library for HTTP applications? Or is this just another type of GuiTesting?

See also relevant stuff at ServletTesting and EjbUnitTest. -- RichardEmerson

It appears that Expect is the most suitable language for HTTP and client/server applications. -- WillSargent

See TclExpect, and the book "ExploringExpect??" on TheTclersWiki (http://mini.net/cgi-bin/wikit/105.html).

"Expect" is a TCL library that's good for driving text-based user interfaces from TCL scripts. It's used to script interaction with servers, and can be used for testing, and many other things.

See also "The DejaGnu Testing Framework" (using TCL/Expect) at ftp://ftp.cygnus.com/pub/dejagnu/dejagnu.html

I wonder if TCL/Expect is the best tool for the job; I'd prefer something that can interpret the HTML as a browser does. -- JeffGrigg

A test suite for Apache is one of the things HP is offering money for at http://www.sourcexchange.com/ -- which has been closed down.

I do this all the time in Perl, using LWP. Generally, I subclass LWP::UserAgent?? to get something like FooTester, and then write code like this: 

  use Test;
  BEGIN { plan test => 14 }
  my $t = new FooTester;
  $t->get('/');
  ok($t->is_success);
  ok($t->content =~ /<TITLE>Front Page</TITLE>/i);


  $t->post('/login', user => 'fred', passwd => 'xsecretx');
  ok($t->is_success);
  ok($t->content =~ /<TITLE>Successful Login</TITLE>/i);
  ok($t->content =~ /fred/i);
  ok($t->content !~ /xsecretx/i);  # don't leak passwords...
-- PeteBevin

At Amazon.com we use a tool that I wrote called Frank. It is a Web testing scripting language that uses a basic grammar to define test cases. It uses Perl regular expressions (although the latest version is written in Java) to define most of the assertions we use. Here is a simple example: 

  GET "/some/url.html";
validate urls;
validate images;
pass if regex m/Some Content/;
the validations generate more tasks like the one above and every task gets a bunch of "implicit tasks" that apply to everything by default. Some can be overridden (like status code checks, by default they check for 200 but can be made to check anything else if explicitly stated).

The latest version is built on top of a distributed task engine that allows us to test an extensive regression test in under 30 minutes. The previous version, which is written in Perl, is single-threaded and I'm trying to get in released as open source. We shall see...

-- RyanDavis

Check out PBP at http://pbp.berlios.de/ --titus.

I rely mostly on my HTML write stream class. It will catch misspelt tags, unbalanced tags, INPUT tags not inside FORMs, and suchlike as the code generates them. I believe it will avoid most of the problems a general parser would find, very cheaply, because it starts from a high level representation. Eg I have methods like #putSubmit:value: so the checker knows a lot about what is going on without having to parse it. And of course it is easier for me to use than typing all those fiddly angle brackets by hand.

I use some other tests, for example asserting that the generated text contains a <BODY> and </BODY> and ends with </HTML>, which gives me some confidence that text was actually produced. Apart from that I mostly care about aesthetics (which I don't know how to test for), and the concrete HTML is changing frequently (so I don't care about byte-for-byte regression), so I don't do any more. -- DaveHarris

I worked on Http Testing at my previous job (they canceled that project but still own the code *sigh*). Fortunately, most of what we used was open-source, so it wouldn't be too hard to reproduce. We would retrieve URLs with standard Java URL class, parse the results using the JTidy (JavaHtmlTidy) HTML parser, and apply tests to the resulting DOM, reporting the results using JUnit. Tests were in terms of expected results, at whatever granularity of the DOM is desired. Once you have the basic framework it is trivial to add features such as load testing, HTML validity testing, link testing, etc. -- DetheElza

Has anyone (in Windows) used JavaScript to invoke InternetExplorer and poke around in the page object model? Would it work?

I use OLEAutomation run from VisualBasicForApplications in an Excel spreadsheet to invoke and poke around in IE. IE now provides a standard DOM model, so once you have the root node it is straight-forward to walk the tree and either capture or compare -- GarethCronin

A new project called Web Testing With Ruby uses Ruby to drive web applications using the IE DOM. Check it out here: http://rubyforge.org/projects/wtr/ and here: http://www.clabs.org/wtr/ -- JonathanKohl

I've been using it too, it's called Watir now (http://wtr.rubyforge.org/), it does actually push buttons and write into text fields in IE, and then reflects the DOM so that you can ask "do I get what I expect here or there?". I find it very effective, both because you don't rely on someone's idea of how should a browser work, and because ruby excels at this sort things. --Loop

Take a look in IeUnit at http://ieunit.sourceforge.net. If you know DHTML & JavaScript you can start coding your web page test cases within minutes. -- JamesXinzhiLi

Another tool is IETest at http://ietest.sourceforge.net. You write tests in any .NET language, like C#. Includes support for testing modeless HTML dialogs. -- LarsThorup

Open Source Projects

HttpUnit - use the JavaUnit framework to write UnitTests in Java.

JwebUnit - builds on HttpUnit; requires less less coding for equivalent tests, in most cases. (Though I find that with recent versions of HttpUnit, I don't need JwebUnit anymore.)

HtmlUnit - alternative to HttpUnit

SimpleTest - Written in PHP, similar in style to JwebUnit.

OpenSta - a GUI based open source test suite. Use InternetExplorer to help make scripts. The product is free but you can buy a boxed version for $250. It supports SNMP (SimpleNetworkManagementProtocol).

CanooWebTest

XmlTestSuite - an xml tool for testing web applications (uses HttpUnit and jUnit)

IeUnit - an open source unit test framework for web based applications.

MaxQ - Can record/playback HTTP test scripts. Scripts are written in Python. (See MaxqTool)

List of Open Source Web Testing Tools: http://java-source.net/open-source/web-testing-tools

Commercial Products (very expensive)

http://www.thewebking.com/products/webking/index.htm - The Web King. I couldn't figure this one out before my trial key ran out. The demo version I tried had a pretty good link checker, but you need more than that to write unit tests. ($3500+ - I think they may have a cheaper version out now)

http://www.soft.com/evalid - eValid. You make scripts in IE and play them back. The demo version can do basic string matching. Not sure what the retail version can do as they haven't sent me a key yet. ($950-$10000)

Here's a recipe for Web testing on the cheap - put a bunch of these in your site makefile: 

  @echo 'test page X..' `(curl -sSL 'http://mysite/pageX' | grep -q 'some valid page X content' && echo ok) || echo FAIL
-- SimonMichael

I just wrote an article for JavaWorld on unit testing Java network client code. Covers two techniques: use InputStreams? intead of Sockets or URLConnections, and use a Java protocol handler to define a new URL namespace so you can test with "testurl:" instead of "http:" and control what happens. The article is titled "Test network code the easy way" and is at http://www.javaworld.com/javaworld/jw-07-2002/jw-0719-networkunittest.html

-- NelsonMinar

I find RegularExpressionMatchAssertions to be be very useful for testing Web output. It's too bad most StarUnit frameworks don't have such a thing built in.

MiniRubyWiki tests everything at the page generation level, down to the individual Wiki markup tags. It comes with a miniature server, but does not test this, nor does it test that your Web browser can download a page. I (PhlIp) keep a notebook hooked up to my desktop with a Web browser in it, and noodle on that after each test run. Not strictly TestFirstUserInterfaces canon, but much more pragmatic than learning HtmlUnit or hacking InternetExplorer.

So, if you use a CGI library that passes around lots of strings, and if you are fluent in RegularExpressionMatchAssertions, then you can code a lot of the Web without looking at it.

I've actually created a program in Python that simulates an Internet user. It's just a series of test cases that goes to specified URL's and downloads the content and runs searches on them. For example http://server/index.jsp?user=test&pwd=test. I then do a simple string search for words like 'exception' or 'error'. That way, if I make any changes in my code I can see how they effect all the different test cases.

Downside? It's slow. It has to contact an actual server. Data also has to be encoded in the URL using POST requests. Anyway, it's useful for small projects and it saved me a lot of time. -- BlakeMason

Tests don't live in a vacuum. See WebApplicationsAndXp for very correct team tips.

Moved from TestFirstDesign

Being new to XP, I'm wondering how i can create tests for ASP.NET web pages. Typically a web page would have some sort of control (e.g. datagrid, textbox) requiring some sort of user interaction. How do you create tests for this? -- WilleusAcuna?

First, you have NUnit for testing classes from the inner side. But, if you want to test them from the outer space, you will have to use something as htmlunit, httpunit, NUnit ASP.NET, JsUnit or something else is. Indeed, I tried to use any of theese, but I'm not satisfied. I didn't find any tool which provides complete functionality you need for testing aspx web pages from the outside. If anyone knows something which can test page content, javascript and has most of the common used http functionality, which does work with .NET, please do a post on the current page. aleksandar.naumovic@saga.co.yu

The acceptance testing FitWiki framework http://fit.c2.com can be easily used to acceptance test HTML pages. All you need is an appropriate fit fixture. Luckily there are 3 already (that I know of) - JwebUnit has one, a fixture called WATT that scripts Internet Explorer is included in FitNesse (http://www.fitnesse.org) - a download and run implementation of the fit framework, which also includes an HTML fixture that can be used to do DOM like element testing on a returned page. -- ChrisBarham

We parse web pages into a DOM tree and use XPath expressions. Our test classes commonly call methods like assertXPathExists(), assertXPathEquals(), or assertXPathContains(). We check that the functional parts of web pages are there - the links are there, the forms exist, the fields exist and are populated, and submit buttons are there - but not the purely visual boilerplate, which is better checked by looking at it.

Parsing the HTML is the slowest part of the unit tests - we're on the lookout for a faster parser.

CategoryTesting

EditText of this page (last edited July 9, 2006) or FindPage with title or text search