An application/script demonstrating bugs in the product, and possibly also how to use the product. It is intended that users edit the application/script in order to demonstrate the bug to developers - it is a form of TestFirst AcceptanceTest.
An example might be that you are building a library for use in an application. A sample application is often built to demonstrate how to use the library, but the 'touchstone' concept takes this as a starting point, and lets users submit patches to the application to show off bugs.
This is similar in concept to automating AcceptanceTests, but it is not required that the actual test is automatable, only the demonstration of the problem.
The term comes from ApacheMaven, http://cvs.apache.org/viewcvs.cgi/jakarta-turbine-maven/src/test/touchstone-build/, but other examples have been seen. Touchstone can mean "a basis for comparison; a reference point against which other things can be evaluated". [http://dictionary.reference.com/search?q=touchstone&r=67]