implement a test harness for 0.9 which does not rely on a build system

[alex]

Dojo 0.9 must include a test suite that anyone can run from the command line or from a web browser without any build system or setup other than a checkout of the repository

[alex]

(In [7524]) refs #2550

[alex]

still needs a browser specialization and a browser harness

[skinner]

It would be great if the new test harness made it easy to test dojo functions that operate asynchronously. Back a few weeks ago we made a tentative decision to change the dojo.data APIs so that they support *only* async access, but the current JUM-based unit tests rely mostly on sync access.

Also, it would be great if the new test harness allowed for grouping a set of tests. Ideally a group of unit tests should be able to share some common helper functions, so we can avoid either (a) having the common helper stuff repeated in 10 different tests DRY, or (b) having the ten different tests glommed together into one mega-test, or (c) having the 10 tests rely on helper functions that are defined in a global namespace. (We accidentally got burned a couple times in the dojo.data unit tests, when two different unit test files both defined the same globally scoped helper function.)

[alex]

(In [7589]) implements async test harness. Refs #2550

[alex]

(In [7624]) adding getTestCallback() method as discussed with Brian Skinner today in IRC (already doc'd in the porting guide). Almost have the test URL story figured out. Refs #2550

[alex]

brian,

so you're looking for something like a test suite concept? I guess I didn't implement it based on a felling of YAGNI since you can register anything (including an object, all of whose methods are tests). Would it be sufficient to give test groups setUp and tearDown methods somehow?

[skinner]

Hey Alex,

Just as an example of what i'm looking for, you might want to check out our existing unit tests for RemoteStore?: http://trac.dojotoolkit.org/browser/trunk/tests/data/core/test_RemoteStore.js

The test_RemoteStore.js file defines 5 functions that are unit tests (the functions with names that start with "test"):

	test_data_core_empty()
	test_data_core_movies_sync()
	test_data_core_movies_async()
	test_data_core_movies_revert()
	test_data_core_movies_save()

The file also includes 3 helper functions, which are used by the unit test functions:

	_test_data_core_newStore()
	_test_data_core_load_movies()
	_test_data_core_movies()

Each of the helper functions is called by two or more of the unit test functions. The helper functions have names that begin with "_" to prevent the test framework from considering them to be test functions and automatically running them.

Unfortunately, the test framework loads this test file into memory at the same time as all the other dojo.data test files, so the helper functions end up defined in the single global namespace that all the dojo unit tests are running in -- to prevent bugs caused by name collisions, the helper functions have painfully long names. The new test harness solves that problem by making it easy to use namespaces like tests.foo.bar.

So what I'm hoping for is just some way of doing the same thing in the new test harness but in a less hackish way -- some way of having a test file that defines 5 actual unit test functions as well as 3 helper functions that are available for the unit test functions to use.

I guess maybe this problem could be solved if a test group had setUp and tearDown functions, as you suggest. Then the setUp function could simply define the 3 helper function. But wouldn't the setUp function end up getting called 5 times, once for each of the unit test functions? That wouldn't be the end of the world, but it would be cleaner if the 3 helper functions only got defined once.

On a related note, here's a guy who argues that you don't really want setUp and tearDown functions at all, just helper functions:

http://www.agileprogrammer.com/dotnetguy/articles/SetupTeardown.aspx

And, also related, here's an article that talks about the usefulness of having a facility for FixtureSetUp? and FixtureTearDown? functions that only get called once for the entire test group, instead of (or in addition to) simple setUp and tearDown functions that get called once per test function. I don't have a ton of experience with unit testing, but to me the idea of FixtureSetUp? and FixtureTearDown? seems more useful than the idea of setUp and tearDown.

[skinner]

Doh, forgot the last link: http://www.codeproject.com/csharp/autp4.asp#Fixture%20Setup%20And%20Tear%20Down15

[alex]

(In [7628]) updating with Brian Skinner's excellent suggestions. Refs #2550

[alex]

(In [7631]) changing the names to assert* from is*. Refs #2550

[alex]

(In [7634]) un-eff the browser harness in IE. Refs #2550

[alex]

(In [7635]) ensuring that logging output shows up in the log tab. Refs #2550

[alex]

(In [7646]) port the command-line tests to run using the most primordial Dojo package loader. Refs #2500. Refs #2550

[alex]

(In [7647]) keep the test system from conflicting w/ Dojo when run in a browser environment. Refs #2550

[alex]

(In [7653]) ensure that all tests get loaded in from tests/_base.js. Refs #2550

[alex]

(In [7654]) ensure that the console is cleared for each test run. Refs #2550

[alex]

(In [7655]) ensure that the play/pause button is in the right state. Refs #2550

[alex]

(In [7697]) start logging error objects directly. Helps us get actual failure info out of exceptions. Huzzah. Refs #2550

[alex]

(In [7702]) fix test-name-finding on Opera. Refs #2550

[alex]

(In [7703]) get debugging output working correctly on the webkit nightlies. Refs #2550

[alex]

(In [7704]) output how many tests need to be run. Refs #2550

[alex]

(In [7847]) adds support for:

• registering URLs
• loading of tests in loaded URLs
• automatic timeouts for loaded URLs
• automatic test creation from code strings
• improvements in use with dojo
• group-level setUp() and tearDown()

Also includes myriad bug fixes. Refs #2550. Still need to document many of these changes in the porting guide.

[alex]

(In [7879]) keep the system from throwing errors when a test page that includes dojo but doesn't have a parent harness loads it. We still get success/failure info at the console. Refs #2550

[alex]

(In [7880]) more fixes to prevent errors with test pages not hosted in harness. Refs #2550

[alex]

(In [7887]) adding sounds so that success and failure sound like...well...success or failure. Refs #2550

[alex]

(In [7889]) we probably don't want selfTest to always be on when not loading Dojo. Refs #2550

[alex]

(In [7890]) ensure that we actually play the "woohoo!" sound if it all goes well. Rate limits sounds. Refs #2550

[alex]

(In [7892]) creating t, f, and is aliases for assertTrue, assertFalse, and assertEqual (respectively). I'm getting really freaking sick of typing the long names. Refs #2550

[alex]

(In [7893]) a basic summarizer. Refs #2550

[alex]

(In [7900]) ensure that we dont' fail when loaded stand-alone. Refs #2550

[alex]

(In [7901]) prevent us from bombing out should we be loaded stand-alone. Refs #2550

[alex]

(In [7919]) ensure that object comparison tests can end successfully. Refs #2550

[alex]

(In [7990]) unfscking log output for Webkit and Opera. Refs #2550

[alex]

(In [7991]) make the log output slightly smaller. Refs #2550

[alex]

(In [7999]) ensure that paused groups resume correctly. Refs #2550

[alex]

Marking fixed. Further improvements to the system should happen under separate bugs.

[alex]

(In [8155]) now that we've got lots of test groups, roll them up but make them expandable. Refs #2550

[anonymous]

Milestone 0.9M1 deleted

[bill]

(In [9775]) Turn off sounds by default. Refs #2550.