Ansi-suite to Lisp-suite

This page was created for the conversion of ansi-suite tests to the CLforJava? lisp-suite tests. This is a process which is time consuming and rather extensive. The lisp-suite is the main suite which runs all the smaller individual suites, which will be discussed below, which was originally created with intent on testing just a few functions, but has now grown as an effort of converting Paul Dietz' Ansi-Suite to a suite that can be run effectively in the CLforJava? environment.

History:
The lisp-suite is actually not quite that old and really has only been truly an actual test suite for about 3-4 months and was built off of the base of the previous deftest-suite. I actually began this project as just a way of adding a few tests from the ansi-suite to our own to test a few functions. This first began with the strings-suite and later I expanded the conses-suite and created the arrays-suite using similar methods. Each of these suites I put together with accessors, functions, macros, etc. from the HyperSpec? in their corresponding divisions. You can see these by looking at the Chapter Index on the HyperSpec? (aka. Conses = Ch. 14, Arrays = Ch. 15, etc.). After developing and getting these suites to run, I would then attempt to fix all errors I encountered.

After developing and expanding the conses, arrays, and strings suites, I decided to take another approach with the development of the numbers-suite and the characters-suite. These suites are entirely derived from the ansi-suite, with a few added tests. The added tests were based off of either the TODO file in the ansi-test section in the CLforJava? project or from the individual test files which I would attempt to re-write to fit the requirements of the CLforJava? lisp-suite. It is from these suites, I developed the idea of *-aux files to contain helper macros/function/variables/etc. and load these in the RunLispTest? .xml at the suite run-time. From here, I have chosen to expand the conses, arrays, strings, and sequences suites to accomplish the same sort of testing regimen. This is an ongoing process and should be expanded to include all sections of the HyperSpec? in the future. The ansi-suite is a very well developed testing suite, however, even Dietz himself has noted that the suite we currently possess is only 85% complete or so. Thus, as I have done with many tests already, there will need to be expansion to include a highly detailed test-suite in order to further the progress of the CLforJava? project.

Current Structure:
The current structure is actually very simple. I have reworked this several times until I was happy with the was things are run. I'll attempt to describe this below as best as possible, but if needed you can look into the following files I will list here. First of all, the RunLispTests? .xml does all the work. All of the *-aux files from the suite are pre-loaded before any of the suites run. This allows the suites to call the needed aux files to utilize their contents. There are also 2 main files which currently allow the suite to run as it does: test-driver.lsp and deftest.lsp. These are also compiled and loaded in RunLispTests? .xml. The test-driver creates the proper output for each test when sent to the outputted text file and the deftest file is the basic macro used to structure each test.

The tests are run in the following order (also following the order of the HyperSpec? chapter ordering):

-Syntax

-Evaluation-And-Compilation

-Types-And-Classes

-Data-And-Control-Flow

-Iteration

-Objects

-Structures

-Conditions

-Symbols

-Packages

-Numbers

-Characters

-Conses

-Arrays

-Strings

-Sequences

-Hash-Tables

-Pathnames

-Files

-Streams

-Printer

-Reader

-System-Construction

-Environment

-Misc

There are representative files for each of these suites which in turn call the other tests for each function/macro/etc. These are found in "test/src/lisp/" folder and are named *-suite.lsp (ex. numbers-suite.lsp). These files contain the names of all the test files in a variable list. There is then a call to a function in the test-driver (run-all-tests). What this does is it calls the specified directory and runs all the test files found in that directory. This bring me to my next point. All the tests for the corresponding tests are found in corresponding directory (ex. "test/src/lisp/numbers/" is the place to find all the numbers-suite tests). Anytime test files are added to this directory, they MUST be added to the variable list in the corresponding *-suite.lsp file.

Next, I will discuss the use of *-aux files. Each suite may or may not have a corresponding *-aux file found in "test/src/lisp/suite-aux-files/". The corresponding aux files are named to match the suite accordingly, ex. Conses-suite == conses-suite-aux.lsp. There is one aux file (lisp-suite-aux.lsp) that is of utter importance. This holds helper functions/macros/etc. for ALL the suites. This is used to hold the more generic helper stuff, whereas the more specific suite-aux files hold things specific to that suite.

How to Approach the translation from Ansi-suite tests to Lisp-suite tests:
There are several differences between the ways we do things in comparison with the ansi-suite. The main difference is the way we test things. For example, in each of our test files, the ONLY functions in these files should be those that use DEFTEST. Anything else will throw out errors. All other things must be place in the corresponding suite-aux file. There are some things that the ansi-suite uses in aux files or that are defined differently that can be translated. For example, if a function in the ansi-suite has an aux file with a specifically defined test, you can just add this INSIDE of a deftest in our files. I'll show you how this works:

Ansi-suite deftest structure:

(deftest fn-name.1 (body) result(s))

Our Lisp-suite deftest structure:

(deftest (fn-name-test-1 (fn-name)) () (body; this must return a boolean value))

NOTE HERE: the ansi-suite deftest compares the body with those results. Our deftest, on the other hand, just returns a simple boolean (t/nil). To make this happen, most tests will use either eq, eql, equal, or any equivalent boolean returning function. As long as the boolean is returned, the test can be determined to have succeeded or failed.

Stuff to Note:
I, myself, am a perfectionist of sorts and have a slight amount of obsessive compulsiveness when it comes to developing these tests. You can see this if you look at any of the suites. They are ALL structured with the same spacing and arrangements and also are all neatly sectioned together. There are also divisions within larger *-aux files. I don't mean to be picky, but to those who decide to follow in my footsteps, please attempt to maintain this consistency. It is easy to follow code when it is consistent and spaced equally. Remember, you are leaving code for others to look at in the future so please be considerate to them.

In regards to testing the accuracy of your tests (which you should do), DO NOT rely upon the accuracies of the ansi-suite. There are some tests that may be done incorrectly or have bad results.

In regards to testing your tests. Using LispWorks? is a great tool to test your tests. However, keep in mind, LispWorks? is NOT a bug free implementation either. Also, there are many, many things on the HyperSpec? that are implementation specific. You may want to ask Prof. Boetje before assuming anything to be the way LispWorks? does it. For example, nil-arrays and nil-vectors are handled COMPLETELY different in LispWorks? than CLforJava? . DO NOT rely solely upon LispWorks? accuracies. Be smart.

In my personal opinion, there should be an experienced lisp programmer working on this section. The ansi-suite is FULL of highly advanced lisp, which can sometimes be overwhelming. Even I do not fully understand everything in there. Take a breath and digest the wealth of information within the ansi-suite. Embrace the madness and use it to your learning advantage. There are some real brain tester and ball busting tests in there.

Suites and Completion Status:

Suite: ------------------------------------------------------------ Completion-status:

Syntax-suite ----------------------------------------------------- Active

Evaluation-and-Compilation-suite ------------------------------ Active

Types-and-Classes-suite ---------------------------------------- Active

Data-and-Control-Flow-suite ------------------------------------ Active

Iteration-suite ---------------------------------------------------- Written-not-active

Objects-suite ------------------------------------------------------ Written-not-active

Structures-suite --------------------------------------------------- Active

Conditions-suite --------------------------------------------------- Written-not-active

Symbols-suite ----------------------------------------------------- Written-not-active

Packages-suite ---------------------------------------------------- Written-not-active

Numbers-suite ---------------------------------------------------- Active

Characters-suite -------------------------------------------------- Active

Conses-suite ------------------------------------------------------ Active

Arrays-suite ------------------------------------------------------- Active

Strings-suite ------------------------------------------------------ Active

Sequences-suite -------------------------------------------------- Active

Hash-Tables-suite ------------------------------------------------ Active

Pathnames-suite -------------------------------------------------- Needs special pathname work

Files-suite --------------------------------------------------------- Needs special pathname work

Streams-suite ----------------------------------------------------- Active

Printer-suite ------------------------------------------------------- Written-not-active

Reader-suite ------------------------------------------------------- Active

System-Construction-suite ---------------------------------------- Active

Environment-suite ------------------------------------------------- Active

Misc-suite ---------------------------------------------------------- Not-written

-- CodyNelson - 18 Aug 2010