Welcome to the Common Lisp for Java (CLforJava) Documentation Web.

This web contains all information related to the research and development of the CLForJava? documentation system. The goal of this project is to take the documentation function defined in the CommonLisp HyperSpec? one step further. The design augments the standard documentation string with a well-formed documentation structure that can be displayed in a variety of ways using XSL transformations. Some possible formats are Portable Document Format, DocBook? , HTML, etc. We include internationalization support by allowing translation of the localized documentation set.

Project Overview

The main functionality of the documentation system in CLforJava is gained through the use of Annotations. Annotations were added to the Java platform as of 5.0. Annotations were chosen because they will allow one solution to handle both Java classes created by the compiler and the pre-existing Java classes necessary for the system to run.

As the compiler parses Lisp code and builds the Java classes it will use the information it gathers to construct annotations and add them to the classes. Once this process is complete an annotation processing tool will "pull out" all the annotations, build the appropriate XML files and store them as resources in JAR files. The alternate case is that the system is doing an in-memory compilation of some Lisp code.

Quoting from Sun's official site, "It (annotation-based development) lets us avoid writing boilerplate code under many circumstances by enabling tools to generate it from annotations in the source code. This leads to a declarative programming style where the programmer says what should be done and tools emit the code to do it."

Documentation Schedule

Task	Due Date	Status	Notes
Update Documentation Web	1/18/2006	Complete	This is ongoing and will be completed as needed (minimum weekly).

Minimum Requirements

As a bare minimum the existing functions defined in the Common Lisp HyperSpec? , along with their functionality, must be maintained.

DOCUMENTATION

HyperSpec
The generic function documentation returns the documentation string associated with the given object if it is available; otherwise it returns nil.

When the Documentation function is called such as (Documentation 'caaaar 'function) the Symbol is checked to see if it has the associated DocType? , such as in this case, Function. If so, the getAnnotations method is used to retrieve any annotations attached to that object. By doing so we are able to extract the unique identifier from that class and use it to look up any documentation. We are able to use that identifier, docUID, to find the associated Symbol within the Lisp system. As shown later in this document that Symbol is associated with the applicable documentation in the DocumentationDom? Variable. Once that Variable is retrieved the Lisp Get function can be used with the List and Symbol as arguments to return the List element immediately following that Symbol. In this case it is the XML Document.

Once the XML Document is obtained it can be transformed using XSL. More can be found on this topic under the section titled "Documentation Display".

DESCRIBE

HyperSpec
Describe displays information about object to stream.

Documentation Scenarios

There are 3 distinct documentation scenarios that must be handled via different but related documentation methods.

Compiled Lisp Code Stored in JAR

Lisp code that is compiled and stored in Java Resource Files during the building of the project offers the most functionality. In this case, annotations are added to the Java Byte Code during the compilation process. These annotations are in-memory object representations of the documentation structure. Once the compilation of the Lisp code is complete a post-compliation annotation tool is used to harvest this data and build XML documents based on a predetermined set of DTDs. These XML documents are then stored in an external Java Resource File. It is important to note that this scenario also offers the best opportunity for internationalization.

Core Java Code

There is certain Lisp code that is represented by Java Classes that make up the core of the CLforJava project. These functions do not go through the Lisp compiler as the other Lisp functions do and thus are not subjected to the same documentation process. As an alternative, the Lisp signatures for these functions are loaded at build time. The signatures contain the name, parameters, and the documentation string. This "empty" signature is passed through the Lisp compiler and the documentation is thus generated the same as functions written in Lisp.

Example:

(defun someFunction (parameters)

 
"Documentation string goes here."

 (... blah ...))

Lisp Code Compiled At Runtime

Lisp code that is compiled after the project has been started is represented in memory. Since the XML document has no secondary storage after on-demand compilation the resulting XML document must be kept in-memory and attached to the Java class via a reference field. This document can be transformed and presented the same as the other 2 scenarios but the benefit of internationalizationis lost because the documentation is not generated until after the project has started.

What Gets Documented

Symbol

Because of the dynamic nature of symbols they are treated differently in the way the documentation is generated. Instead of attaching the documentation to the symbol via annotations, a single annotation holding an XML template is attached to the SymbolImpl? class. When the Documentation function is called on a Symbol the information is generated at that moment and the XML template is filled in with as much information as possible. This is important because information about Symbols, such as current package, change often.

Some examples of data that is collected for Symbols:
Name
Home Package
Is Special
Is Constant

Some other questions that can be answered are ones such as where is the symbol interned? Is it accessible in the current package? Is it accessible in other packages? Is it internal, exported, or inherited?

Doc String

The doc string for Symbols is handled by adding 2 fields to the common.type.Symbol class that all Symbols are derived from. The first field called docString is used to store a doc string that is added using the setf function. The second field, docStringRef, is used to refer to a doc string that exists in the jar file that is generated when the project is built. This allows the doc string to be stored in the same facilities as functions and variables. Also, it allows the doc string to be translated into other languages. If the documentation function is called and the doc string is retrieved from a resource it is copied into the docString field so that we can gain a speed advantage through caching.

Function

Function documention includes the function name in Lisp as well as the Java system name (including fully qualified package name). It also includes the UniqueID? generated during the Annotation process, a Java timestamp when that documentation was created, and most importantly, the DocString? included during the function declaration.

Package

Package documentation includes the package name and the number of symbols that are within the package. It will also report, if applicable, the packages that use the documented package as well as the packages that it uses.

Documentation Discovery

Architecture

Intermediate Code Generator

The ICG is where the documentation details are associated with the Java class to which they relate. Using ASM the annotations are tacked onto the class using the Emitter instance. Methods have been added to the Emitter to accomodate Annotations.

newAnnotation()
emitAnnotationField()
endAnnotation()

These method calls in an example:
emitter.newAnnotation("Llisp/system/compiler/DocStringAnn"); //Note the fully qualified package name.
emitter.emitAnnotationField("DocUID", "docUID_512345641235");
emitter.endAnnotation();

The resulting ASM code is:
AnnotationVisitor av0;
av0 = cw.visitAnnotation("Llisp/system/compiler/DocStringAnn;", true);
av0.visit("DocUID", new String("docUID_512345641235"));
av0.visitEnd();

At this point the compiled Java class has the necessary information associated with it to generate documentation when that class is loaded into memory. To add new Documentation you must either create new Annotation classes or add fields to existing Annotation classes. You must then emit the annotations and corresponding fields using the methods above. Be sure to emit in the proper location. All annotations should be directly after the Class declaration. There are certain stipulations that each Annotation class must meet. Please refer to the section titled "Annotation Class Design".

CompileFile (Function)

The CompileFile function is where the Annotations are pulled from the Java classes and built into an XML Document.

For each class that CompileFile comes across, the following occurs:

1. The ByteArray is used to create an ASM ClassReader.
2. A custom AnnotationCollector is run on that ClassReader using the accept method.
3. Any annotation fields found are returned as a HashTable
4. For each field that is found in the HashTable using an iterator a node is appended  
   to the document. This is where domParentNode is so important - it used to determine
   the nodes location in the tree. The only node that is treated special is docUID.
   docUID which is the unique identifier for every piece of documentation is add as an
   attribute to the root node of that document, docInstance. All the docInstance sub-trees
   are children of the root element, documentation.
5. The resulting XML document is tranformed into a Java String using the TransformerFactory.
6. Using the Main Class name the XML String is turned into bytes and added to the final JAR
   file as a resource name. By storing it in the JAR file we will have easy access when the JAR
   file is loaded.

Example XML Document:
<?xml version="1.0" encoding="UTF-8"?>
<documentation>
 <docInstance docUID="docUID_114714269625899">
  <generated>1147142696258</generated>
  <docString>Here is an example of a documentation string.</docString>
  <javaName>lisp/common/function/Caaaar</javaName>
 </docInstance>
<documentation>

Load (Function)

For Lisp code that is compiled at project build time and stored in JAR files we load that documentation into the system using the function Load. Using the ClassLoader? we can pass the XML file that we are looking for and it will find it (assuming it exists) in the appropriate JAR file. Using the getResourceAsStream method we are able to build the XML Stream back into our original document created in CompileFile.

The next step is to identify and isolate each docInstance element and process them seperately. For every docInstance discovered the docUID is retrieved and a new Interned Symbol is created with the name being that of the docUID value; such as Package.Compiler.intern(String.Factory("docUID_765436546")).

A Lisp List is maintained which contains all the documentation in the system. It is in the form (docUID domTree docUID domTree docUID domTree) where docUID is an interned symbol and domTree is of type Document. Every time an instance of docInstance is processed successfully it is added to this list. Once the processing has completed the List is attached to an Entry in lisp.common.type.Variable called DocumentationDom? . This is done so that we can get at this information from the Documentation function, it is important because this Variable is accessible system wide. The fact that the docUID is an interned symbol becomes very important later as it simplifies the process of extracting the XML document from the DocumentationDom? List.

Annotations

Annotations are the method that is used to "decorate" the Java byte code that is created by the Lisp compiler. The structure of the annotation classes closely model the resulting XML document that is produced and stored in the JAR file or in memory.

Sun Annotations Language Guide
JavaWorld Annotation Introduction
Java Lobby: Creating Source Files Using APT

By using the ASMifierClassVisitor? class in the ASM library we are able to generate the ASM code that would generate the original annotations. This is a quick and easy method for mastering the ASM calls.

Class Design

The following is a simplified example of a fully qualified Annotation class:

package lisp.system.compiler;

import java.lang.annotation.ElementType;
import java.lang.annotation.Target;

@Target(Element.TYPE) //Here we annotate our annotation!
@Retention(RetentionPolicy.RUNTIME)

public @interface DocStringAnn {
    String domParentNode();
    String docString();
}

A very important field is domParentNode. This value is used during XML document construction. Because Annotation classes cannot be subclassed we cannot use that natural relationship to build our DOM tree. Instead, we include a value of the PARENT NODE along with the Annotation. This way we can determine later in the load process where in the Document tree we would like to put these fields. The name of the field in the Annotation represents the name that will exist in the resulting XML Document. At this point there is no support in the system for Annotation with a field type of Annotation. Field values should be primitive.
Also very important is docUID. This special field receives a Symbol named "docUID_" + System.currentTimeMillis(). This is used to uniquely identify every "thing" that is documented. It only exists in DocStringAnn? , which is the main Annotation class of any documentation set.

For example, the above Annotation would result in:

<docString>Here is a documentation string.</docString>

Documentation Storage

XML

Documentation that is generated during compile time or runtime is stored in an XML structure that is dictated by the object that is being documented. They currently include function, symbol, and package.

Resource File

Documentation that is created during compile time within the ICG is stored in a JAR file as a resource. This allows the XML file to be recalled at any time later during system operation using a ClassLoader? on the CLforJava instance.

Documentation Display

XSLT

Naming Convention

The XSLT is discovered using a combination of the type of transformation and the DocType? passed to the Documentation function. If the transformation type is HTML and the DocType? is function then the Documentation function will attempt to look up the XSL file called HTML_Function.xsl. It then can use the file it finds to transform the Document into the appropriate format. If no transformation is found it is transformed into a simple \n separated Java String. This can then be returned to the reader as a Lisp String.

It was decided that any transformation, regardless of the type, will be returned as a Lisp String. This is done because the Lisp programmer can simple redirect that String to the output location of their choice.

Available Transformations

Proposed:

• Command Line (minimum requirement)
• DocBook?
• HTML
• PDF
• JavaDoc?
• GUI Based

Internationalization

There are 2 main components that must be translated to achieve internationalization of the CLForJava? documentation system. Not only does the actual documentation need translation but also the information that makes up the presentation layer. For example, when displaying symbol information, we would label the Home Package field in whatever language was current. It is not necessary to translate the underlying XSL or XML structures, only the data it contains.

---

Notes: This web also contains all of the documentation for the CLforJava (Common Lisp for Java) product. This documentation includes the User's Guide and the Programmer's Guide along with any other shippable documentation such as Release Notes etc.

Other related places:

• Common Lisp
  • http://clforjava.cs.cofc.edu/HyperSpec/HyperSpec

• You are currently in the Documentation web. The color code for this web is this background, so you know where you are.
• If you are not familiar with the TWiki collaboration platform, please visit WelcomeGuest first.

Documentation Web Utilities

• WebTopicList - all topics in alphabetical order
• WebChanges - recent topic changes in this web
• WebNotify - subscribe to an e-mail alert sent when topics change
• WebRss, WebAtom - RSS and ATOM news feeds of topic changes
• WebStatistics - listing popular topics and top contributors
• WebPreferences - preferences of this web
•   - advanced search