Modernizing Common Lisp: Recommended Extensions
John C. Mallery, Howard E. Shrobe, Robert Laddaga,
Thomas F. Knight, Kalman Reti, Paul Robertson,
Andrew J. Blumberg, Susan Felshin, and Patrick H. Winston
Artificial Intelligence Laboratory
Massachusetts Institute of Technology
July 7, 1999
The document suggests enhancements to Common Lisp that modernize the language in light of developments since that last ANSI standardization round. These elements can be incorporated as changes to the:
* Suggested "yellow pages" that implementations or repositories may supply as libraries
* Experimental components intended to explore areas not yet stable enough for inclusion within the language proper or the suggested libraries.
The presentation is as an outline of topics in order to keep the document short and facilitate discussion. It is intended that separate two-page outlines will be developed by proponents for each of the substantial topics for more careful consideration. If the community decides to move forward on a specific recommendation, we expect proponents to develop a detailed proposal for critical review, a working implementation, and ultimately, a standards specification.
The most important recommendation with the most leverage is the recommendation to develop a PORTABLE BASE LANGUAGE FOR COMMON LISP.
This strategic move provides:
* An opportunity to reengineer the core language;
* A minimal base to port to new platforms;
* Opportunities for new hardware and compiler efforts (base-to-platform & portable-to-base);
* Increased sharing of higher-level code.
How do we identify the evolutionary trajectory for Common Lisp? One approach is to identify what features a modern programming language requires by examining at other recent languages (eg, Java, Dylan) and selecting features that merit refinement and generalization. Another approach analyzes the deficits of contemporary programming languages and environments and proposes new facilities to address the identified gaps.
These recommendations use both approaches but are tempered by recognition of the constraints of previous standardization rounds and the need to maintain continuity. Although some of the suggestions below may seem beyond the conventional purview of a standards body, we believe that language leadership is not possible without reference to the kinds of facilities that a modern programming environment must offer. Feedback from independent efforts to develop libraries and experimental components will provide a critical input into the language evolution effort.
The entries below adhere to the following style for clarity of presentation:
1. Title: A telegraphic title.
2. Abstract: Three sentences describing the feature.
3. Category: Extension, Library, Experimental
4. Impact: Low, Medium, High
5. Examples: Are examples available in LISP or other languages?
6. Implementation: Are existing implementations available?
7. Longevity: How long will this feature be relevant?
8. Difficulty: Easy, moderate, hard.
9. Benefits: What specific benefits does the feature provide?
10. Approach: How can this feature be incorporated?
A. Core Language Extensions
* Package System: CL should provide relative package names and associated operators. This feature is important for smooth evolution of large Lisp systems. Language, Examples, Implementations, Easy, Low. SF, JM, KR.
* Declarations: Add the ability to declare returned values and their types for functions and methods. Provide a standard arglist function to return the arguments and values of a function. Language, Examples, Implementation, Easy, Low. SF, JM, KR.
* Modern Character Sets: CL should support Unicode, which is becoming the standard character set for multi-language applications. Extension, Examples, Implementations, Moderate. JM, KR
* Unit Awareness: Provide extensions to numeric operations that support conservation and coercion of units. Extension, low Impact, Easy. TK, JM
* APL Array Operations: Provide a standard array package analogous to the one found in APL but building on top of current CL array substrate. Implementations should provide highly optimized specializations for functions. Library, Low Impact, Examples, Implementations, Moderate. TK, JM
B. Networking Components
* Uniform Process Model: A standard API for working with processes (aka threads) should be developed and provided as a standard part of CL. It should include locks, store-conditional, and timers. The design should allow applications to exploit multiple processor hardware. Extension, High Impact, Examples, Implementations, Moderate. JM, HES
* Uniform Foreign Function Model: A single, standard API should be provided by CL for calling foreign functions. The facility should provide some basic operators for marshalling and unmarshalling arguments that can be used by applications to support FF over other transports (eg HTTP) in either direction. Extension, High Impact, Examples, Implementations, Moderate. JM, HES
* Standard Streams: CL should provide a standard API for streams in order to allow stream code to be portable. The grey stream proposal from CLIM is a good starting point, however, the actual standard should consider the requirements of networked applications, graphics display, and streamed video/audio. Streams should define generic buffer-in-place operations and provide a set of generic "internal" operations which enable, for example, encapsulation or metering during OPEN or other standard stream operations. Extension, High Impact, Examples, Implementations, Moderate. JM, HES, KR
* Networking Substrate: Incorporate a standard API for performing generic networking operations. Modern programming languages need to be network-aware. Java provides an example. A networking substrate assumes a thread model, streams, and locking mechanisms. Crucially, the networking model should provide a hierarchy of conditions relevant for networked applications. The networking API should be generic and capable of spanning multiple different networking standards, e.g. Internet, Chaos, AppleTalk, ATM. Networking implementations should keep track of the costs associated with sending and receiving data over specific links, so that data flow can be optimized by user code based on standardized metrics. Library, High Impact, examples, implementations, moderate. JM, HES
* Uniform Resource Locators: The IETF URL standard should be incorporated as an extension to physical pathnames in order to allow lisp programs to refer to data stored remotely via multiple access protocols (eg FTP, HTTP). URL have received wide deployment in the World Wide Web and will remain relevant for the foreseeable future. Incorporation involves making "URL:" a privilege host name that serves to invoke the URL parser. "URI:" and "URN:" should be reserved for use future use. A standard locator scheme should be provided to allow reference to specific versions of definitions (see Defsystem). Extension, High Impact, Examples, Implementations, Moderate. JM, HES
* Uniform Event Model: The Design should include support for externally generated events (OS generated). It should allow applications to exploit window systems I/O devices with high performance by mapping naturally onto their event model. The design should extend the condition system in a way that is more targeted at non-exceptional circumstances (event driven programming). Extension, High Impact, Examples, Implementations, Moderate. JM, PR, RL
* CORBA: A standard CORBA API should be provided as a library for CL. Library, Examples, Implementations, Moderate.
* Java CLIM Binding: CLIM should provide a Java binding along with support within CLIM lisp so that applications can drive clients on conventional web browsers Presentations and graphics are most important here. Library, High Impact, Examples, Implementations, Moderate. JM, HES
C. Descriptive and Organizational Tools
* Defsystem: CL should provide a two tiered system definition model. There should be a base system definition language and data structure that can be automatically produced by Lisp environments, but be human readable and editable. The base model should be definition centric, and use the persistence model (see below). A separate tool layer should provide extended capabilities, including
2) a patching and versioning facility (this facility should be URL aware so that resources can be drawn from multiple locations over the network);
3) a module system that clearly groups functionality supporting specific external interfaces;
4) a functional dependency tracking facility (Apple Dylan) that allows coherent system versioning based on definition versioning and supports portable "tree-shaking". Library, High Impact, Examples, Implementations, Moderate. JM, HES, PR, RL
* Documentation Facilities: Enhance CL to support automatic generation of documentation for definitions. A series of generic functions (or a single one with parameterization) should write definition components to a stream. Components should include the definition name, argument list, summary documentation, full documentation, and returned values. An assertion interface for making statements about definitions should be provided as a standard, but left to implementations to determine its implementation. A default module grouping scheme can use the assertion interface to support documentation units smaller than package internal or external symbols, or completely orthogonal to the package system. Library, Examples, Implementations, Easy, Moderate. SF, JM
D. Advanced Capabilities
* CL Portable Base Language: Develop a base language to which all extant Common Lisp implementations can be targeted. This language should include locatives and basic OS operations like processes, file I/O, networking. This base layer will increase portability of lisp applications and enable work to improve performance with specialized hardware or compilers. This layer should make efforts to support real-time and parallel dialects. Experimental, Very High Impact, Examples, Implementations, Hard. KR, RL, JM
* CL Virtual Machine: Develop a virtual machine capable of efficiently executing CL, multimethod dispatch, multiple inheritance, and GC. Library, Examples, High Impact, Implementation, Hard. AJB, HES, JM
* Persistence: CL should provide a persistence binding for CLOS. The facility should provide transactions and caching suitable for low latency operation. There should be a native object binding for all Lisp forms, so that they can be persistently stored. Library, High Impact, Examples, Implementation, Moderate. JM, HES, PR, RL
* Mobility: Provide a framework for mobile LISP that allows computations to be stopped and moved to different machines. Include a layered, encapsulating security model that allows environments in which a computation occurs to assign selectable levels of trust. Use the type system to enforce the trust policy. Deploy a browser plug-in that supports client-side execution and exploits standard browser facilities. Experimental, High Impact, Examples, Implementations, hard. PHW, JM, HES
* Debugger API: Development of a low-level debugger API would allow developers to debug LISP applications remotely and via different, perhaps third part interfaces. The API should provide for stack movement, restarts, access of data from stack frames or the environment. Experimental, Moderate Impact, Examples, Implementations, Moderate. AJB, KR, JM
* Java Integration: Compile Java into Lisp and execute. Library, High Impact, Examples, Implementations, Moderate. TK, PHW, JM
E. Experimental Facilities
* Compiler Advice: Provide a general mechanism for communication between the compiler, the application, and the programmer. Clean up the declaration mechanism: 1) provide accurate scoping of declarations; 2; Separate declarations effecting semantics from those effecting efficiency. Provide an assertion facility orthogonal to the code base that allows the programmer to provide advice to the compiler and for that advice to be easily separated according to operating environments (eg Dylan Compiler, CMU Python Compiler). Experimental, Moderate Impact, Examples, Hard. AJB, JM, RL, HES, TK
* Profiling and Test Vectors: Provide portable operator for determining the amount of consing within functions and the speed with which they execute. These facilities should work in multiple process and multiple processor contexts. Facilities for asserting test vectors for definitions should be provided and higher level facilities should run the defined test vectors. Automatic fault localization could be explored. These facilities should provide the substrate for developing compilers able to exploit runtime feedback from the application. Library/Experimental, High Impact, Examples, Implementations, Moderate/Hard. TK, JM, HES, KR
* Adaptive Programming Model: An example of this sort of model is a multi-level categorization of functionality, such as the generic network system under Genera. In such a system, high level functionality are characterized as Services, where services are implemented by alternative protocols, and similarly for protocols. Thus a declarative method exists for specifying alternative implementations, along with implementation of glue code to allow the interoperation (read plug-and-play) of alternatives. The design should include support for at least three levels of service specification, and glue code to implement the specification layers, and utilization of the condition system to control switching between alternatives. Experimental, Very High Impact, Examples, Implementations, Moderate. PR, RL, JM, HES, AJB
* Transactional Memory: Provide a facility for atomic and coherent computations in a multiple process environment. This may provide coherence for persistent or volatile storage. Experimental, High Impact, Examples, Hard. TK, KR, AJB, JM
* Low Latency Garbage Collection: Networked applications in CL can suffer serious performance degradations when the main GC is tripped. Ideas for low latency GC should be explored, possibly in conjunction with the portable base CL language. Experimental, Examples, Implementations, Hard. JM, KR
* Parallel Dialect: Develop a dialect as close to ANSI CL as possible that is suitable for MIMD parallel processing. Iteration constructs should be deprecated and replaced with higher level abstractions that prevent the programmer from making commitments about the underlying computational model. Numeric operations should be vectorized. Experimental, High Impact, Examples, Implementations, Hard. TK, JM, KR
|Last update : 23/10/2001|