2. Overview of the Ceilidh system

We will describe here the general layout of the Ceilidh file system, and the function of the main directories and commands.

The support provided by the Ceilidh system falls into the four distinct levels

System		Department-wide view, all courses and papers can be viewed.
Course		One course (teaching module). Certain users will be authorised for teacher permissions related to a particular course, and certain for tutor facilities.
Unit		Documentation and exercises for one course unit of a course.
Exercise		Assessment definition for one item of coursework.

Filestore layout

The current filestore structure is headed by a directory which we will refer to as ~ceilidh. The menu commands (dumb terminal menus, one command for each category of user at each relevant level, all implemented as shell commands) are in the directory ~ceilidh/bin.mnu, the X-windows version commands are in the directory ~ceilidh/bin.x, the command line interface commands (one command for each student menu command, see the CLI document [ref 5] [content] for more details) themselves in the directory ~ceilidh/bin.cli, the general tools (shell scripts and programs called by all the command systems, menus, command line interface commands and X versions of Ceilidh, those implemented as shell scripts have a name started with "C", executable programs have a name starting with a lower case letter) in the directory ~ceilidh/Tools.

If you run multiple architectures, there will be additional directories for each architecture's executables, with names such as ~ceilidh/bin.sun4.3 where the name incorporates specification of the machine hardware and operating system obtained using the uname command.

Additional textual information is stored as follows.

Below the ~ceilidh directory each course has a directory such as ~ceilidh/course.pr1 for the course "pr1" (Introductory C++ programming). This contains a directory ~ceilidh/course.pr1/bin for any special marking commands for this course. Below the course directory are directories for each course unit, typically ~ceilidh/course.pr1/unit.3 to contain the third unit of the "pr1" course. The unit name is always numeric, and indicates the sequencing of the units. Below this are directories for each item of coursework, such as ~ceilidh/course.pr1/unit.3/ex.1 for the first exercise or example. Exercise names can be any string of up to three characters*

All course, unit and exercise directories include a file title containing a one-line ASCII summary title of the course, unit or exercise. The course directory should also contain a summary file typically containing a summary of the lectures, times, courseworks set and hand-in dates, kept up-to-date by the teacher using the teacher administrative facilities.

The unit summaries are assumed to be set up by the course developer, and the course summary to be the responsibility of the teacher administrator.

The contents of the directories and the commands available at each of these levels are described below.

3. Administrator commands at the system level

We will describe now the commands, all at the system level, intended for the Ceilidh administrator. Both the adminstrators and course developers need general write permission to the Ceilidh file system, Ceilidh does not disctinguish between them. All the administrator facilities are available to anyone logging in as Ceilidh.

In response to the command

ceilidh

the menu for a developer or system administrator should appear as shown in figure 1.

The ordinary user items (see the Student Guide to Ceilidh) appear at the top, and are followed by additional menu items, one for the tutor (see the Tutor Guide), and others relevant to the system administrator.

If the extra items do not appear, you are not logged in as the special username allocated for this purpose, normally ceilidh, but possibly another name set during the initialisation process.

The additional administrative facilities include the following.

Edit system message-of-the-day (~ceilidh/motd)
Edit help files (all of the help files are offered)
Create a new course
Amend the values of environment variables
Set up the default course
Perform auditing operations

These commands will now be described in detail.

Default course

Using the rdc command, the system administrator can set up the default course which will be set when a new user calls Ceilidh at the top level for the first time. Thereafter the user will pick up as default the course they most recently used.

Within a course, it is the teacher (as part of the "new week" nw command) who sets up the default unit and exercise numbers.

Edit help files ("eh" command)

The help files live in the directory ~ceilidh/help. These are the files which are listed when the user types the "help" command on any menu. You may wish to amend them to insert pointers to local documents, staff or phone numbers. The names of help files are of the form

<category>.<level>

where the <category> is

sys	for system level help
cse	for course and unit level help
ex...	for exercise level help

dev	for developer's help
tch	for teacher's help
tut	for tutor's help
usr	for student help

Thus the teacher's course level help (the information printed when the help command is called on the teacher menu) is in the file cse.tch.

At the student exercise level, there are several files

expr1.usr	help when no compilable program source exists
exin1.usr	help when no interpretable program source exists
expr2.usr	help when source but no executable exists
expr3.usr	help if source and executable both exist
exes.usr	student help for text submission exercises
extxt.dev	developer help for text submission exercises
exmc.usr	student help for text question-answer exercises
exqa.dev	developer help for text question-answer exercises

Miscellaneous other help files include

sys.aud	audit facilities
en.dev	edit notes
enms.dev	edit master notes

The papers directory

All documents viewed by students using the vp command are stored in this directory.

Each document has three associated files. The document Student (the student guide) for example has a file Student.ttl, containing a single line ASCII title (this is displayed by the vp command), Student.cat, a file of ASCII contents viewable through the $PAGER viewer, and a file Student.ps of PostScript, printable to a PostScript printer, and viewable through a PostScript viewer such as ghostview if available.

To add any new local documents for viewing, at least the first two of these files should be inserted by hand in the ~ceilidh/papers directory. ,LP There may also be Student.ohp, a file of overheads in PostScript.

Edit shell variables ("esv" command)

Each Ceilidh shell environment variable (in the file ~ceilidh/bin.mnu/CShlVars set up during the installation process) can be set to a new value. These are the global variables; variable relevant to a particular course are store in a similar way in the course type ot type.tch files.

The variable names are displayed in order; a new value can be given, or "return" used to leave the current value set. The new values take effect for users upon their next entry to CEILIDH. A summary of the relevant variables is shown in figure 2.

Maintaining audit trails ("audit" command)

There are minimal elementary auditing facilities built into the system from release 2.1. There are no audit trail analysis facilities provided, just their generation. Suggestions for appropriate analyses would be welcome.

Separate trails can be kept of all use of certain Ceilidh operations (e.g. printing or compiling), or of all use of Ceilidh by a particular user. These separate trails can each be switched on or off at will. If a trail file of the appropriate name exists, the trail information will be appended to it whenever necessary. It must be remembered that such facilities can result in the collection of large quantities of data; this must be archived and removed regularly. The files containing the trails are stored in the directory ~ceilidh/audit.

The audit administrator command calls up a separate auditing menu. The auditing menu allows the developer/administrator to perform the following functions.

course:unit:exercise:user:date-time:user:facility:information

These follow the same format as mark information. If the appropriate trail files exist, an entry will be appended to both the facility file and the student file whenever that event occurs.

The Error audit should always be switched on, and should be checked regularly.

Create a new course ("nc" command)

We consider the creation of the skeleton of a new course to be an administrator function. When at some future date we distinguish more carefully the functions of administration from those of course development, the developer will have access to a particular course after it has been created.

The "new course" command requests a course abbreviation suitable for use as a directory suffix (e.g. the C++ programming course "pr1" is contained in the directory ~ceilidh/course.pr1 ) and a one-line course title. The course name should be at most three characters, with no spaces, to conform to MS-DOS practice. The command then requests the login names and full names of staff who will be permitted to administrate this course.

It sets up the course directory, its bin and bin/SOURCE sub-directories, and the title and staff and tutor files. The course bin directory may eventually contain compilation and marking shell scripts and programs specific to this course, with their source and Makefile in bin/SOURCE .

If there is a command CCompile in the course bin directory, it is used for all compilations in preference to the main command in ~ceilidh/Tools/CCompile. It is called with two arguments as in

CCompile <source> <executable>

and expects a "true" exit to indicate successful compilation. The actual compiler called is defined by the

CC=

environment variable set in the course or exercise type or type.tch file.

Similar considerations apply to CVCompile (the verbose compilation option).

If you require a debugging command for the students, there must be a CDebug command in the "bin" directory. This will be called as

<course directory>/bin/CDebug <course> <unit> <exercise>

When the marking process (see later) requests a program such as typog_c for C typographic metrics (or typog_C for C++) it searches first the course "bin" directory for an executable typog_c, and if this is absent uses the ~ceilidh/Tools/typog_c version.

The nc command then moves into the newly created course, and offers the developer's course menu, see below for details.

4. Developer commands at the course and unit level

The developer may move to this level from the system level either by typing

sc pr1

or will more normally enter the course "pr1" for example directly, by calling

ceilidh -c pr1

either directly or using a preset alias. We normally set an alias pr1 for this command. There will always be a currently set default unit and exercise name.

To use course developer facilities, choose the

develop

command on the menu*.

ceilidh -c pr1 -x develop

to enter the developer's menu directly on the course "pr1". We normally set an alias pr1dev for this command.

The menu is now as shown in figure 3.

The menu commands can be divided into course-related items, unit-related items, and exercise-related items.

Course related menu commands

Unit related menu commands

Exercise related menu commands

Course and unit notes

Most of the supporting notes for a course are divided among the course units. For courses we distribute, in each unit there are files

notes.cat	ASCII copy, viewable by "$PAGER" (e.g. "more")
notes.ps	PostScript notes for printing or viewing
notes.ohp	PostScript notes for overhead projectors

We generate these from a master copy kept at Nottingham, which is in roff format. Copies could perhaps be distributed if volunteers were prepared to improve them significantly.

We also keep at Nottingham corresponding .dvi files in device independent troff (ditroff) form if required. These can be viewed through xditview or a similar program.

We are in the process of adding .pdf files.

The student generally accesses the notes on-line through the vn command, which calls a PostScript viewer if the user is at an X-terminal and the PostScript viewer has been set, or the user's command defined by the PAGER environment variable to view the .cat version of the notes on the screen otherwise.

The command pn prints the PostScript notes notes.ps to a default printer.

The overhead projector slides are in file notes.ohp in PostScript, and must be printed by Unix commands if required.

The en developer command displays an extra menu, which allows the following operations.

If you have the master roff copy of the notes in file notes.ms you will obtain a different menu allowing editing of the master file, and the re-creation of all the other files.

We now have another set of notes files at the course level, in files such as ~ceilidh/course.pr1/notes.cat to include an overview of the course, and coursewide information such as as discussion of the marking process, and the recommended typographic layout for the course. There is no menu entry for editing these course-level files; use raw Unix.

Annual tidying of courses

Two of the above commands ( ar and zap ) relate to this activity. They are currently in the developer's course level menu, but should perhaps eventually be in the teacher's menu. Unix file permissions make this difficult.

5. Developer commands at the exercise level

To work as developer on a given exercise, you may either first move to the developer course level as above and then type the command

sx 5

for example, to work on exercise 5 of the current unit, or you can move into the exercise as an ordinary user and then type

develop

which should appear as an additional menu item.

Each exercise is stored in a directory such as

~ceilidh/course.pr1/unit.2/ex.5

for course "pr1", unit "2", exercise "5". Working as developer on the exercise by either of the above approaches moves you into the exercise directory, and offers you a new menu, the developers exercise menu. The particular menu will vary depending on the type of exercise.

In addition to the commands offered on the menu, the usual additional facilities of replying with an exclamation mark followed by a Unix command can be particularly useful when working as developer, for example

!ls -l

to see the names of the files in the exercise, or a command such as

!emacs model.q

to edit one of them directly, or simple

!

to be put into a shell in the exercise directory.

The exercise may currently be of several basic types:

Question/answer exercise
Text submission (essay) exercise
Compiled programming exercise
Interpretive programming exercise

The type is defined by a file type in the course or exercise directory, in which the various details of the exercise type and requirements are specified using Bourne shell notation. The type file in the course directory sets course-wide defaults, individual exercises may vary these parameters with a local type file.

After submission of work by students, the subdirectory ex.<exercise>/solns will contain the students' submitted work in files starting with their username (program sources in files named <username>.c or <username>.C for programming exercises, perhaps files <username>.txt etc for essays), and their mark histories (in a single file ex.<exercise>./solns/marks containing all marks for that exercise) for any marked exercises.

The marks file consists of one line entries with colon-separated fields. Each line is of the form

<course>:<unit>:<ex>:<student>:<date-time>:<entered by>:<total mark>:<other marks>

All marks are kept, each date- and author-stamped by the person who entered it. Only the last entry for a particular student is used in vm summaries. The "entered by" field contains the student username for work submitted by the student, or the tutor's username for work remarked by the tutor.

Submitted text files (such as program sources, essays ...) are normally overwritten at each submission. There are now facilities to keep all submissions using RCS or SCCS, see later for details.

At the exercise level, both the user and developer see one of several different menus depending on the type of the exercise, and the contents of the directory. Programming exercises have a separate major section below; other types of exercise will be discussed more briefly.

Question/answer exercises

This type of exercise is useful both for marked question/answer or multiple choice exercises, and for end-of-course questionnaires where the results are collected for analysis, but not marked.

The type file here will be simply

TYPE="QA"; export TYPE
SUFF="mc"; export SUFF

The exercise directory for a "QA" exercise will contain (in addition to the usual title file with a one-line title, and model.q containing the question in ASCII) a question/answer script file model.mc containing the vital information for display to the student, and for checking the responses. The key program here is the question/answer program (see the Ceilidh document [ref 6] [content] for more details of the capabilities and file formats) ~ceilidh/Tools/multi.

The model.mc file will contain the main driving script, which contains all the display and marking information. The script may contain any number of questions for the student. For each question it will include the text to be displayed to the student, and (if marking is required) details of the required answer in the form of the name of a particular recognising program (the generic name for a program intended for recognising meaning in text is an oracle program) and the data required by the chosen oracle. The answers may be checked using a variety of recognisers, all stored in the ~ceilidh/Tools directory. There are ones for checking multi-choice answers ( Tools/mchoice checks single character replies) or numeric answers ( Tools/numeric checks sequences of numeric values from the student against specified ranges) or more general textual recognisers (the main Ceilidh Tools/oracle program, based on regular expressions, see the Ceilidh Oracle document by Zin and Foxley [ref 7] [content] for details). There is also an English language semantic answer recogniser at Nottingham, but its distribution is restricted for copyright reasons. It is hoped that general distribution will be possible at some stage.

The student answers to the questions in the model.mc script are combined into a single file using the standard Unix ar (archive) command, and stored in a file <username>.ar in the solns directory.

When a student uses cks to check submitted information, the separate responses are extracted from the archive file.

Note that the script file model.mc should have no public read access, since it may also contain the oracles which check the student's answers to the questions.

The developer's exercise level menu for QA exercises offers

editing the title file
editing the question file
editing of the script file

and rerunning the multi program in a verbose marking mode.

Text submission exercises

Ceilidh exercises can be set up purely for the collection of textual material by students. This can usefully simplify the administration of coursework from large classes, but the submitted work still has to be marked (by hand) and the marks entered using the em Ceilidh command. The students must, of course, prepare the essay in an on-line file (typically with a name such as prog<unit><ex>.txt or some such) before submission and it all essays must be in some consistent form for the teacher to print them or view them for marking.

It is also possible to add text (writeup) submission to programming exercises, so that the student completes both a program and a written explanation. This will be described along with the setting up of programming exercises.

The type file here will be of the form

TYPE="ESSAY"; export TYPE
SUFF="txt"; export SUFF

The menu commands for the student are

Read or print question
Obtain skeleton of essay in local file
Edit local essay file
Submit local essay file

The developer commands are

Edit exercise title
Edit question
Edit skeleton file

The exercise directory will contain the title file, the question file model.q defining the problem to be solved, and an essay skeleton in a file such as model.sk. The exercise is purely (at present) for text submission purposes. When the texts have been submitted, they are stored in the usual way in the exercise solutions directory, and the developer can (directly under Unix) print out the files, or look at them on-line. When essay marking by program is realistic, it will be installed! It would be expected that the human generated marks would then be entered by hand using the teacher's em (enter marks) command. On-line viewing, annotation and marking of essays may be provided later.

A student can edit text files (program source or essays), of course, outside the Ceilidh system if preferred. The use of Ceilidh to administer essay questions can considerably simplify the collection and administration of work, even though there are no formally offered essay marking facilities yet. Any submitted essays appear in the solns directory, under names such as <username>.txt for each student.

The developer's menu offers editing of the question or the skeleton, or of the setup actions file setup.act. The latter file (used as the basis when the student types set at the beginning of solving an exercise) will be of the form

Copy model.sk prog56.txt

This will copy the file model.sk in the exercise directory to prog56.txt in the student directory. We find it helpful to use the skeleton file as an essay outline, with major subheadings for the student report or essay indicated. This ensures that they remember to address all of the major points.

There will always be a mark.act file, which in text submission exercises will consist of the single line

Save o txt

to save a copy of the student file prog56.txt for unit 5 exercise 6 in the file <username>.txt in the solutions directory.

Programming exercises

For program development (our major application so far) the user and developer menus are considerably larger.

We will outline some of the principles here, but leave most of the detail to the next section. The developer's menu for a programming exercise is as shown in figure 4.

The type file will typically be of the form

TYPE="COMP"; export TYPE
SUFF="C"; export SUFF
CC="g++"; export CC

specifying the exercise type as "compiled programming", the program source file suffix, and the compiler command. These can be defaulted by the type file in the course directory, but a local type file must exist. The commands are set using the ety command. There is a number of optional extra entries which can be specified in either of the type files on a per-course or per-exercise basis.

MAXSUB=20

MINGAP=300

OUTOF=10

GRADE=E

SAVEOUT=yes

VDATA=no

VDATA=no

in the type file. An intelligent student could, of course, write a program which echoes its input to a file, and find the data that way by submitting the program.

C_ORACLE=my_oracle

The compilation commands on all menus of "COMP"-type exercises will call a command CCompile in the exercise directory if it exists, or in the course bin directory if it exists, or the one in the ~ceilidh/Tools directory otherwise.

The standard Ceilidh CCompile shell script uses $CC for its compilations, and is called with arguments

CCompile <source> <executable>

See below for details of the compilation process.

6. Amending an existing programming exercise

This important activity will now be described in detail. Bear in mind that choosing a particular exercise in developer mode moves you to that exercise directory. This means that you can now perform direct Unix operations in that directory by either executing each command preceded by an exclamation mark as in

!ls -l

or by obtaining a local shell as in

!

or

!bash

and logging out from the shell to return to Ceilidh.

For programming assignments the student will generally start by first reading or printing the question, then calling a setup command to set up any skeleton or header files in the local directory. They will then develop the program for the solution of the given problem (which can be done outside this system if required), and will then mark and submit it.

6.1. The question

The developer command

eq

allows you to edit the student question with your default editor.

The question is stored in the file model.q and will be in ASCII.

We hope to add other (PostScript, dvi, pdf, HTML) options later.

Experience has shown the value of using a standard structure outline for the question file, such as

Question:
Method:
Notes/Hints:
Mark scheme:
Typical input:
Typical output:

You can, of course, make the question as helpful or unhelpful as you wish. Don't forget to describe the files which are set up by set, and the marking scheme. The last entry above ("Typical output") may be most easily added after the model solution has been developed, using actual model program output.

6.2. Setup

You may wish to change the effect of the student set command in order to provide a more helpful skeleton, a less helpful skeleton, or some additional files. The student executes the setup command once before solving the exercise. The command uses a file in the exercise directory named setup.act containing lines such as

Copy model.sk prog72.C
Copy header.h

Any line with two filenames causes the exercise directory file such as model.sk in the above example to be copied to the student's directory under the name prog72.C. If the line contains only one name, the source and target names are assumed to be the same. You may wish to add a line to copy a file of test data to the student directory

Copy model.d1 TestData

or to copy an additional write-up skeleton by

Copy model.wsk prog72.txt

The setup actions file is edited using the esa developer command, and the skeleton file model.sk using the es command. Other files such as a write-up skeleton would have to be edited by hand using, for example,

!emacs model.wsk

If no mark actions file exists, the esa command gives a default file to copy the program skeleton, which may be adequate in simple cases. Note that execution of the setup function by the student will not overwrite existing files in their directory.

During any copying performed as part of the setup activity, the string $USER in the source file is replaced by the user's login name as the file is copied, $DATE by the current date, and $NAME by the user's full name (extracted from the course students file).

If no program skeleton file exists, the es command starts the skeleton with a copy of the model solution; it is sensible to set up the skeleton after the model solution has been completely developed. It is assumed that the developer will take the solution program, and then edit out all the parts which the student has to create. There may be a file such as ../../box (in the course directory) containing a comment box such as

//////////////////////////////////////////
//
// Put your program instructions here
//
//////////////////////////////////////////

for C++ programs, which you may choose to insert with your editor at appropriate points. Consistent use of features such as this in the skeleton makes the student's understanding of what they have to do much clearer.

6.3. Program development

This stage of student activity can be performed outside the Ceilidh system if so desired; we anticipate that the student will work completely within Ceilidh during the early stages of the course, but outside during the later stages.

The student will have a source such as prog57.C and executable such as prog57. In the exercise directory, the source will be linked to model.C and the executable to model.

The commands available to the student for development of their programs with Ceilidh include the following.

CCompile first looks for a Makefile in the exercise directory. If it finds a Makefile containing a dependence such as

prog72 :

in the unit 7 exercise 2 directory, for example, it compiles using that entry with a

make prog72

command. (The Makefile can also contain an entry

model : ...

to assist the course developer; this will not be usable by the student.)

If there is no Makefile in the exercise directory, or no

prog72 :

entry in it, the system uses one of the CCompile shell scripts and the $CC command, which would typically default to

$CC -o prog72 prog72.C

The value of the CC variable is set coursewide in the course type file, over-ridable on a per-exercise basis in the exercise type file.

Refusing compilation

There is a particular Makefile available which will put the user back into a shell for compilation. This is offered to the developer if no Makefile exists when using the

em

(edit makefile) command, and takes the form

$PROG : $PROG.C
     rm -f $PROG
     cat $$C_LIB/NoComMsg
     $$SHELL

where the file NoComMsg contains

There are no automatic compilation facilities in this exercise.
The system will now put you into a shell. Execute the required
compilation commands, then quit the shell. To do this,
    if you are using the C-shell, type "exit",
    if you are using the Bourne shell, type "control-D".

You will then be returned to the point where you left the Ceilidh
system.

Now wait for the shell prompt .....

The run command runs the student's executable such as prog57 interactively from the terminal.

The student can also use a rut command to run their executable against test data supplied by the developer (it uses the first of the data sets provided for the marking process, see below).

Students have a vtd command to enable them to view the files of test data used in dynamic tests during the marking process. This can be turned off by the developer or teacher by setting the value of the variable VDATA to "no" in the ety (edit type file) developers command.

If the developer leaves an executable program in the exercise directory, the student can use further commands to run the developer's executable interactively, and to run the developer's executable against the same test data. This allows the students to compare their answers with those presumably expected by the developer. When a course is set up (by un-shar-ing the distributed shar file) there are no executables. If you require executables to specific exercises to enable students to run them, you must use the developer's compile command to create them as required. Generally the C++ executables we get are quite large, so they are created only when required.

The extent to which we should assist the student's program development (by repeated submission, by releasing the test data, by offering an executable, etc) is very open to debate; comments and suggestions are always welcome. The system can easily be adapted to offer more or less help on different exercises.

6.4. Marking and submission

The student calls the sub command to mark the program which she or he has developed. The submission/marking process is driven by a file mark.act (mark actions) in the exercise directory, and expects to find in the current (student's) directory a source in a file such as prog72.c or prog72.C and a corresponding executable*

The submission/marking process appends a one-line summary of the marks to the marks file, and a copy of the current version of the student program source to the ex.<exercise>/solns directory; if the lecturer wishes, lines of the user's source code starting #include which refer to a local file may be substituted by that file if it is local and readable. In general, only the latest copy of the source is retained.

The mark.act file

If a line such as

Save o C

exists in the mark.act file, the intention is that the student' source (from a file such as prog72.C with name ending ".C", the last argument of the "Save" line) will be saved at each submission, overwriting (the "o" specifies "overwrite", specify "a" if each submission is to be appended rather than overwritten) previous copies. To save a student essay (in file prog72.txt ) and header file (in prog72.h ) add lines such as

Save o txt
Save o h

to the file.

The following generalisation is an enhancement in release 2.2 onwards.

A line such as "Save o C" will first search the course.<course>/bin and ~ceilidh/Tools directories for an executable CSav_C (the name is "CSav_" followed by the given suffix). If such a file is found, it is executed with arguments

CSav_C -o <course> <unit> <exercise>

If it is not found, the student's source is saved directly as described earlier. It is thus possible now to write special save commands such as CSav_... as shell scripts if required. Each such command needs a corresponding CVw_... command which is called when the student executes a cks command to enable them to view their submitted work.

Three new CSav_... commands are provided in the Tools directory of the current release as follows.

Save o ar

in conjunction with a line

ARFILES="prog56.C header.h hotel.C"

in the type file will cause the named files from the student directory to be combined using the Unix ar archive command into a single stored file

<username>.ar

in the solns directory. This facility is implemented by the command CSav_ar in the Tools directory for performing the saves, and CVw_ar to enable the student to view the stored files.

Save o rcs

saves the student source in a file <username>.rcs in the solutions directory using the RCS (Berkeley "Revision Control System", a source code control program) command. All versions of the program submitted by the student are thus available to the teacher afterwards; the student just sees the most recent version during a cks (check submission) command. Viewing the RCS files has not been built-in as a teacher command; the demand for it is uncertain. The RCS file would enable a teacher or researcher to follow the student's progress through an exercise; this could be useful for research into student program development, and makes obvious the occasions when a student copies another program instead of editing the existing one. The shell script to perform the RCS collection is in the file ~ceilidh/Tools/CSav_rcs. You must have the RCS command on your system. A similar system could surely be built using SCCS as an alternative.

Save o exe

in the mark.act file causes the student executable to be saved in the solns directory under the name <username>.exe The command CSav_exe looks for either of the files prog<unit><exercise> or prog<unit><exercise>.exe in the student directory. Beware that executables can be very large; storing large numbers of them can consume vast areas of disc.

The mark details (one line) are appended to the file marks at each submission, they are never overwritten. The submission/marking process cannot be called by one user more frequently that a general time interval set by the developer; this is currently set to 600 seconds, and is set in the source of the controlling program "ccef.c" (in file ~ceilidh/Tools/SOURCE/ccef.c ) as the "#define MIN_GAP" constant (in seconds). You must then re-make the program. The value can be reset on a per-exercise basis by a line

MINGAP=60

(in seconds) in the exercise type file.

The number of submissions can be limited by a line

MAXSUB=10

in the type file. The student will receive messages informing them how many submissions have already been made, and what the upper limit is, typically

You have now made 4 out of a maximum of 6 submissions.

If a negative value is set as in

MAXSUB=-5

the student will be informed of the number of submissions made after (in this case) the 5th, but no upper limit is set.

The overall mark awarded is made up from a number of sub-marks, the weights and sub-tests being specified in the mark.act file. A typical mark.act file might be

60 Dynamic: CDynCorr -v1 $C $U $E
25 Typographic: typog_C -F1 < prog$U$E.C
15 Features: CFeature $C $U $E
Save o C

The "Save" line has already been explained. The other lines have significance as follows.

The first three lines specify that the overall mark is to be calculated from three tests, dynamic, typographic and complexity. The dynamic tests are to be scaled out of 60, the typographic test out of 25, and the features test out of 15. If these marks do not add up to 100, they are further scaled during the marking process.

Each sub-test will also include weights for its sub-components, of which details will be given later.

The overall mark awarded is formed from a number of components, of which the following are provided.

Dynamic correctness
Dynamic efficiency (C on SUNs only)
Typographic style
Complexity level
Features of the program
Structure of the program

Other assessment programs can be added in the course.<course>/bin or Tools directories, and called in the same way. The only restriction is that they MUST produce as their last line a line such as

Score 85

to indicate the mark awarded. These components are then combined to form a single percentage mark.

The mark.act file is set up when the exercise is created initially with one line such as

Save o C

for saving the program source. Other lines are added as the tests are set up by the developer, or can be added by hand using the ema (edit mark actions) command.

Any marking program must produce output ending with a line such as

Score 75

as a percentage.

The variables in the mark.act file introduced by dollar symbols have significance as follows.

$C	course abbreviation	"c" or "pr1")
$U	unit abbreviation	"1" or "2" or
$E	exercise abbreviation	"1" or "hal"
$X	the exercise directory full pathname
$S	the program file suffix	"c" or "C"

The default call of the marking program will show the student the sub-totals for each marking sub-program, and the final weighted total. If in the type file there is a line

MARKVERB=0

(marking verbosity = 0) none of these will be displayed, although they will be stored as usual in the collected marks file.

MARKVERB=1

shows the student the total only.

MARKVERB=2

gives the subtotals and the total as at present. The default value is 2.

We now describe how to set up marking using the various built-in marking programs.

6.4.1. Dynamic correctness

These are the most difficult tests to set up. The vital line in the mark actions file is

50 Dynamic: CDynCorr -v2 -F0 $C $U $E

The flags to CDynCorr control the amount and type of information displayed to the student, and are as follows.

-v0: no information
-v1: overall total score out of ...
-v2: individual test scores (plus the previous)
-v3: individual RE scores (plus the previous)
-F0: no information
-F2: messages about "incorrect exit code", "core dumped", ...
-F3: all REs failing (plus the previous)

The student executable program is run against various sets of test data (or shell scripts) provided by the developer, and an oracle searches the program output for signs of correctness in each test. For details of how the oracle program works, and how its marking weights are set, see the oracle [ref 8] [content] document.

All the tests and associated information can be set up using the developer's sd (set dynamic) command.

Each test must have a short title, and a total mark. These are specified in the model.dv file, which might take the form

Dynamic Correctness
25 Simple data
20 Zero denominator
15 No data
35 Longer test

This specifies four tests (the first line must be there, but can otherwise be ignored): the first test (worth 25 marks) using simple data, the second (worth 20 marks) involving care over a zero denominator, the third (15 marks) requiring a check for the existence of data, and the last (35 marks) being a substantial test. If the marks do not add up to 100, the final total is scaled appropriately. This percentage total is then scaled for the overall mark awarded as specified in the mark.act file. The chosen titles can be as helpful (as above) or unhelpful ("Test 1", "Test 2") as you require.

The test data for the first test will be in the file model.d1, for the second in model.d2 etc. The oracle data (a set of regular expressions to search for the required information in the output of the program under test, see below) for the first test will be in the file model.k1, for the second in model.k2 etc. The data should have public read permission, the oracle files should not.

The default oracle program is in the Tools directory; another oracle can be specified by inserting, for example

C_ORACLE=search

in the type file.

The default oracle works (by calling awk ) using sets of regular expressions (REs) devised by the developer; these will usually be designed to recognise alternative output formats which are close to what the developer asked for, but cannot in the end recognise every semantic possibility. Each RE can be preceded by a mark (it is otherwise given a default mark of 10), and/or a limitation on the number of occurrences (full marks are otherwise awarded if the RE is found at least once in the program output). The percentage mark awarded for each dynamic test is based on the sum of the marks for the REs found divided by the total of all possible marks.

Suppose we wish to create a file of REs for the oracle to check output which should read roughly

A temperature of 124.6 degrees F converts to 32.7 degrees C

The "oracle" file of REs might be

10:32.7
5:124.6
2:degree
1:temp
==0:20:[Ee]rror
~10:Fail

This awards 10 marks if the result contains the string 32.7 (probably to most important test of correctness), a further 5 marks if it contains the string 124.6 (at this stage students were asked to print out copies of all entered data), and so on. If the output contained the string Error or error then 20 marks would not be awarded (read the "==0:" as "there must be exactly 0 occurrences of ...").

The last line will subtract 10 marks if a line containing "Fail" is found; the maximum possible mark for scaling purposes does not include any of the negative marks. The total awarded is then scaled out of the total of the possible positive marks, 38 in this case, and given as a percentage. The full stops in the above REs (representing decimal points) should strictly be expressed as

10:32\.7
5:124\.6

since a full stop in an awk RE represents any one character, whereas an escaped full stop represents a full stop.

The REs could be further extended (they are passed to an awk program) if the lecturer so wishes, to specify that the numeric values are surrounded by non-numerics, for example, using the standard awk RE notation such as

[^0-9]18[^0-9]

and alternatives given as in

cms|centimetres

The possibility of rounding errors may necessitate the RE

3\.141[56]

to allow for either "3.1416" or "3.14159". See the original awk reference [ref 9] [content] or the relevant section in a book such as Bourne's [ref 10] [content] for exact details of permitted REs, and the "oracle" document for further details of the oracle program.

The marks for each dynamic test can thus be built up as a series of weighted RE scores.

If the program under test fails catastrophically at run time (core dump, or stuck in a loop) useful results may be difficult to guarantee. The system is designed to award zero for that dynamic test, and continue trying all remaining dynamic tests. The program is killed after 5 seconds of processing if it is still running, using the ~ceilidh/Tools/run command (source in Tools/SOURCE/run.c if required). This is a configurable parameter; to change it, edit the defined value for MAX in the source in the source file run.c and remake.

Each dynamic test needs a file of REs for the oracle, the files being named model.k1, model.k2 etc for successive tests. Each test is driven either from a file of test data, named model.d1, model.d2 etc (the executable is run from the test data), or from a shell script, named model.s1, model.s2 etc (in which case the shell script is executed as given).

Other oracles can be used by assigning

C_ORACLE=my_oracle

in the exercise type file, which will look for a program ~ceilidh/Tools/my_oracle and will call it with the keywords file as argument, and the student output as standard input. It will expect the output of the oracle to be of the form

Score 95

out of 100.

The separate dynamic tests could by given explicitly in the mark.act file by lines such as

10 Dyn test 1 : prog45 < $X/model.d1 | oracle $X/model.k1

but beware of problems of the executable getting stuck in a loop.

6.4.2. Dynamic efficiency

Run-time efficiency is not generally an important consideration in early programming courses. The C system on SUNs (not the C++ system) examines the number of times each line of the code is executed (using tcov on the SUNs), and compares the maximum line execution count with those for the developer's model solution. If this test has been set up using the

se

(set efficiency) developer's command, the maximum line execution count on the student program is compared with that on the model program. If the student's count is less than the model count, 100% is awarded; if it is equal, 99% is awarded; if it is greater,

100 * model count / student count

is awarded. See the shell script Tools/CEff_c for details.

6.4.3. Typographic analysis

The program typog_c reads C program source ( typog_C reads C++ source) and computes various statistics associated with maintainability and readability of the source, such as

Average characters per line	% blank lines
Average spaces per line
Average function length	% good function
Average identifier length	% names with good length
% define's
% number comments	% chars in comments
% indentation	% indent errors

The students can obtain the system definition of features such as good indentation from the help facility, where it is defined as

Either

or the following three conditions must hold.

The present system also insists (since it is recommended in early lectures) that closing curly brackets must be followed on the same line by a comment if more than 10 lines after the corresponding opening curly.

Each of the general metric factors is measured, and compared with set values. There will be a range of values within which full marks are awarded, and a wider range within which part marks will be awarded. Beyond the outer range, no marks are awarded.

The relevant line in the mark actions file is of the form

25 Typographic: typog_c -v0 -F1 < prog$U$E.c

where the flags again control the information display as in The "F" (feedback) flag gives

-F0: no feedback
-F1: reason for the biggest loss of marks
-F2: reasons for all marks lost (plus the previous ...)
-F3: the position of all indenting errors (plus the previous ...)

The "-v" (verbosity) flag as an alternative gives

-v0: no information
-v1: a total score (out of 100)
-v2: a score for each metric (plus the previous)
-v3: all the parameters for all the metrics (plus the previous)
-v4: other calculated values (plus the previous)

Thus by using the "F" flag, the student can be given English language help about the program weaknesses. It is obviously (in general) bad to give too much detailed help. Output is jumbled if both "F" and "v" flags are used; they are intended for different applications.

The marks and range parameters for each of the factors involved either take on default values from data initialisations built into the typog_c etc programs (to change the defaults, edit the source and remake), or can be supplied on a per-exercise basis from a file model.tv in the exercise directory using the call

25 Typographic: typog_c -v0 -F1 -f $X/model.tv < prog$U$E.c

to name the file containing the typographic weights. A skeleton model.tv file can be generated in the current directory by calling

typog_c -w

The developer may choose to vary these parameters to emphasize different aspects of the marking at different points in the course. Each line of the model.tv file is of the form

AIDL   10    1    4   10   15 Average identifier length
%NGL    1    0   45  100  100 % names with good length

where the entries are

A 4-letter code
The maximum mark for this feature

The next four numbers define five ranges
If the program value is less than the first number,
no marks are awarded.
If it is between the first and second numbers,
part interpolated marks are awarded.
If it is between the second and third numbers,
full marks are awarded.
If it is between the third and fourth numbers,
part interpolated marks are awarded.
If it is above the fourth number,
no marks are awarded.

A title for this metric

The developer's

st

(set typographic) command shows the default score for the developer's source, and if the mark awarded is not 100% (it rarely is!) offers repeatedly editing of either the program source, or of a local file of typographic parameters.

If this is the first developer's call of st the system will ask for a total mark to be awarded, and will add an appropriate entry to the mark.act file.

To change the default values for the whole course, edit the source in Tools/SOURCE/typowgts.c, where there is a table of the above values held in a structure array, and remake.

The student sees only the overall typographic mark. Tutors and teachers can see a more detailed breakdown if they wish using the mt (mark typographic) tutor command.

A call of

cat source.C | typog_c -p

will produce a local model.tv file with parameters tweaked to give the named program a typographic score of 100%.

6.4.4. Complexity analysis

This follows a similar pattern to the typographic analysis, but in this case the metrics for the student program are not compared with absolute values, but with those for the developer's model. The factors involved include the following.

Number of reserved words	Number of includes
Number of gotos	Number of conditionals
Number of loops	Depth of loops
Number of operators
Number of braces	Max depth of braces
Number of square brackets	Max depth of square brackets
Number of round brackets	Max depth of round brackets
Number of function calls
Number of numeric denotations

The line in the mark actions file is

25 Complexity: compl_c -v0 -F0 -x $X/model.cm -f $X/model.cv < prog$U$E.c

and the flags have the same effect as in the typographic program.

The metrics of the model program set up by the developer must first be created in file model.cm in the exercise directory using

cat source.C | compl_c -m

When the student's source is analysed by

cat source.C | compl_c <model-metrics-file>

the student program metrics must be within factors for each metric set by the developer (usually between 50% and 200%) of those of the model solution*.

cat source.C | compl_c -f model.cv <model-metrics-file>

The developer's command

sc

(set complexity) operates in a similar way to the "set typographic" command.

If there are significantly different possible solution programs to a given problem (e.g. a recursive and an iterative solution), it should be possible for the developer to supply more than one model, and for the system to compare the student program with all of these models, and to choose the best fit. This is not yet implemented.

6.4.5. Features of the program

When assessing program sources by hand/eye, one typically looks at the general layout (the typographic marks above), and for particular problem dependent features such as

The marking system allows the setting up of files of REs in the exercise directory, in files such as model.ft which will then be marked using the oracle against the program source with comments and strings removed. Such "features" files can be set up using the sf (set features) developer's command. The command performing this marking is Tools/CFeature shell script.

The features to be found can be placed in several files. The file

model.ft

is typically used for a single features oracle.

If the entry in the mark.act file is

20 Features: CFeature -v0 -F0 $C $U $E

the flags are interpreted as

-v0: no scores given
-v1: total score given
-v2: sub-scores given (plus the previous)
-v3: individual RE scores given (plus the previous)

Extra features of the oracle useful at this point are the limitations on the number of occurrences of each RE. A line for the oracle can be of the form

<=1:3.141[56]

meaning that the RE after the colon must occur less than or equal to once.

To require that a program involving the conversion factor 60 uses that factor only once (presumably declared as a constant), and the value 59 not at all, use the oracle file

==1:60
==0:59

Experience can be gained by looking critically by eye at student solutions to a given problem, and the features oracle can then be improved for future years.

6.4.6. Program structure

To look for structural weaknesses in the student's program, it can be re-compiled in the present GNU C++ compiler with

g++ -c -Wall

or for C programs we can use the lint command. Certain warning messages (such as variables declared and not used) are picked up by an oracle, and a score awarded.

The developer command

ss

(set structure) allows the developer to set up such an oracle file model.st in the current exercise. If it is not set up, a standard file model.st in the course directory is used. The standard file also includes scaling, so that a few errors can produce a significant drop in the marks awarded.

The commands performing the work are Tools/CStr_C for C++ and Tools/CStr_c for C.

6.4.7. Summary of files in the exercise directory

In a simple one module C++ programming exercise, the files which must be created in the exercise directory before the students access the exercise would be as shown in figure 5. There may be other sources, objects and header files and perhaps a Makefile in more complex examples.

7. Setting up a new programming exercise

The developer will normally use commands from the developer's menu to set up a new exercise in approximately the order shown in figure 6.

With the "sd" command to set up the dynamic tests, typically type commands in order as shown in figure 7.

We now summarise in more detail the actions needed to set up a new exercise. Suppose that the chosen exercise is a program to convert seconds into minutes and seconds, within unit 2 of the course "pr1". All the appropriate files could be set up by hand in Unix. We assume that the developer exercise menu will be used.

Choose an identifier for the new exercise, such as "9".

Login as "ceilidh", and move to the defined existing course and unit and go into "developer" mode. Type

sc pr1
su 2
develop

Then create the new exercise with "nx", which will ask for the exercise number and title.

Command?	nx
Ex number?	9
Title?	Seconds to minutes

We are now in the new exercise, in developer mode, and have to set up the files listed in figure 8. The exact order is not vital, but that suggested above is suggested.

Model solution program

Use "ep" to edit the program; you can edit, compile and run the program to develop it under Ceilidh using "cm" and "run", or outside if you wish. You would do the latter by moving to the directory

~ceilidh/course.pr1/unit.2/ex.9

or by simply obtaining a shell within Ceilidh by typing

!

The source should be in "model.C" (or "model.c" ...) linked to "prog29.C" (or ".c" ...), and the executable should be in "model" linked to "prog29".

The marking process

The dynamic tests are the most complex to develop. This stage requires an executable program in the exercise directory. Type sd (set dynamic), which will offer you the following sequence.

Another test?

Type "c" to create a new test. Each dynamic test requires a file of test data (or a driving shell script), and an oracle file of recognisers (regular expressions) for the output. The sequence of prompts for a new test is:

Data file or shell script?

reply "d" if you wish to run the test from a data file. The editor is then called automatically; use it to create an appropriate data file (or shell script). The data file in this case might be just a single integer

2345

If you chose a shell script, it might be

prog29 << ++++
2345
++++

The system then runs the program against this data file, and its output is displayed. You are then invited to edit the file of regular expressions (REs) for the oracle which seeks to find out whether the output is correct. The file is initialised to a copy of the program output, so that you can simply edit out the unimportant parts. This will then be entered into the oracle output recogniser file. It is suggested that the file should be edited in the following stages.

• For a simple test, just leave the essential output values which should be in the output, each on a separate line. If the number in the output should be "123", it is prudent to give the RE as

  [^0-9]123[^0-9]
  

  With floating point values, allowance should be made for rounding, or for different precisions. The value "1.42857" might be recognised using

  [^0-9]1\.428[56]
  

  If the output is expected to be two numbers such as "12\ 23", observe that the single expression

  [^0-9]12  *[^0-9]23[^0-9]
  

  will offer only full marks or no marks. If you use instead

  [^0-9]12[^0-9]
  [^0-9]23[^0-9]
  

  the student will get half marks if only one of the two expressions is found.

• For later tests, enter also recognisers for ancillary text, to recognise "12 minutes" use

  [^0-9]12[^0-9]
  [Mm]ins|[Mm]inutes
  

  or a variant thereof, permitting the user to use "Mins", "minutes" etc as alternative words. To give more weight to the numeric value than to the surrounding text, use the mark specification facility of the oracle as in

  10:[^0-9]12[^0-9]
  5:[Mm]in(ute|)s
  

  in which each RE is preceded by a numeric mark and a colon. It is unwise to form the complete requirement into a single RE, since if you specify, for example

  [^0-9]12[^0-9] *[Mm]in(ute|)s
  

  the student obtains zero or full marks, with no possible intermediate values.

When you leave the editor after creating the oracle RE file, the system runs the program against this oracle, and prints the mark awarded as a percentage.

You are then asked if you want to repeat the cycle

edit test data
edit oracle REs
look at the resulting mark

The default is to repeat the cycle if the mark is not 100%. If you indicate the end of this test, you are asked for

Marks for this test?

Title of test?

Edit model.dv file?

Control then returns to the "Create another dynamic test?" query. Type q to quit the loop. You will then be asked for the overall "dynamic" total to be entered into the mark.act file.

If you re-enter the sd command, you can skip the earlier tests by typing s .

Typographic marks

Use the command "st" to set the typographic marking. If the immediate result is not 100%, you may choose to stick with the system default metrics, and edit your program to improve its typographic marks relative to the built-in default values (the test shows you a breakdown of each individual metric), or set up an exercise-specific metrics model.tv file and insert adjusted metrics; the latter should not often be necessary. In each case the cycle

look at metrics
edit either the program or the weights

repeats until you are satisfied with the mark awarded.

The system then asks for a total typographic mark for the mark.act file.

Set complexity marks

The "sc" command to set complexity operates in a similar way to the typographic marking, except that a file storing the complexity metrics of the model program must be created. This is done each time that the "sc" command is entered.

The command then follows the pattern of the "set typographic" command.

Set features mark

If you wish, an oracle can be set up to recognise particular good or bad features which might appear in the program source for this particular problem. This facility is set up by the "sf" command, and is typically used for features such as one might look for by eye if hand marking program sources. This oracle usually requires the extra facility for specifying limits on the number of occurrences of given features.

==0:59

to specify zero occurrences of "59".

<=1:2\.54
<=1:0\.39

or perhaps

==1:2\.54|0\.39

to specify exactly one occurrence of either "2.54" or its inverse.

The text in comments and strings in a program is normally extracted before it is passed to this oracle. See the oracle document for further examples.

You will be offered the oracle file to edit, then the model solution will be marked against it. This process can be repeated.

Finally, you are asked for a total features mark, to be entered into the mark.act file.

Complete marking

A complete marking should then be performed with the "mk" command. The ratios of the subtotals for the separate aspects of the marking are adjusted using the ema (edit marking actions) command. Check the weights, and delete any lines corresponding to marking components which you wish to ignore.

Setup actions

The "esa" ("edit setup actions") command allows actions which take place when the user types "set" at the start of solving a particular exercise. A typical setup actions file is

Copy model.sk prog29.C
Copy header.h

This causes the skeleton program to be copied to the user's area as "prog29.C", and the header file "header.h" to be copied to the same name in the user area. An appropriate default file will be created automatically. During copying, the string "$USER" is replaced by the user's login, "$NAME" by the user's full name, and "$DATE" by the current date.

The question

Finally, the question file should be edited and checked. It is often easy at this stage to insert an example of the model program's output (by running the model against the first test data) and perhaps of an input file (by inserting the first test data). The question should be reasonably specific in all aspects.

Leaving the exercise mode

When you type "q" to quit the exercise mode as a developer, checks are made that the essential files are all there. You will be asked whether you wish to leave an executable in the directory; if you do, the "run model solution" commands become available to the student.

8. A complete example from Ceilidh

For the C++ "pr1" unit 2 exercise 5 the non-obvious files are as follows:

Program ("model.C"):

#include <stream.h>
// Convert feet and inches to centimetres
// Written by EF 1988
const int INSPERFT = 12;
const float CMSPERIN = 2.54;
main ()
{
    float cms, ins;
    int   ft;
    cout << "Type feet and inches: ";
    cin >> ft >> ins;
    if ( ins < 0 || ins >= INSPERFT ) {
        cout << "Error: Inches out of range\n";
        exit( 0 );
    }
    if ( ft > 0 ) {
// Avoid explicit constants anywhere
        cms = ( ft * INSPERFT + ins ) * CMSPERIN;
    } else {
        cms = ( ft * INSPERFT - ins ) * CMSPERIN;
    }
    cout << ft << " feet "
        << ins << " inches is "
        << cms << " cms\n";
} // end of program

Skeleton ("model.sk"):

#include <stream.h>
// Convert feet and inches to centimetres
// Written by EF 1988
const int INSPERFT = 12;
const float CMSPERIN = 2.54;
main ()
{
    float cms, ins;
    int   ft;
    cout << "Type feet and inches: ";
    cin >> ft >> ins;
//////////////////////////////////////////
//
// Put your program instructions here
//
//////////////////////////////////////////
    cout << ft << " feet "
        << ins << " inches is "
        << cms << " cms\n";
}

Test data 1 ("model.d1"):

12 9.5

Oracle RE 1 ("model.k1"):

389.89

Test data 2 ("model.d2"):

-27 6.5

Oracle RE 2 ("model.k2"):

-839.470

Test data 2 ("model.d3"):

9 1.2

Oracle RE 2 ("model.k3"):

9
ft|feet
1.2
ins|inches
277.368

Overall marking ("model.dv"):

Dynamic Correctness
50 Simple test
0 Negative distance
50 Check "feet" "ins"

Features ("model.ft"):

==1:12
==1:2.54

We have no model.tv or model.cv files, but are using the default metrics.

Setup actions ("setup.act")

Copy model.sk prog25.C

Marking actions ("mark.act")

60 Dynamic: CDynCorr -v1 $C $U $E
25 Typographic: typog_C -F1 < prog$U$E.C
15 Features: CFeature $C $U $E
Save o C

The complexity metrics file model.cm is set up if required by the system.

References

1: Steve Benford, Edmund Burke, Eric Foxley, Student's Guide to the Ceilidh System (2.4), LTR Report, Computer Science Dept, Nottingham University (1995)
2: Steve Benford, Edmund Burke, Eric Foxley, N H Gutteridge, A Mohd Zin, Tutor's Guide to the Ceilidh System (2.4), LTR Report, Computer Science Dept, Nottingham University (1995)
3: Steve Benford, Edmund Burke, Eric Foxley, Teacher's Guide to the Ceilidh System 2.4, LTR Report, Computer Science Dept, Nottingham University (1995)
4: Steve Benford, Edmund Burke, Eric Foxley, The Design Document for Ceilidh Version 2, LTR Report, Computer Science Dept, Nottingham University (1993)
5: Steve Benford, Edmund Burke, Eric Foxley, The Command Line Interface to Ceilidh, LTR Report, Computer Science Dept, Nottingham University (1993)
6: Eric Foxley, Question/answer exercises in Ceilidh, LTR Report, Computer Science Dept, Nottingham University (1993)
7: Abdullah Mohd Zin, Eric Foxley, The Oracle program, LTR Report, Computer Science Dept, Nottingham University (1992)
8: Abdullah Mohd Zin, Eric Foxley, The Oracle program, LTR Report, Computer Science Dept, Nottingham University (1992)
9: Alfred V Aho, Brian W Kernighan, Peter J Weinberger, Awk - A Pattern Scanning and Processing Language, Programmer's, Manual A T & T Bell Laboratories (1985)
10: Steve R Bourne, Unix System V, Addison-Wesley (1989)
11: Abdullah Mohd Zin, Eric Foxley, Automatic Program Quality Assessment System, Proceedings of the IFIP Conference on Software Quality S P University, Vidyanagar, INDIA (March 1991)

Notes converted from troff to HTML by an Eric Foxley shell script, email errors to me!