Developer's Guide to Ceilidh S D Benford, E K Burke, E Foxley ltr @ cs.nott.ac.uk Learning Technology Research Computer Science Department University of Nottingham NOTTINGHAM NG7 2RD, UK ====================================================================== Note: Revision for Ceilidh 3.4 October 23, 1996 ====================================================================== Contents Introduction to the developer's role Overview of the Ceilidh system Administrator commands at the system level Developer commands at the course and unit level Developer commands at the exercise level Amending an existing programming exercise Setting up a new programming exercise A complete programming example from Ceilidh 1. Introduction to the developer's role The users of the Ceilidh system fall into the classes Students Those using the system as a learning tool. They have read permission only to public documents, and to their own sub- mitted work and marks. Tutors Those with access to student progress monitoring. They essentially have read access to all student work and marks for a particular course. Teacher Course administration/management, set- ting the maximum number of submissions, opening and closing exercises, etc. They have write access to selected items in the file system for a particular course. Developers Course material creation and management. They essentially have write access to all material for a particular course. In addition, there is a system support level, usually pro- vided by the staff who run the computing service on which Ceilidh is running, and who installed the Ceilidh system. This level is not at present distinguished from developer facilities. This guide is for course developers, and should be read in conjunction with the other Ceilidh guides, the student guide[4] , the tutor guide[5] , the teacher guide[6] and the design document[2] . The course developer The course teacher has the facility (using the Unix SUID permissions for security) to amend a limited number of Ceilidh parameters related to information in any course for which their username is authorised. Further facilities involving writing to the Ceilidh file system are the pro- vince of the course developer; these include o changing detail such as the test data for an existing exercise; o setting up new exercises in an existing course (both of these are relatively straightforward operations once you are familiar with the principles involved); o adding new marking tools to an existing course, and incorporating them into the marking process; and o setting up a completely new course (a complete new set of marking tools will need to be written). Straightforward operations can be performed using Ceilidh itself, through menus or the X interface. Other more unusual operations will require direct commands in Unix, so the general principles of operation must be understood. Access to developer facilities To use the developer's facilities, you will have to log in as "ceilidh." It is recommended that this is normally be done indirectly using "rlogin" or "xlogin" rather than directly by using a password. At the moment, there is no distinction between different users logged in as "ceilidh." Course developer facilities and system administration facilities will both be available, and both are described in this document. This will be rec- tified at a later stage. Most administrative and developer facilities are covered by menu items as described below. However, some specialised operations (such as the installation of new marking pro- grams) will need to be done by hand through the normal Unix filestore mechanisms. You will thus need to understand the main principles of operation of the system, and the layout of the filestore. In this document we will summarise information in the fol- lowing order. (i) Overview of the Ceilidh system. We describe the file- store layout, and the general principles of operation. (ii) We describe the system administrator commands available at the Ceilidh system level. These are available to anyone logging in under the "ceilidh" username. (iii)We describe the developer commands available at the course level. (iv) We describe the developer commands at the exercise level in the general case. (v) Amending an existing programming exercise. You may wish to amend the text of the question, or the test data (in which case you must amend the corresponding output recogniser) or the weighting of the different components of the mark. (vi) Creating a complete new programming exercise. This is not a difficult operation once the principles are known. (vii)We give a complete example of setting up a programming exercise. 2. Overview of the Ceilidh system We will describe here the general layout of the Ceilidh file system, and the function of the main directories and com- mands. The support provided by the Ceilidh system falls into the four distinct levels System Department-wide view, all courses and papers can be viewed. Course One course (teaching module). Certain users will be authorised for teacher permissions related to a particular course, and certain for tutor facili- ties. Unit Documentation and exercises for one course unit of a course. Exercise Assessment definition for one item of coursework. Filestore layout The current filestore structure is headed by a directory which we will refer to as "~ceilidh." The menu commands (dumb terminal menus, one command for each category of user at each relevant level, all implemented as shell commands) are in the directory "~ceilidh/bin.mnu," the X-windows ver- sion commands are in the directory "~ceilidh/bin.x," the command line interface commands (one command for each stu- dent menu command, see the CLI document[3] for more details) themselves in the directory "~ceilidh/bin.cli," the general tools (shell scripts and programs called by all the command systems, menus, command line interface commands and X ver- sions of Ceilidh, those implemented as shell scripts have a name started with "C", executable programs have a name starting with a lower case letter) in the directory "~ceilidh/Tools." If you run multiple architectures, there will be additional directories for each architecture's executables, with names such as "~ceilidh/bin.sun4.3" where the name incorporates specification of the machine hardware and operating system obtained using the "uname" command. Additional textual information is stored as follows. o Help information in the directory "~ceilidh/help." o Ceilidh guides and research papers in the directory "~ceilidh/papers." o Miscellaneous data files in "~ceilidh/lib." o Audit trails in the directory "~ceilidh/audit." Below the "~ceilidh" directory each course has a directory such as "~ceilidh/course.pr1" for the course "pr1" (Intro- ductory C++ programming). This contains a directory "~ceilidh/course.pr1/bin" for any special marking commands for this course. Below the course directory are directories for each course unit, typically "~ceilidh/course.pr1/unit.3" to contain the third unit of the "pr1" course. The unit name is always numeric, and indicates the sequencing of the units. Below this are directories for each item of course- work, such as "~ceilidh/course.pr1/unit.3/ex.1" for the first exercise or example. Exercise names can be any string of up to three characters* ====================================================================== Note: * At the moment we generally use numeric names for exercises in some courses, and mnemonic strings in others, but mnemonics appear more convenient since they can reflect relationships between exercises in different units. ====================================================================== and exercises are not necessarily in a particular sequence. All course, unit and exercise directories include a file "title" containing a one-line ASCII summary title of the course, unit or exercise. The course directory should also contain a "summary" file typically containing a summary of the lectures, times, courseworks set and hand-in dates, kept up-to-date by the teacher using the teacher administrative facilities. The unit summaries are assumed to be set up by the course developer, and the course summary to be the responsibility of the teacher administrator. The contents of the directories and the commands available at each of these levels are described below. 3. Administrator commands at the system level We will describe now the commands, all at the system level, intended for the Ceilidh administrator. Both the adminstra- tors and course developers need general write permission to the Ceilidh file system, Ceilidh does not disctinguish between them. All the administrator facilities are avail- able to anyone logging in as Ceilidh. In response to the command ceilidh the menu for a developer or system administrator should appear as shown in figure 1. _____________________________________________________________________ |Ceilidh system top level menu: | |lc list course titles | sc move to named course | |vp view papers | pp print papers | |clp change printer | h for more help | |co make a comment to teacher | q quit this session | |fs find student | ft find tutees | |=========================================== | |Additional tutor menu: | |ss summarise one student | |=========================================== | |Additional administrator menu: | |em edit ceilidh system motd | eh edit help files | |nc create new course | audit administer audit trails | |rdc reset default course | esv edit shell variables | |=========================================== | |System level command: | | | | Figure 1 : Ceilidh administrator menu | |____________________________________________________________________| The ordinary user items (see the Student Guide to Ceilidh) appear at the top, and are followed by additional menu items, one for the tutor (see the Tutor Guide), and others relevant to the system administrator. If the extra items do not appear, you are not logged in as the special username allocated for this purpose, normally "ceilidh," but possibly another name set during the initial- isation process. The additional administrative facilities include the follow- ing. o Edit system message-of-the-day (~ceilidh/motd) o Edit help files (all of the help files are offered) o Create a new course o Amend the values of environment variables o Set up the default course o Perform auditing operations These commands will now be described in detail. Default course Using the "rdc" command, the system administrator can set up the default course which will be set when a new user calls Ceilidh at the top level for the first time. Thereafter the user will pick up as default the course they most recently used. Within a course, it is the teacher (as part of the "new week" "nw" command) who sets up the default unit and exer- cise numbers. Edit help files ("eh" command) The help files live in the directory "~ceilidh/help." These are the files which are listed when the user types the "help" command on any menu. You may wish to amend them to insert pointers to local documents, staff or phone numbers. The names of help files are of the form . where the "" is sys for system level help cse for course and unit level help ex... for exercise level help and the "" is dev for developer's help tch for teacher's help tut for tutor's help usr for student help Thus the teacher's course level help (the information printed when the "help" command is called on the teacher menu) is in the file "cse.tch." At the student exercise level, there are several files expr1.usr help when no compilable program source exists exin1.usr help when no interpretable program source exists expr2.usr help when source but no executable exists expr3.usr help if source and executable both exist exes.usr student help for text submission exercises extxt.dev developer help for text submission exercises exmc.usr student help for text question-answer exercises exqa.dev developer help for text question-answer exercises Miscellaneous other help files include sys.aud audit facilities en.dev edit notes enms.dev edit master notes The papers directory All documents viewed by students using the "vp" command are stored in this directory. Each document has three associated files. The document "Student" (the student guide) for example has a file "Student.ttl," containing a single line ASCII title (this is displayed by the "vp" command), "Student.cat," a file of ASCII contents viewable through the "$PAGER" viewer, and a file "Student.ps" of PostScript, printable to a PostScript printer, and viewable through a PostScript viewer such as "ghostview" if available. To add any new local documents for viewing, at least the first two of these files should be inserted by hand in the "~ceilidh/papers" directory. ,LP There may also be "Student.ohp," a file of overheads in PostScript. Edit shell variables ("esv" command) Each Ceilidh shell environment variable (in the file "~ceilidh/bin.mnu/CShlVars" set up during the installation process) can be set to a new value. These are the global variables; variable relevant to a particular course are store in a similar way in the course "type" ot "type.tch" files. The variable names are displayed in order; a new value can be given, or "return" used to leave the current value set. The new values take effect for users upon their next entry to CEILIDH. A summary of the relevant variables is shown in figure 2. ______________________________________________________________________________ |Variable Purpose Typical value | |DITVIEW Default viewer for ditroff files xditview | |EDITOR Default editor emacs, vi | |LPR_ASCII Printing command for ASCII files lpr | |LPR_PS Printing command for PostScript files lpr | |PAGER Default pager more | |PRINTER_ASCII Default printer for ASCII print command | |PRINTER_PS Default printer for PostScript print command | |PRINT_PAPER Permit users to print papers (yes or no) no | |PSVIEW Default viewer for PostScript files ghostview | | | | Figure 2 : Course developer shell variables | |_____________________________________________________________________________| Maintaining audit trails ("audit" command) There are minimal elementary auditing facilities built into the system from release 2.1. There are no audit trail analysis facilities provided, just their generation. Suggestions for appropriate analyses would be welcome. Separate trails can be kept of all use of certain Ceilidh operations (e.g. printing or compiling), or of all use of Ceilidh by a particular user. These separate trails can each be switched on or off at will. If a trail file of the appropriate name exists, the trail information will be appended to it whenever necessary. It must be remembered that such facilities can result in the collection of large quantities of data; this must be archived and removed regu- larly. The files containing the trails are stored in the directory "~ceilidh/audit." The audit administrator command calls up a separate auditing menu. The auditing menu allows the developer/administrator to perform the following functions. o Check the state of the currently set audit trails. This command gives the name and size of the audit trail file for each facility and student currently switched on. Files related to facilities always start with an upper case letter, those for students are names with the student's login name. Sizes and dates can be checked to see how big the files are, and which files need to be archived. We could also give the number of entries in each trail if required. There are separate commands for showing separately the feature trails set, and the student trails set. o Show a list of those facilities not currently switched on. These are the facilities which could be added. o Switch on the auditing of a new trail (either a par- ticular facility or a named user). An empty trail file of the appropriate name is created. o Switch off the auditing of a particular facility or student. The named trail file will be removed. The user will first be invited to save any existing stored information. If required, it will be appended to a trail in the directory "~ceilidh/audit/archive." See below for storing the archived files elsewhere. o A particular audit file can be viewed using the stan- dard pager defined by the environment variable "$PAGER." Each record is a single line of the form course:unit:exercise:user:date-time:user:facility:information These follow the same format as mark information. If the appropriate trail files exist, an entry will be appended to both the facility file and the student file whenever that event occurs. o The user can edit an audit trail file, perhaps to remove unwanted information after it has been inspected, but leave other entries intact. o An existing trail can be archived, i.e. appended to a file of the same name in the directory "~ceilidh/audit/archive." To use a different archive directory, edit the setting of the "ARCHIVE" environ- ment variable in the command "~ceilidh/Tools/CAudit." o The archived trails can be viewed or edited as for the current trails. The "Error" audit should always be switched on, and should be checked regularly. Create a new course ("nc" command) We consider the creation of the skeleton of a new course to be an administrator function. When at some future date we distinguish more carefully the functions of administration from those of course development, the developer will have access to a particular course after it has been created. The "new course" command requests a course abbreviation suitable for use as a directory suffix (e.g. the C++ pro- gramming course "pr1" is contained in the directory "~ceilidh/course.pr1" ) and a one-line course title. The course name should be at most three characters, with no spaces, to conform to MS-DOS practice. The command then requests the login names and full names of staff who will be permitted to administrate this course. It sets up the course directory, its "bin" and "bin/SOURCE" sub-directories, and the "title" and "staff" and "tutor" files. The course "bin" directory may eventually contain compilation and marking shell scripts and programs specific to this course, with their source and "Makefile" in "bin/SOURCE" . If there is a command "CCompile" in the course "bin" direc- tory, it is used for all compilations in preference to the main command in "~ceilidh/Tools/CCompile." It is called with two arguments as in CCompile and expects a "true" exit to indicate successful compila- tion. The actual compiler called is defined by the CC= environment variable set in the course or exercise "type" or "type.tch" file. Similar considerations apply to "CVCompile" (the verbose compilation option). If you require a debugging command for the students, there must be a "CDebug" command in the "bin" directory. This will be called as /bin/CDebug When the marking process (see later) requests a program such as "typog_c" for C typographic metrics (or "typog_C" for C++) it searches first the course "bin" directory for an executable "typog_c," and if this is absent uses the "~ceilidh/Tools/typog_c" version. The "nc" command then moves into the newly created course, and offers the developer's course menu, see below for details. 4. Developer commands at the course and unit level The developer may move to this level from the system level either by typing sc pr1 or will more normally enter the course "pr1" for example directly, by calling ceilidh -c pr1 either directly or using a preset alias. We normally set an alias "pr1" for this command. There will always be a currently set default unit and exercise name. To use course developer facilities, choose the develop command on the menu*. ====================================================================== Note: * If this command does not appear, you are not logged in as "ceilidh," or the main login username has not been set correctly in the installation process. ====================================================================== You can use instead the command or alias ceilidh -c pr1 -x develop to enter the developer's menu directly on the course "pr1". We normally set an alias "pr1dev" for this command. The menu is now as shown in figure 3. ____________________________________________________________ |Developer's course (pr1) and unit (1) level menu: | |ect edit course title | eut edit unit title | |eus edit unit summary | en edit notes | |cx make copy of exercise | mx rename exercise | |lu list units | lx list exercises | |su set unit number | sx move to exercise (1) | |nu create new unit | nx create new exercise | |ar archive course | zap remove student work | |h help | q return to user menu | |=========================================== | |Type developer course command: | | | | Figure 3 : Developer's course level menu | |___________________________________________________________| The menu commands can be divided into course-related items, unit-related items, and exercise-related items. Course related menu commands ect Edit the course one-line title file. All edit opera- tions are, of course, implemented using the editor specified in the "$EDITOR" environment variable. Defaults for these variables are set up at installation time, or can be changed by the administrator using the "esv" (edit system variables) command. ety Edit the course-level "type" file. The determines the course defaults for variables such as the compiler to be used, the maximum number of submissions, etc. Each variable is offered as for the "esv" administrator com- mand above. lu List the titles of all units in the course. ar This makes an archive copy of the course directory and all its contents (including student solutions and mark files, and weights and mark scaling files), using a standard Unix "shar" command. The resulting file will be large if there is much coursework submitted. At Nottingham we are required to do this to preserve a copy of the coursework for inspection by the external examiner at a later stage. The Ceilidh group at Not- tingham would appreciate receiving copies of any such files for statistical analysis purposes. zap This removes all student submitted work and marks from the current course, and resets permissions as though the course had been reinstalled. You will normally have archived the material first for future reference. Unit related menu commands su Set the currently selected unit number. eut Edit the one-line title of the currently set unit, in the file "title" in the unit directory. eus Edit the unit summary file "summary" in the unit direc- tory. en Edit notes. See the section on notes below for details. nu Create a new unit for the course. This requests a numeric identifier for the unit, and a one-line title in a similar way to the "new course" command. It sets up a new unit directory "unit." and "title" file under it. At the end you will be offered the removal of the new unit just created. Exercise related menu commands mx Move an existing exercise in the current unit. This requests the exercise identifier, and a new unit number and exercise identifier. The exercise and all its files are moved to the new position, and any files internal to the exercise are edited to reflect the change where possible. You will need to perform hand edits if the notes or other exercises cross refer to "exercise ... of unit ..." which has been renamed. cx Copy an exercise which already exists in the current unit. This requests the exercise identifier, and a new unit number and exercise identifier. The exercise and all its files are copied to the new position, and edited as required. This is often an easy way of set- ting up a sequence of related exercises in different units. nx Create a new exercise for the current unit. This will request an identifier (up to three characters) for the new exercise, and a one-line title. It then requests an exercise type (question/answer, text submission, programming) and details such as the compiler to be used, and the file suffix required. It will then move into the exercise menu (see below) for the new exer- cise. lx List all exercises in the currently set unit. sx Move to a named exercise in current unit. This gives the exercise menu appropriate to the exercise type, see below. Course and unit notes Most of the supporting notes for a course are divided among the course units. For courses we distribute, in each unit there are files notes.cat ASCII copy, viewable by "$PAGER" (e.g. "more") notes.ps PostScript notes for printing or viewing notes.ohp PostScript notes for overhead projectors Any diagrams will not appear correctly in the ".cat" ver- sion. We generate these from a master copy kept at Nottingham, which is in "roff" format. Copies could perhaps be distri- buted if volunteers were prepared to improve them signifi- cantly. We also keep at Nottingham corresponding ".dvi" files in device independent troff (ditroff) form if required. These can be viewed through "xditview" or a similar program. We are in the process of adding ".pdf" files. The student generally accesses the notes on-line through the "vn" command, which calls a PostScript viewer if the user is at an X-terminal and the PostScript viewer has been set, or the user's command defined by the "PAGER" environment vari- able to view the ".cat" version of the notes on the screen otherwise. The command "pn" prints the PostScript notes "notes.ps" to a default printer. The overhead projector slides are in file "notes.ohp" in PostScript, and must be printed by Unix commands if required. The "en" developer command displays an extra menu, which allows the following operations. You can edit the ASCII notes in "notes.cat." You can copy new named files of your own into the "notes.cat" and "notes.ps" files. You can view or print either version of the notes. If you have the master "roff" copy of the notes in file "notes.ms" you will obtain a different menu allowing editing of the master file, and the re-creation of all the other files. We now have another set of notes files at the course level, in files such as "~ceilidh/course.pr1/notes.cat" to include an overview of the course, and coursewide information such as as discussion of the marking process, and the recommended typographic layout for the course. There is no menu entry for editing these course-level files; use raw Unix. Annual tidying of courses Two of the above commands ( "ar" and "zap" ) relate to this activity. They are currently in the developer's course level menu, but should perhaps eventually be in the teacher's menu. Unix file permissions make this difficult. 5. Developer commands at the exercise level To work as developer on a given exercise, you may either first move to the developer course level as above and then type the command sx 5 for example, to work on exercise 5 of the current unit, or you can move into the exercise as an ordinary user and then type develop which should appear as an additional menu item. Each exercise is stored in a directory such as ~ceilidh/course.pr1/unit.2/ex.5 for course "pr1", unit "2", exercise "5". Working as developer on the exercise by either of the above approaches moves you into the exercise directory, and offers you a new menu, the developers exercise menu. The particular menu will vary depending on the type of exercise. In addition to the commands offered on the menu, the usual additional facilities of replying with an exclamation mark followed by a Unix command can be particularly useful when working as developer, for example !ls -l to see the names of the files in the exercise, or a command such as !emacs model.q to edit one of them directly, or simple ! to be put into a shell in the exercise directory. The exercise may currently be of several basic types: Question/answer exercise Text submission (essay) exercise Compiled programming exercise Interpretive programming exercise The type is defined by a file "type" in the course or exer- cise directory, in which the various details of the exercise type and requirements are specified using Bourne shell nota- tion. The "type" file in the course directory sets course- wide defaults, individual exercises may vary these parame- ters with a local "type" file. After submission of work by students, the subdirectory "ex./solns" will contain the students' submitted work in files starting with their username (program sources in files named ".c" or ".C" for program- ming exercises, perhaps files ".txt" etc for essays), and their mark histories (in a single file "ex../solns/marks" containing all marks for that exercise) for any marked exercises. The "marks" file consists of one line entries with colon- separated fields. Each line is of the form ::::::: All marks are kept, each date- and author-stamped by the person who entered it. Only the last entry for a particular student is used in "vm" summaries. The "entered by" field contains the student username for work submitted by the stu- dent, or the tutor's username for work remarked by the tutor. Submitted text files (such as program sources, essays ...) are normally overwritten at each submission. There are now facilities to keep all submissions using RCS or SCCS, see later for details. At the exercise level, both the user and developer see one of several different menus depending on the type of the exercise, and the contents of the directory. Programming exercises have a separate major section below; other types of exercise will be discussed more briefly. Question/answer exercises This type of exercise is useful both for marked question/answer or multiple choice exercises, and for end- of-course questionnaires where the results are collected for analysis, but not marked. The "type" file here will be simply TYPE="QA"; export TYPE SUFF="mc"; export SUFF The exercise directory for a "QA" exercise will contain (in addition to the usual "title" file with a one-line title, and "model.q" containing the question in ASCII) a question/answer script file "model.mc" containing the vital information for display to the student, and for checking the responses. The key program here is the question/answer pro- gram (see the Ceilidh document[8] for more details of the capabilities and file formats) "~ceilidh/Tools/multi." The "model.mc" file will contain the main driving script, which contains all the display and marking information. The script may contain any number of questions for the student. For each question it will include the text to be displayed to the student, and (if marking is required) details of the required answer in the form of the name of a particular recognising program (the generic name for a program intended for recognising meaning in text is an oracle program) and the data required by the chosen oracle. The answers may be checked using a variety of recognisers, all stored in the "~ceilidh/Tools" directory. There are ones for checking multi-choice answers ( "Tools/mchoice" checks single charac- ter replies) or numeric answers ( "Tools/numeric" checks sequences of numeric values from the student against speci- fied ranges) or more general textual recognisers (the main Ceilidh "Tools/oracle" program, based on regular expres- sions, see the Ceilidh Oracle document by Zin and Foxley[10] for details). There is also an English language semantic answer recogniser at Nottingham, but its distribution is restricted for copyright reasons. It is hoped that general distribution will be possible at some stage. The student answers to the questions in the "model.mc" script are combined into a single file using the standard Unix "ar" (archive) command, and stored in a file ".ar" in the "solns" directory. When a student uses "cks" to check submitted information, the separate responses are extracted from the archive file. Note that the script file "model.mc" should have no public read access, since it may also contain the oracles which check the student's answers to the questions. The developer's exercise level menu for QA exercises offers editing the title file editing the question file editing of the script file and rerunning the "multi" program in a verbose marking mode. Text submission exercises Ceilidh exercises can be set up purely for the collection of textual material by students. This can usefully simplify the administration of coursework from large classes, but the submitted work still has to be marked (by hand) and the marks entered using the "em" Ceilidh command. The students must, of course, prepare the essay in an on-line file (typi- cally with a name such as "prog.txt" or some such) before submission and it all essays must be in some con- sistent form for the teacher to print them or view them for marking. It is also possible to add text (writeup) submission to pro- gramming exercises, so that the student completes both a program and a written explanation. This will be described along with the setting up of programming exercises. The "type" file here will be of the form TYPE="ESSAY"; export TYPE SUFF="txt"; export SUFF The menu commands for the student are Read or print question Obtain skeleton of essay in local file Edit local essay file Submit local essay file The developer commands are Edit exercise title Edit question Edit skeleton file The exercise directory will contain the title file, the question file "model.q" defining the problem to be solved, and an essay skeleton in a file such as "model.sk." The exercise is purely (at present) for text submission pur- poses. When the texts have been submitted, they are stored in the usual way in the exercise solutions directory, and the developer can (directly under Unix) print out the files, or look at them on-line. When essay marking by program is realistic, it will be installed! It would be expected that the human generated marks would then be entered by hand using the teacher's "em" (enter marks) command. On-line viewing, annotation and marking of essays may be provided later. A student can edit text files (program source or essays), of course, outside the Ceilidh system if preferred. The use of Ceilidh to administer essay questions can considerably sim- plify the collection and administration of work, even though there are no formally offered essay marking facilities yet. Any submitted essays appear in the "solns" directory, under names such as ".txt" for each student. The developer's menu offers editing of the question or the skeleton, or of the setup actions file "setup.act." The latter file (used as the basis when the student types "set" at the beginning of solving an exercise) will be of the form Copy model.sk prog56.txt This will copy the file "model.sk" in the exercise directory to "prog56.txt" in the student directory. We find it help- ful to use the skeleton file as an essay outline, with major subheadings for the student report or essay indicated. This ensures that they remember to address all of the major points. There will always be a "mark.act" file, which in text sub- mission exercises will consist of the single line Save o txt to save a copy of the student file "prog56.txt" for unit 5 exercise 6 in the file ".txt" in the solutions directory. Programming exercises For program development (our major application so far) the user and developer menus are considerably larger. _____________________________________________________________________________ |CEILIDH developer's exercise menu (Course pr1, unit 1, ex 1): | |et to edit title | ety to edit the exercise "type" file | |ep to edit program | eq to edit question | |es to edit skeleton | em to edit Makefile | |esa to edit setup actions file | ema to edit mark actions file | |cm to compile | run to run | |sd to set dynamic mark scheme | sf to set "features" mark | |st to set typographic weights | sc to set complexity weights | |se to set efficiency test | ss to set structure test | |mk to mark the model soln | mv to mark more verbosely | |mks to mark a studs soln | mka to remark all studs | |ch to do a complete check | | |h help | q to return to outer level | |=========================================== | |Type developer's exercise command: | | | | Figure 4 : Developer's exercise level menu | |____________________________________________________________________________| We will outline some of the principles here, but leave most of the detail to the next section. The developer's menu for a programming exercise is as shown in figure 4. The "type" file will typically be of the form TYPE="COMP"; export TYPE SUFF="C"; export SUFF CC="g++"; export CC specifying the exercise type as "compiled programming", the program source file suffix, and the compiler command. These can be defaulted by the "type" file in the course directory, but a local type file must exist. The commands are set using the "ety" command. There is a number of optional extra entries which can be specified in either of the "type" files on a per-course or per-exercise basis. "MAXSUB=20" This will limit any student's maximum number of submis- sions to 20. The student will be warned at each sub- mission of the upper limit, and of the number of sub- missions made so far. A value such as -5 sets no limit, but produces messages after 5 submissions tel- ling the student how many submissions have been made. "MINGAP=300" This sets the minimum time gap between submissions to 300 seconds. "OUTOF=10" This sets all mark totals seen by the student (except for individual dynamic tests) to be scaled out of 10, instead of as a percentage. Such coarser granularity of marking should reduce the student temptation to obtain higher marks by tweaking. "GRADE=E" The user can either set to a simple subdivision of the marks into grades using defaults, or the borderlines and grade names can be set. Setting "GRADE=E" causes the student mark to be graded as a letter (e.g. "A", "B", "C") instead of as numeric value. The letter ("E") specifies the number of divisions, but the bord- erlines for the divisions (e.g. 0, 60, 80, 90, 95, 100 for five divisions) and the corresponding strings (e.g. "E", "D", "C", "B", "A" or "Fail", "Poor", "Ave", "Good", "VG") are stored in C code in the source module "~ceilidh/Tools/SOURCE/grade.c" . Alternatively, set GRADE to a value such as that shown in the Teacher guide to specify borderlines and strings. If the course default is "GRADE=E", use "GRADE=" to cancel it for a particular exercise. "SAVEOUT=yes" This will cause the output from the runs of the student program against test data to be saved in a file ".out" in the solutions directory. "VDATA=no" The "view test data" student command lists all of the test data files in turn to the screen, and can be turned off by putting VDATA=no in the "type" file. An intelligent student could, of course, write a program which echoes its input to a file, and find the data that way by submitting the pro- gram. "C_ORACLE=my_oracle" to specify a particular oracle. This program will be searched for first in the "~ceilidh/course.<...>/unit.../ex..." directory, then in the "~ceilidh/course.<...>/bin" directory, and then in the "~ceilidh/Tools" directory. The compilation commands on all menus of "COMP"-type exer- cises will call a command "CCompile" in the exercise direc- tory if it exists, or in the course "bin" directory if it exists, or the one in the "~ceilidh/Tools" directory other- wise. The standard Ceilidh "CCompile" shell script uses "$CC" for its compilations, and is called with arguments CCompile See below for details of the compilation process. 6. Amending an existing programming exercise This important activity will now be described in detail. Bear in mind that choosing a particular exercise in developer mode moves you to that exercise directory. This means that you can now perform direct Unix operations in that directory by either executing each command preceded by an exclamation mark as in !ls -l or by obtaining a local shell as in ! or !bash and logging out from the shell to return to Ceilidh. For programming assignments the student will generally start by first reading or printing the question, then calling a setup command to set up any skeleton or header files in the local directory. They will then develop the program for the solution of the given problem (which can be done outside this system if required), and will then mark and submit it. 6.1. The question The developer command eq allows you to edit the student question with your default editor. The question is stored in the file "model.q" and will be in ASCII. We hope to add other (PostScript, dvi, pdf, HTML) options later. Experience has shown the value of using a standard structure outline for the question file, such as Question: Method: Notes/Hints: Mark scheme: Typical input: Typical output: You can, of course, make the question as helpful or unhelp- ful as you wish. Don't forget to describe the files which are set up by "set," and the marking scheme. The last entry above ("Typical output") may be most easily added after the model solution has been developed, using actual model pro- gram output. 6.2. Setup You may wish to change the effect of the student "set" com- mand in order to provide a more helpful skeleton, a less helpful skeleton, or some additional files. The student executes the setup command once before solving the exercise. The command uses a file in the exercise directory named "setup.act" containing lines such as Copy model.sk prog72.C Copy header.h Any line with two filenames causes the exercise directory file such as "model.sk" in the above example to be copied to the student's directory under the name "prog72.C." If the line contains only one name, the source and target names are assumed to be the same. You may wish to add a line to copy a file of test data to the student directory Copy model.d1 TestData or to copy an additional write-up skeleton by Copy model.wsk prog72.txt The setup actions file is edited using the "esa" developer command, and the skeleton file "model.sk" using the "es" command. Other files such as a write-up skeleton would have to be edited by hand using, for example, !emacs model.wsk If no mark actions file exists, the "esa" command gives a default file to copy the program skeleton, which may be ade- quate in simple cases. Note that execution of the setup function by the student will not overwrite existing files in their directory. During any copying performed as part of the setup activity, the string "$USER" in the source file is replaced by the user's login name as the file is copied, "$DATE" by the current date, and "$NAME" by the user's full name (extracted from the course "students" file). If no program skeleton file exists, the "es" command starts the skeleton with a copy of the model solution; it is sensi- ble to set up the skeleton after the model solution has been completely developed. It is assumed that the developer will take the solution program, and then edit out all the parts which the student has to create. There may be a file such as "../../box" (in the course directory) containing a com- ment box such as ////////////////////////////////////////// // // Put your program instructions here // ////////////////////////////////////////// for C++ programs, which you may choose to insert with your editor at appropriate points. Consistent use of features such as this in the skeleton makes the student's understand- ing of what they have to do much clearer. 6.3. Program development This stage of student activity can be performed outside the Ceilidh system if so desired; we anticipate that the student will work completely within Ceilidh during the early stages of the course, but outside during the later stages. The student will have a source such as "prog57.C" and exe- cutable such as "prog57." In the exercise directory, the source will be linked to "model.C" and the executable to "model." The commands available to the student for development of their programs with Ceilidh include the following. The edit commands calls the student's or developer's chosen editor as defined by the environment variable "EDITOR" with the user's program source as argument. The compile commands use the Ceilidh shell script "~ceilidh/course.???/bin/CCompile" if it exists, or by default "~ceilidh/Tools/CCompile" to compile the source from a file such as "prog72.C" into the corresponding executable file "prog72." "CCompile" first looks for a "Makefile" in the exercise directory. If it finds a "Makefile" containing a dependence such as prog72 : in the unit 7 exercise 2 directory, for example, it compiles using that entry with a make prog72 command. (The "Makefile" can also contain an entry model : ... to assist the course developer; this will not be usable by the student.) If there is no "Makefile" in the exercise directory, or no prog72 : entry in it, the system uses one of the "CCompile" shell scripts and the "$CC" command, which would typically default to $CC -o prog72 prog72.C The value of the "CC" variable is set coursewide in the course "type" file, over-ridable on a per-exercise basis in the exercise "type" file. If there is no appropriate "Makefile" entry, and the example involves more than one module, then the student must create the executable by hand outside Ceilidh. Refusing compilation There is a particular "Makefile" available which will put the user back into a shell for compilation. This is offered to the developer if no "Makefile" exists when using the em (edit makefile) command, and takes the form $PROG : $PROG.C rm -f $PROG cat $$C_LIB/NoComMsg $$SHELL where the file "NoComMsg" contains There are no automatic compilation facilities in this exercise. The system will now put you into a shell. Execute the required compilation commands, then quit the shell. To do this, if you are using the C-shell, type "exit", if you are using the Bourne shell, type "control-D". You will then be returned to the point where you left the Ceilidh system. Now wait for the shell prompt ..... The "run" command runs the student's executable such as "prog57" interactively from the terminal. The student can also use a "rut" command to run their exe- cutable against test data supplied by the developer (it uses the first of the data sets provided for the marking process, see below). Students have a "vtd" command to enable them to view the files of test data used in dynamic tests during the marking process. This can be turned off by the developer or teacher by setting the value of the variable "VDATA" to "no" in the "ety" (edit type file) developers command. If the developer leaves an executable program in the exer- cise directory, the student can use further commands to run the developer's executable interactively, and to run the developer's executable against the same test data. This allows the students to compare their answers with those presumably expected by the developer. When a course is set up (by un-shar-ing the distributed shar file) there are no executables. If you require executables to specific exer- cises to enable students to run them, you must use the developer's compile command to create them as required. Generally the C++ executables we get are quite large, so they are created only when required. The extent to which we should assist the student's program development (by repeated submission, by releasing the test data, by offering an executable, etc) is very open to debate; comments and suggestions are always welcome. The system can easily be adapted to offer more or less help on different exercises. 6.4. Marking and submission The student calls the "sub" command to mark the program which she or he has developed. The submission/marking pro- cess is driven by a file "mark.act" (mark actions) in the exercise directory, and expects to find in the current (student's) directory a source in a file such as "prog72.c" or "prog72.C" and a corresponding executable* ====================================================================== Note: * Earlier versions of the marking process recompiled the source before marking to produce a guaranteed up-to-date executable; the extension to numbers of object modules made this difficult and undesirable. Client-server versions of course need to re-compile if the marking is done on the server. ====================================================================== in a file "prog72" . The submission/marking process appends a one-line summary of the marks to the "marks" file, and a copy of the current version of the student program source to the "ex./solns" directory; if the lecturer wishes, lines of the user's source code starting "#include" which refer to a local file may be substituted by that file if it is local and readable. In general, only the latest copy of the source is retained. The mark.act file If a line such as Save o C exists in the "mark.act" file, the intention is that the student' source (from a file such as "prog72.C" with name ending ".C", the last argument of the "Save" line) will be saved at each submission, overwriting (the "o" specifies "overwrite", specify "a" if each submission is to be appended rather than overwritten) previous copies. To save a student essay (in file "prog72.txt" ) and header file (in "prog72.h" ) add lines such as Save o txt Save o h to the file. The following generalisation is an enhancement in release 2.2 onwards. A line such as "Save o C" will first search the "course./bin" and "~ceilidh/Tools" directories for an executable "CSav_C" (the name is "CSav_" followed by the given suffix). If such a file is found, it is executed with arguments CSav_C -o If it is not found, the student's source is saved directly as described earlier. It is thus possible now to write spe- cial save commands such as "CSav_..." as shell scripts if required. Each such command needs a corresponding "CVw_..." command which is called when the student executes a "cks" command to enable them to view their submitted work. Three new "CSav_..." commands are provided in the "Tools" directory of the current release as follows. ar A line Save o ar in conjunction with a line ARFILES="prog56.C header.h hotel.C" in the "type" file will cause the named files from the student directory to be combined using the Unix "ar" archive command into a single stored file .ar in the "solns" directory. This facility is implemented by the command "CSav_ar" in the "Tools" directory for performing the saves, and "CVw_ar" to enable the stu- dent to view the stored files. The command "CSav_ar" has code included such that if the student has a file "save.lst" containing the names of files to be saved, this list is used in preference to the "$ARFILES" list. This should enable student to design their own split into separate files. rcs A line Save o rcs saves the student source in a file ".rcs" in the solutions directory using the RCS (Berkeley "Revi- sion Control System", a source code control program) command. All versions of the program submitted by the student are thus available to the teacher afterwards; the student just sees the most recent version during a "cks" (check submission) command. Viewing the RCS files has not been built-in as a teacher command; the demand for it is uncertain. The RCS file would enable a teacher or researcher to follow the student's pro- gress through an exercise; this could be useful for research into student program development, and makes obvious the occasions when a student copies another program instead of editing the existing one. The shell script to perform the RCS collection is in the file "~ceilidh/Tools/CSav_rcs." You must have the RCS com- mand on your system. A similar system could surely be built using SCCS as an alternative. exe The line Save o exe in the "mark.act" file causes the student executable to be saved in the "solns" directory under the name ".exe" The command "CSav_exe" looks for either of the files "prog" or "prog.exe" in the student directory. Beware that executables can be very large; storing large numbers of them can consume vast areas of disc. The mark details (one line) are appended to the file "marks" at each submission, they are never overwritten. The submission/marking process cannot be called by one user more frequently that a general time interval set by the developer; this is currently set to 600 seconds, and is set in the source of the controlling program "ccef.c" (in file "~ceilidh/Tools/SOURCE/ccef.c" ) as the "#define MIN_GAP" constant (in seconds). You must then re-make the program. The value can be reset on a per-exercise basis by a line MINGAP=60 (in seconds) in the exercise "type" file. The number of submissions can be limited by a line MAXSUB=10 in the "type" file. The student will receive messages informing them how many submissions have already been made, and what the upper limit is, typically You have now made 4 out of a maximum of 6 submissions. If a negative value is set as in MAXSUB=-5 the student will be informed of the number of submissions made after (in this case) the 5th, but no upper limit is set. The overall mark awarded is made up from a number of sub- marks, the weights and sub-tests being specified in the "mark.act" file. A typical "mark.act" file might be 60 Dynamic: CDynCorr -v1 $C $U $E 25 Typographic: typog_C -F1 < prog$U$E.C 15 Features: CFeature $C $U $E Save o C The "Save" line has already been explained. The other lines have significance as follows. The first three lines specify that the overall mark is to be calculated from three tests, dynamic, typographic and com- plexity. The dynamic tests are to be scaled out of 60, the typographic test out of 25, and the features test out of 15. If these marks do not add up to 100, they are further scaled during the marking process. Each sub-test will also include weights for its sub- components, of which details will be given later. The overall mark awarded is formed from a number of com- ponents, of which the following are provided. Dynamic correctness Dynamic efficiency (C on SUNs only) Typographic style Complexity level Features of the program Structure of the program Other assessment programs can be added in the "course./bin" or "Tools" directories, and called in the same way. The only restriction is that they MUST pro- duce as their last line a line such as Score 85 to indicate the mark awarded. These components are then combined to form a single percentage mark. The "mark.act" file is set up when the exercise is created initially with one line such as Save o C for saving the program source. Other lines are added as the tests are set up by the developer, or can be added by hand using the "ema" (edit mark actions) command. Any marking program must produce output ending with a line such as Score 75 as a percentage. The variables in the "mark.act" file introduced by dollar symbols have significance as follows. $C course abbreviation "c" or "pr1") $U unit abbreviation "1" or "2" or $E exercise abbreviation "1" or "hal" $X the exercise directory full pathname $S the program file suffix "c" or "C" The default call of the marking program will show the stu- dent the sub-totals for each marking sub-program, and the final weighted total. If in the "type" file there is a line MARKVERB=0 (marking verbosity = 0) none of these will be displayed, although they will be stored as usual in the collected "marks" file. MARKVERB=1 shows the student the total only. MARKVERB=2 gives the subtotals and the total as at present. The default value is 2. We now describe how to set up marking using the various built-in marking programs. 6.4.1. Dynamic correctness These are the most difficult tests to set up. The vital line in the mark actions file is 50 Dynamic: CDynCorr -v2 -F0 $C $U $E The flags to "CDynCorr" control the amount and type of information displayed to the student, and are as follows. -v0: no information -v1: overall total score out of ... -v2: individual test scores (plus the previous) -v3: individual RE scores (plus the previous) -F0: no information -F2: messages about "incorrect exit code", "core dumped", ... -F3: all REs failing (plus the previous) The student executable program is run against various sets of test data (or shell scripts) provided by the developer, and an "oracle" searches the program output for signs of correctness in each test. For details of how the "oracle" program works, and how its marking weights are set, see the oracle[10] document. All the tests and associated information can be set up using the developer's "sd" (set dynamic) command. Each test must have a short title, and a total mark. These are specified in the "model.dv" file, which might take the form Dynamic Correctness 25 Simple data 20 Zero denominator 15 No data 35 Longer test This specifies four tests (the first line must be there, but can otherwise be ignored): the first test (worth 25 marks) using simple data, the second (worth 20 marks) involving care over a zero denominator, the third (15 marks) requiring a check for the existence of data, and the last (35 marks) being a substantial test. If the marks do not add up to 100, the final total is scaled appropriately. This percen- tage total is then scaled for the overall mark awarded as specified in the "mark.act" file. The chosen titles can be as helpful (as above) or unhelpful ("Test 1", "Test 2") as you require. The test data for the first test will be in the file "model.d1," for the second in "model.d2" etc. The oracle data (a set of regular expressions to search for the required information in the output of the program under test, see below) for the first test will be in the file "model.k1," for the second in "model.k2" etc. The data should have public read permission, the oracle files should not. The default oracle program is in the "Tools" directory; another oracle can be specified by inserting, for example C_ORACLE=search in the "type" file. The default oracle works (by calling "awk" ) using sets of regular expressions (REs) devised by the developer; these will usually be designed to recognise alternative output formats which are close to what the developer asked for, but cannot in the end recognise every semantic possibility. Each RE can be preceded by a mark (it is otherwise given a default mark of 10), and/or a limitation on the number of occurrences (full marks are otherwise awarded if the RE is found at least once in the program output). The percentage mark awarded for each dynamic test is based on the sum of the marks for the REs found divided by the total of all pos- sible marks. Suppose we wish to create a file of REs for the oracle to check output which should read roughly A temperature of 124.6 degrees F converts to 32.7 degrees C The "oracle" file of REs might be 10:32.7 5:124.6 2:degree 1:temp ==0:20:[Ee]rror ~10:Fail This awards 10 marks if the result contains the string "32.7" (probably to most important test of correctness), a further 5 marks if it contains the string "124.6" (at this stage students were asked to print out copies of all entered data), and so on. If the output contained the string "Error" or "error" then 20 marks would not be awarded (read the "==0:" as "there must be exactly 0 occurrences of ..."). The last line will subtract 10 marks if a line containing "Fail" is found; the maximum possible mark for scaling pur- poses does not include any of the negative marks. The total awarded is then scaled out of the total of the possible positive marks, 38 in this case, and given as a percentage. The full stops in the above REs (representing decimal points) should strictly be expressed as 10:32\.7 5:124\.6 since a full stop in an "awk" RE represents any one charac- ter, whereas an escaped full stop represents a full stop. The REs could be further extended (they are passed to an awk program) if the lecturer so wishes, to specify that the numeric values are surrounded by non-numerics, for example, using the standard awk RE notation such as [^0-9]18[^0-9] and alternatives given as in cms|centimetres The possibility of rounding errors may necessitate the RE 3\.141[56] to allow for either "3.1416" or "3.14159". See the original "awk" reference[1] or the relevant section in a book such as Bourne's[7] for exact details of permitted REs, and the "oracle" document for further details of the oracle program. The marks for each dynamic test can thus be built up as a series of weighted RE scores. If the program under test fails catastrophically at run time (core dump, or stuck in a loop) useful results may be diffi- cult to guarantee. The system is designed to award zero for that dynamic test, and continue trying all remaining dynamic tests. The program is killed after 5 seconds of processing if it is still running, using the "~ceilidh/Tools/run" com- mand (source in "Tools/SOURCE/run.c" if required). This is a configurable parameter; to change it, edit the defined value for "MAX" in the source in the source file "run.c" and remake. Each dynamic test needs a file of REs for the oracle, the files being named "model.k1," "model.k2" etc for successive tests. Each test is driven either from a file of test data, named "model.d1," "model.d2" etc (the executable is run from the test data), or from a shell script, named "model.s1," "model.s2" etc (in which case the shell script is executed as given). Other oracles can be used by assigning C_ORACLE=my_oracle in the exercise "type" file, which will look for a program "~ceilidh/Tools/my_oracle" and will call it with the key- words file as argument, and the student output as standard input. It will expect the output of the oracle to be of the form Score 95 out of 100. The separate dynamic tests could by given explicitly in the "mark.act" file by lines such as 10 Dyn test 1 : prog45 < $X/model.d1 | oracle $X/model.k1 but beware of problems of the executable getting stuck in a loop. 6.4.2. Dynamic efficiency Run-time efficiency is not generally an important considera- tion in early programming courses. The C system on SUNs (not the C++ system) examines the number of times each line of the code is executed (using tcov on the SUNs), and com- pares the maximum line execution count with those for the developer's model solution. If this test has been set up using the se (set efficiency) developer's command, the maximum line exe- cution count on the student program is compared with that on the model program. If the student's count is less than the model count, 100% is awarded; if it is equal, 99% is awarded; if it is greater, 100 * model count / student count is awarded. See the shell script "Tools/CEff_c" for details. 6.4.3. Typographic analysis The program "typog_c" reads C program source ( "typog_C" reads C++ source) and computes various statistics associated with maintainability and readability of the source, such as Average characters per line % blank lines Average spaces per line Average function length % good function Average identifier length % names with good length % define's % number comments % chars in comments % indentation % indent errors For further discussion and justification of the typographic and complexity metrics, see Zin and Foxley[9] The students can obtain the system definition of features such as good indentation from the help facility, where it is defined as Either the opening and closing curlies are on the same line, or the following three conditions must hold. (i) Each closing curly bracket must relate to the line containing the corresponding opening curly, and the closing curly must be in a column between the first visible character on that line and the actual opening curly. (ii) A line containing the opening curly must not be indented more than the one following it. and (iii)A line containing the closing curly must not be indented more than the one preceding it. The present system also insists (since it is recommended in early lectures) that closing curly brackets must be followed on the same line by a comment if more than 10 lines after the corresponding opening curly. Each of the general metric factors is measured, and compared with set values. There will be a range of values within which full marks are awarded, and a wider range within which part marks will be awarded. Beyond the outer range, no marks are awarded. The relevant line in the mark actions file is of the form 25 Typographic: typog_c -v0 -F1 < prog$U$E.c where the flags again control the information display as in The "F" (feedback) flag gives -F0: no feedback -F1: reason for the biggest loss of marks -F2: reasons for all marks lost (plus the previous ...) -F3: the position of all indenting errors (plus the previous ...) The "-v" (verbosity) flag as an alternative gives -v0: no information -v1: a total score (out of 100) -v2: a score for each metric (plus the previous) -v3: all the parameters for all the metrics (plus the previous) -v4: other calculated values (plus the previous) Thus by using the "F" flag, the student can be given English language help about the program weaknesses. It is obviously (in general) bad to give too much detailed help. Output is jumbled if both "F" and "v" flags are used; they are intended for different applications. The marks and range parameters for each of the factors involved either take on default values from data initialisa- tions built into the "typog_c" etc programs (to change the defaults, edit the source and remake), or can be supplied on a per-exercise basis from a file "model.tv" in the exercise directory using the call 25 Typographic: typog_c -v0 -F1 -f $X/model.tv < prog$U$E.c to name the file containing the typographic weights. A skeleton "model.tv" file can be generated in the current directory by calling typog_c -w The developer may choose to vary these parameters to emphasize different aspects of the marking at different points in the course. Each line of the "model.tv" file is of the form AIDL 10 1 4 10 15 Average identifier length %NGL 1 0 45 100 100 % names with good length where the entries are A 4-letter code The maximum mark for this feature The next four numbers define five ranges If the program value is less than the first number, no marks are awarded. If it is between the first and second numbers, part interpolated marks are awarded. If it is between the second and third numbers, full marks are awarded. If it is between the third and fourth numbers, part interpolated marks are awarded. If it is above the fourth number, no marks are awarded. A title for this metric The developer's st (set typographic) command shows the default score for the developer's source, and if the mark awarded is not 100% (it rarely is!) offers repeatedly editing of either the program source, or of a local file of typographic parameters. If this is the first developer's call of "st" the system will ask for a total mark to be awarded, and will add an appropriate entry to the "mark.act" file. To change the default values for the whole course, edit the source in "Tools/SOURCE/typowgts.c," where there is a table of the above values held in a structure array, and remake. The student sees only the overall typographic mark. Tutors and teachers can see a more detailed breakdown if they wish using the "mt" (mark typographic) tutor command. A call of cat source.C | typog_c -p will produce a local "model.tv" file with parameters tweaked to give the named program a typographic score of 100%. 6.4.4. Complexity analysis This follows a similar pattern to the typographic analysis, but in this case the metrics for the student program are not compared with absolute values, but with those for the developer's model. The factors involved include the follow- ing. Number of reserved words Number of includes Number of gotos Number of conditionals Number of loops Depth of loops Number of operators Number of braces Max depth of braces Number of square brackets Max depth of square brackets Number of round brackets Max depth of round brackets Number of function calls Number of numeric denotations The line in the mark actions file is 25 Complexity: compl_c -v0 -F0 -x $X/model.cm -f $X/model.cv < prog$U$E.c and the flags have the same effect as in the typographic program. The metrics of the model program set up by the developer must first be created in file "model.cm" in the exercise directory using cat source.C | compl_c -m When the student's source is analysed by cat source.C | compl_c the student program metrics must be within factors for each metric set by the developer (usually between 50% and 200%) of those of the model solution*. ====================================================================== Note: * The developer's model program will thus always be awarded 100% for complexity. ====================================================================== The marks and parameters for each of the factors operate in the same way as the typog_c weights, with each line contain- ing a maximum mark, and four values defining the five ranges of marks awarded. The command either uses default values built into the "complexity" program source, or can be sup- plied from a file "model.cv" in the example directory called by cat source.C | compl_c -f model.cv The developer's command sc (set complexity) operates in a similar way to the "set typo- graphic" command. If there are significantly different possible solution pro- grams to a given problem (e.g. a recursive and an iterative solution), it should be possible for the developer to supply more than one model, and for the system to compare the stu- dent program with all of these models, and to choose the best fit. This is not yet implemented. 6.4.5. Features of the program When assessing program sources by hand/eye, one typically looks at the general layout (the typographic marks above), and for particular problem dependent features such as occurrences of particular numeric denotations within the code which should really be set as constants (in C++) or "#define"s (in C); the use of "<= 59" where "< 60" is better practice; the use of 64 as a specific ASCII code rather than the correct character denotation 'A'. The marking system allows the setting up of files of REs in the exercise directory, in files such as "model.ft" which will then be marked using the oracle against the program source with comments and strings removed. Such "features" files can be set up using the "sf" (set features) developer's command. The command performing this marking is "Tools/CFeature" shell script. The features to be found can be placed in several files. The file model.ft is typically used for a single features oracle. If the entry in the "mark.act" file is 20 Features: CFeature -v0 -F0 $C $U $E the flags are interpreted as -v0: no scores given -v1: total score given -v2: sub-scores given (plus the previous) -v3: individual RE scores given (plus the previous) Extra features of the oracle useful at this point are the limitations on the number of occurrences of each RE. A line for the oracle can be of the form <=1:3.141[56] meaning that the RE after the colon must occur less than or equal to once. To require that a program involving the conversion factor 60 uses that factor only once (presumably declared as a con- stant), and the value 59 not at all, use the oracle file ==1:60 ==0:59 Experience can be gained by looking critically by eye at student solutions to a given problem, and the features ora- cle can then be improved for future years. 6.4.6. Program structure To look for structural weaknesses in the student's program, it can be re-compiled in the present GNU C++ compiler with g++ -c -Wall or for C programs we can use the lint command. Certain warning messages (such as variables declared and not used) are picked up by an oracle, and a score awarded. The developer command ss (set structure) allows the developer to set up such an ora- cle file "model.st" in the current exercise. If it is not set up, a standard file "model.st" in the course directory is used. The standard file also includes scaling, so that a few errors can produce a significant drop in the marks awarded. The commands performing the work are "Tools/CStr_C" for C++ and "Tools/CStr_c" for C. 6.4.7. Summary of files in the exercise directory ________________________________________________________________________ |model.q the question | |prog83.C the model source (no public read permission) | | linked to "model.C" | |model.sk the skeleton program for the user to start from | |prog83 the executable (no public read)(optional) | | linked to "model" | |setup.act the files to be copied before the student starts | |mark.act the components of the marking | |model.d1 the first file of test data | |model.k1 the oracle for "model.d1" output (no public read) | |model.d2 the second file of test data (optional) | |model.k2 the oracle for "model.d2" (optional, no public read) | |model.d3 further tests and oracles as required (optional) | |model.k3 | |model.dv the dynamic test weights and titles | |model.tv the typographic weights (optional) | |model.cm the model complexity metrics | |model.cv the complexity weights (optional) | |model.ft the source code features oracle (optional, no public read) | |model.st the structure oracle (optional) | | | | Figure 5 | |_______________________________________________________________________| In a simple one module C++ programming exercise, the files which must be created in the exercise directory before the students access the exercise would be as shown in figure 5. There may be other sources, objects and header files and perhaps a "Makefile" in more complex examples. 7. Setting up a new programming exercise The developer will normally use commands from the developer's menu to set up a new exercise in approximately the order shown in figure 6. __________________________________ |ep edit program source | |cm compile program | |run test run program | |ema edit mark actions | |sd set up dynamic marking | |st set up typographic metrics | |sc set up complexity metrics | |sf set program feature oracle | |ss set structure marking | |mk for complete marking | |eq edit question | |esa edit setup actions | |es edit skeleton | | | | Figure 6 | |_________________________________| With the "sd" command to set up the dynamic tests, typically type commands in order as shown in figure 7. ___________________________________________________________________ |c create new test | |d data file rather than shell script | | then edit the data file | | observe program output | | edit oracle file | | default initial value is a copy of the program output | | observe the oracle score | |y if oracle and data OK | | if not, alter data or oracle (you are invited to edit both) | | Then give a mark and a title for this test | |q quit creating tests | |mk to perform complete test marking | | | | Figure 7 | |__________________________________________________________________| We now summarise in more detail the actions needed to set up a new exercise. Suppose that the chosen exercise is a pro- gram to convert seconds into minutes and seconds, within unit 2 of the course "pr1". All the appropriate files could be set up by hand in Unix. We assume that the developer exercise menu will be used. Choose an identifier for the new exercise, such as "9". Login as "ceilidh", and move to the defined existing course and unit and go into "developer" mode. Type sc pr1 su 2 develop Then create the new exercise with "nx", which will ask for the exercise number and title. Command? nx Ex number? 9 Title? Seconds to minutes _______________________________ |model solution program | | source "prog29.c" | | and executable "prog29" | |mark actions | |dynamic tests | | test data | | oracles | |typographic metrics | |complexity metrics | |features metrics | |program skeleton | |setup actions | |question | | | |Figure 8 : Files to be set up | |______________________________| We are now in the new exercise, in developer mode, and have to set up the files listed in figure 8. The exact order is not vital, but that suggested above is suggested. Model solution program Use "ep" to edit the program; you can edit, compile and run the program to develop it under Ceilidh using "cm" and "run", or outside if you wish. You would do the latter by moving to the directory ~ceilidh/course.pr1/unit.2/ex.9 or by simply obtaining a shell within Ceilidh by typing ! The source should be in "model.C" (or "model.c" ...) linked to "prog29.C" (or ".c" ...), and the executable should be in "model" linked to "prog29". The marking process The dynamic tests are the most complex to develop. This stage requires an executable program in the exercise direc- tory. Type "sd" (set dynamic), which will offer you the following sequence. Another test? Type "c" to create a new test. Each dynamic test requires a file of test data (or a driving shell script), and an oracle file of recognisers (regular expressions) for the output. The sequence of prompts for a new test is: Data file or shell script? reply "d" if you wish to run the test from a data file. The editor is then called automatically; use it to create an appropriate data file (or shell script). The data file in this case might be just a single integer 2345 If you chose a shell script, it might be prog29 << ++++ 2345 ++++ The system then runs the program against this data file, and its output is displayed. You are then invited to edit the file of regular expressions (REs) for the oracle which seeks to find out whether the output is correct. The file is ini- tialised to a copy of the program output, so that you can simply edit out the unimportant parts. This will then be entered into the oracle output recogniser file. It is suggested that the file should be edited in the following stages. For a simple test, just leave the essential output values which should be in the output, each on a separate line. If the number in the output should be "123", it is prudent to give the RE as [^0-9]123[^0-9] With floating point values, allowance should be made for rounding, or for different preci- sions. The value "1.42857" might be recog- nised using [^0-9]1\.428[56] If the output is expected to be two numbers such as "12\ 23", observe that the single expression [^0-9]12 *[^0-9]23[^0-9] will offer only full marks or no marks. If you use instead [^0-9]12[^0-9] [^0-9]23[^0-9] the student will get half marks if only one of the two expressions is found. For later tests, enter also recognisers for ancillary text, to recognise "12 minutes" use [^0-9]12[^0-9] [Mm]ins|[Mm]inutes or a variant thereof, permitting the user to use "Mins", "minutes" etc as alternative words. To give more weight to the numeric value than to the surrounding text, use the mark specification facility of the oracle as in 10:[^0-9]12[^0-9] 5:[Mm]in(ute|)s in which each RE is preceded by a numeric mark and a colon. It is unwise to form the complete requirement into a single RE, since if you specify, for example [^0-9]12[^0-9] *[Mm]in(ute|)s the student obtains zero or full marks, with no possible intermediate values. When you leave the editor after creating the ora- cle RE file, the system runs the program against this oracle, and prints the mark awarded as a per- centage. You are then asked if you want to repeat the cycle edit test data edit oracle REs look at the resulting mark The default is to repeat the cycle if the mark is not 100%. If you indicate the end of this test, you are asked for Marks for this test? Reply with the maximum marks for this test. It is convenient if the total of the marks for all tests add up to 100. Title of test? This is the title which will appear on the student marking output. Edit model.dv file? This file contains the marks and titles of all of the tests (following an initial title line, which is ignored). The file can be amended if you wish to adjust the marks or titles of the tests. Control then returns to the "Create another dynamic test?" query. Type "q" to quit the loop. You will then be asked for the overall "dynamic" total to be entered into the "mark.act" file. If you re-enter the "sd" command, you can skip the ear- lier tests by typing "s" . Typographic marks Use the command "st" to set the typographic marking. If the immediate result is not 100%, you may choose to stick with the system default metrics, and edit your program to improve its typographic marks relative to the built-in default values (the test shows you a breakdown of each individual metric), or set up an exercise-specific metrics "model.tv" file and insert adjusted metrics; the latter should not often be necessary. In each case the cycle look at metrics edit either the program or the weights repeats until you are satisfied with the mark awarded. The system then asks for a total typographic mark for the "mark.act" file. Set complexity marks The "sc" command to set complexity operates in a similar way to the typographic marking, except that a file storing the complexity metrics of the model program must be created. This is done each time that the "sc" command is entered. The command then follows the pattern of the "set typo- graphic" command. Set features mark If you wish, an oracle can be set up to recognise particular good or bad features which might appear in the program source for this particular problem. This facility is set up by the "sf" command, and is typically used for features such as one might look for by eye if hand marking program sources. This oracle usually requires the extra facility for specifying limits on the number of occurrences of given features. To forbid the appearance of a number such as 59 in a problem essentially involving the value 60, use ==0:59 to specify zero occurrences of "59". To insist that a particular constant is used only once, and is not repeated at several points in the program, use <=1:2\.54 <=1:0\.39 or perhaps ==1:2\.54|0\.39 to specify exactly one occurrence of either "2.54" or its inverse. The text in comments and strings in a program is normally extracted before it is passed to this oracle. See the ora- cle document for further examples. You will be offered the oracle file to edit, then the model solution will be marked against it. This process can be repeated. Finally, you are asked for a total features mark, to be entered into the "mark.act" file. Complete marking A complete marking should then be performed with the "mk" command. The ratios of the subtotals for the separate aspects of the marking are adjusted using the "ema" (edit marking actions) command. Check the weights, and delete any lines corresponding to marking components which you wish to ignore. Setup actions The "esa" ("edit setup actions") command allows actions which take place when the user types "set" at the start of solving a particular exercise. A typical setup actions file is Copy model.sk prog29.C Copy header.h This causes the skeleton program to be copied to the user's area as "prog29.C", and the header file "header.h" to be copied to the same name in the user area. An appropriate default file will be created automatically. During copying, the string "$USER" is replaced by the user's login, "$NAME" by the user's full name, and "$DATE" by the current date. The question Finally, the question file should be edited and checked. It is often easy at this stage to insert an example of the model program's output (by running the model against the first test data) and perhaps of an input file (by inserting the first test data). The question should be reasonably specific in all aspects. Leaving the exercise mode When you type "q" to quit the exercise mode as a developer, checks are made that the essential files are all there. You will be asked whether you wish to leave an executable in the directory; if you do, the "run model solution" commands become available to the student. 8. A complete example from Ceilidh For the C++ "pr1" unit 2 exercise 5 the non-obvious files are as follows: Program ("model.C"): #include // Convert feet and inches to centimetres // Written by EF 1988 const int INSPERFT = 12; const float CMSPERIN = 2.54; main () { float cms, ins; int ft; cout << "Type feet and inches: "; cin >> ft >> ins; if ( ins < 0 || ins >= INSPERFT ) { cout << "Error: Inches out of range\n"; exit( 0 ); } if ( ft > 0 ) { // Avoid explicit constants anywhere cms = ( ft * INSPERFT + ins ) * CMSPERIN; } else { cms = ( ft * INSPERFT - ins ) * CMSPERIN; } cout << ft << " feet " << ins << " inches is " << cms << " cms\n"; } // end of program Skeleton ("model.sk"): #include // Convert feet and inches to centimetres // Written by EF 1988 const int INSPERFT = 12; const float CMSPERIN = 2.54; main () { float cms, ins; int ft; cout << "Type feet and inches: "; cin >> ft >> ins; ////////////////////////////////////////// // // Put your program instructions here // ////////////////////////////////////////// cout << ft << " feet " << ins << " inches is " << cms << " cms\n"; } Test data 1 ("model.d1"): 12 9.5 Oracle RE 1 ("model.k1"): 389.89 Test data 2 ("model.d2"): -27 6.5 Oracle RE 2 ("model.k2"): -839.470 Test data 2 ("model.d3"): 9 1.2 Oracle RE 2 ("model.k3"): 9 ft|feet 1.2 ins|inches 277.368 Overall marking ("model.dv"): Dynamic Correctness 50 Simple test 0 Negative distance 50 Check "feet" "ins" Features ("model.ft"): ==1:12 ==1:2.54 We have no "model.tv" or "model.cv" files, but are using the default metrics. Setup actions ("setup.act") Copy model.sk prog25.C Marking actions ("mark.act") 60 Dynamic: CDynCorr -v1 $C $U $E 25 Typographic: typog_C -F1 < prog$U$E.C 15 Features: CFeature $C $U $E Save o C The complexity metrics file "model.cm" is set up if required by the system.