The Ceilidh System

The Ceilidh System

A General Overview 1996

Eric Foxley, Colin Higgins, Cleveland Gibbon

Learning Technology Research

Computer Science Department
University of Nottingham
NOTTINGHAM NG7 2RD, UK
email : ltr @ cs.nott.ac.uk
[Document in file /ltr/system/papers/docs/Overview96.ms, PostScript generated Thu Nov 7 15:30:39 GMT 1996.]

Abstract

This document summarises the state of the Ceilidh system as at Autumn 1996.

Contents

1. Overview

Ceilidh is a courseware system designed for the assessment of student coursework in Computer Science and the administration of the corresponding courses.

1.1. Functionality

The three main areas involved in what we refer to as courseware are

The Ceilidh project aims eventually to cover all of these areas, but at present covers only the first two. The third area is being actively pursued as part of the current project, but at present the system gives administration and assessment support for a course of lectures in certain subjects.

The areas of administration and assessment both increase in proportion to student load, and which can absorb significant quantities of manpower effort in a large department. The thrid area bove (delivery of information) does not necessaril;y increase in proportion to student load, especiall if it involves large class teaching. It is therefore important that we provide computer assistance in the first two areas.

System versus courses

We must distinguish carefully between the Ceilidh system itself, and the courses which run under it. The system is a general collection of software for course administration, and contains certain general purpose assessment tools. A number of courses are available separately (see below) and any of these can be installed as required; each course includes its own specialised assessment tools, and a collection of exercises and their assessment details.

1.2. History

The Ceilidh system has been in use at Nottingham since 1988, assisting in both C and C++ courses to classes of up to 200 students. The major current version runs on UNIX systems.

It is now distributed to about 200 sites in 30 countries.

1.3. Marking and assessment

The original version of Ceilidh was developed at Nottingham mainly for marking programs written in C. The student would (on-line)

  • (i) read this week's question (a file of ASCII text describing what the program has to do, and the format of input and output required),
  • (ii) obtain a skeleton outline of the solution program (and any associated header files etc),
  • (iii) develop a solution program, and
  • (iv) submit the program for marking.

    The last two steps could be repeated, so that the student could have several attempts at a solution; after each attempt the system provided feedback about the weaknesses in submitted work.

    The marking was done using a number of metrics

    Originally (when machines were not so powerful as now) the marking was done overnight; the student submitted the work during the day, the marking was done (and results emailed to the student) overnight. This implied at most one attempt per day.

    Now we mark interactively. This provides instant feedback to the student on the mark awarded, and on the major areas where they have lost marks. The marks are stored, and are made available also to the course teacher and the student's tutor. In addition a copy of the student program is stored for possible future reference.

    1.4. Course administration

    The system was then extended to assist the teacher more in the administration of the course, and to broaden the scope of the student activities. Marks generated by staff can be entered by hand; these marks may be either amendments to existing marks (overriding the computer assessments), or marks for associated work (such as essays or reports) which would be marked by hand.

    In addition work in the form of essays/reports can be submitted on-line by the students (having been generated using a word processor), stored on the system, marked by hand on or off the machine, and the marks then entered by hand. The essays sometimes form part of a programming exercise, and at other times are independent exercises.

    The teacher can then look at the mark statistics for a class, exercise or student, find who hasn't submitted (and perhaps email them and/or their tutors), look at overall class program metrics, and check for plagiarism in submitted work. The overall metrics for a given exercise are useful in keeping the teacher in touch with the current performance of the class as a whole; this is more important when the teacher is not hand marking student work. The plagiarism pattern over a series of exercises can be significant; in general the known presence of plagiarism tests acts as a considerable deterrent to copying.

    A tutor can look at the progress of tutees, and look at their submitted work. We find it better to email results regularly to the students' tutor rather than expect the tutor to log in and look it up!

    2. Experience of the use of Ceilidh

    Experience of use over a number of years at several sites has produced the following observations.

    Modifications to the style

  • It has proved easy to modify the courses as distributed, to change them and add new features. Different institutions use Ceilidh in different ways (some do not use skeletons at all, while some use almost complete programs as skeletons; some use many small exercises, others set fewer larger problems).
  • A number of other sites have written new courses (and their associated tools) to run under Ceilidh. The list of courses below shows the origin of each course.

    Simpler marking

  • We have generally moved to simpler marking metrics as experience develops. Originally the thought was that, to mimic human marking, we must have a very complicated marking process involving equivalents of all the human marking processes we could find. We now think that it is better for the students to have much simpler onjectives; and it must be possible for the teacher to both explain and justify these objective to the class. It is implicit therefore that the teacher understands the marking process; this is a non-trivial taks for new users.
  • There were originally six main marking tools (dynamic correctness, dynamic efficiency, typorgraphioc layout, program structure, program complexity, and program features). We now concentrate on three ((dynamic correctness, typorgraphioc layout, and program features) This is fairer to students; they have clear objectives

    The use of Ceilidh marks

  • The marks produced by Ceilidh can be used in different ways. Some courses are 100% ceilidh assessed; the marks produced by Ceilidh need to be scaled to fit with the institution's generally accepted borderlines. Facilities are available to do this.
  • Ceilidh marks can be used in other ways: All of these save considerable human effort when compared with traditional marking systems.

    The use of essays.

  • We make much more use of essays now. These are intended so that the student can demonstrate their thinking behind their program design and testing.
  • The essays could be marked on-line, but in general We print and take home to mark for each student a single document containing:

    The essays are then marked by hand.

    Innovative marking.

  • Many other possibilities arise for student assessment in Ceilidh.
  • We have a new "Software Tools" course, which teaches general software principles. It is a challenging new course requiring a new approach to marking. A typical exercise might be to "write a print spooler"; we assess their solutions using simulation. Other exercises involve Inter Process Communcation (IPC) techniques; these can be marked by driving the programs from a shell script. We may need to start several processes to run concurrently.
  • For teaching the use of publishing tools, we need to be able to assess graphical (ASCII or PostScript) output to check that they are inserting headers and footers as specified.
  • This course also included units on the Unix Bourne shell and on "awk" programming. These were poth treated as interpreted programming languages, and tested using techniques related to those for compiled languages.

    Other ways of marking

  • For some exercises, the stiudents are required to design and submit a set of test data for the [problem under consideration. We then (outside Ceilidh) compile and run all of the student programs against all of the test datas. The students get a better mark if

    Miscellaneous

  • Simple things like tutors appreciate if I email student marks to them even though they COULD find out for themselves.

    3. Educational considerations

    We will look at the educational implications of the essential student process under Ceilidh.

    The student reads the question

  • The question must be complete, non-ambiguous, consistent This is surprisingly hard, even in relatively simple examples. Floating point calculations cause particular problems. A simple exercise, hurredly implemented after a trip abroad, involved currency conversion between twi denominations. The basic formula is simple:

    At which points does the rounding occur? The question MUST be specific.

  • In general, the more specific the question, the easier the marking process.

    The student "setup"

  • The course developer decides how helpful the skeleton is to be. Some teachers don't allow any skeletons (the students must always start from scratch). We usually just miss out a few key sections of the model solution which which involve this week's topic. We believe that in industry programmers usually start from a skeleton of some sort. Some developers give complete working examples as a skeleton, for the students to explore.
  • As well as a skeleton program, it may be useful for Ceilidh to set up:

    The development (edit/compile/run) cycle

  • Nowadays many PC environments (Borland etc) are excellent for program development. Future versions of Ceilidh will make it an option to leave Ceilidh for this stage. This relieves the load on the central system, although there may be incompatibility problems between compilers.

    Submission

  • The use of MAXSUB MINGAP. The use of a large value for MAXSUB enables student to work towards a standard; bright students if they wish work to the best grade (grade A), others can work to achieve a pass grade (grade C) only. Students are then encouraged not to spend further time once they have reached their chosen standard.
  • A small MAXSUB or a large MINGAP makes student think more carefully before re-submitting.
  • With a small MAXSUB the results show student's ability to be right first time the student's ability to be right first time. In the Mathematics courses at Nottingham we set one exercise each week with large weight but only one submission (this assesses their ability) and others each week with a small weight but several submissions.

    4. Administrative considerations

    For legal and moral reasons to make the system acceptable to students, we think that

  • the student must be able to see how the marking works; and
  • the student must be able to see data about them stored in the system (including marks and submitted programs and essays).

    For academic reasons at our institution we must keep an archive copy of the complete course including solutions and marking schemes for

  • an external examiner to check borderline candidates at later examiners' meetings; and
  • in case of an appeal by a student at a later stage, when we would need to be able to produce copies of all that student's work.

    Ceilidh can keep audit trails of

    We have found this useful in monitoring monitor defaulting students, and in analysing the used of Ceilidh.

    5. Funding

    The original system was developed and used locally at Nottingham in both the Computer Science (supporting C++ teaching) and Mathematics (supporting C teaching) departments. The Computer Science Department then received a small grant from the local "Enterprise in Higher Education" initiative. This was used to support a graduate student (Adrian Bullock) over the summer of 1992, who tidied the system, and and arranged the distribution of the system with C and C++ courses to other departments and sites.

    A consortium of Polytechnics and Universities then jointly produced an application for funds to the TLTP (the "Teaching and Learning Technology Programme" run by the "Higher Education Funding Council for England") and was successful in obtaining funding for a 3-year project. Most of the funding was for the integration into Ceilidh of software which was to be developed at various sites using other funding arrangements.

    That grant has now ended. A smaller continuation grant has been obtained from TLTP, and this is being used together with grants obtained by Ngee Ann Polytechnic in Singapore to generate a new and improved system.

    6. The current state

    The final Nottingham release is available over FTP, and contains the following material.

    Unix Ceilidh

    The final Unix version of Ceilidh (release 2.5) has a number interfaces.

    The WWW version can be tried by going to the page

    and selecting the "ceilidh" pointer.

    The courses available with Unix Ceilidh are as follows.

    A statistics package is available for viewing details of course statistics.

    PC Ceilidh

    A stand-alone single-user version of Ceilidh without the administrative facilities was implemented by Lancaster University. It supports the C and C++ courses. It also supports a hypertext language for developing CAL-type notes, written by Manchester Metropolitan University, but the notes have not yet been re-written to use this facility.

    7. Current developments

    The major development currently taking place at Nottingham University (UK) is the production of revised enhanced versions of Ceilidh and its courses as follows. The dates given are approximate.

    Some parallel developments are taking place at other universities, some a on-going research projects. Other suggestions and contributions would be very welcome. They include the following.

  • Course administration:

  • Other courses:
  • Other platforms:

    Questionnaires completed by the students (every course has an on-line questionnaire as the last "exercise" of the course) do not necessarily produce unbiassed results. The results analysed so far indicate that they agree overwhelmingly system is helpful.

    In addition to the questionnaires, we have had discussions with students at the end of each course. The general impression given is that they find the system very supportive.

    Some students express concern at having difficulty in finding the last few percentage marks to raise their total from the mid-nineties to the high nineties. Others say that they use Ceilidh to develop their mark up to a certain figure (80%?) and then leave the problem completely.

    9. Other considerations

    Educational problems

    There are many areas of educational interest.

  • How much feedback should we give to the student ?
  • How should we control the number of attempts?

    Practice would be different if Ceilidh were being used for training/quality control in industry.

    Reading programs

    Some teachers attach importance to teaching programming through the reading of programs.

    The skeleton programs from which the students start provide this experience. The skeletons for some exercises form an almost complete program; they may provide a complete module for which a linking module has to be written. All this gives good experience in reading code.

    In addition, a number of exercise themes follow through the course, and students are obliged to read and develop code they wrote earlier in the course.

    Availability

    We would like to see a wider availability of

    Distance learning

    There are a number of distinct scenarios for the use of Ceilidh.

  • (i) Centralised.
    All processes run on a central system. Students perform their development in the system under Ceilidh, so that Ceilidh can monitor all processes. This is the way we run the early stages of our introductory course.
  • (ii) Local.
    We use a central system, but program development is not necessarily done under Ceilidh. The student reads the question and obtains a skeleton answer under Ceilidh, leaves Ceilidh to develop a solution, and returns to Ceilidh to submit and mark. This is the system we use in our second C++ course.
  • (iii) Close
    A close coupled system relies on the use of a network. All of the notes, questions and marking are centralised, but student development work is done on the peripheral machines. These machines could be UNIX systems, or based on other platforms such as PCs or Macintoshes.
  • (iv) Loose
    We could have a loose client-server approach, in which the server supplies the requested information and marking facilities, but does not control or co-ordinate the marks.
  • (v) Standalone.
    There are no centralised functions, and thus Ceilidh becomes a self-paced self-teaching tool. There are no secret solutions, or overall marking and control.

    Testing operator practice

    At the moment, the marking process runs from input, through a program, checking the resulting output for validity. The program is the item under test.

    It is possible to imagine a scenario in which the program is fixed, but the student's task is to create the input to it.

  • The program might be a database; the problem might be accessing certain information.
  • The program might be a reservation system; the problem is to perform a certain series of transactions.

    All this could be monitored by a modified version of Ceilidh, and marked. It could be used to assess trainee computer terminal operators.

    Courses completely assessed by Ceilidh

    Some early C and C++ courses at Nottingham are completely assessed by Ceilidh. The results of the Ceilidh exercise assessments are weighted, scaled and submitted as the returned mark for the module. There are certain implications when this happens.

  • (i) A complete dump of the file system is taken at the end of the course, to be made available to external examiners, or in the event of an appeal. Any later changes to the on-line information will not affect the archive.
  • (ii) The marks and submitted programs must be made available to the student. The students cannot then claim that "That isn't the version I submitted". There are two Ceilidh commands to assist in this: cks allows students to check all of their submitted marks and programs as seen by the system; and vm lets them view their marks, complete with weighting and scaling factors which will be applied as part of the examining process.

    10. Student participation

    Students are encouraged to send comments to the teacher at many points in the system. These comments vary from

  • I think my program is perfect, why doesn't Ceilidh give it 100%
  • I'm sorry it was late, I was unavoidably detained.

    It is hoped to develop a help desk within the Ceilidh system.

    11. Using Ceilidh

    The software is available by FTP over the Internet.

    It is vital that anyone planning to use the course should be familiar with the marking metrics, and should adjust them to suit local standards.

    12. Contributing universities

    Ceilidh is the result of collaboration from a number of universities. Their help is appreciated, and is hereby acknowledged.

    Institution Contribution Available from Nottingham?
    _
    Lancaster PC version y
    Heriot Watt SML course y
    Evaluation
    Manchester Metropolitan PC hypertext system y
    Liverpool SQL course y
    Royal Holloway Pascal course y
    Luton Evaluation
    Loughborough Macintosh interface n
    Nottingham
    Abdullah Zin X interface
    Cleve Gibbon X interface y
    Neil Gutteridge Statistics package y

    13. World Wide Web

    Further details of the Ceilidh project are available on the WWW at

    14. References

    Recommended reading material includes the following Ceilidh documents (distributed with Ceilidh, and available over the WWW): the Student Guide [ref 1] [content] , the Tutor Guide [ref 2] [content] , the Teacher Guide [ref 3] [content] , the Developer Guide [ref 4] Steve Benford, Edmund Burke, Eric Foxley, Course Developer's Guide to the Ceilidh System, LTR Report, Computer Science Dept, Nottingham University (January 1994)

    The following journal papers describe new aspects of the system: from the Software Quality Journal [ref 5] Steve Benford, Edmund Burke, Eric Foxley, A System to Teach Programming in a Quality Controlled Environment, The Software Quality Journal (1993) 2; pp 177-197 , the Association for Learning Technology Journal [ref 6] and the Journal Education Technology [ref 7] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Modh Zin, Ceilidh as a Course Management Support System, Journal of Educational Technology Systems 22; 3; (September 1993)

    The following conferences papers summarise various aspects of Ceilidh: The CBLIS conferences at Kiev [ref 8] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Modh Zin, The Ceilidh Courseware System, Proceedings of the International Conference on Computer Technologies in Education 1993, Kiev, Ukraine (September 1993) , Dublin [ref 9] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Modh Zin, Experience using the Ceilidh System, Proceedings of the All Ireland Conference on delivering the Computer Curriculum, Dublin (September 1993) , Amsterdam [ref 10] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Modh Zin, Integrating Software Quality Assurance into the Teaching of Programming, Proceedings of the CSR 10th Annual Workshop "Applications of Software Metrics and Quality Assurance in Industry", Amsterdam (1993) , CBLIS at Vienna [ref 11] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Mohd Zin, Ceilidh: A course administration and marking system, Proceedings of the International Conference on Computer Based Learning in Science, Vienna (December 1993) and [ref 12] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Mohd Zin, Experiences with the Ceilidh System, Proceedings of the International Conference on Computer Based Learning in Science, Vienna (December 1993) , York [ref 13] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Mohd Zin, CEILIDH - a management and marking system, Assessment in Higher Education Computing, York (June 1993) [ref 14] Steve Benford, Edmund Burke, Eric Foxley, Neil Gutteridge, Abdullah Modh Zin, Ceilidh as a Course Management Support System, Journal of Educational Technology Systems 22; 3; (September 1993)

    [ref 15]

    References

    Notes converted from troff to HTML by an Eric Foxley shell script, email errors to me!