Tutor's Guide to Ceilidh S D Benford, E K Burke, E Foxley, N Gutteridge, A Mohd Zin ltr @ cs.nott.ac.uk Learning Technology Research Computer Science Department University of Nottingham NOTTINGHAM NG7 2RD, UK ====================================================================== Note: Revision 2.4 October 23, 1996 ====================================================================== 1. Introduction Ceilidh is an on-line coursework administration and auto- marking facility designed to help both students and staff with programming courses. It helps students by informing them of the coursework required of them, and by permitting them to submit their work on the computer, instead of having to print things out and hand them in. It also marks pro- grams directly, and informs the student and teacher of the mark awarded. The marking uses a comprehensive variety of static and dynamic metrics to assess the quality of submit- ted programs. Ceilidh also provides students with on-line access to notes, examples and solutions, and provides tutors with extensive course monitoring and tracking facilities. This document is a guide for tutor users of the Ceilidh sys- tem. For more details of the student view of the system please see the student[1] guide. 2. Overview of Ceilidh The Ceilidh system acts in a number of ways for students, tutors and teachers, and can support a variety of different courses. There are facilities for students (reading notes and course- work definitions, looking at examples, developing programs, marking programs, submitting work), and tutors (observing submitted work and marks, checking for plagiarism) and for teachers (amending course material, setting up exercises). The appropriate facilities are offered to appropriate users by the Ceilidh system itself, which takes note of the login identification of the user and checks this against lists stored in the system. To obtain access to the tutor facili- ties described below, your name must have been added to the appropriate list by a teacher for this course, or the Ceilidh system administrator. Ceilidh teachers and developers automatically have access to tutor facilities. 3. Using Ceilidh as a Tutor Upon issuing the command ceilidh you will be greeted with a standard Ceilidh system level menu such as that shown in figure 1. - Ceilidh system - Type ________________________________________________________________ | lc list course titles | sc move to named course | | vp view papers | pp print papers | | clp change printer | h for more help | | co make a comment to teacher | q quit this session | |_______________________________________________________________| Figure 1 : System Level Ceilidh Menu ====================================================================== Note: - Example menus are shown in this document. Menus seen in practice may vary slightly from those shown, since the actual menu you are offered reflects only those facili- ties available at the time. ====================================================================== You may see a menu with minor differences; if the printing of papers by students is forbidden, for example, they will not see the "pp" option. Tutors will always see this option, but in the form print papers (tutor) if it is not available to students. If the tutor then selects a particular course (using the "sc" command) a menu such as that shown in figure 2 is displayed. Course and unit menu for course "pr1" unit "1" ________________________________________________________________________ | lu list unit titles | su set unit code | | lx list unit exercise titles | sx move to named exercise (1) | | lux list units and exercises | state current exercise state | | vn view notes on the screen | pn print notes on letter13 | | csum read course summary | usum read unit summary | | vm view all marks | | | clp change printer | h for more help | | co make a comment to teacher | q quit | | tutor tutor administration | |_______________________________________________________________________| =========================================== Unit command: Figure 2 : Course and Unit Level Ceilidh Menu This differs from the student menu by having the tutor option at the bottom; if you do not see this option, you are not a registered tutor for this course, see your Ceilidh course teacher if you wish to be added to the list. As a tutor, in addition to having the additional "tutor" option, you will be permitted to override certain student restrictions such as the printing of notes and the submis- sion of coursework on ordinary student menus irrespective of the teacher's settings of limited number of submissions and minimum time intervals for the course. Tutors access their own special facilities using a menu obtained by typing the command tutor A menu such as that in figure 3 will then be displayed. Tutor's (course pr1 unit 2 ex 5) menu ___________________________________________________________________________________ | vr view course register | pr print register | | fs find named student entry | ft find tutees' register entry | | rs register student | em enter marks | | ss summarise named student's work | st summ tutees' work in this ex | | su set unit | sx set exercise | | lu list units | lx list exercises | | vs view named student's work | vt view tutees' work in this ex | | mi search for missing students | uk search for unknown students | | sv save missing etc lists | vm view mark summaries | | plag search for plagiarism (slow) | met overall exercise metrics | | su set unit | sx set exercise | | mt mark typographic | mc mark complexity | | md mark dynamic | mf mark features | | em enter marks | vo view oracle files in this ex | | mss mark student source | | rms remark student | rmas remark all students | | h help | H View tutor document | |__________________________________________________________________________________| =========================================== Type tutor command: Figure 3 : Tutor Level Menu This menu gives the tutor access to commands involving read access to confidential Ceilidh information, which perform the following. o Find students registered on the current course o Allow tutee's work to be monitored o Help determine areas of weakness in a tutee's solution program o Examine the metrics used to mark student's programs o Identify students who have not submitted work for an exercise o Identify students who have submitted work and are not registered A summary of these commands is as follows 4. General commands su This command allows you to change the currently set unit and sx allows you to change the currently set exercise. There is no change of menu after this command. These com- mands MUST be used before later options which relate to a particular exercise. lu List units in the current course. lx List exercises in the current unit. 5. Student registrations vr This allows the tutor to view* ====================================================================== Note: * All output of this type is fed through the viewer defined by the environment variable PAGER, typi- cally "more" or "less". ====================================================================== the course register for the current course, or just your own tutees. Typical output is cxa:Claire Ackerton (ef):95.02.16-12.58.00 mja:Matthew James Duncan Alderpeak (hugh):95.02.16-12.58.05 rjb:Robin Jonathan Badcock (cah):95.02.16-12.58.13 mqb:Maria Bagiokou (ef):95.02.16-12.59.13 or whatever you have stored in the "student" register files (only the first two fields are essential). If a paper copy is needed, the "pr" command can be used to send the register to a printer. Individual students registered on courses can be found using the fs command. This looks for a student in all the course registers using a name or substring. If the first letter of the substring is a capital letter, the system assumes that this is a student's name and will look for a student whose name contains this substring. If the first letter is lower case, the system assumes that the string is a username. Output might be Name or substring? Eric aeh : Hill, Alan Eric (leon) : course cua ef : Eric Foxley : course c ef : Eric Foxley : course pr1 ef : Eric Foxley : course pr2 exd : Erick Deschamps (dge) : course pr2 seb : Bamford, Stephen Eric (djc) : course cua ft This asks for a tutor's login name and returns the names of his/her tutees, and the courses they are tak- ing. rs This command can be used to register students on a course. It asks repeatedly for a student's login name, full name and tutor's login name. Once all students have been entered, type "q" to quit this process. 6. Viewing submitted work and marks ss This command summarises a named student's marks. It first prints one line for every attempt at every exer- cise ordered by exercise (ordered by date and time within exercise), each line giving the unit and exer- cise number, the date and time of the attempt, whether it was late, the mark awarded, and whether it was marked by the student or the teacher. Typical output is Unit 1, ex 1: date 93.10.19-16.33.46, grade E Unit 2, ex 1: date 93.10.26-16.41.57, grade C Unit 2, ex 1: date 93.10.28-10.49.55, grade A Unit 2, ex 7: date 93.10.28-10.02.52, grade E Unit 2, ex 8: date 93.10.28-10.27.59, grade E Unit 2, ex 8: date 93.10.28-10.40.01, grade A Unit 3, ex 8: date 93.11.09-11.25.24, grade E Unit 3, ex 8: date 93.11.09-11.43.49, grade C Unit 3, ex 9: date 93.11.09-10.18.55, grade E Unit 4, ex 3: date 93.11.10-11.51.59, grade E Unit 4, ex 3: date 93.11.14-13.46.11, grade E Unit 4, ex 3: date 93.11.15-10.08.06, grade E Unit 4, ex 4: date 93.11.09-10.21.13, grade A Unit 4, ex 7: date 93.11.14-15.37.04, grade E Unit 4, ex 8: date 93.11.09-11.23.12, grade E Then for each of the formally set exercises, it lists the unit and exercise number, the mark awarded (starred if submitted late), the class average mark, and the weight to be awarded to that exercise in the final total. The summary line at the end gives the weighted total mark, the scaled mark (using scaling factors set by the teacher), the number of exercises submitted out of the total possible (and the number of late submissions in parentheses), and the total number of attempts. An example of the final summary can be seen below. gxb Student Gary Bingham Unit 1, ex 1: grade E , class avge A, weight 1 Unit 2, ex 1: grade A , class avge A, weight 2 Unit 2, ex 7: grade E , class avge B, weight 2 Unit 2, ex 8: grade A , class avge B, weight 2 Unit 3, ex 8: grade C , class avge B, weight 3 Unit 3, ex 9: grade E , class avge B, weight 3 Unit 4, ex 3: grade E , class avge B, weight 4 Unit 4, ex 4: grade A , class avge A, weight 4 Unit 4, ex 7: grade E , class avge C, weight 4 Unit 4, ex 8: grade D , class avge C, weight 4 Unit 4, ex 9: grade E , class avge C, weight 4 Unit 5, ex 1: grade A , class avge A, weight 5 Unit 5, ex 4: grade C , class avge B, weight 5 Unit 5, ex 7: grade E , class avge C, weight 5 Unit 6, ex 2: grade A , class avge B, weight 5 Unit 6, ex 4: grade A , class avge A, weight 6 Unit 6, ex 6: grade B , class avge B, weight 6 Unit 7, ex 8: grade A , class avge A, weight 6 Unit 7, ex 9: grade E , class avge C, weight 7 Unit 7, ex bg2: grade E*, class avge B, weight 10 Unit 7, ex n1: grade D , class avge A, weight 10 Unit 7, ex n2: absent, class avge C, weight 10 gxb Gary Bingham ( 124-th in class) Grade E, Submitted 21( 1 late) of 22, Attempts 39 Marks marked with an asterisk were late; on the last summary line, the number in parentheses is the number of exercises submitted late. The class average is a useful comparison point. st This is similar to the above, except that it summarises the work of all of a specified tutor's tutees. vs This command allows the tutor to examine every piece of coursework submitted by a named student. The student solution to each exercise is offered in turn; you may view the solution or skip it. After each exercise, you can continue to view the next exercise solution or quit. vt This allows the tutor to examine submitted coursework by each of his/her tutees in the currently set unit and exercise. 7. Finding who has submitted mi Lists the students who have not submitted work for the currently set exercise. A list is printed of login names of students in the register (the file "students" in the course directory) but who have not submitted work, for example In student list, but no C file: akp cag ef pcx Total 4 uk Looks for students who have submitted work yet are not registered on the course (i.e. do not appear in the "students" file). The above lists can be saved for later processing using the "sv" command. 8. Marking and submitting work. mss To mark and submit a student's source directly (a tutor might wish to do this after the hand-in date if the student has justifiable reasons for being late) use "mss" . First change directory to the student's direc- tory by using "cd" outside or inside Ceilidh, and select the appropriate unit and exercise. This option should then appear on the menu. The command checks that the student's login name occurs as a component of the current working directory path. This is a security check. The student must give public read permission to the source, and public execute to the executable. The "mss" command then marks and submits the student source, appending an entry such as pr1:7:bg2:dxw:94.12.15-10.26.15:ef:100: Dyn 100: Typ 100: Fea 100: Str 100: to the marks file; the fields are course:unit:exercise:student:date-time:person submitting:mark:mark details so that any such marks are attributable to the tutor who performed the marking. The above record was added for student "dxw" by tutor "ef" . The student should then be told to remove the public permissions from the program files. rms and rmas These commands recompile and remark an already submit- ted source (unlike "mss" above which works from the student's own source and executable), and can be per- formed at any time by the tutor after the student has performed a submission. The current directory must be writable (because the system is going to have to recom- pile the program), and must not contain a file of the name required for the source of the current program (e.g. "prog3ssr.c"). With "rms" , the login name of a student is requested; that student's submitted source will be copied from the solutions directory of the chosen exercise, copied into a file such as "prog3ssr.c" in the current directory, compiled into a file such as "prog3ssr", and marked. The "rmas" command goes through all the submitted work in the solutions directory for the chosen exercise, and remarks each one in turn. It is, of course, very slow if 200 exercises have to be compiled and marked. This is normally performed only if the marking scheme has been amended (by the developer). On both of these, there is an option to email the new marking results to the student or students; this is recommended. The marking outputs are in any case appended to a file __.rms in the current directory so that they can be inspected. 9. Viewing marks vm This command allows you to view or print mark summaries for a student, the current exercise or the current course. It first asks whether the marks are to be viewed on the screen, stored (in which case it asks for a filename) or printed. It then asks whether the marks requested are those for a particular student (it will request a login name), or for one exercise (the exer- cise currently set by "su" and "sx" ), or for the whole (currently set) course. For a student, the output is as for the "ss" command. For a course, the output might be cxa Claire Ackerton ( 56-th in class) Grade A, Submitted 22( 1 late) of 22, Attempts 64 mja Matthew James Duncan Alderpeak ( 81-st in class) GradeC, Submitted 21( 1 late) of 22, Attempts 46 rjb Robin Jonathan Badcock ( 120-th in class) Grade D, Submitted 20( 1 late) of 21, Attempts 43 mqb Maria Bagiokou ( 4-th in class) Grade A, Submitted 22( 1 late) of 22, Attempts 73 thb Theo Hallam Banks ( 46-th in class) Grade A, Submitted 22( 1 late) of 22, Attempts 52 dgb Daveran Giles Baulch ( 108-th in class) Grade C, Submitted 21( 1 late) of 22, Attempts 25 gxb Gary Beard ( 124-th in class) Grade E, Submitted 21( 1 late) of 22, Attempts 39 lxb Louise Beard ( 92-nd in class) Grade C, Submitted 22( 1 late) of 22, Attempts 45 cag Cleveland A Gibbon ( 134-th in class) Grade E, Submitted 3( 0 late) of 21, Attempts8 Mark analysis 135 students Mark Scaled Range Range CummCumm range range total % total % 0 <= mk < 70 0 <= sc < 40 14 10 14 10 70 <= mk < 80 40 <= sc < 45 11 8 25 18 80 <= mk < 90 45 <= sc < 50 25 18 50 37 90 <= mk < 95 50 <= sc < 70 25 18 75 55 95 <= mk < 100 70 <= sc < 100 60 44 135100 10. Plagiarism and metrics plag This command will carry out a search for plagiarism within the current exercise. It is very slow and machine demanding. Read the departmental policy docu- ment[2] on plagiarism to find our use of plagiarism results. It looks for pairs of similar solution pro- grams among those submitted so far. It lists pairs of login names in order with the most similar pair first. The dates and sizes of the programs are also shown. Typical out is as follows. Ceilidh system plagiarism check: course pr1 unit 8 ex 3 Fri Jun 11 09:21:01 BST 1993 , 94 files ***** Difference 1 ***** kjc Dave Prentice (oklee) 6863 Jan 24 15:32 kjc.C ktc Steve Earwicker (prs) 6748 Jan 20 13:00 ktc.C nhx Lucy Barwell (oklee) 6855 Jan 7 17:10 nhx.C bmz Jane Wilkinson (oklee) 6884 Jan 20 10:51 bmz.C wcw Les Shaw (msg) 6833 Jan 24 15:26 wcw.C ktc Steve Earwicker (prs) 6748 Jan 20 13:00 ktc.C ***** Difference 2 ***** kjc Dave Prentice (oklee) 6863 Jan 24 15:32 kjc.C wcw Les Shaw (msg) 6833 Jan 24 15:26 wcw.C ***** Difference 4 ***** bmz Jane Wilkinson (oklee) 6884 Jan 20 10:51 bmz.C kjc Dave Prentice (oklee) 6863 Jan 24 15:32 kjc.C bmz Jane Wilkinson (oklee) 6884 Jan 20 10:51 bmz.C ktc Steve Earwicker (prs) 6748 Jan 20 13:00 ktc.C bmz Jane Wilkinson (oklee) 6884 Jan 20 10:51 bmz.C wcw Les Shaw (msg) 6833 Jan 24 15:26 wcw.C ***** Difference 5 ***** nhx Lucy Barwell (oklee) 6855 Jan 7 17:10 nhx.C kjc Dave Prentice (oklee) 6863 Jan 24 15:32 kjc.C nhx Lucy Barwell (oklee) 6855 Jan 7 17:10 nhx.C wcw Les Shaw (msg) 6833 Jan 24 15:26 wcw.C nlx Jean Best (mjg) 5018 Jan 25 12:09 nlx.C cla Claire L Aspell (sdb) 5317 Jan 20 16:22 cla.C ***** Difference 6 ***** mcs Mark C Willoughby (pmc/ANO) 4692 Jan 23 14:54 mcs.C pdr Paul D Dimbleby (pmc/ANO) 4412 Jan 25 23:02 pdr.C ***** Difference 13 ***** akx Naji Kumar (pmc/rbh) 2753 Jan 25 11:14 akx.C pmj Peter M Jamithorp (leon) 2578 Jan 22 21:05 pmj.C ***** Difference 14 ***** tsx Tom J Johnson (pmc/ANO) 3325 Jan 25 11:24 tsx.C pmj Peter M Jamithorp (leon) 2578 Jan 22 21:05 pmj.C ***** Difference 15 ***** akx Arun Kumar (pmc/rbh) 2753 Jan 25 11:14 akx.C tsx Tom J Johnson (pmc/ANO) 3325 Jan 25 11:24 tsx.C ***** Difference 16 ***** tsx Tom J Johnson (pmc/ANO) 3325 Jan 25 11:24 tsx.C agx Andrew Bramman (mfd) 2478 Jan 22 14:00 agx.C met This produces overall metrics for work submitted for the current exercise. This is useful to the teacher in determining in which areas the class is weak. See the "mt" command below for a description of the output. 11. Checking details of student marks The tutor has the facility to examine the details of marking in various areas of a student's program (which must be stored in a file such as "prog21.C" [for the solution to unit 2 exercise 1 in a C++ course] in the current directory) in more detail using the following commands. Students should NOT be encouraged to look at these detailed break- downs, as that will encourage them to tweak results. The tutor will first need to change directory to the one con- taining the student source and executable. This can be done within Ceilidh by a command such as cd /stud/ug/fred mt to examine typographic marks in more detail. The result may be Typographic Analysis factor:value: mark:out of: lost Average characters per line: 15.0: 10.0: 10 : 0.0 % blank lines: 27.8: 10.0: 10 : 0.0 % space per line: 18.5: 10.0: 10 : 0.0 Average identifier length: 4.1: 5.4: 10 : 4.6 % names with good length: 95.7: 5.0: 5 : 0.0 % lines as comments: 16.7: 10.0: 10 : 0.0 % chars in comments: 15.4: 10.0: 10 : 0.0 % indentation: 19.5: 10.0: 10 : 0.0 % indent errors: 0.0: -0.0: -20 : 0.0 % [] indent errors: 0.0: -0.0: -20 : 0.0 % () indent errors: 0.0: -0.0: -20 : 0.0 % uncommented }: 0.0: -0.0: -20 : 0.0 Largest typographic loss of marks was: "Average identifier length too small" Total for Typographic analysis is 93.9 out of 100 Score 93 For each significant typographic factor, the columns show the value achieved by the program, the mark awarded, the maximum mark, and the number of marks lost. The last column is the most useful. Metrics marked out of zero are not included. For more details of the metrics used and the way in which the marking is performed, read the "ASQA" (Automatic Software Quality Assessment) paper[3] in Ceilidh. Note: Tutors should learn the Ceilidh rules for typo- graphic layout rather than recommend their own prefer- ences to students. The official recommendations should be listed in the course notes. mc This examines the complexity metrics in a similar way. md This shows the details of the dynamic marking. Each dynamic test is performed in turn, and the output inspected by an oracle. You should also read the Ceilidh oracle document[4] by Eric Foxley. Output might be test min max cnt mrk oof cum oof lost RE 1 1 1 1 10 10 10 10 0 radius 2 1 1 1 10 10 20 20 0 30 3 1 1 1 10 10 30 30 0 area 4 1 1 1 10 10 40 40 0 2827.43 Awarded 40 marks out of max 40 marks 100 test min max cnt mrk oof cum oof lost RE 1 1 1 0 0 10 0 10 10 [Nn]egative 2 1 1 1 10 10 10 20 0 [Nn]ot Awarded 10 marks out of max 20 marks 50 This shows two tests. For each test, the table shows what was found, and the marks awarded. The columns in the tables include minimum and maximum number of times the expression should be found, the actual number of occurrences found ("cnt"), the mark awarded ("mrk") out of a maximum ("oof"), cumulative marks so far ("cum") out of cumulative total ("oof"), the marks lost on this expression, and the regular expression itself. Finally an overall percentage mark for the dynamic test is shown. The first test here involved searching for the strings "radius", "30", "area", and "2827.43", and all were found, 100% awarded. The second test looked for the string "[Nn]egative" (not found, 10 marks lost), and then "[Nn]ot" (found), 10 marks out of 20 or 50% awarded. mf This performs the "features" marking with the oracle. The oracle output is as described above for the dynamic tests. The process is to change in the next release. The "features" oracles are items which the teacher has prescribed relevant to the student's program source. An example of tutor output is test min max cnt mrk oof cum oof lost RE 1 1 1 1 10 10 10 10 0 define.*3.14159 2 1 1 1 10 10 20 20 0 =1:3.14159 3 4 4 5 10 10 30 30 0 >3:radius Awarded 30 marks out of max 30 marks Score 100 The teacher expected the string "define.*3.14159" to occur at least once, the string "3.14159" to occur exactly once, and the string "radius" to occur more than three times. 12. Miscellaneous commands em For hand marked examples, this command allows the tutor to enter marks by hand. It assumes the currently set course and unit, and asks for an exercise number. If the exercise already exists (it should in most cases), marks will be appended to the existing marks file. If not, the named new exercise is set up. You can then enter marks in three ways. o You type a name, the computer checks for unique- ness, you then type the mark. o The computer lists each name from the register in register order. o The computer lists just your tutee's names from the register. vo Allows a tutor to look at all the oracle files from which the student solutions are marked. You will be shown the oracles for checking the dynamic test output, and the "features" oracle if it exists. To understand these, you will need to look at the Oracle[4] document referred to above. See the "md" (mark dynamic) and "mf" (mark features) commands above. h Gives general help on tutor commands. q Returns the user to the previous menu (typically the course level student menu). EDITOR=vi The values of environment variables can be temporarily change by commands of this type. cd You can change directory from with Ceilidh by using the "cd" command. !command Any Unix command can be executed, or a shell called, if required. 13. Note The may be some activities which require more privilege than the tutor is permitted, perhaps involving access to files as the "ceilidh" user. You will have to ask someone with "teacher" or "developer" privilege to do these things for you. 14. Command line arguments You can call Ceilidh at the system level ceilidh or directly at the course level ceilidh -c pr1 or directly as tutor ceilidh -c pr1 -x tutor It is generally worth setting an alias for the last of these if you act as a tutor regularly. c Eric Foxley, Nottingham 1996