By Frank Vahid, Prof. of CS&E, Univ. of California, Riverside (zyBooks co-founder). Last update: 9/21/17.

After creating labs in zyLabs for a while now, I’ve learned some things about creating auto-graded labs that seem to increase student learning and minimize issues.

  • Develop in the zyBook. Have at least your first few assignments use the “Students develop in the zyBook” option, rather than uploading a file. You can then postpone introducing an IDE until later, which makes your first weeks a lot smoother (last time I taught, I just did the whole course using the in-book approach).
  • Avoid prompts. When developing interactive programs, prompts are essential. In contrast, auto-graded programs just read input and generate output. Prompts like “Enter an integer: ” can confuse beginning students, who don’t quite understand how to interpret an auto-grader’s output that shows prompts but not the user’s input.
  • Avoid unnecessary output. If you want students to learn something like expressions or branches, don’t also have them worry about getting all the output text exactly right. Just output the computed values. Keep output to a bare minimum. The interesting and challenging parts of programming don’t lie in generating lengthy text output.
  • Start SIMPLE. Define simple programs that do one thing, like “Read an integer and output that integer raised to the power 2, 3, and 4”. Don’t create multi-step programs until much later in the term. Now don’t misinterpret this as “Make your class easy”. Rather, make your assignments really focus on the core subject you are teaching that week, and eliminate unnecessary other stuff.
  • Start with exact input/output tests. Don’t try to get fancy with “starts with” or “contains” — those are actually harder. I see many instructors tripping over themselves by trying too many options. If you keep output simple, students can quickly master matching output exactly. And programming is all about precision, so I recommend you make students realize that quickly, by requiring exact output matching.
  • Make many small programs. Give students more smaller programs rather than one larger program. Smaller programs are less intimidating, and give them more practice specifically on what you want them to learn (e.g., branches). Before auto-grading, more programs meant more time spent grading, but that’s not the case now. I give 7 programs a week: 2 easy, 3 medium, and 2 harder (but not too hard). Each only takes about 30 min for me or my TA to make with the zyLabs platform.
  • Use a points threshold. Because auto-grading gives students their score immediately, you can now tell students they get full credit just for meeting some threshold. I give 7 programs per week, each with 10 possible points, with earned points depending on how many test cases they pass (I usually make 4-5 test cases). I then say “50 points in a week is full credit for the week”. Students LOVE this. If they get stuck, they can move on (and maybe come back). Turns out my students completed 85% anyways — same as when we had one-program-per-week. (Or, if you use one assignment a week, you can tell them “8 points is full credit” even though the assignment might have 10 points.
  • Don’t meter or limit early. Let students work without worry early in the term. If you feel it’s important to encourage student thought and testing and reduce over-reliance on the auto-grader, introduce metering or limits later in a term, and don’t be too strict — maybe set the metering to 2 or 3 minutes, or set a high limit like 50.
  • Focus on data. Before auto-grading, our programs did all sorts of things, like interactive tic-tac-toe, graphical moon lander, and more. Those don’t work with an auto-grader. Instead, focus on data analysis and manipulation, like “Read a list of numbers and output the max” or “Read a line of text and replace multiple spaces by one space”. I know those doesn’t sound as sexy, but I inform students that writing such programs may be useful in their future jobs (whether majors or non-majors), and they see that. Plus, they tell me that they enjoy figuring out the logic (even non-majors say that); programming can be inherently fun, without having to dress it up very much.

I’ve found that an easy-to-use auto-grader is having a more profound (positive) impact on how I teach introductory programming than I previously anticipated. Across all four of us who teach CS 1 at UCR, student performance has improved, including dramatically fewer fails/withdrawls than before, and course evaluations have gone up too and are quite high. Our CS 1 is now rated in the top 10% of all classes at UCR; not bad for a difficult, required class! But like any tool, an auto-grader needs to be used right for it to help. I hope the above advice helps you get off to a good start!

Have fun helping your students achieve great goals!