go-autograder

CS 134 Autograder, made by Alec Machlis (@burturt). Portions of this code is based off of https://github.com/nthnluu/go-autograder, with heavy modifications.

Using the Autograder

This autograder is intended to be used by checking out the autograder in a go-autograder submodule folder inside of the git repository containing the configuration folders. A template is available at https://github.com/ucla-progsoftsys/go-autograder-template with the configuration files already set up.

How the Autograder works

During setup, the specified go version will be installed, in addition to running any shell commands specified in a custom_setup.sh script.

This autograder works by running each test by name and folder specified in autograder.config.json, and giving credit based on the exit code status of the test case to determine whether it passed. Only tests that you specify in autograder.config.json are run and sent to Gradescope.

The autograder also applies a built-in exponential late penalty: $2^{n-1}%$ deduction for $n$ days late (rounded up), capped at 100%.

Multi-run exponential scoring

When a test is configured with count > 1, the autograder runs it multiple times and awards partial credit using a smooth exponential decay formula: points × 0.5^((1 - passRate) / 0.1), where pass rates below 70% receive 0 points.

Pass Rate	Points Awarded
100%	100%
95%	≈ 70.7%
90%	50%
80%	25%
70%	12.5%
< 70%	0%

Values between anchor points are smoothly interpolated. This is the default behavior. To revert to the previous all-or-nothing scoring (any failure = 0 points), set "allOrNothing": true on the test case.

When the autograder runs, the student's submission will be copied into /autograder/source/submission, all existing _test.go files will be deleted (to prevent students from providing their own test cases), and then files inside of replacement_files will be overlayed over the student's submission, such as your own test_test.go files. You can make any necessary changes to a student's submission not possible with this folder -- such as ensuring parts of a file are unchanged -- before the autograder runs by adding shell commands to custom_run_autograder.sh.

File hierarchy

• setup.sh - A setup (Bash) script that installs all your dependencies.
• run_autograder - An executable script, in any language (with appropriate #! line), that compiles and runs your autograder suite and produces the output in the correct place.
• src/test_runner - A Go module containing the code responsible for running go test on a student's submission, parsing the results from stdout, and returning a results.json file in Gradescope's specified format.
• create_zip.sh - A shell utility to be run within a template repository to generate the zip file to upload to Gradescope

Config files

autograder.config.json

This JSON file is where you will configure your autograder for your particular assignment. In this file, you must specify the names of the tests you want to use for grading, along with associated point values.

{
    "visibility": "visible", // Optional: visibility setting for autograder results: visible, hidden, after_due_date, after_published
    "uploader": "bashupload.com", // Deprecated and disabled. Do not use.
    "score_message": "Please note: the automatically generated autograder score when you submit is not your final score. We will rerun the autograder once after submission closes on your active submission to determine your actual project score.", // Optional: A message to be appended to the final autograder output. This is displayed at the top of the student's screen when viewing the Autograder output.
    "ratelimit": { // Optional: Defines ratelimit of log uploading. 
        "count": 2, // Max number of submissions allowed within the time window
        "minutes": 20 // Time window in minutes
    },
    "tests": [
        {
            "name": "TestAddTwoNumbers",  // The name of the test (must match the test name as defined in test files)
            "number": "1.1", // Optional (will just be numbered in order of array if no number given)
            "points": 5, // The point value of the test case (supports floats)
            "visibility": "visible", // Optional: visibility setting for test case: visible, hidden, after_due_date, after_published
            "folder": "main", // Optional: directory to run go test in, relative to root folder of submission files
            "timeout": "600s", // Optional: test timeout for go test command - fails if it goes beyond this time
            "count": 10, // Optional: total number of times to run test case. When count > 1, exponential decay scoring is used by default (see above)
            "parallelCount": 2, // Optional: max number of runs of this test case to execute at once while satisfying count
            "race": true, // Optional: specify whether to run test case with -race flag
            "allOrNothing": false // Optional: set to true to disable exponential scoring and use all-or-nothing (any failure = 0 points) when count > 1
        },
        {
            "name": "TestAddTwoNegativeNumbers",
            "number": "1.2",
            "points": 5,
            "visibility": "visible"
        },
        {
            "name": "TestAddNums",
            "number": "2.1",
            "points": 5,
            "visibility": "visible"
        },
        {
            "name": "TestAddNumsOne",
            "number": "2.2",
            "points": 5,
            "visibility": "visible"
        }
    ]
}

replacement_files/

This folder's contents will be overlayed on top of students' submissions before running tests. For example, if students should have a main/test_test.go file, make the file replacement_files/main/test_test.go, which will replace (or add) that file in the student's code but keep any other files inside of the main folder submitted.

custom_setup.sh

This shell script is run (using source custom_setup.sh) during autograder build time. Set the GO_VERSION variable here (e.g., GO_VERSION=1.24.13) to specify which Go version to install. You can also add any additional setup commands here.

custom_run_autograder.sh

This shell script is run after the student's submission is copied into /autograder/source/submission and had their files overlayed, but before running the test cases. This can be used to, for example, check integrity of parts of files in the submission, verify file structure, check for extraneous/missing files, or search for known suspicious strings.

required_files.txt

Put one path per line (relative to the submission root) to a file that must exist. If any file path does not exist, the autograder will reject the submission and give 0 points.