Creating a Custom Checker

Last updated: December 23, 2024

Custom Checker Code

Custom checker code consists of 3 parts. Two non-editable parts are referred to as a head & tail section. The editable part of the custom checker is called the body section. The body section has a method named run_custom_checker(). This method takes two arguments: a TestStructure object and a ResultStructure object. This method is executed after a candidate's submission is run. Each test case evaluates the candidate's submission results.

Test Structure

testcase_id	[int] ID of the test-case
testcase_input_path	[str] File path to test-case input
testcase_output_path	[str] File path to test-case output generated by the problem solver
testcase_expected_output_path	[str] File path to test-case expected output to be matched with
metadata_file_paths	[list<str>] File paths to Question metadata (Extra files usually used for defining training sets)
submission_code_path	[str] File path to submission source code
testcase_result	[bool] Set to True if test-case output matches test-case expected output. Matching is done line by line
testcase_signal	[int] Exit code of the test-case process
testcase_time	[float] Time is taken by the test-case process in seconds
testcase_memory	[int] Peak memory of the test-case process determined in bytes
data	[str] <Future use>

Result Structure

result	[bool] Assign test-case result. True determines success. False determines a failure
score	[float] Assign test-case score. Normalized between 0 to 1
message	[str] Assign test-case message. This message is visible to the problem solver.

The test case result is determined by setting values for the ResultStructure object.

Result: Set this to True if a minimum cutoff score is achieved, otherwise, False. This decides whether you have cleared this test case or not.
Score: This is a normalized score, where one would fetch the total score, and 0.5 would fetch half the score.
Message: This is a customizable message that can be used to convey a message regarding a test case result, e.g., “Failed because the cutoff was not reached."

Note: Please don’t print anything to STDOUT in the run_custom_checker function.

Custom-Checker for list of 5 prime numbers under 100

import re
# Function to determine if the given integer is a prime numberdef is_prime_number(x):
    if x >= 2:
        for y in range(2, x):
            if not x % y:
                return False
    else:
        return False
    return True

def run_custom_checker(t_obj, r_obj):
    result_data = ''
    try:
        result_data = open(t_obj.testcase_output_path, 'r').read()
    except IOError:
        r_obj.result = False
        r_obj.score = 0
        r_obj.message = 'Error reading result file'
        return

    # Read contents of the result file
    values = re.split(' |\n', result_data)

    # Make sure all the values are unique
    uniq_values = set(values)

    # Count the number of primes
    correct_values = 0
    for value in uniq_values:
        if value and is_prime_number(int(value)):
                correct_values = correct_values + 1

    # Cutoff score to determine success
    if correct_values > 3:
        r_obj.result = True
    else:
        r_obj.result = False
    r_obj.score = correct_values / 5
    r_obj.message = str(correct_values) + ' out of 5 values are correct'

Result

Here is a similar custom checker in Java -https://gist.github.com/patilarpith/7b17b5afb9218f770ebc528747017d25

Scoring An Approximate Solution Question

Approximate Solution questions are automatically scored based on a custom checker. They are scored like coding questions without a custom checker, using test cases to determine the final score. The question setter defines individual test cases with input-output pairs for coding questions. Each test case has an assigned score, and the total possible score is the sum of all test case scores. When a custom checker is used, it runs once for each test case. During the test, the candidates' code is tested with each test case input. The custom checker evaluates and scores each test case; the candidate's final score is the sum of these scores.

Related Article:

📄 Creating an Approximate Solution Question