moodle-coderunner — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited moodle-coderunner (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Generate Moodle CodeRunner questions. Two modes depending on your environment.
Before doing anything else, determine your mode:
Run this command:
echo "jobe-check"FALLBACK MODE (no bash access)
>
When you cannot run bash commands (Claude.ai, ChatGPT, API without tools): 1. Generate the same JSON structure as Section 6, but you MUST include"expected"in each test case by carefully tracing the solution through the test code. 2. Trace character-by-character. Pay special attention to: trailing spaces, newlines, floating-point precision, array formatting. 3. Apply all per-language rules from Section 2 and auto-fix rules from Section 8 manually. 4. Wrap in XML using the template from Section 7. 5. Add this warning to the user: "These questions were generated WITHOUT Jobe server validation. Before using them, import into Moodle and ensureValidate on saveis enabled (it is by default). Moodle will run the solution against all test cases on import and flag any mismatches." 6. All other sections (question design, type rules, XML template, checklist) still apply.
Full Pipeline mode is strongly preferred. It eliminates the #1 source of errors (hallucinated expected output, which causes 35% of all failures).
Choose the type FIRST -- it determines everything else.
| Type | Student Writes | Test Mechanism | Stdin? | Reliability |
|---|---|---|---|---|
java_class | A class | Test code calls methods | No | HIGH |
java_method | Static method(s) ONLY | Test code calls methods | No | HIGH |
java_program | Full program with main | Empty test, stdin input | Yes | LOW -- EOF issues |
python3 | Function(s) or class | Test code calls with print() | No | HIGH |
python3_w_input | Full program | Empty test, stdin input | Yes | MEDIUM -- EOF issues |
c_function | Function(s) + headers | Test code has main() | No | HIGH |
c_program | Full program with main | Empty test, stdin input | Yes | MEDIUM |
cpp_function | Function(s) + headers | Test code has main() | No | MEDIUM -- Werror |
cpp_program | Full program with main | Empty test, stdin input | Yes | MEDIUM -- Werror |
nodejs | Function(s) | Test code uses console.log() | No | HIGH |
Default choice: Prefer java_class > java_method > java_program. Prefer function types over program types -- they avoid stdin/EOF problems entirely.
These rules are derived from 1000+ validated questions. Every rule exists because questions failed without it.
| Rule | Applies To | What To Do |
|---|---|---|
| Scanner EOF guard | java_program | ALWAYS call hasNextLine()/hasNextInt() before EVERY Scanner read. #1 Java failure cause. |
| Class name | java_program | Class MUST be named Answer |
| Method-only solution | java_method | Solution MUST be ONLY the method(s) -- NO class wrapper, NO main(), NO import statements. The buildSource step wraps it in public class Answer { <methods> main() { testcode } }. Including public class or main() causes compilation errors. |
| Class qualifier in tests | java_class | Test code runs in a separate __Tester__ class. Call static methods as ClassName.method(). Create objects as ClassName obj = new ClassName(); obj.method(). |
| Rule | Applies To | What To Do |
|---|---|---|
| EOF guard | python3_w_input | Use import sys; data = sys.stdin.read().strip() with if data: guard. Never bare input(). |
| Function-only solution | python3 | Solution is function/class definitions only. Test code calls them with print(). |
| Rule | Applies To | What To Do |
|---|---|---|
-Werror | ALL C types | Jobe compiles with -Werror. ALL warnings are fatal. No unused variables, no implicit declarations. |
#include in answer | c_function | ALL headers (<stdio.h>, <ctype.h>, <string.h>, <math.h>, <stdlib.h>) MUST be in the solution, not just test code. |
scanf return check | c_program | Always: if (scanf("%d", &n) == 1) { ... }. Unchecked scanf on empty stdin = uninitialized variable. |
| Test code wrapper | c_function | Test code MUST include #include <stdio.h> and int main(void) { ... return 0; } |
| Array printing | ALL C types | Use if (i) printf(" "); printf("%d", arr[i]); -- NOT printf("%d ", arr[i]) (trailing space fails). |
| Function-only solution | c_function | Solution is function(s) with #include headers only. NO main(). Test code provides main(). |
| Rule | Applies To | What To Do |
|---|---|---|
-Werror | ALL C++ types | Same as C -- all warnings fatal. |
size_t for .size() | ALL C++ types | for (size_t i = 0; i < vec.size(); i++) -- int vs size_t is -Werror=sign-compare fatal. |
| Member init order | cpp_function | Declare class members in SAME order as constructor initializer list. -Werror=reorder is fatal. |
#include in answer | cpp_function | Include <vector>, <string>, <algorithm>, <sstream> etc. in the solution, not just test code. |
| Test code wrapper | cpp_function | Test code MUST include #include <iostream>, using namespace std;, and int main() { ... } |
| Array printing | ALL C++ types | if (i) cout << " "; cout << v[i]; -- no trailing space. |
| Rule | Applies To | What To Do |
|---|---|---|
| Deterministic output | nodejs | Use JSON.stringify(result) for objects/arrays. Do NOT use JSON.stringify(obj, replacer). |
| No npm packages | nodejs | Only Node.js built-in features. No require() of external modules. |
| Server | URL | Notes |
|---|---|---|
| Primary | https://jobe2.cosc.canterbury.ac.nz | Canterbury University public Jobe. No API key needed. |
| Backup | http://3.252.250.94 | EC2 backup with CORS proxy. |
Try the primary server first. If it fails (timeout or HTTP error), fall back to the backup.
| Question Types | Jobe language_id |
|---|---|
python3, python3_w_input | python3 |
java_class, java_method, java_program | java |
c_function, c_program | c |
cpp_function, cpp_program | cpp |
nodejs | nodejs |
Use this exact curl pattern for every Jobe call:
curl -s -w "\n%{http_code}" \
https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
-H "Content-Type: application/json" \
-d '{
"run_spec": {
"language_id": "LANGUAGE_ID",
"sourcecode": "FULL_SOURCE_CODE",
"input": "STDIN_OR_EMPTY",
"sourcefilename": "FILENAME",
"parameters": {"cputime": 10, "memorylimit": 256000}
}
}'Response format:
{"outcome": 15, "cmpinfo": "", "stdout": "42\n", "stderr": ""}Outcome codes:
15 = success. Use stdout as the expected output.11 = compilation error. Read cmpinfo for the error.12 = runtime error. Read stderr for the error.13 = time limit exceeded.17 = memory limit exceeded.Important: Stdin MUST end with \n. If the input does not end with a newline, append one before sending.
Important: Expected output: strip the trailing newline from stdout before using it as expected output. CodeRunner strips trailing newlines before comparing, so the <expected> tag should NOT include a trailing newline.
The sourcecode and input fields are inside a JSON string inside a bash command. You MUST properly escape:
\ becomes \\" becomes \"\n\tRecommended pattern: Write the JSON payload to a temp file, then curl with --data @file:
# Write payload to temp file (avoids all escaping issues)
cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
"run_spec": {
"language_id": "python3",
"sourcecode": "def add(a, b):\n return a + b\n\nprint(add(2, 3))",
"input": "",
"sourcefilename": "test.py",
"parameters": {"cputime": 10, "memorylimit": 256000}
}
}
PAYLOAD
curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
-H "Content-Type: application/json" \
--data @/tmp/jobe_payload.jsonFor complex source code (multi-line, quotes, special chars), ALWAYS use the temp file approach.
Before calling Jobe, assemble the full source code from the solution and test code. The assembly rules depend on the question type.
#### java_method Wrap the solution methods inside a class, add main() with the test code:
public class Answer {
SOLUTION_METHODS
public static void main(String[] args) throws Exception {
TESTCODE
}
}Answer.java#### java_class Strip public from the student's class declaration, append a tester class:
class ClassName {
// solution with 'public class' changed to 'class'
}
public class __Tester__ {
public static void main(String[] args) throws Exception {
TESTCODE
}
}__Tester__.javaClassName.method() or ClassName obj = new ClassName();#### java_program Use the solution as-is. Extract the public class name for the filename.
{ClassName}.java (extracted from public class X, default Answer)#### python3 Concatenate solution and test code:
SOLUTION
TESTCODEtest.py#### python3_w_input Use solution as-is, pass stdin separately.
test.py#### c_function / cpp_function Concatenate solution and test code:
SOLUTION
TESTCODEtest.c or test.cpp#include directives and int main().#### c_program / cpp_program Use solution as-is, pass stdin separately.
test.c or test.cpp#### nodejs Concatenate solution and test code:
SOLUTION
TESTCODEtest.jsEvery question MUST include:
Micro-Assessment - Descriptive Title (unique, not "Question 1")<p><strong>Penalty Regime:</strong></p>
<ul>
<li>You have a total of <strong>3 attempts</strong> without any penalties.</li>
<li>Starting from the <strong>4th attempt onwards</strong>, a penalty of <strong>10%</strong> will be applied.</li>
</ul>useasexample="1", display SHOW)useasexample="0", display HIDE)allornothing="0"): marks MUST sum to 1.0This workflow assumes you passed the mode detection check and can run bash commands. If you cannot, follow Fallback Mode described at the top.
Ask the user for:
For each question, generate a JSON object with this structure:
{
"name": "Micro-Assessment - Descriptive Title",
"qtype": "java_class",
"question_text": "<h4>Title</h4><p>Instructions with examples...</p><p><strong>Penalty Regime:</strong></p><ul><li>You have a total of <strong>3 attempts</strong> without any penalties.</li><li>Starting from the <strong>4th attempt onwards</strong>, a penalty of <strong>10%</strong> will be applied.</li></ul>",
"feedback": "<p>Explanation of the correct approach.</p>",
"solution": "complete model solution code",
"preload": "skeleton code for students",
"test_cases": [
{"testcode": "System.out.println(obj.method(arg));", "stdin": "", "visible": true},
{"testcode": "System.out.println(obj.method(arg2));", "stdin": "", "visible": false},
{"testcode": "System.out.println(obj.method(edge));", "stdin": "", "visible": false}
]
}Do NOT include `expected` in test_cases. Jobe will compute it.
For program types (java_program, python3_w_input, c_program, cpp_program):
testcode is empty string ""stdin contains the input dataFor each question, for each test case:
15: strip the trailing newline from stdout and save it as the expected output for this test case15: the test case failed -- see Step 4Example -- validating a python3 question:
# Question: def add(a, b): return a + b
# Test case: print(add(2, 3))
# Built source: "def add(a, b):\n return a + b\n\nprint(add(2, 3))"
cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
"run_spec": {
"language_id": "python3",
"sourcecode": "def add(a, b):\n return a + b\n\nprint(add(2, 3))",
"input": "",
"sourcefilename": "test.py",
"parameters": {"cputime": 10, "memorylimit": 256000}
}
}
PAYLOAD
curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
-H "Content-Type: application/json" \
--data @/tmp/jobe_payload.jsonResponse: {"outcome":15,"cmpinfo":"","stdout":"5\n","stderr":""}
Expected output for this test case: 5 (stdout with trailing newline stripped).
Example -- validating a java_class question:
# Solution: public class Calculator { public int add(int a, int b) { return a + b; } }
# Test case: Calculator c = new Calculator(); System.out.println(c.add(2, 3));
# Built source (java_class assembly):
# class Calculator { public int add(int a, int b) { return a + b; } }
# public class __Tester__ {
# public static void main(String[] args) throws Exception {
# Calculator c = new Calculator(); System.out.println(c.add(2, 3));
# }
# }
cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
"run_spec": {
"language_id": "java",
"sourcecode": "class Calculator { public int add(int a, int b) { return a + b; } }\n\npublic class __Tester__ {\n public static void main(String[] args) throws Exception {\n Calculator c = new Calculator(); System.out.println(c.add(2, 3));\n }\n}",
"input": "",
"sourcefilename": "__Tester__.java",
"parameters": {"cputime": 10, "memorylimit": 256000}
}
}
PAYLOAD
curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
-H "Content-Type: application/json" \
--data @/tmp/jobe_payload.jsonIf Jobe returns a non-15 outcome for any test case:
cmpinfo (compile errors) or stderr (runtime errors)If a question still fails after 3 retries, report it to the user and move on.
| Error | Fix |
|---|---|
implicit declaration | Add missing #include to solution |
sign-compare | Change int i to size_t i in .size() loops |
reorder | Reorder class members to match constructor initializer list |
illegal start of expression (java_method) | Solution must be raw method(s) only -- remove class wrapper, main(), imports |
| Trailing space in output | Use if(i) printf(" ") pattern for array printing |
NoSuchElementException | Add hasNext() guard before Scanner read |
EOFError | Use sys.stdin.read() instead of input() |
cannot find symbol (java_class) | Test code must use class name qualifier: ClassName.method() |
non-static (java_method) | Add static keyword to method |
control reaches end of non-void | Add return statement at end of function |
reached end of file / expected } | Count { and } -- add missing closing brace |
unnamed classes are a preview | Wrap code in public class Answer { } |
ArrayIndexOutOfBounds | Add empty array check: if (arr.length == 0) return default; |
Once ALL test cases for ALL questions have passed Jobe validation, wrap everything in the Moodle XML template from Section 7.
For each test case, the <expected> value is the stdout captured from Jobe (with trailing newline stripped).
{module}_{topic}_coderunner.xml<?xml version="1.0" encoding="UTF-8"?>
<quiz>
<question type="category">
<category><text>$course$/Category Name</text></category>
</question>
<!-- Repeat this block for each question -->
<question type="coderunner">
<name><text>QUESTION_NAME</text></name>
<questiontext format="html"><text><![CDATA[QUESTION_HTML]]></text></questiontext>
<generalfeedback format="html"><text><![CDATA[FEEDBACK_HTML]]></text></generalfeedback>
<defaultgrade>1</defaultgrade>
<penalty>0</penalty>
<hidden>0</hidden>
<idnumber></idnumber>
<coderunnertype>QTYPE</coderunnertype>
<prototypetype>0</prototypetype>
<allornothing>1</allornothing>
<penaltyregime>0, 0, 0, 10, 20, ...</penaltyregime>
<precheck>0</precheck>
<hidecheck>0</hidecheck>
<showsource>0</showsource>
<answerboxlines>18</answerboxlines>
<answerboxcolumns>100</answerboxcolumns>
<answerpreload><![CDATA[PRELOAD_CODE]]></answerpreload>
<globalextra></globalextra>
<useace></useace>
<resultcolumns></resultcolumns>
<template></template>
<iscombinatortemplate></iscombinatortemplate>
<allowmultiplestdins></allowmultiplestdins>
<answer><![CDATA[MODEL_SOLUTION]]></answer>
<validateonsave>1</validateonsave>
<testsplitterre></testsplitterre>
<language></language>
<acelang></acelang>
<sandbox></sandbox>
<grader></grader>
<cputimelimitsecs></cputimelimitsecs>
<memlimitmb></memlimitmb>
<sandboxparams></sandboxparams>
<templateparams></templateparams>
<hoisttemplateparams>1</hoisttemplateparams>
<extractcodefromjson>1</extractcodefromjson>
<templateparamslang>None</templateparamslang>
<templateparamsevalpertry>0</templateparamsevalpertry>
<templateparamsevald>{}</templateparamsevald>
<twigall>0</twigall>
<uiplugin></uiplugin>
<uiparameters><![CDATA[{"live_autocompletion": true}]]></uiparameters>
<attachments>0</attachments>
<attachmentsrequired>0</attachmentsrequired>
<maxfilesize>10240</maxfilesize>
<filenamesregex></filenamesregex>
<filenamesexplain></filenamesexplain>
<displayfeedback>1</displayfeedback>
<giveupallowed>0</giveupallowed>
<prototypeextra></prototypeextra>
<testcases>
<!-- Repeat for each test case -->
<testcase testtype="0" useasexample="USE_AS_EXAMPLE" hiderestiffail="0" mark="MARK_VALUE">
<testcode><text>TESTCODE</text></testcode>
<stdin><text>STDIN</text></stdin>
<expected><text>EXPECTED_FROM_JOBE</text></expected>
<extra><text></text></extra>
<display><text>DISPLAY_VALUE</text></display>
</testcase>
</testcases>
</question>
</quiz>| Field | Visible test (first) | Hidden test (remaining) |
|---|---|---|
useasexample | 1 | 0 |
display | SHOW | HIDE |
mark | 1.0000000 (all-or-nothing) | 1.0000000 (all-or-nothing) |
For partial credit: distribute marks so they sum to 1.0 (e.g., 4 tests = 0.2500000 each).
| Feature | Tag | Values | Effect |
|---|---|---|---|
| Precheck | <precheck>1</precheck> | 0/1 | Students can test before submitting |
| Give up | <giveupallowed>1</giveupallowed> | 0/1 | Students can reveal model answer |
| CPU limit | <cputimelimitsecs>5</cputimelimitsecs> | seconds | Enforce time limit |
| Memory limit | <memlimitmb>64</memlimitmb> | MB | Enforce memory limit |
Before sending code to Jobe, apply these fixes to the solution and test code. These catch the most common AI generation mistakes.
| # | Fix | Applies To | What To Do |
|---|---|---|---|
| 1 | Strip class/main/imports | java_method | Remove any public class wrapper, main() method, and import statements. java_method solutions must be bare static methods only. |
| 2 | Wrap in public class | java_class | If solution has no class keyword, wrap in public class Answer { ... } |
| 3 | Rename class to Answer | java_program | If solution has a class not named Answer, rename it |
| 4 | Add Scanner hasNext guards | java_program | Wrap scanner.nextInt() with scanner.hasNextInt() ? scanner.nextInt() : 0 and scanner.nextLine() with scanner.hasNextLine() ? scanner.nextLine() : "" |
| 5 | Remove accidental main() | c_function, cpp_function | Strip int main(...) block from function-type solutions |
| 6 | Add missing main() | c_program, cpp_program | Append int main(void) { return 0; } if no main() exists |
| 7 | Auto-close braces | java_*, c_*, cpp_*, nodejs | Count { vs } and append missing closing braces |
| 8 | Wrap input() in try/except | python3_w_input | If solution uses bare input() without sys.stdin or try, wrap in try: ... except EOFError: pass |
| 9 | Add missing C headers | c_function, c_program | Detect function usage and add missing #include: stdio.h, string.h, stdlib.h, math.h, ctype.h, limits.h |
| 10 | Add missing C++ headers | cpp_function, cpp_program | Detect usage and add: iostream, vector, string, algorithm, map, set, stack, queue, sstream |
| 11 | Fix int to size_t | cpp_function, cpp_program | Rewrite for (int i = 0; i < v.size() to for (size_t i = 0; i < v.size() |
| 12 | Cast to size_t | cpp_function, cpp_program | Rewrite if (x < v.size() to if ((size_t)x < v.size() |
| # | Fix | Applies To | What To Do |
|---|---|---|---|
| 1 | Wrap C test code | c_function | If test code lacks int main, wrap with #include <stdio.h> + int main(void) { ... return 0; } |
| 2 | Wrap C++ test code | cpp_function | If test code lacks int main, wrap with #include <iostream> + using namespace std; + int main() { ... } |
| 3 | Fix unmatched parens | All types | If closing parens exceed opening parens, trim trailing ) |
Apply these fixes programmatically before building the source and calling Jobe.
Here is a complete end-to-end example generating one Python question.
{
"name": "Micro-Assessment - Sum of Even Numbers",
"qtype": "python3",
"question_text": "<h4>Sum of Even Numbers</h4><p>Write a function <code>sum_evens(nums)</code> that takes a list of integers and returns the sum of all even numbers in the list.</p><h5>Example</h5><pre>sum_evens([1, 2, 3, 4, 5, 6]) -> 12</pre><p><strong>Penalty Regime:</strong></p><ul><li>You have a total of <strong>3 attempts</strong> without any penalties.</li><li>Starting from the <strong>4th attempt onwards</strong>, a penalty of <strong>10%</strong> will be applied.</li></ul>",
"feedback": "<p>Use a list comprehension or loop to filter even numbers (n % 2 == 0) and sum them. The built-in <code>sum()</code> function works well with a generator expression.</p>",
"solution": "def sum_evens(nums):\n return sum(n for n in nums if n % 2 == 0)",
"preload": "def sum_evens(nums):\n # Write your code here\n pass",
"test_cases": [
{"testcode": "print(sum_evens([1, 2, 3, 4, 5, 6]))", "stdin": "", "visible": true},
{"testcode": "print(sum_evens([]))", "stdin": "", "visible": false},
{"testcode": "print(sum_evens([1, 3, 5]))", "stdin": "", "visible": false},
{"testcode": "print(sum_evens([-2, -1, 0, 1, 2]))", "stdin": "", "visible": false}
]
}Build source (python3 = solution + "\n\n" + testcode):
def sum_evens(nums):
return sum(n for n in nums if n % 2 == 0)
print(sum_evens([1, 2, 3, 4, 5, 6]))cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
"run_spec": {
"language_id": "python3",
"sourcecode": "def sum_evens(nums):\n return sum(n for n in nums if n % 2 == 0)\n\nprint(sum_evens([1, 2, 3, 4, 5, 6]))",
"input": "",
"sourcefilename": "test.py",
"parameters": {"cputime": 10, "memorylimit": 256000}
}
}
PAYLOAD
curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
-H "Content-Type: application/json" \
--data @/tmp/jobe_payload.jsonResult: {"outcome":15,"stdout":"12\n",...} -- expected output: 12
sum_evens([]) -- stdout 0\n -- expected: 0sum_evens([1, 3, 5]) -- stdout 0\n -- expected: 0sum_evens([-2, -1, 0, 1, 2]) -- stdout 0\n -- expected: 0Fill in the XML template from Section 7:
QUESTION_NAME = Micro-Assessment - Sum of Even NumbersQTYPE = python3QUESTION_HTML = the question_text HTMLFEEDBACK_HTML = the feedback HTMLMODEL_SOLUTION = the solution codePRELOAD_CODE = the preload codeTESTCODE, STDIN, EXPECTED_FROM_JOBE, USE_AS_EXAMPLE, DISPLAY_VALUEBefore delivery, every question must satisfy:
<name> tag uses descriptive title starting with Micro-Assessment -<coderunnertype> for the task<answer> contains complete, compilable model solution<answer> for java_method is raw method(s) only (no class wrapper)<answerpreload> provides compilable skeleton<generalfeedback> explains correct approach<validateonsave>1</validateonsave><penaltyregime>0, 0, 0, 10, 20, ...</penaltyregime><questiontext>Validate on save enabled{module}_{topic}_coderunner.xmloop1_polymorphism_coderunner.xml.xml file~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.