CSTester API¶
- class CSTester(argv: List[str] | None = None)[source]¶
Bases:
objectMain class to drive testing Computer Science submissions
- __init__(argv: List[str] | None = None) None[source]¶
Initialize the CSTester instance
- Parameters:
argv (Optional[List[str]]) – Optional command-line arguments
If arguments are given, we expect argv[1] to be a configuration file
- scr¶
The active
CursesScreenobject
- name¶
The actual name of this class
- cfg¶
An instance of
ConfigManager, this class provides an abstraction ofCSTesterConfig
- refreshallgroups(title: str, extracted: bool | None = False) None[source]¶
Refresh the group numrange strings for all groups in the phase directory
- getgrouplists() Tuple[List[str], List[str], List[str]][source]¶
Return included, excluded, and all group lists (expanded)
- getfilteredgroups() List[str][source]¶
Return the filtered list of groups after applying include/exclude rules
- Returns:
Filtered group identifiers as strings
- Return type:
List[str]
- makeclean() None[source]¶
Remove files matching the configured clean patterns within group directories
- help_runtests(title: str) None[source]¶
Display an informational window describing the runtests view
- runtests() None[source]¶
Run tests for each group using the configured test command
Failures are collected and reported
- Navigation and control commands available:
Previous group: a A b B
Next group: d D n N
Continue in direction: <SPACE>
Quit: <ESC>
- runsearch() None[source]¶
Search for configured string patterns within files (visual grep)
Uses
searchstrsandsearchfilesfromcfgto select files and patterns; files are scanned line-by-line as text for search patterns
- runpreparation() None[source]¶
Run a sequence of preparation commands in each group directory
Commands run in order, on failure subsequent commands are skipped
Failures are collected and reported
- help_lessreadmes(title: str) None[source]¶
Display an informational window describing the lessreadmes view
- lessreadmes() None[source]¶
Display README files from each group directory
- Navigation and control Commands available:
Previous group: a A b B
Next group: d D n N
Continue in direction: <SPACE>
Quit: <ESC>
- freezefiles() None[source]¶
Create a frozen snapshot of matching files
Matching files are archived (zip) and their hashes stored, performed before modifications can allow patch generation
- mkpatches() None[source]¶
Detect modification of frozen files and generate diff patches
Patches can be applied with the
patchcommand or distributed to show changes, e.g., fixes required for compilation
Display an informational window describing the phasemenu view
Menu to configure and setup the phase directory from a phase zip
The phase zip contains group submission zips
- Options:
keyfile: optional mapping to associate zip-filename parts with groups group regex: required pattern to extract group number zip include/exclude patterns with target locations
Display an informational window describing groupmenu’s view
Menu for managing group directories: include/exclude groups, freeze files, create patches, and clean files.
Include/exclude lists use comma-separated numbers/ranges, e.g., 1,4-9
Display an informational window describing the evalmenu view
Menu for running evaluation tasks: run tests, prepare, clean, search, and view READMEs
- help_configure(title: str) None[source]¶
Display an informational window describing the configure view
- configure() None[source]¶
Configuration menu to display and modify configuration values
Accepted value types include strings, directory or file paths, lists, and dicts.
Validation may include regex compilation, number/range parsing, and command verification.
Display an informational window describing the mainmenu view
Main menu entry point that links to phase, groups, evaluation, and configuration submenues. Saving and Loading from YAML is supported
- class CSTesterConfig[source]¶
Bases:
objectA dataclass which holds the actual configuration values for
CSTesterThis class collects configuration options which can be saved/loaded
- zipinclude_strs: Dict[str, str]¶
Regex patterns of files to put in specific locations from submitted zips
- expdir: str = ''¶
Path containing expected output files The basenames of expected output/input test cases should match
- testcmd: str = ''¶
Command to test their program, shlex.split(testcmd)[0] is their program input test cases are either: passed via argument with @case@ or, if @case@ is not present in testcmd, the case is piped to stdin
- prepcmds: List[str]¶
A list of commands to run, in order, from each group directory (if a command fails, later commands aren’t ran) this can be used to simulate a make command
- readmename: str = 'writeup.txt'¶
A document name that should be within each submission this filename, these files can be quickly read for all groups
- to_yaml() str[source]¶
- Returns:
configuration
- Return type:
A YAML string which can be written to a file
yaml.dump(asdict(self))
- classmethod from_yaml(filename: str) CSTesterConfig[source]¶
- Parameters:
filename (str) – Path to a YAML file
- Returns:
new instance
- Return type:
- class ConfigManager(scr: CursesScreen, configfile: str | None = '')[source]¶
Bases:
objectUtility class providing a managed interface to the
CSTesterConfigdataclass- __init__(scr: CursesScreen, configfile: str | None = '') None[source]¶
- Parameters:
scr (CursesScreen) – The running curses screen which provides window functions to display windows and accept input
configfile (str, Optional) – Path to a saved YAML configuration file
- scr¶
The active
CursesScreenobject
- conf¶
Our
CSTesterConfig, the dataclass holding configuration
- prompts¶
Dictionary of prompts to use in
modifyconf()
- resetattr(attr: str) None[source]¶
Reset
conf.``attr`` to its default value- Parameters:
attr (str) – Attribute name to reset
- get(attr: str) str | List[str] | Dict[str, str][source]¶
Get
conf.``attr``- Parameters:
attr (str) – Attribute name to retrieve
- Returns:
type str, list of str, or dict[str, str]:
conf.``attr``- Raises:
AttributeError – if
conf.``attr`` doesn’t exist
- set(attr: str, val: str | List[str] | Dict[str, str]) None[source]¶
Set
conf.``attr`` toval- Parameters:
attr (str) – Attribute name to set
val – type str, list of str, or dict[str, str]: Value to assign
- __getattr__(attr: str) str | List[str] | Dict[str, str][source]¶
Forward member requests from the dot operator to
conf
- load(filename: str = '') None[source]¶
Create an instance of
CSTesterConfigfrom a YAML configuration file- Parameters:
filename (str) – Path to YAML file
Sets
conf
- verifiedregex(val: str | None | Tuple[str, str | None]) str | None | Tuple[str, str | None][source]¶
Validate
valusingre.compile()- Parameters:
val – type Optional[str], or Tuple[str, Optional[str]]: Input value
- Returns:
Verified regex string or None
- Return type:
Optional[str]
- verifiednumrange(val: str | None) str | None[source]¶
Validate that
valis a valid number/range list
- verifyval(key: str, val: str | None | Tuple[str, str | None]) str | None | Tuple[str, str | None][source]¶
Dispatch verification for a field based on its key
- Parameters:
key (str) – Configuration key
val – type Optional[str], or Tuple[str, Optional[str]]: Value to verify
- Returns:
type Optional[str], or Tuple[str, Optional[str]]: Verified value
- __add__(keyval: Tuple[str, str | None | Tuple[str, str | None]]) ConfigManager[source]¶
Add a key/value pair to the configuration using the + operator
- Parameters:
keyval – type Tuple[str, [Optional[str], or Tuple[str, Optional[str]]]]: Tuple or mapping with key and value
- Returns:
self (after modification)
- Return type:
- modifylist(key: str, title: str) None[source]¶
Prompt the user to obtain and set a list value for a key
- modifydict(key: str, title: str) None[source]¶
Prompt the user to obtain and set a dict value for a key
- __annotate_func__ = None¶
- __annotations_cache__ = {}¶
- class WinOpt(*values)[source]¶
Bases:
FlagFlags for drawing windows can be combined with
|(OR)Flag values are listed in order of precedence
- NOSCROLL = 1¶
Disable vertical scrolling – only prints (optionally) title, errorlines, bodylines, footerline – returns immediately, as with
RETURNIMM–NOSCROLLignores all options besides theTAILoptions
- TAILBOTH = 2¶
When not scrollable, split real-estate between error and body (by default, body may be hidden if error lines fill the screen)
- TAILBODY = 4¶
When not scrollable prefer the body (error lines may then be hidden)
- RETURNIMM = 8¶
Return None immediately (without input)
- RETURNANY = 16¶
Return the any key
- RETURNKEY = 32¶
Return on a provided key or
CursesScreen.returnkeys
- RETURNMUL = 64¶
Return a list of choices from the body – provide a confirmation prompt or use
choices[0]to confirm
- RETURNDEL = 128¶
Include the delete key as a returnkey, returns
str('KEY_DC')
- USEHELP = 256¶
Use
CursesScreen.helpkeysas helpkeys
- TEXTBOX = 512¶
Body is 1 “textbox” block – no highlight and collective scroll
- SHOWCURS = 1024¶
Show the cursor at the end of the footerline (if possible)
- class CursesScreen[source]¶
Bases:
objectCurses helper class to initialize and manage a curses screen
- tabsize = 2¶
when tabs are replaced, replace with this number of spaces
- inputtimeout = 10¶
set inputtimeout = an int (milliseconds); NOTE: input timeout temporarily set to 10ms between inputs to combine rapid chars or paste, you may possibly need to adjust the timeout
- returnkeys = ('KEY_ENTER', '\r', '\n')¶
Default return keys are
curses.KEY_ENTER, ‘\r’, ‘\n’
- cancelkeys = ('\x1b', 'Q', 'q')¶
Default cancel keys are ‘\x1b’ (escape), ‘Q’, ‘q’
- helpkeys = ('?', 'H', 'h')¶
Default help keys are ‘?’, ‘H’, ‘h’
- refcount¶
Reference counter for safe deletion with nested references
- itemcolor¶
Regular item color, green on black
- activecolor¶
Item color with bold
- standoutcolor¶
Item color with standout
- titlecolor¶
Title color, bold white on black
- errorcolor¶
Error color, bold red on black
- disabledcolor¶
Disabled color, bold black on black
- bordercolor¶
Border color (with whitespace), dim black on cyan
- initscr() None[source]¶
Increments
refcountand returns ifscris already initializedOtherwise, initializes
scrand preparescursescurses.windowmay not return the correctmaxyxafter window resize events. This was reported in this issue (yes, opened shortly after GitHub was founded) and is still present. There was a PR made 17 years later to solve the problem, but it hasn’t been merged. A workaround is to set and delete theLINESandCOLUMNSenvironment variables, which is done here, otherwise resize events won’t update values obtained fromcurses.window.getmaxyx()- Setup includes:
curses.cbreak()– immediately respond to keypressescurses.curs_set()– set curs to0to hide the cursorcurses.noecho()– suppress echo of keypressescurses.nonl()– leave newline mode, we handle all keyscurses.raw()– we really do handle all of the keys!curses.set_escdelay()– “remove” escape delaycurses.set_tabsize()– set number of spaces per tabcurses.window.leaveok()– don’t move the cursor on updatescurses.window.keypad()– enable additional curses keys
- clearkeys() None[source]¶
call
curses.flushinp()to clear input buffers
- statuswindow(title: str, status: str, body: List[str] | None = [], err: List[str] | None = []) None[source]¶
Method aimed toward status/progress windows
Prints the title, followed by the error, followed by the body
If the length of the error/body go beyond the screen height, only the last height lines are printed.
The final line printed is the status string
This method takes no input and immediately returns
- getinput(title: str, prompt: str, val: str | None = '') str | None[source]¶
Prompt the user for input using a modal input menu
- window(opts: WinOpt, title: str = '', helpkeys: Tuple[str] | None = (), returnkeys: Tuple[str] | None = (), helpstr: str | None = 'Help: {@keys@}', err: List[str] | None = [], body: List[str] | None = [], choices: List[str] | None = [], footer: str | None = '', disabled: List[int] | None = [], chosen: List[str] | None = [], top: int | None = 0, hpos: int | None = 0) Tuple[int, int, str | List[str]][source]¶
Display a scrollable choice menu and return the user’s selection
- Parameters:
opts (WinOpt) – Menu options
title (str) – Menu title shown on the first line
helpkeys (Optional[Tuple[str]]) – Keys to recognize as help/exit keys
returnkeys (Optional[Tuple[str]]) – Keys to return, default is enter
helpstr (Optional[str]) – Format string for help display
body (Optional[List[str]]) – Lines shown between title and choices
choices (Optional[List[str]]) – List of choice strings
disabled (Optional[List[int]]) – Indices that cannot be selected
chosen (Optional[List[str]]) – Chosen items (for multi-select)
top (int) – Index of the top visible choice
hpos (int) – Index of the currently highlighted choice
- Returns:
- Tuple[int, int, Union[str, List[str]]]
topline (for across-call consistency)
activeline
input triggering return
- This method is used to print a text menu using
scr The title is drawn on the first line
An empty line separates the title from the body
The body is a list of strings
The remaining lines are “choice” lines which can be scrolled
The current selection at hpos will be highlighted
- Lines which can’t be “active” lines and can’t be chosen:
Empty strings
Strings matching an index in disabled
- Returns on either:
A key in
returnkeys(orCursesScreen.returnkeys)A key in
CursesScreen.cancelkeysA key in
helpkeys, if a corresponding help method not called
- getdir(title: str, prompt: str | None = 'Select a directory', path: Path | None = None, allownew: bool | None = True) str | None[source]¶
Show a directory selection menu and return the chosen path
- Parameters:
title (str) – Window title
prompt (str) – Prompt shown to the user
path (Optional[pathlib.Path]) – Starting directory (default is pwd)
allownew (Optional[bool]) – Whether the directory must already exist
- Returns:
Selected directory as a string, or None if cancelled
- Return type:
Optional[str]
- getfile(title: str, prompt: str | None = 'Select a file', path: Path | None = None, perm: Literal[4, 2, 1] | None = 4, filere: str | None = '') str | None[source]¶
Show a file-selection menu and return the chosen filename
- Parameters:
- Returns:
Selected filename as a string, or None if cancelled
- Return type:
Optional[str]
- drawsplitpane(lhs: List[str], lpos: List[int], rhs: List[str], rpos: List[int], highlight: bool, paneshmt: int, ltitle: str, rtitle: str, linenums: bool, helpstr: str | None = '') None[source]¶
The screen is divided vertically into 2 segments
lhs and rhs are lists of strings with titles ltitle and rtitle
- lpos and rpos determines which row/col is the top left of each pane
{l,r}pos[0] = first row, [1] = first col
last row = {l,r}pos[0] + height - 1
- The division is shifted by paneshmt
where 0 is vertical bar at width/2 – neg/pos shifts left/right
- With linenums=True, a line number can be printed to the left of a line
(if {l,r}pos[1] is negative)
The screen is cleared, strings added to screen, then refreshed
- help_diffwindow(title: str) None[source]¶
help_diffwindow displays an infowindow with diffwindow’s docstr
- diffwindow(args: Tuple[str, str, List[str], str, List[str]]) None[source]¶
Diff Window¶
- Commands available while the diff view is active:
Toggle match highlighting: d D
Toggle left/right pane lock: <SPACE>
Toggle left/right pane scrolling: <TAB>
Move pane separator left/right: + -
Reset pane separator shift: =
Toggle line-number printing: n N
Quit: <ESC> q Q
- class Extractor(scr: CursesScreen | None = None)[source]¶
Bases:
objectClass to extract submissions
- __init__(scr: CursesScreen | None = None) None[source]¶
This class may either be given a screen or will make its own
- getgroupfiles(tempdir: str, keyfile: str, groupre: str, steps: List[str]) Tuple[List[str], Dict[str, str]][source]¶
This method will return a tuple which maps groups to zip files
The best way to get a group number from a zip filename is with a regex
In case the filename isn’t correct, we accept an optional keyfile
- Keys are extracted from the keyfile with the regex:
^([0-9]+),.*:([^:]+)$
When a group number can’t be inferred, the user is prompted for input
Decisions are recorded in the extraction log
Steps performed in this method:
Obtain any keys from the optional keyfile
Iterate through files in tempdir
skip (but log) non-zip files
Attempt a group regex match
If regex match fails, attempt to use the keyfile
If unable to automatically determine group, prompt for input
Use of keyfile/prompts recorded in logs
Verify a single submission for each group
Groups with multiple submissions prompt for which to keep
Return the logs and group->file mapping
- extract(tempdir: str, phasedir: str, keyfile: str, groupre: str, include: Dict[str, str], _exclude: List[str], steps: List[str]) List[str][source]¶
This method extracts individual zip files to respective group directories
We get a group->file mapping from
getgroupfiles, then, for each group,The extraction path is
phasedir/group_n
where n is a non-negative integer, i.e.,
[0-9]+Iterate through the zipfile’s member list
Exclude symlinks and files matching an exclude pattern
Refuse to extract zips with an unrealistic compression ratio
Extract selected members to a temporary directory
Set owner-only permissions RWX for directories and RW for files
Ensure directories exist rather than filenames with directory parts
Ensure no single top-level directory (move files up)
Track paths for any file wanted in a specific location
Don’t save duplicate files
Move files into destination group directory
Save extraction log and original zip in group directory
- extractphasezip(phasedir: str, phasezip: str, keyfile: str, groupre: str, include: Dict[str, str], exclude: List[str], title: str = 'Extract Phase Dir') Literal[-1, 0, 1][source]¶
This is the base extraction method for a zip of zips
The extraction log is stored in the output directory,
phasedirIf there was a prior extraction, prior decisions made can be repeated
- Parameters:
phasedir (str) – The output directory for all group files
phasezip (str) – The filename of the zip of zips
keyfile (str, Optional) – A keyfile to aid association of filename->group
groupre (str) – Regex to use to associate filename->group
include – (dict[str, str]): Explicit map of files to destination
exclude – (list[str]): patterns for files to not extract
title – (str): The title for prompts
- Returns:
(status)
- Return type:
Literal[-1, 0, 1]
0 = success
-1 = failure
1 = aborted
utils.py¶
- collapsenumrange(val: str) str | None[source]¶
Order and collapse a number/range string into its optimal form
- expandnumrange(val: str) List[str][source]¶
Expand a number/range string into a list of numbers as strings
- removecommonprefix(left: str, right: str) Tuple[str, str][source]¶
This method is used to the common prefix from 2 files
- Parameters:
- Returns:
left and right stripped of their common prefix
If left and right were the same then the returnvalue will be the basename of each prefixed with ‘a/’ and ‘b/’
- runprocess(cmd: List[str], filename: str | None = '', getout: bool | None = True, geterr: bool | None = True) Tuple[str, str, int][source]¶
Instantiate an instance of
subprocess.Popen, usesubprocess.Popen.communicate()with an optional input, and return its stdout, stderr and return code- Parameters:
- Returns:
(stdout, stderr, return_code)
- Return type:
stdout: Text written to stdout (may be empty)
stderr: Text written to stderr (may be empty)
return_code: Process exit code (0 normal, > 0 error, < 0 signal)
testOutput.py¶
testoutput command
Given a directory with test cases (casedir), run each case through a program and optionally compare output against expected files in expdir
Input handling: - Piped to stdin or
Passed as an argument where
@case@is replaced with the case filename
Behavior: - If there is an expected output file, compare actual output to expected and display differences when they differ
If no expected output exists, display input and actual output
Report cases that end with error or exception
When output matches expected output report success
- Example usage with the program:
./obj/prog (program expects an input filename as argument)
./obj/prog --input=@case@
- dotests(cases: Dict[str, str | None], runstr: str) Dict[str, str][source]¶
Execute tests for each cases in
casesand report results- Parameters:
- Returns:
[Error testcases, error strings]
- Return type:
Runs the program for each test case, capturing output, and
If an expected output file exists, compares outputs and displays diff
If no expected output exists, displays input and output
Tracks non-normal exits to display following the final test case
diffwin.py¶
- class DiffWindow(argv: List[str] | Tuple[str, str, List[str], str, List[str]] | None = None)[source]¶
Bases:
objectA window that displays a side-by-side diff of two files for comparison
- __init__(argv: List[str] | Tuple[str, str, List[str], str, List[str]] | None = None) None[source]¶
Initialize the DiffWindow
- When given arguments
argv=[program, leftfile, rightfile]
argv=(title, ltitle, lhs, rtitle, rhs)
Will setup, call
diffwindow, and cleanup- Parameters:
argv – (Optional[Union[List[str], Diffargs]]): Optional arguments
Display an informational window with the main menu help
Show the diff view and provide navigation/commands for comparison
The view compares lines with leading/trailing whitespace stripped, converts tabs to spaces, highlights matching lines, and can show line numbers.
Open help while in the ‘diff view’ for a command listing