The release date for the 2010 version of AMazeBot is quickly approaching; I expect that in 2 to 3 weeks the website will be updated and the devkits available for download. We who are developing it are quite excited, and we’ve heard that a number of students are also eager to get started on their new bots. I can’t yet announce any new features or changes we’ve made in the ruleset, but I can reveal some improvements to the batch testing system.
When I first wrote the Batch Module for 2008 I didn’t have much time to polish it. As a result, although it was functional it was somewhat unfriendly. It’s an important tool for the students however, perhaps used more often than the Developer Module. I had always intended to make a number of improvements, but failed to do so for 2009. Now for the upcoming 2010 competition I have finally given it the attention it deserves. Here are a few things that I’ve changed.
Cleaner Arguments
One nuisance with the old batch mode was the format of the command-line arguments. All the options were order dependent, so specifying one meant you had to first specify all the previous ones, even if you wanted to leave those other options at their defaults. The new format is much nicer; here is the usage information given by the program itself:
Usage: java -jar AMazeBot2010.jar batch <botname> [options] Where options can be any combination of: -i <iterations> Default 50,000. -s <sequence> Default is random. -m <maze type> One of GR, NS, RG, PR. Default is Automatic. -d Produce detailed action counts in csv output. Note that the botname is case-sensitive; options are not.
Any items in angle brackets are required, and those in square brackets are optional.
Action Counts
In the early AMazeBots the GUI showed the number of looks, moves, and turns that the bot performed. This was deemed not particularly useful for students and removed in the 2008 rewrite. However we (the Professor and I) realized that we could actually make use of that information for planning purposes.
Mostly we wanted to know how often students were using the lookFarAhead function. This is an advanced function that allows the bot to determine the number of open cells in the direction it’s facing, as opposed to the regular look function which only indicates whether a single adjacent cell is open. If the cost for this function is too high then it’s never worthwhile to use it, so knowing the frequency of use could help us in adjusting its cost appropriately. Thus I’ve added the counts of all the various bot actions to the batch log. Perhaps in this easier-to-access format students may come to find it of some use after all.
By default all the actions of a particular type are combined—for example all the successful moves are summed together, regardless of whether they were forward or backward. Likewise for blocked moves, looks and turns. However in some rare cases it may be necessary to know the counts for each individual action; perhaps you suspect your algorithm is behaving oddly and doesn’t like to turn left, as an example. For discovering things like this the “detailed action count” option is provided.
CSV Output
The old module printed its output to the console in a human-readable table, like this:
Batch testing Bot "MotionTrout (alias "MotionTrout") On 10 mazes from sequence 605023665 of type (Automatic) ========================================================== Maze Best Final Number Maze Type Score Result Score Perf ---------- ---------------- ----- --------- ----- ---- 1701561105 Pure Random 2165 FOUNDGOAL 9615 4.4 331234850 Recursive Growth 2090 FOUNDGOAL 7410 3.5 1964692999 Nested Squares 2050 FOUNDGOAL 12870 6.3
This output could be redirected to a file for later inspection, and could also be imported into a spreadsheet for more elaborate analysis. The problem is that the fixed-width columns can’t be automatically parsed by the spreadsheet, requiring manual setup. Since this raw data is rarely inspected directly by humans, the new module produces output in CSV (comma separated values) format and furthermore always saves the output to a file. This makes it very easy to load the data into a spreadsheet, and a separate text file containing the summary only is produced for humans.
Interruptible Batches
Although previously a batch run could be interrupted with the usual Ctrl-C key combo, it would just end abruptly. No summary would be given of the mazes run so far, although if you were redirecting output to a file you could at least get the raw data up to that point. If one was doing this redirection, the program would periodically print a single-line progress report to stderr (which would still appear on the console when stdout went to a file). This showed the number of mazes run (both absolute and as a percentage of iterations requested), elapsed time and estimated time remaining:
Completed: 10 (3%) Elapsed: 1 s ETR: 56 s
Completed: 20 (6%) Elapsed: 4 s ETR: 56 s
Completed: 30 (10%) Elapsed: 5 s ETR: 47 s
Completed: 40 (13%) Elapsed: 7 s ETR: 48 s
Redirection is no longer necessary since a log file is always saved. However the progress indicator still remains, and I realized that it would be better if this included the average score and performance achieved so far. Also it would be easier to read if it remained on a single line rather than continually printing new lines. Both of these changes have been implemented, and additionally when interrupting a batch a summary is produced for the mazes run so far.
Previously because of the poor support for batch interruption the default number of iterations was set at a very low 10. Now the default is a huge 50,000 (since common spreadsheet programs are limited to 65K rows). You can watch the averages change as the batch proceeds, and whenever you’re satisfied that the numbers have settled down you can simply stop the batch and get the full data and summary up to that point. No need to specify the number of iterations ahead of time.
Conclusion & Future
I think the Batch Module is quite nicely polished now, being more functional and friendlier. I hope the students find it makes the task of performance-tuning their bots easier.
I will mention one feature that I had in mind but did not implement this time: multi-core support. It isn’t of great importance, both because batch testing isn’t a huge time burden and because the average computer has only 2.2 cores. But it’s always nice to have things run faster, and I may eventually use this as a reason to properly learn the Java concurrency package.
