\documentclass[nojss]{jss} \usepackage{thumbpdf} %% need no \usepackage{Sweave} \shortcites{exams:Blesius+Moreno-Ger+Neumann:2007,exams:Agea+Crespo-Garcia+Delgado-Kloos:2009} %% for internal use \newcommand{\fixme}[1]{\emph{\marginpar{FIXME} (#1)}} \newcommand{\readme}[1]{\emph{\marginpar{README} (#1)}} %% new environments for exams \newenvironment{question}{\item \textbf{Problem}\newline}{} \newenvironment{solution}{\textbf{Solution}\newline}{} \author{Achim Zeileis\\Universit\"at Innsbruck \And Nikolaus Umlauf\\Universit\"at Innsbruck\\ \And Friedrich Leisch\\Universit\"at f\"ur\\Bodenkultur Wien} \Plainauthor{Achim Zeileis, Nikolaus Umlauf, Friedrich Leisch} \title{Flexible Generation of E-Learning Exams in \proglang{R}: \pkg{Moodle} Quizzes, \pkg{OLAT} Assessments, and Beyond} \Plaintitle{Flexible Generation of E-Learning Exams in R: Moodle Quizzes, OLAT Assessments, and Beyond} \Shorttitle{Flexible Generation of E-Learning Exams in \proglang{R}} \Keywords{exams, e-learning, multiple choice, arithmetic problems, \code{Sweave}, \proglang{R}, {\LaTeX}, HTML, XML, IMS QTI, \pkg{Moodle}, \pkg{OLAT}} \Plainkeywords{exams, e-learning, multiple choice, arithmetic problems, Sweave, R, LaTeX, HTML, XML, IMS QTI, Moodle, OLAT} \Abstract{ This introduction to version~2 of the \proglang{R}/\pkg{exams} package is a (slightly) modified version of \cite{exams:Zeileis+Umlauf+Leisch:2014}, published in the \emph{Journal of Statistical Software}. Since its publication, the \pkg{exams} package has been extended in numerous ways which are not reflected in this vignette. Examples include: Exercises in \proglang{R}/Markdown format (instead of \proglang{R}/{\LaTeX}), the \code{exams2nops} interface for exams that can be automatically scanned and evaluated, export to \pkg{Canvas} or \pkg{Blackboard} etc. For a current and more general introduction to the \proglang{R}/\pkg{exams} package see the materials at the official web page: \url{https://www.R-exams.org/}. The capabilities of the package \pkg{exams} for automatic generation of (statistical) exams in \proglang{R} are extended by adding support for learning management systems: As in earlier versions of the package exam generation is still based on separate \code{Sweave} files for each exercise -- but rather than just producing different types of PDF output files, the package can now render the \emph{same} exercises into a wide variety of output formats. These include HTML (with various options for displaying mathematical content) and XML specifications for online exams in learning management systems such as \pkg{Moodle} or \pkg{OLAT}. This flexibility is accomplished by a new modular and extensible design of the package that allows for reading all weaved exercises into \proglang{R} and managing associated supplementary files (such as graphics or data files). The manuscript discusses the readily available user interfaces, the design of the underlying infrastructure, and how new functionality can be built on top of the existing tools. } \Address{ Achim Zeileis, Nikolaus Umlauf\\ Universit\"at Innsbruck\\ Department of Statistics\\ Universit\"atsstr.~15, 6020 Innsbruck, Austria\\ E-mail: \email{Achim.Zeileis@R-project.org}, \email{Nikolaus.Umlauf@uibk.ac.at}\\ URL: \url{https://www.zeileis.org/}, \url{https://nikum.org/}\\ Friedrich Leisch\\ Universit\"at f\"ur Bodenkultur Wien\\ Institute of Applied Statistics and Computing\\ Peter Jordan-Str.~82, 1190 Wien, Austria\\ E-mail: \email{Friedrich.Leisch@R-project.org}\\ URL: \url{https://boku.ac.at/rali/stat/personen/leisch-friedrich} } \begin{document} \SweaveOpts{engine = R, eps = FALSE, keep.source = TRUE} %\VignetteIndexEntry{Flexible Generation of E-Learning Exams in R: Moodle Quizzes, OLAT Assessments, and Beyond} %\VignetteDepends{stats, graphics, tools, utils, tth, base64enc} %\VignetteKeywords{exams, e-learning, multiple choice, arithmetic problems, Sweave, R, LaTeX, HTML, XML, IMS QTI, Moodle, OLAT} %\VignettePackage{exams} <>= options(width = 76, prompt = "R> ", continue = "+ ") library("exams") library("base64enc") library("tth") @ \section{Introduction} \label{sec:intro} The design for version~1 of the \pkg{exams} package was conceived eight years ago (in 2006) when the original authors \citep{exams:Gruen+Zeileis:2009} were involved in a redesign of the introductory statistics lecture at WU Wirtschaftsuniversit\"at Wien. Back then the main goal was to be able to produce exams along with associated self-study materials as PDF (portable document format) files. Thus, the main focus was still on printable materials for classic classroom exams. Although e-learning systems started to become available more easily back at that time, they were still not very widely used and, more importantly, rather few easy-to-use standards for specifying e-learning exams were available (e.g., WU Wien used a partially self-developed e-learning system based on \pkg{.LRN}, see \citealp{exams:Blesius+Moreno-Ger+Neumann:2007}). However, since 2006 the situation has clearly changed: E-learning systems are now abundant with many universities offering one (ore more) e-learning system(s) in which all students are readily registered. Consequently, many lecturers routinely offer online exams (or tests, quizzes, assessments) for large-lecture courses -- either as self-study materials or as (part of) the main assessment of the course. Among the more popular choices of learning management systems are the open-source systems \pkg{Moodle} -- developed by \cite{exams:Moodle:2014} and supported by a large world-wide user community -- or \pkg{OLAT} (for online learning and training) -- originally developed by \cite{exams:OLAT:2012} and with the recent fork \pkg{OpenOLAT} being developed by \cite{exams:OpenOLAT:2014} and a support community. A popular proprietary learning management system is \pkg{Blackboard} developed by \cite{exams:Blackboard:2010}. Standards for specifying and exchanging e-learning exams/assessements have also emerged \citep[see][for an overview]{exams:Agea+Crespo-Garcia+Delgado-Kloos:2009}. While \pkg{Moodle} specifies its own \pkg{Moodle} XML format (but can import and export several other formats), \pkg{OLAT}/\pkg{OpenOLAT} and \pkg{Blackboard} employ certain subsets of the international QTI (question \& test interoperability) standard, version~1.2, maintained by the \cite{exams:QTI12:2002}. The successor formats are QTI~2.0 and the current QTI~2.1 which is for example employed in the \pkg{ONYX} testsuite \citep{exams:BPS:2014} that also offers interfaces to \pkg{OLAT} and \pkg{Blackboard}. Therefore, although the PDF exams produced by version~1 of the \pkg{exams} package as introduced by \cite{exams:Gruen+Zeileis:2009} are still useful for many types of courses, it would also be highly desirable to have support for generating e-learning exams \emph{from the same pool of exercises}. In fact, this became an apparent need when the authors of the present manuscript took over new large-lecture statistics and mathematics courses at their respective institutions (Universit\"at Innsbruck and Universit\"at f\"ur Bodenkultur Wien, BOKU, respectively). For example, the new ``Mathematics~101'' lecture at Universit\"at Innsbruck is currently attended by about 1,600~students (mostly first-year business and economics students) and accompanied by biweekly online exams conducted in the university's \pkg{OLAT} learning management system. This was a strong incentive to start developing version~2 of the \pkg{exams} package that is presented here and offers an extensible toolbox for generating e-learning exams, including easy-to-use functions for \pkg{Moodle} quizzes and \pkg{OLAT} assessments\footnote{Meanwhile, various configurations of exams in \pkg{OLAT} and \pkg{OpenOLAT} are supported. See the convenience function \code{exams2openolat()} for some more details.}. The new version of the \pkg{exams} package for the \proglang{R} system for statistical computing \citep{exams:R:2014} is available from the Comprehensive \proglang{R} Archive Network at \url{https://CRAN.R-project.org/package=exams}. Like prior versions it employs ideas and technologies from literate programming and reproducible research \citep[see e.g.,][]{exams:Knuth:1992,exams:deLeeuw:2001,exams:Leisch+Rossini:2003,exams:Kuhn:2014} by using \code{Sweave()} \citep{exams:Leisch:2002a} to combine data-generating processes in \proglang{R} with corresponding questions/solutions in {\LaTeX} \citep{exams:Knuth:1984,exams:Lamport:1994}. But in addition to producing exams in PDF format, the new version of \pkg{exams} includes extensible tools for generating other output formats \emph{without} having to modify the pool of exercises. Thus, the design principles of the \pkg{exams} package are only somewhat extended compared to version~1: \begin{itemize} \item Each exercise template (also called ``exercise'' for short) is a single \code{Sweave} file (\code{.Rnw}) interweaving \proglang{R} code for data generation and {\LaTeX} code for describing question and solution (possibly containing mathematical notation in {\LaTeX}). \item Exams can be generated by randomly drawing different versions of exercises from a pool of such \code{Sweave} exercise templates. The resulting exams can be rendered into various formats including PDF, HTML, \pkg{Moodle} XML, or QTI~1.2\footnote{An implementation of the newer QTI~2.1 standard is also available in the package, see \code{?exams2qti21}. However, at the time of writing the function still needed more testing and hence was not discussed here.} (for \pkg{OLAT}/\pkg{OpenOLAT}). \item Maintenance is simple as exercises are separate standalone files. Thus, large teams can work jointly on the pool of exercises in a multi-author and cross-platform setting because each team member can independently develop and edit a single exercise. \end{itemize} The remainder of this paper consists of two major parts: First, we illustrate in Section~\ref{sec:usage} how to use both the old and new exam-generating functions that are readily available in the package. This serves as a first introduction and is sufficient for getting a good overview of the available features and how to get started. Second, Section~\ref{sec:design} provides details about the design underlying the toolbox for the new infrastructure. This section -- as well as the subsequent Section~\ref{sec:extensions} showing how to extend the toolbox -- may be skipped upon first reading but it contains many important details that are likely to be required when actually starting to create course materials with the package. Finally, a discussion in Section~\ref{sec:discussion} concludes the paper. \section[Using the exams package]{Using the \pkg{exams} package} \label{sec:usage} In this section we provide an overview of the most important user interfaces provided by the \pkg{exams} package. This serves as a first introduction, assuming only (basic) knowledge of \code{Sweave} \citep{exams:Leisch:2008a,exams:Leisch:2008}. First, the format of the exercise \code{Sweave} files is reviewed along with the old (version~1) \code{exams()} function. Subsequently, the new (version~2) functions of type \code{exams2}\emph{xyz}\code{()} are introduced: \code{exams2pdf()} and \code{exams2html()} produce one PDF or HTML file for each exam, respectively. In case of just a single exam, this is shown interactively in a viewer/browser. \code{exams2moodle()} and \code{exams2qti12()} generate \pkg{Moodle} and QTI~1.2 exams, i.e., just a single XML or ZIP file, respectively, which can be easily uploaded into \pkg{Moodle} and \pkg{OLAT}. \subsection[Version 1: PDF exams() from Sweave exercises]{Version~1: PDF \code{exams()} from \code{Sweave} exercises} Exercise templates (or just ``exercises'' for short) are essentially separate standard \code{Sweave} files \citep{exams:Leisch:2008a,exams:Leisch:2008}. They are composed of the following elements: \begin{itemize} \item \proglang{R} code chunks (as usual within \verb|<<>>=| and \code{@}) for random data generation. \item Question and solution descriptions contained in {\LaTeX} environments of corresponding names. Both can contain \proglang{R} code chunks again or include data via \verb|\|\verb|Sexpr{}|. \item Metainformation about the exercise type (numeric, multiple choice, \dots), its correct solution etc. All metainformation commands are in {\LaTeX} style but are actually commented out and hidden in the final output file. \end{itemize} The underlying ideas are eplained in more detail by \cite{exams:Gruen+Zeileis:2009} and Section~\ref{sec:design} provides more technical details. Here, we focus on illustrating how different output formats can be generated from such exercises. %% from Rnw \begin{figure}[p!] \centering \makebox[\textwidth]{\hrulefill} <>= invisible(file.copy(system.file("exercises", "tstat.Rnw", package = "exams"), "tstat.Rnw")) Rnw <- readLines("tstat.Rnw") cat(c("\\begin{verbatim}", Rnw, "\\end{verbatim}"), sep = "\n") @ \makebox[\textwidth]{\hrulefill} \caption{\label{fig:Rnw} A simple \code{Sweave} exercise: \code{tstat.Rnw}.} \end{figure} %% to LaTeX \begin{figure}[t!] \centering \makebox[\textwidth]{\hrulefill} <>= set.seed(1090) Sweave("tstat.Rnw") tex <- readLines("tstat.tex") file.remove(c("tstat.Rnw", "tstat.tex")) @ <>= cat(c("\\begin{verbatim}", tex, "\\end{verbatim}"), sep = "\n") @ \makebox[\textwidth]{\hrulefill} \caption{\label{fig:tex} {\LaTeX} output of \code{Sweave("tstat.Rnw"}).} \end{figure} %% to PDF/HTML \begin{figure}[p!] \centering \makebox[\textwidth]{\hrulefill} \begin{enumerate} <>= cat(tex, sep = "\n") @ \end{enumerate} \makebox[\textwidth]{\hrulefill} \caption{\label{fig:pdf} Display of a \code{tstat} exercise as PDF via \code{exams()} or \code{exams2pdf()}.} \vspace*{0.5cm} \includegraphics[width=\textwidth]{screenshots/exams2html-tstat.png} \caption{\label{fig:exams2html-tstat} Display of a \code{tstat} exercise as HTML via \code{exams2html()}. MathML is employed for mathematic equations, as rendered by a \pkg{Firefox} browser.} \end{figure} In Figure~\ref{fig:Rnw}, the \code{Sweave} file for a simple exercise asking students to compute a one-sample $t$~test statistic is shown for illustration \citep[as already used by][]{exams:Gruen+Zeileis:2009}. The \proglang{R} chunk for the data-generating process (DGP), the \code{question} and \code{solution} environments, and the metainformation can be easily distinguished. The {\LaTeX} file resulting from an \code{Sweave()} call is shown in Figure~\ref{fig:tex}, and Figure~\ref{fig:pdf} shows the final compiled PDF output generated by <>= library("exams") set.seed(1090) exams("tstat.Rnw") @ Here, the \code{exams()} function looks for the exercise template \code{tstat.Rnw} first in the local working directory and then within the installed \pkg{exams} package where this file is provided. Then it copies the exercise \code{.Rnw} to a temporary directory, calls \code{Sweave()} to generate the \code{.tex}, and includes this in the default {\LaTeX} template for exams before producing the \code{.pdf}. As, by default, just a single \code{.pdf} exam is produced and no output directory is specified, a PDF viewer pops up and displays the resulting exam (as in Figure~\ref{fig:pdf}). \begin{table}[t!] \centering \begin{tabular}{lp{13cm}} \hline Argument & Description \\ \hline \code{file} & A (list of) character vector(s) specifying the (base) names of the \code{Sweave} exercise files. \\ \code{n} & The number of exams to be generated from the list of exercises. Default: 1.\\ \code{nsamp} & The number of exercise files sampled from each list element of \code{file}. Default: One for each list element.\\ \hline \code{dir} & Path to output directory. Default: Single PDF or HTML files are shown directly in a viewer/browser (i.e., \code{exams}/\code{exams2pdf}/\code{exams2html} with \texttt{n~=~1}). In all other cases the current working directory is used by default.\\ \code{edir} & Path to the directory in which the exercises in \code{file} are stored. Default: Working directory (or within the \pkg{exams} installation). \\ \code{tdir} & Path to a temporary directory in which \code{Sweave()} is carried out. Default: New \code{tempfile()}. \\ \code{sdir} & Path to the directory in which supplementary files (e.g., graphics or data files) are stored (except for \code{exams()}). Default: New \code{tempfile()}. \\ \hline \code{name} & Name prefix for the resulting exam files.\\ \code{template} & Character specifying the (base) names of a {\LaTeX}, HTML, or XML file template for the exam (except for \code{exams2moodle()}). Default: A function-specific template provided within the \pkg{exams} installation.\\ \code{encoding} & Character specifying the encoding to be used (in version~2 interfaces only).\\ \code{verbose} & Should progress information be displayed (in version~2 interfaces only)?\\\hline \end{tabular} \caption{\label{tab:args} Common arguments of the main user interfaces for generating exams: \texttt{exams()}, \texttt{exams2pdf()}, \texttt{exams2html()}, \texttt{exams2moodle()}, \texttt{exams2qti12()}. The first group of arguments pertains to the specification of the exam(s), the second group to the handling of input/temporary/output directories, and the last group to name and setup for the resulting files. For further function-specific arguments and more details/examples, see the corresponding manual pages.} \end{table} While applying \code{exams()} to just a single exercise is very useful while writing/programming an exercise, a full exam will typically encompass several different exercises. Also, it may require suppressing the solutions, including a title page with a questionnaire form, etc. The former can be achieved by supplying a (list of) vector(s) of exercises while the latter can be accomodated by using different templates: <>= myexam <- list( "boxplots", c("confint", "ttest", "tstat"), c("anova", "regression"), "scatterplot", "relfreq") odir <- tempfile() set.seed(1090) exams(myexam, n = 3, dir = odir, template = c("exam", "solution")) @ The \code{myexam} list contains five exercises: the first one is always \code{boxplots.Rnw} while the second exercise is randomly drawn from \code{confint.Rnw}, \code{ttest.Rnw}, \code{tstat.Rnw}, and so on for the remaining exercises. Then, \code{exams()} is used to draw \code{n = 3} random exams and produce one exam and one solution PDF for each. The \code{template} argument takes names of {\LaTeX} files which provide the {\LaTeX} headers and footers. These templates can be used to create a title page with a questionnaire form (for student name, id, signature, etc.), show or suppress solutions, and set further formatting details. All involved \code{.Rnw} files (with exercises) and \code{.tex} templates employed in the example above are provided in the \pkg{exams} source package and its installed versions. The resulting output files are stored along with the extracted metainformation in the output directory: <>= dir(odir) @ More details on basic usage and more advanced customization of this function are provided by the vignette \code{vignette("exams", package = "exams")} which is an updated version of \cite{exams:Gruen+Zeileis:2009}. The vignette actually uses the new and somewhat more flexible \code{exams2pdf()} function rather than \code{exams()} and discusses the enhancements made for the former (see also below). \subsection[Version 2: Producing PDF, HTML, or XML for Moodle or OLAT]{Version~2: Producing PDF, HTML, or XML for \pkg{Moodle} or \pkg{OLAT}} The new infrastructure added to the \pkg{exams} package on the road to version~2 is providing more flexibility and enables a much broader variety of output formats while keeping the specification of the exercise templates fully backward compatible and only slightly extended. While the design of the underlying workhorse functions is rather different (see Section~\ref{sec:design}), the new user interfaces are very similar to the old one, sharing most of its arguments (see Table~\ref{tab:args}). Hence, for users of the previous version of the package, it is easy and straightforward to adapt to the new facilities. \subsubsection[Producing PDF documents: exams2pdf()]{Producing PDF documents: \code{exams2pdf()}} As mentioned above, the function \code{exams2pdf()} is a more flexible reimplementation of \code{exams()} using the new extensible infrastructure of the \pkg{exams} package. For the user virtually nothing changes: <>= set.seed(1090) exams2pdf("tstat.Rnw") @ pops up the same PDF as shown in Figure~\ref{fig:pdf}. We refrain from further discussion of customization of the PDF output because this is discussed in \code{vignette("exams", package = "exams")} with details about {\LaTeX} master templates, additional auxiliary files, showing/hiding solutions etc. Here we only point out the main difference between the old \code{exams()} function and the new \code{exams2pdf()}: The latter not only returns the metainformation from the exercise but additionally also the {\LaTeX} code for the question and solution environments as well as paths to supplementary materials (such as graphics or data files). Section~\ref{sec:design} explains the structure of the return values in more detail and illustrates how this can be used.\footnote{To obtain the same type of return value as from the \texttt{exams()} function, \texttt{exams\_metainfo(exams2}\emph{xyz}\texttt{(...))} can be used.} \subsubsection[Producing HTML documents: exams2html()]{Producing HTML documents: \code{exams2html()}} As a first step towards including exams generated from \code{Sweave} files into e-learning exams, it is typically necessary to be able to generate an HTML version of the exams. Hence, the function \code{exams2html()} is designed analogously to \code{exams()}/\code{exams2pdf()} but produces HTML files. In case of just a single generated exam, this is displayed in a browser using base \proglang{R}'s \code{browseURL()} function\footnote{In \pkg{RStudio} \citep{exams:RStudio:2014}, versions prior to~0.97.133, the \code{"browser"} option is set to a function that cannot browse local HTML files on some platforms. Recent versions of \pkg{RStudio} have resolved this problem and \code{?exams2html} also provides workarounds for older \pkg{RStudio} versions.}. Again, this is particularly useful while writing/programming a new exercise template. For example, <>= set.seed(1090) exams2html("tstat.Rnw") @ generates the HTML file shown in Figure~\ref{fig:exams2html-tstat} which corresponds directly to the PDF file from Figure~\ref{fig:pdf}. Note that for properly viewing the formulas in this HTML file, a browser with MathML support is required. This is discussed in more detail in Section~\ref{sec:transform_html}. Here, the \pkg{Firefox} browser is used (in Debian Linux's rebranded \pkg{Iceweasel} version) which has native MathML support. To transform the {\LaTeX} questions/solutions to something that a web browser can render, three options are available: translation of the {\LaTeX} to (1)~plain HTML, (2)~HTML plus MathML for mathematical formulas (default), or conversion of the corresponding PDF to (3)~HTML with one embedded raster images for the whole question and solution, respectively. The former two options are considerably faster and more elegant -- they just require the \proglang{R}~package \pkg{tth} \citep{exams:Hutchinson+Leisch+Zeileis:2013} that makes the `{\TeX}-to-HTML' converter \pkg{TtH} \citep{exams:Hutchinson:2012} easily available in~\proglang{R}. Also, by default, the \pkg{base64enc} package \citep{exams:Urbanek:2012} is employed for embedding graphics in Base64 encoding. More details on this approach are provided in Section~\ref{sec:transform_html}. The HTML files produced with approaches~(1) and~(2) can also easily contain hyperlinks to supplementary files. For example, if the \proglang{R} code in the \code{Sweave} file generates a file \code{mydata.rda}, say, then simply including \verb|\url{mydata.rda}| in the question/solution will result in a suitable hyperlink. The supplementary data files for each random replication of the exercise is managed fully automatically and a copy of the data is created in an (exam-specific) sub-directory of the output directory. Run \code{exams2html("boxhist.Rnw")} for such an example. Just like \code{exams()}/\code{exams2pdf()}, \code{exams2html()} can also generate multiple replications of randomly drawn exams via \code{exams2html(myexam, n = 3, dir = odir)}. Also multiple versions of the same replications can be generated by providing several templates, e.g., for showing/suppressing solutions. \subsubsection[Producing Moodle XML: exams2moodle()]{Producing \pkg{Moodle} XML: \code{exams2moodle()}} To incorporate exams generated from \code{Sweave} exercises into learning management systems, such as \pkg{Moodle}, two building blocks are typically required: (1)~questions/solutions are available in plain text or HTML format, and (2)~questions/solutions can be embedded along with the metainformation about the possible and correct solutions into some exam description format. (1)~can be accomplished as outlined in the previous subsection for \code{exams2html()} and for \pkg{Moodle} (2)~requires embedding everything into \pkg{Moodle} XML format. Both steps can be easily carried out using the \code{exams2moodle()} function\footnote{Meanwhile, the default \code{converter} for Moodle is \code{"pandoc-mathjax"}. Here, we use \code{"ttm"} instead which was the default at the time of writing.}: <>= set.seed(1090) exams2moodle(myexam, n = 3, dir = odir, converter = "ttm") @ This draws the same three random exams from the \code{myexam} list that were already generated in PDF format above. The output file, stored again in \code{odir}, is a single XML file. <>= dir(odir) @ This XML file \code{moodlequiz.xml} can be easily imported into a \pkg{Moodle} quiz and then further customized: First, the XML file is imported into the question bank in \pkg{Moodle}. Then, all replications of each exercise can be added as ``random'' questions into a quiz (and potentially further customized). Figure~\ref{fig:exams2moodle-boxplots} shows the first random draw of the \code{boxplots} exercise in the resulting \pkg{Moodle} quiz (again rendered by a \pkg{Firefox} browser). More details on how \pkg{exams}-generated questions can be integrated in \pkg{Moodle} are provided in Section~\ref{sec:discussion}. \begin{figure}[t!] \centering \includegraphics[width=0.95\textwidth]{screenshots/exams2moodle-boxplots.png} \caption{\label{fig:exams2moodle-boxplots} Display of exercise~1 (\code{boxplots}) from \code{myexam} in \pkg{Moodle} (as rendered by a \pkg{Firefox} browser).} \end{figure} The corresponding solutions are displayed upon completion of the exam in \pkg{Moodle}. As before, selected supplementary files are automatically managed and can easily be included using \verb|\url{}| in the underlying {\LaTeX} code. To be able to include all these supplements in a single XML file, Base64 encoding is employed for all supplements. See the manual page for the list of all supported supplement file formats. \subsubsection[Producing QTI 1.2 XML (for OLAT): exams2qti12()]{Producing QTI 1.2 XML (for \pkg{OLAT}): \code{exams2qti12()}} \begin{figure}[t!] \centering \includegraphics[width=0.95\textwidth]{screenshots/exams2qti12-boxplots.png} \caption{\label{fig:exams2qti12-boxplots} Display of exercise~1 (\code{boxplots}) from \code{myexam} in \pkg{OLAT} (as rendered by a \pkg{Firefox} browser).} \end{figure} The generation of QTI~1.2 assessments (for \pkg{OLAT}) proceeds essentially in the same way as for the \pkg{Moodle} quizzes, by default using \code{ttm} for transformation of the text to HTML\footnote{It may be of interest to \pkg{OLAT} users that we experienced problems with the display of MathML matrices in \pkg{OLAT}. The columns were not separated by spaces and we have not been able to adapt our \pkg{OLAT} installation to avoide this problem. Hence, if we want to display matrices in \pkg{OLAT}, we generate them with extra empty columns. The \code{cholesky.Rnw} exercise template has code that can automatically do this, if enabled.}. The same three random draws of exams from \code{myexam} can be prepared in QTI~1.2 format via: <>= set.seed(1090) exams2qti12(myexam, n = 3, dir = odir) @ This produces a single ZIP file \code{qti.zip}, again written to \code{odir}. <>= dir(odir) @ The ZIP file can again be easily imported into an \pkg{OLAT} test configuration where further customization can be performed\footnote{While customization of the features of the overall assessment was always possible for us, \pkg{OLAT} typically did not allow for modification of the individual exercise items. We were not able to track down which part of the QTI~1.2 XML specification causes this.}. The first \code{boxplots} exercise from the exam generated above is shown in \pkg{OLAT} in Figure~\ref{fig:exams2qti12-boxplots} (again as rendered by a \pkg{Firefox} browser). The corresponding solutions are displayed in \pkg{OLAT} immediately after \emph{incorrectly} completing an individual exercise. The display of solutions can also be suppressed completely by setting \code{solutionswitch = FALSE} in \code{exams2qti12()}.\footnote{In our e-learning exams, we typically employ these default settings (i.e., \code{maxattemps = 1} and \code{solutionswitch = TRUE}). Alternatively, we give the students an unlimited number of attempts to solve an exercise (\code{maxattempts = Inf}) but then suppress solutions completely (\code{solutionswitch = FALSE}) because otherwise the correct solution would be displayed after the first incorrect attempt.} The main difference of the generated ZIP file for QTI~1.2, compared to the \pkg{Moodle} XML output, is that in addition to the \code{qti.xml} file further supplementary files can be included. Hence, supplements in all potential formats can be easily included and uploaded in one go into \pkg{OLAT}. Therefore, by default, Base64 is employed only for graphics but not for other files (such as data sets etc.) but either approach can be used optionally for all types of supplements. The QTI~1.2 standard allows for rather fine control of the properties of the exercises (also known as items in QTI~1.2) and the exams (also known as assessments). Hence, \code{exams2qti12()} provides a variety of options for controlling the appearance of exam/exercises, see the manual page \code{?exams2qti12} for details. Also, the underlying XML template can be adapted. \subsection{Creating the first exam} \label{sec:skeleton} When creating the first ``real'' exam with \pkg{exams}, i.e., when starting to prepare course materials with the help of the package, it is our experience that it works best to start (almost) from scratch with some simple examples. The package provides a wide range of examples for typical exercises (see Table~\ref{tab:exercises} in the appendix for an overview) which can serve as a starting point and it is often useful to copy parts of these exercises to create new ones. In particular, we recommend to keep the formatting as simple as possible for two reasons: (1)~The resulting exercises are typically more robust and work well with different \code{exams2}\emph{xyz}\code{()} interfaces (especially both in {\LaTeX} and in HTML). (2)~Some formatting issues require attention to technical details, e.g., as discussed in Section~\ref{sec:design}. Thus, we recommend to start with exercises taking inspiration from the available examples and only using basic {\LaTeX} markup for mathematical notation and formatting. To aid this process, \pkg{exams} provides the function \code{exams_skeleton()} (or equivalently \code{exams.skeleton()}) which creates a directory with a copy of the exercises from Table~\ref{tab:exercises} (in the appendix) and the required templates (e.g., for {\LaTeX}, HTML, or XML) along with a `\code{demo.R}' script illustrating the use of the various \code{exams2}\emph{xyz}\code{()} interfaces. As an illustration, assume that we are interested in using \code{exams2moodle()} and \code{exams2pdf()} for generating both \pkg{Moodle} and PDF exams (for printout). To test that the {\LaTeX}-to-HTML conversion for \pkg{Moodle} actually works for all exercises, we additionally want to inspect the results of \code{exams2html()}. Hence, the code below calls \code{exams_skeleton()} specifying these three writers. Here, we employ a temporary directory but users may set the \code{dir} argument to something like \code{"C:/myexam/"} or \code{"~/myexam/"} etc. <>= dir.create(mydir <- tempfile()) exams_skeleton(dir = mydir, absolute = TRUE, writer = c("exams2html", "exams2pdf", "exams2moodle")) dir(mydir) @ The directory then contains the files `\code{demo-all.R}' which can be opened in any editor for \proglang{R}~scripts -- as well as separate `\code{demo-*.R}' scripts for the different interfaces. These illustrate how to create various kinds of output using the \code{exams2html()}, \code{exams2pdf()}, and \code{exams2moodle()} functions based on the exercises and templates in the subdirectories of the same name. Absolute paths are employed in the script to refer to these subdirectories (while the default \code{absolute = FALSE} would result in relative paths being used). The function \code{exams_skeleton()} always copies all exercise files to the directory but the `\code{demo-*.R}' scripts only employ some selected files for each exercise type. It is easy to modify the `\code{demo-*.R}' scripts, omitting or adding exercises that are readily available in the subdirectory. Finally, to illustrate how different encodings can be used, \code{exams_skeleton()} can also be called with an \code{encoding} argument, e.g., setting \code{encoding = "UTF-8"}. This modifies the \code{demo-*.R} scripts as well as the HTML and {\LaTeX} templates accordingly. The \code{encoding}s \code{"latin1"} (or \code{"ISO-8859-1"}) and \code{"latin9"} (or \code{"ISO-8859-15"}) have also been tested. As usual, employing \code{Sweave} files in a particular encoding can be very convenient for special characters (such as accents or umlauts) but might also lead to problems if they are used in different locales (e.g., on different operating systems). An alternative route (employed by the authors of the \pkg{exams} package) is to employ \code{Sweave} ASCII files only, using {\LaTeX} commands for special characters. \section{Design} \label{sec:design} All the new \code{exams2}\emph{xyz}\code{()} interfaces for generating exams in different formats (with currently \emph{xyz} $\in$ \{\code{pdf}, \code{html}, \code{moodle}, \code{qti12}\}) are built by combining the modular building blocks provided by version~2 of \pkg{exams}. All functions have the same goal, i.e., to write exam files in a certain format to the hard disk. The approach is that the \code{Sweave} exercises are first \emph{weaved} to {\LaTeX}, \emph{read} into \proglang{R}, potentially \emph{transformed} (e.g., to HTML), and then \emph{written} to suitable output file formats on the disk. Different customizable driver functions (or even driver-generating functions) for performing the weave/read/transform/write steps are available in \pkg{exams}. Internally, all the \code{exams2}\emph{xyz}\code{()} interfaces choose certain drivers and then call the new function \code{xexams()} (for extensible exams) that handles all temporary files/directories and suitably executes the drivers. In the following subsections, all these building blocks are introduced in detail. \subsection{Extended specification of exercises} \label{sec:exercises} As discussed in Section~\ref{sec:usage} and illustrated in Figure~\ref{fig:Rnw}, each exercise is simply an \code{Sweave} file containing \proglang{R} code for data generation, \code{{question}}/\code{{solution}} environments with {\LaTeX} text, and metainformation about the type of exercise and the correct solution etc. This design was introduced by \cite{exams:Gruen+Zeileis:2009} but is slightly extended in the new version to provide some more options for the generation of e-learning exams. See Table~\ref{tab:metainfo} for an overview for a list of exercise types and corresponding metainformation commands. \begin{table}[t!] \centering \begin{tabular}{lp{11.8cm}} \hline Command & Description \\ \hline \verb|\extype{}| & Specification of the type of exercise (required): \code{num} for questions with a numeric answer, \code{mchoice} for questions with multiple-choice answers, \code{schoice} for questions with single-choice answers (i.e., multiple-choice with exactly one correct solution), \code{string} for questions with a (short) text answer, or \code{cloze} for cloze solutions (i.e., combinations of the above). \\ \verb|\exname{}| & Short name/description (to be used for printing within \proglang{R}). \\ \verb|\extitle{}| & Pretty longer title (for \pkg{Moodle}). \\ \verb|\exsection{}| & Section of the exercise (for \pkg{Moodle}, with slashes for subsections as in a URL). \\ \verb|\exversion{}| & Version of the exercise. \\ \hline \verb|\exsolution{}| & Correct solution (required). It must contain a numeric solution for \code{num}, a string of zeros/ones for \code{mchoice}/\code{schoice}, or a character string of \code{string}. For \code{cloze} a combination of these can be specified, e.g., \verb+\exsolution{1.23|001|glm}+. \\ \verb|\extol{}| & Tolerance for \code{num} solutions or a vector of tolerances (expanded if necessary) for \code{cloze} solutions. If unspecified the tolerance is 0. \\ \verb|\exclozetype{}| & List of types for the elements of a \code{cloze} exercise, e.g., \verb+\exclozetype{num|schoice|string}+ for the example above. \\ \verb|\expoints{}| & Points for (fully) correct solution. Default is 1.\\ \hline \verb|\exextra[]{}| & Additional metainformation to be read and stored, e.g., for new custom interfaces. The default storage type is character, e.g., \verb|\exextra[myinfo]{1.23}| yields a metainformation element \code{myinfo} of \code{"1.23"}. The type can also be numeric or logical, e.g., \verb+\exextra[myinfo,logical]{FALSE|FALSE|TRUE}+. \\ \hline \end{tabular} \caption{\label{tab:metainfo} Overview of metainformation commands in exercises. The commands in the first section allow for a general description, those in the second section for question/answer specification. %%and those in the last section for customization in e-learning environments. Only \texttt{extype} and \texttt{exsolution} are always required (but \texttt{exname} is recommended additionally for nice printing in \proglang{R}).} \end{table} Each exercise must specify at least an \verb|\extype{}| and an \verb|\exsolution{}| and should typically also have a short \verb|\exname{}|. There are now five different \code{extype}s. Two types that have a single question and answer: \begin{itemize} \item \code{num} for questions with a numeric answer, e.g., \verb|\exsolution{1.23}|. \item \code{string} for questions with a (short) text answer, e.g., \verb|\exsolution{glm}|. \end{itemize} Three types have a list of questions (or statements): \begin{itemize} \item \code{mchoice} for multiple-choice questions where each element of the question/statement can either be true or false, e.g., \verb|\exsolution{01011}|. \item \code{schoice} for single-choice questions where exactly one of the questions/statements is true and all others are false, e.g., \verb|\exsolution{01000}|. \item \code{cloze} for a combination of questions/statements with \code{num}, \code{string}, or \code{mchoice} answers. Thus, each element of the question has either a numeric, short text, or single/multiple-choice answer, e.g., \verb+\exsolution{1.23|001|glm}+. To specify the individual \code{cloze} types, a \code{clozetype} has to be given, e.g., \verb+\exclozetype{num|schoice|string}+. \end{itemize} The types \code{schoice} and \code{cloze} have been newly introduced. The purpose of the former is mainly to allow for different processing of options (e.g., for assigning points to correct/wrong results) between \code{mchoice} and \code{schoice}. The \code{cloze} type was introduced because both \pkg{Moodle} and QTI~1.2 have support for it (albeit in slightly different ways, for details see below). Possible evaluation strategies (with/without partial credits and/or with/without negative points) are discussed below for \code{exams2moodle()} and \code{exams2qti12()} and in Appendix~\ref{sec:eval} for further functionality within \proglang{R}. For the three types with lists of questions (\code{mchoice}, \code{schoice}, \code{cloze}), the \code{{question}} and \code{{solution}} environments should each contain at the end an \code{{answerlist}} environment. In the \code{{question}} this should list an \verb|\item| for each question/statement and in the \code{{solution}} the corresponding answers/explanations can be provided (if any). The \code{{answerlist}} environment can either be written as usual ``by hand'' or by using the \code{answerlist()} function provided by the \pkg{exams} package. For illustration, we set up a multiple-choice question with three statements about Switzerland. First, we generate an \code{{answerlist}} with statements for the \code{{question}}. <>= qu <- c("Zurich is the capital of Switzerland.", "Italian is an official language in Switzerland.", "Switzerland is part of the European Union (EU).") answerlist(qu) @ Then the corresponding \code{{answerlist}} for the \code{{solution}} is set up. <>= sol <- c(FALSE, TRUE, FALSE) ex <- c("The capital of Switzerland is Bern.", "The official languages are: German, French, Italian, Romansh.", "Switzerland is part of the Schengen Area but not the EU.") answerlist(ifelse(sol, "True", "False"), ex) @ For more examples see the exercise files in the \code{inst/exercises} directoy of the \pkg{exams} source package. There are various multiple-choice questions with and without figures and/or verbatim \proglang{R} output (e.g., \code{anova}, \code{boxplots}, \code{cholesky}, among others). The files \code{tstat} and \code{tstat2} illustrate how the same type of exercise can be coded as a \code{num} or \code{schoice} question, respectively. The \code{cloze} type is employed in, e.g., \code{boxhist} or \code{fourfold} (with more flexible formatting specifically for \pkg{Moodle} in \code{boxhist2} and \code{fourfold2}, respectively). See also Table~\ref{tab:exercises} in the appendix for an overview and Appendix~\ref{sec:cloze} for more details on cloze exercises. \subsection[The xexams() wrapper function]{The \code{xexams()} wrapper function} To avoid recoding certain tedious tasks -- such as copying/reading files and handling temporary directories -- for each of the user interfaces introduced in Section~\ref{sec:usage}, the new \pkg{exams} package provides a modular and extensible framework for building new exam-generating functions. This framework is tied together by the \code{xexams()} function which is typically not called by users directly but forms the basis for all new \code{exams2}\emph{xyz} interfaces. To accomplish this, \code{xexams()} also takes the arguments listed in Table~\ref{tab:args} (except \code{name} and \code{template}), draws exams from the exercise \code{file} list, and does all the necessary file/directory handling. Furthermore, it takes a \code{driver} argument that needs to be a list of four functions \code{driver = list(sweave, read, transform, write)}. These are utilized as follows: \begin{enumerate} \item \emph{Weave:} For each of the selected exercise files (within all \code{n} exams) \code{driver$sweave(file)} is run to weave the \code{.Rnw} file into a \code{.tex} file. If \code{sweave = NULL} (the default), the standard \code{Sweave()} function is used. If \code{sweave = list(...)} is a list, e.g., \code{list(pdf = FALSE, png = TRUE)}, this is passed as arguments to \code{Sweave()}. \item \emph{Read:} Each resulting \code{.tex} file is read into \proglang{R} using \code{driver$read(file)}. By default (\code{read = NULL}), the function \code{read_exercise()} is used (see below), resulting in a list of character vectors with the {\LaTeX} code for question/solution plus metainformation. \item \emph{Transform:} Each of these exercise-wise list objects can subsequently be transformed by \code{driver$transform(object)} which can be leveraged for transformations from {\LaTeX} to HTML etc. By default (\code{transform = NULL}), no transformation is applied. \item \emph{Write:} The (possibly transformed) lists of exercises, read into \proglang{R} for each exam object, can be written out to one ore more files per exam in an output directory via \code{driver$write(object, dir, info = list(id, n))}. By default (\code{write = NULL}), no files are written. \end{enumerate} After performing each of the driver functions, \code{xexams()} returns invisibly a nested list object (currently unclassed) as illustrated in Figure~\ref{fig:xexam}. It is a list of \emph{exams} (of length \code{n}), each of which is a list of \emph{exercises} (whose length depends on the length of \code{file} and \code{nsamp}), each of which is a list (whose length/contents depends on \code{driver$read}). When used with the default \code{read_exercise()}, each exercise is a list of length~6 containing the question/solution texts, metainformation, and paths to supplementary files. These will be introduced in more detail in the next section. \begin{figure}[t!] \hrulefill \begin{Code} list( ## of 'exams', length: n list( ## of 'exercises', length: k list( ## of exercise content, length: 6 question, questionlist, solution, solutionlist, metainfo, supplements ) ) ) \end{Code} \hrulefill \caption{\label{fig:xexam} Structure of the return value of \texttt{xexams()}, when used with the default \code{read} driver \texttt{read\_exercises()}.} \end{figure} All of the interfaces introduced in Section~\ref{sec:usage} employ the standard \code{Sweave()} function for the weaving step (possibly with custom arguments) and the \code{read_exercise()} function for the reading step. They mainly differ in the transformation and writing step. \code{exams2pdf()} needs no transformation and the writer first sets up a \code{.tex} file for each exam, calls \code{texi2dvi(pdf = TRUE)}, and then copies the resulting \code{.pdf} file to the output \code{dir}. \code{exams2html()} on the other hand uses a {\TeX}-to-HTML transformation and the writer then sets up a \code{.html} file for each exam and copies it to the output \code{dir}. Finally, \code{exams2moodle()} and \code{exams2qti12()} both also use a transformation to HTML but have no writer. The reason for this is that they do not write one file per exam (i.e., with only one replication per exercise) but rather need to produce XML files that include all different replications of each exercise. Hence, they take the list returned by \code{xexams()} and process it subsequently in different ways. The details for all these steps are explained in the subsequent subsections. \subsection[The read driver: read_exercise() and read_metainfo()]{The \code{read} driver: \code{read\_exercise()} and \code{read\_metainfo()}} The function \code{read_exercise()} reads the weaved exercises, i.e., files like that shown in Figure~\ref{fig:tex}. It simply extracts the text lines from the \code{{question}} and \code{{solution}} environments and stores them in vectors of the same name. If these environments contain \code{{answerlist}} environments, these are extracted and stored separately in \code{questionlist} and \code{solutionlist} vectors, respectively. Finally, the metainformation is extracted using \code{read_metainfo()} which not only stores character vectors but also transforms them to suitable types (depending on the \code{extype}) and performs some sanity checks. The resulting metainformation is a list with elements essentially corresponding to the commands from Table~\ref{tab:metainfo}. For illustration, we run \code{xexams()} to select the same three exams as used in the \pkg{Moodle} and \pkg{OLAT} examples above. However, using the default \code{driver} specification, \code{xexams()} just performs the weaving and reading steps (and has no transformation or writing step): <>= set.seed(1090) x <- xexams(myexam, n = 3) @ The resulting object is a nested list as shown in Figure~\ref{fig:xexam} with \Sexpr{length(x)}~exams of \Sexpr{length(x[[1]])}~exercises each (drawn from the \code{myexam} list). Using \code{x[[i]][[j]]}, the \code{j}-th exercise of the \code{i}-th exam can be accessed. Here, we explore the first exercise (\code{boxplots}, a multiple-choice question) from the first exam that is also shown in Figures~\ref{fig:exams2moodle-boxplots} and~\ref{fig:exams2qti12-boxplots}. Its general question text (in {\LaTeX}) is printed below -- it requires a graphic which is stored in a supplementary file in a temporary directory. <>= writeLines(x[[1]][[1]]$question) x[[1]][[1]]$supplements @ The corresponding list of statements about the graphic is stored separately. It is shown below along with the most important metainformation elements. <>= x[[1]][[1]]$questionlist x[[1]][[1]]$metainfo[c("file", "type", "solution")] @ In summary, \code{xexams()} combined with the default readers is relatively straightforward to use in other progams (such as the \code{exams2}\emph{xyz} functions). The return value is somewhat ``raw'' as it is not classed and has no dedicated methods for subsetting etc. However, we refrained from using a more elaborate structure as this function is not meant to be called by end-users while we expected other developers to find the current structure sufficiently simple to use in their programs. \subsection[LaTeX-to-HTML transform driver generator]{{\LaTeX}-to-HTML \code{transform} driver generator} \label{sec:transform_html} When embedding statistical/mathematical exercises into web pages or learning management systems, the exercises' {\LaTeX} text -- typically containing mathematical notation -- has to be transformed in some way so that it can be rendered by a browser. Until relatively recently, this posed the notorious problem of how to display the mathematical formulas and often the only good answer was to embed raster images of the formulas. However, this situation has clearly changed \citep[see e.g.,][]{exams:Vismor:2012} and there are now various convenient options: e.g., using the mathematical markup language MathML \citep{exams:W3C+MathML:2014,exams:Wiki+MathML:2014} or keeping {\LaTeX} formulas in the web page and embedding some \proglang{JavaScript} for rendering them. Especially the display of MathML in web pages has become very easy: \pkg{Firefox} long had native support for it and for the Microsoft \pkg{Internet Explorer} the \pkg{MathPlayer} plugin \citep{exams:MathPlayer:2013} has long been available. More recently, other major browsers like \pkg{Opera} or \pkg{Safari} also added support for MathML \citep[see][Section~1.2]{exams:Vismor:2012}. Google \pkg{Chrome} briefly enabled MathML support but disabled it again due to instabilities. Furthermore, \pkg{MathJax} \citep{exams:MathJax:2013}, an open-source \proglang{JavaScript} engine, can be used to render MathML (or {\LaTeX}) formulas on a server rather than in the local browser. Therefore, the new \pkg{exams} package offers functionality for automatically transforming the {\LaTeX} exercises to HTML within \proglang{R} and by default employs MathML for all mathematical notation (e.g., as demonstrated in Figure~\ref{fig:exams2html-tstat}). More specifically, the package provides the driver generator \code{make_exercise_transform_html()}. It returns a function suitable for plug-in into the \code{transform} driver in \code{xexams()} which then replaces the {\LaTeX} code in \code{question}/\code{questionlist} and \code{solution}/\code{solutionlist} with HTML code. For illustration, we set up a particular function \code{trafo()} below and apply it to the first exercise in the first exam within the object \code{x} that we had considered before: <>= trafo <- make_exercise_transform_html(converter = "ttm", base64 = FALSE) writeLines(trafo(x[[1]][[1]])$question) @ It can be seen that the resulting exercise employs HTML text, e.g., uses \code{} instead of \verb|\emph| or \code{} instead of \verb|\includegraphics|.\footnote{It may be noteworthy to that the conversion (a)~assumes the graphics to be in \texttt{.png} format and (b)~does not resolve the figure reference at the beginning of the text correctly. For (a), we just need to make sure that the \texttt{sweave} driver in \texttt{xexams()} has \texttt{png = TRUE} (and \texttt{pdf = FALSE}) which is accounted for in \texttt{exams2html()} etc. Issue (b), however, needs to be avoided by formulating the underlying \texttt{.Rnw} differently (or by tolerating the missing number).} Internally, \code{make_exercise_transform_html()} can leverage three different \code{converter}s: \code{ttm} (default), \code{tth}, or \code{tex2image}. The former two come from the \proglang{R} package \pkg{tth} \citep{exams:Hutchinson+Leisch+Zeileis:2013} and internally call the two \proglang{C}~functions \code{tth} ({\TeX} to HTML) and \code{ttm} ({\TeX} to HTML/MathML) taken from the \pkg{TtH} suite of \cite{exams:Hutchinson:2012}. The last option, \code{tex2image}, is a function provided by the \pkg{exams} package itself. It proceeds by first running \code{texi2dvi(pdf = TRUE)} from the base \proglang{R} package \pkg{tools} and subsequently converting the resulting PDF to a raster image in a \code{system()} call to \pkg{ImageMagick}'s \code{convert} function \citep{exams:ImageMagick:2014}. Thus, for this function \pkg{ImageMagick} is assumed to be installed and in the search path. All three converters have their benefits and drawbacks: \begin{itemize} \item \code{tth} is typically preferable if there is no or only very simple mathematical notation. The resulting HTML can then be rendered in any modern browser. \item \code{ttm} is preferable if there is some standard mathematical notation (e.g., fractions or equation arrays etc.). As argued above this can still be easily displayed in suitable browsers or by employing \pkg{MathJax} in the web page. \item \code{tex2image} is the ``last resort'' if neither of the two previous approaches work, e.g., if more complex {\LaTeX} commands/packages need to be used which are not supported by \code{tth}/\code{ttm}. It is fairly slow while \code{tth}/\code{ttm} are typically even faster than calling {\LaTeX}. \end{itemize} To explore the differences of the results, the converters can also be called directly on character strings containing {\LaTeX}. Below we use two simple code lines for which \code{tth()} would probably be sufficient: <>= tex <- c("This is \\textbf{bold} and this \\textit{italic}.", "Points on the unit circle: $x^2 + y^2 = 1$.") ttm(tex) tth(tex) @ %% avoid evaluation of tex2image() in order to have no system dependency on ImageMagick <>= (tex2image(tex, dir = odir, show = FALSE)) @ <>= t2i <- file.path(odir, "tex2image_1.png") foo <- file.create(t2i) print(t2i) @ Note that \code{tex2image(tex)} returns the path to a raster image file which by default is also shown directly in the browser. Finally, our illustration of \code{make_exercise_transform_html()} also employed a second option, \code{base64 = FALSE}, which still deserves more detailed explanation. After converting an exercise from {\LaTeX} to HTML code (using either of the three converters above), the HTML code may contain references to supplementary files (e.g., in \code{} tags). Optionally, by using the default \code{base64 = TRUE}, these images can be embedded directly into the HTML code in Base64 encoding \citep[via the \pkg{base64enc} package in \proglang{R},][]{exams:Urbanek:2012} and thus waiving the need for having them as supplementary files. \subsection[PDF and HTML write driver generators]{PDF and HTML \code{write} driver generators} In the first three steps of \code{xexams()}, exams are randomly drawn and weaved, read into \proglang{R}, and potentially transformed from {\LaTeX} to HTML (or some other format). However, so far, no output files have been generated. The original idea of \cite{exams:Gruen+Zeileis:2009} was to produce one or more output files for each of the \code{n} generated exams. To do so in \code{xexams()} a \code{write} driver can be specified. The package provides several generating functions for suitable drivers, especially for generating PDF and HTML files. As before, the idea is to pass customization options to the driver generator which can then be plugged into \code{xexams()}. For PDF output files, the following driver generator is available: \begin{Code} make_exams_write_pdf(template = "plain", name = NULL, inputs = NULL, header = list(Date = Sys.Date()), quiet = TRUE, control = NULL) \end{Code} This is employed in \code{exams2pdf()} and proceeds in the same way as described by \cite{exams:Gruen+Zeileis:2009} for the \code{exams()} function. It includes the \code{question}/\code{questionlist} and \code{solution}/\code{solutionlist} in a {\LaTeX} \code{template}, then runs \code{texi2dvi(pdf = TRUE)} from the base \pkg{tools} package, and finally copies the resulting PDF files to a desired output directory. The default \code{plain.tex} template is provided within the \pkg{exams} package and also more than one template can be employed as illustrated in Section~\ref{sec:usage}. Details about the remaining customization arguments are provided on the manual page and in \cite{exams:Gruen+Zeileis:2009}. \begin{figure}[t!] \hrulefill \begin{Code} Exam ##ID##

Exam ##ID##

##\exinput{exercises}## \end{Code} \hrulefill \caption{\label{fig:plain.html} Default HTML \code{template} file \texttt{plain.html} employed in \texttt{make\_exams\_write\_html()}. Elements marked by \texttt{\#\#...\#\#} are being replaced in each replication of the exam.} \end{figure} For HTML output files, a similar driver generator is available: \begin{Code} make_exams_write_html(template = "plain", name = NULL, question = "

Question

", solution = "

Solution

", mathjax = FALSE) \end{Code} This is employed in \code{exams2html()} and is also based on a \code{template}. By default the \code{plain.html} file is used that is provided within \pkg{exams} and shown in Figure~\ref{fig:plain.html}. This contains placeholders marked with \verb|##...##| that are to be replaced in each randomly drawn exam. The \verb|##ID##| is simply replaced with a numeric ID ($1, \dots, \mathtt{n}$) and \verb|##\exinput{exercises}##| is replaced by an ordered list (\code{
    }) containing the question/solution. If the \code{question} and \code{solution} arguments to \code{make_exams_write_html()} are character strings, these are added as titles in the list. Alternatively, either argument can also be set to \code{FALSE} which avoids inclusion of the corresponding element of the exercise in the resulting HTML file. As an additional convenience setting \code{mathjax = TRUE} includes the \code{