{0 How to Drive [odoc]}

This 'live' document describes how to use [odoc] to produce the documentation of [odoc] itself. The aim is
to show a short, simple example of how [odoc] can be used, covering most of the important features.
The document built here includes not only the documentation of [odoc] itself, but it also builds the
docs for a subset of [odoc]'s dependent libraries to show how this may be done. For a much more
complete and comprehensive use of [odoc], see the {{: https://github.com/ocaml-doc/voodoo} Voodoo project}, the tool that is being used to build
the package docs for
{{: https://ocaml.org/packages} ocaml.org/packages}. The information in this page is specific to
odoc version 2.3 or later. For earlier
versions see the [driver.md] or [driver.mld] files in the corresponding source distribution.

First, we need to initialise MDX with some libraries and helpful values:

{[
(* Prelude *)
#require "bos";;
#install_printer Fpath.pp;;
#print_length 655360;;
#print_depth 10;;
open Bos;;
let (>>=) = Result.bind;;
let (>>|=) m f = m >>= fun x -> Ok (f x);;
let get_ok = function | Ok x -> x | Error (`Msg m) -> failwith m
let relativize p = Fpath.(v ".." // p) (* this driver is run from the [doc] dir *)

(* Whether to instrument with landmarks. Result for each commands will be saved
   to directory [_build/default/doc/landmarks]. *)
let instrument = false
]}

{1 Desired Output}

[odoc] produces output files (HTML or others) in a structured directory tree, so before running [odoc], the structure of the output must be decided. For these docs, we want the following structure:
{ul {- [odoc/index.html] : main page
}{- [odoc/{odoc_for_authors.html,...}] : other documentation pages
}{- [odoc/odoc_model/index.html] : [odoc] model library subpage
}{- [odoc/odoc_model/Odoc_model/index.html] : Module page for the module [Odoc_model]
}{- [odoc/odoc_model/Odoc_model/...] : Further pages for the submodules of [Odoc_model]
}{- [odoc/odoc_.../index.html] : other [odoc] library pages
}{- [odoc/deps/stdlib/index.html] : stdlib main page
}{- [odoc/deps/stdlib/Stdlib/index.html] : Module page for the module [Stdlib]
}{- [odoc/deps/astring/index.html] : astring main page
}{- [odoc/deps/...] : other dependencies
}{- [odoc/source/...] : rendered source files
}}

The [odoc] model for achieving this is that we have {e pages} ([.mld] files) that have {e children} which are either {e further pages} ([.mld] files), {e modules} (from [.cmti] files), or
a {e source parent}. This {{!page-parent_child_spec}parent/child relationship} is specified on the command line. Parent pages must be {e compiled} by [odoc] before their children. Then compiling a page [mypage.mld] will produce the file [page-mypage.odoc].

In the example below, there will be a file [odoc.mld] that corresponds with the top-level directory [odoc/]. It will be compiled as follows:

{@sh skip[
odoc compile odoc.mld --child page-odoc_model --child deps 
  --child src-source ...
]}

The file [deps.mld] which corresponds with the sub-directory [odoc/deps/], will be compiled as follows:

{@sh skip[
odoc compile deps.mld -I . --parent page-odoc --child page-stdlib --child page-astring ...
]}

The file [odoc_model.mld] will have a child module [Odoc_model]. It will be compiled as follows:

{@sh skip[
odoc compile odoc_model.mld -I . --parent page-odoc --child module-Odoc_model
]}

The last type of page contains a list of paths to the source files that should be rendered as HTML. The output will be found as a tree underneath this page. This will be compiled in the following way:

{@sh skip[
odoc source-tree source.map -I . --parent page-odoc
]}

where the first few lines of [source.map] are:

{@text[
src/xref2/utils.ml
src/xref2/type_of.ml
src/xref2/tools.ml
]}

indicating the desire for the rendered source of [utils.ml] to be found as the file [odoc/source/src/xref2/utils.ml.html].

When compiling any [.mld] file, the parent and all children must be specified. Parents can only be pages from other [.mld] files, and children may be pages (from [.mld] files) or modules (from [.cmti]/[.cmt] or [.cmi] files).

The parent page must exist before the child page is created, and it must have had the child specified when it was initially compiled.

{1 Document Generation Phases}

Using [odoc] is a three-phase process:
{ol {- Compilation: odoc compile
}}

This takes as input either [.mld] files containing pure odoc markup, or the output from the compiler in the form of [.cmti], [.cmt], or [.cmi] files (in order of preference). For [.mld] files, this step simply translates them into [odoc]'s internal format and writes the corresponding file. For example, given the input [foobar.mld], [odoc] will output [page-foobar.odoc]. There are no dependencies for compiling [.mld] files beyond the parent as outlined above.

For modules, compilation is the point where [odoc] performs some initial expansion and resolution operations, a process that usually introduces dependencies. For a given input [/path/to/file.cmti] it will output the file [/path/to/file.odoc] unless the [-o] option is used to override the output file. If there were [.cmi] dependencies required for OCaml to compile a particular module, then there will be equivalent [.odoc] dependencies needed for the [odoc compile] step. [odoc] will search for these dependencies in the paths specified with the [-I] directive on compilation. [odoc] provides a command to help with this: [odoc compile-deps].

As an example we can run [odoc compile-deps] on the file [../src/xref2/.odoc_xref2.objs/byte/odoc_xref2__Compile.cmti]:

{@text[
$ `odoc` compile-deps ../src/xref2/.odoc_xref2.objs/byte/odoc_xref2__Compile.cmti | tail -n 5
Stdlib__result 2ba42445465981713146b97d5e185dd5
Stdlib__seq d6a8de25c9eecf5ae9420a9f3f8b2e88
Stdlib__set 5d365647a10f75c22f2b045a867b4d3e
Stdlib__uchar ab6f1df93abf9e800a3e0d1543523c96
Odoc_xref2__Compile e0d620d652a724705f7ed620dfe07be0
]}

From this, we see it's necessary to run [odoc compile] against several [Stdlib] modules before we can compile [odoc_xref2__Compile.cmti]
{ol {- Linking: odoc link
}}

This takes the [odoc] files produced during the compilation step and performs the final steps of resolution for both pages and modules, and expansion for modules only. It is during this phase that all the references in the documentation comments are resolved. In order for these to be resolved, everything that is referenced must have been compiled already, and their [odoc] files must be on the
include path as specified by the [-I] arguments to [odoc link]. In this example, we achieve that by compiling all modules and [.mld] files before linking anything. The output of the
link step is an [odocl] file, which is in the same path as the original [odoc] file by default.

Please note: it's only necessary to link the non-hidden modules (i.e., without a double underscore).
{ol {- Generation: odoc html-generate
}}

Once the compile and link phases are complete, the resulting [odocl] files may be rendered in a variety of formats. In this example we output HTML.

{1 [odoc] Documentation}

In this section [odoc] is used to generate the documentation of [odoc] and some of its dependent packages. We can make a few simplifying assumptions here:
{ol {- Since we're working with one leaf package, we can assume that there can be no module name clashes in the dependencies. As such, we can afford to put all of our [.odoc] files into one directory and then hard-code the include path to be this directory. When using [odoc] in a context where there may be module name clashes, it requires more careful partitioning of output directories.
}{- We'll do all of the compiling before any linking.
}}

Let's start with some functions to execute the three phases of [odoc].

Compiling a file with [odoc] requires a few arguments: the file to compile, an
optional parent, a list of include paths, a list of children for [.mld] files,
optional parent and name for source implementation, and an output path. Include
paths can be just ['.'], and we can calculate the output file from the input
because all of the files are going into the same directory.

Linking a file with [odoc] requires the input file and a list of include paths. As
for compile, we will hard-code the include path.

Generating the HTML requires the input [odocl] file, an optional implementation
source file (passed via the [--source] argument), and an output path. We will
hard-code the output path to be [html/].

Using the [--source] argument with an [.odocl] file that was not compiled with
[--source-parent-file] and [--source-name] will result in an error, as will omitting [--source] when generating HTML of an [odocl] that was
compiled with [--source-parent-file] and [--source-name].

In all of these, we'll capture [stdout] and [stderr] so we can check it later.

{[
let odoc = Cmd.v "../src/odoc/bin/main.exe" (* This is the just-built odoc binary *)

let compile_output = ref [ "" ]

let link_output = ref [ "" ]

let generate_output = ref [ "" ]

type executed_command = {
  cmd : Cmd.t;
  time : float; (** Running time in seconds. *)
  output_file : Fpath.t option;
}

(* Record the commands executed, their running time and optionally the path to
   the produced file. *)
let commands = ref [ ]

let instrument_dir =
  lazy (
    let dir = Fpath.v "landmarks" in
    OS.Dir.delete dir |> get_ok;
    OS.Dir.create dir |> get_ok |> ignore;
    dir
  )

(* Environment variables passed to commands. *)
let env = OS.Env.current () |> get_ok

let run ?output_file cmd =
  let t_start = Unix.gettimeofday () in
  let env =
    if instrument then
      let lazy instrument_dir = instrument_dir in
      let instrument_out =
        match output_file with
        | Some outf ->
            Fpath.(/) instrument_dir (Fpath.basename outf ^ ".json")
            |> Fpath.to_string
        | None -> "temporary:" ^ Fpath.to_string instrument_dir
      in
      Astring.String.Map.add "OCAML_LANDMARKS"
        ("time,allocation,format=json,output=" ^ instrument_out)
        env
    else env
  in
  let r = OS.Cmd.(run_out ~env ~err:OS.Cmd.err_run_out cmd |> to_lines) |> get_ok in
  let t_end = Unix.gettimeofday () in
  let time = t_end -. t_start in
  commands := { cmd; time; output_file } :: !commands;
  r

let add_prefixed_output cmd list prefix lines =
  if List.length lines > 0 then
    list :=
      !list
      @ Bos.Cmd.to_string cmd :: List.map (fun l -> prefix ^ ": " ^ l) lines

let compile file ?parent ?(output_dir = Fpath.v "./")
    ?(ignore_output = false) ?source_args children =
  let output_basename =
    let ext = Fpath.get_ext file in
    let basename = Fpath.basename (Fpath.rem_ext file) in
    match ext with
    | ".mld" -> "page-" ^ basename ^ ".odoc"
    | ".cmt" | ".cmti" | ".cmi" -> basename ^ ".odoc"
    | _ -> failwith ("bad extension: " ^ ext)
  in
  let output_file = Fpath.(/) output_dir output_basename in
  let open Cmd in
  let source_args =
    match source_args with
    | None -> Cmd.empty
    | Some (source_name, source_parent_file) ->
        Cmd.(
          v "--source-name" % p source_name % "--source-parent-file"
          % p source_parent_file)
  in
  let cmt_arg =
    let cmt_file = Fpath.set_ext ".cmt" file in
    if Fpath.get_ext file = ".cmti" then
    match Bos.OS.File.exists cmt_file with
    | Ok true -> Cmd.(v "--cmt" % p cmt_file)
    | _ -> Cmd.empty
    else Cmd.empty
  in
  let cmd =
    odoc % "compile" % Fpath.to_string file %% source_args %% cmt_arg
    % "-I" % "." % "-o" % p output_file
    |> List.fold_right (fun child cmd -> cmd % "--child" % child) children
  in
  let cmd =
    match parent with
    | Some p -> cmd % "--parent" % ("page-\"" ^ p ^ "\"")
    | None -> cmd
  in
  let lines = run ~output_file cmd in
  if not ignore_output then
    add_prefixed_output cmd compile_output (Fpath.to_string file) lines

let link ?(ignore_output = false) file =
  let open Cmd in
  let output_file = Fpath.set_ext "odocl" file in
  let cmd = odoc % "link" % p file % "-o" % p output_file % "-I" % "." in
  let cmd = if Fpath.to_string file = "stdlib.odoc" then cmd % "--open=\"\"" else cmd in
  let lines = run ~output_file cmd in
  if not ignore_output then
    add_prefixed_output cmd link_output (Fpath.to_string file) lines

let html_generate ?(ignore_output = false) ?(assets = []) ?(search_uris = []) file source =
  let open Cmd in
  let source =
    match source with None -> empty | Some source -> v "--source" % p source
  in
  let assets =
    List.fold_left
      (fun acc filename -> acc % "--asset" % filename)
      empty
      assets
  in
  let search_uris =
    List.fold_left
      (fun acc filename -> acc % "--search-uri" % p filename)
      empty
      search_uris
  in
  let cmd =
    odoc % "html-generate" %% source % p file %% assets %% search_uris % "-o" % "html"
    % "--theme-uri" % "odoc" % "--support-uri" % "odoc"
  in
  let lines = run cmd in
  if not ignore_output then
    add_prefixed_output cmd generate_output (Fpath.to_string file) lines

let support_files () =
  let open Cmd in
  let cmd = odoc % "support-files" % "-o" % "html/odoc" in
  run cmd
]}


We'll now make some library lists. We have not only external dependency
libraries, but [odoc] itself is also separated into libraries. These two
sets of libraries will be documented in different sections, so we'll keep them
in separate lists. Moreover, [odoc] libraries will include the source code, via
a hardcoded path.

Additionally we'll also construct a list containing the extra documentation pages. Finally let's create a list mapping the section to its parent, which matches
the hierarchy declared above.

{[
let dep_libraries_core = [
    "odoc-parser";
    "astring";
    "cmdliner";
    "fpath";
    "result";
    "tyxml";
    "fmt";
    "stdlib";
    "yojson";
];;

let extra_deps = [
    "base";
    "core_kernel";
    "bin_prot";
    "sexplib";
    "sexplib0";
    "base_quickcheck";
    "ppx_sexp_conv";
    "ppx_hash";
    "core";
]

let dep_libraries =
    match Sys.getenv_opt "ODOC_BENCHMARK" with
    | Some "true" -> dep_libraries_core @ extra_deps
    | _ -> dep_libraries_core

let odoc_libraries = [
    "odoc_xref_test"; "odoc_xref2"; "odoc_odoc"; "odoc_html_support_files";
    "odoc_model_desc"; "odoc_model"; "odoc_manpage"; "odoc_loader";
    "odoc_latex"; "odoc_html"; "odoc_document"; "odoc_examples"; "odoc_parser";
    "ocamlary"; "odoc_search" ; "odoc_html_frontend" ; "odoc_json_index" ];;

let all_libraries = dep_libraries @ odoc_libraries;;

let extra_docs = [
    "interface";
    "driver";
    "parent_child_spec";
    "features";
    "odoc_for_authors";
    "cheatsheet";
    "dune";
    "ocamldoc_differences";
    "api_reference";
]

let parents =
    let add_parent p l = List.map (fun lib -> (lib, p)) l in
    (add_parent "deps" dep_libraries) @ (add_parent "odoc" odoc_libraries);;

]}

[odoc] operates on the compiler outputs. We need to find them for both the files compiled by Dune within this project and those in libraries we compile against.
The following uses [ocamlfind] to locate the library paths for our dependencies. Since [ocamlfind] gives
us the absolute path, we also have a short function here to relativize it based on our current working
directory to ensure the log of commands we collect is as reproducible as possible.

{[
let ocamlfind = Cmd.v "ocamlfind"

let reach t ~from =
  let rec loop t from =
    match (t, from) with
    | a :: t, b :: from when a = b -> loop t from
    | _ -> List.fold_right (fun _ acc -> ".." :: acc) from t
  in
  let v s = String.split_on_char '/' s in
  loop (v t) (v from) |> String.concat "/"

let relativize_path =
  let pwd = Sys.getcwd () in
  fun p -> reach p ~from:pwd

let lib_path lib =
  let cmd = Cmd.(ocamlfind % "query" % lib) in
  run cmd |> List.hd |> relativize_path

let lib_paths =
  List.fold_right
    (fun lib acc ->
      (lib, lib_path lib) :: acc)
    dep_libraries []
]}

We need a function to find [odoc] inputs from the given search path. [odoc]
operates on [.cmti], [.cmt], or [.cmi] files, in order of preference, and the following
function finds all matching files starting from the given path. Then it returns an [Fpath.Set.t]
that contains the [Fpath.t] values representing the absolute file path, without its extension.

{[
let find_units p =
  OS.Dir.fold_contents ~dotfiles:true
    (fun p acc ->
      if List.exists (fun ext -> Fpath.has_ext ext p) [ "cmt"; "cmti"; "cmi" ]
      then p :: acc
      else acc)
    [] (Fpath.v p)
  >>|= fun paths ->
  let l = List.map Fpath.rem_ext paths in
  let l =
    List.filter
      (fun f ->
        not @@ Astring.String.is_infix ~affix:"ocamldoc" (Fpath.to_string f))
      l
  in
  List.fold_right Fpath.Set.add l Fpath.Set.empty;;
]}

Since the units returned by this function have their extension stripped, we need
function to find the best file to use with this basename.

{[
let best_file base =
  List.map (fun ext -> Fpath.add_ext ext base) [ "cmti"; "cmt"; "cmi" ]
  |> List.find (fun f -> Bos.OS.File.exists f |> get_ok)
]}

Many of the units will be 'hidden', meaning that Dune will mangle their name
in order to namespace them. This is achieved by prefixing the namespace module and
a double underscore, so we can tell by the existence of a double underscore that
a module is intended to be hidden. The following predicate tests for that condition:

{[
let is_hidden path = Astring.String.is_infix ~affix:"__" (Fpath.to_string path)
]}

To build the documentation, we start with these files. With the following function, we'll call [odoc compile-deps] on the file to
find all other compilation units upon which it depends:

{[
type compile_deps = { digest : Digest.t; deps : (string * Digest.t) list }

let compile_deps f =
  let cmd = Cmd.(odoc % "compile-deps" % Fpath.to_string f) in
  let deps = run cmd in
  let l = List.filter_map (Astring.String.cut ~sep:" ") deps in
  let basename = Fpath.(basename (f |> rem_ext)) |> String.capitalize_ascii in
  match List.partition (fun (n, _) -> basename = n) l with
  | [ (_, digest) ], deps -> Ok { digest; deps }
  | _ -> Error (`Msg "odd")
]}

For each compiled odoc file, we'll need to remember some options given at
[odoc compile]-time. An example of this is the source code rendering: when we
enable the feature at compile time, we need to provide the source file at html
generation.

{[
type unit = {
  file : Fpath.t;
  ignore_output : bool;
  source : Fpath.t option;
  assets : string list;
}
]}

For [odoc] libraries, we infer the implementation and interface source file path
from the library name. We list them in a file, passed to [odoc source-tree], to
generate [src-source.odoc]. This file contains the source hierarchy, and will be
linked and passed to [html-generate] just as other pages and compilation units.

It is used as the [source-parent] for all units for which we could provide
sources.

{[
let source_tree_output = ref [ "" ]

let source_tree ?(ignore_output = false) ~parent ~output file =
  let open Cmd in
  let parent = v "--parent" % ("page-\"" ^ parent ^ "\"") in
  let cmd = odoc % "source-tree" % "-I" % "." %% parent % "-o" % p output % p file in
  let lines = run cmd in
  if not ignore_output then
    add_prefixed_output cmd source_tree_output (Fpath.to_string file) lines

let odoc_source_tree = Fpath.v "src-source.odoc"

let source_dir_of_odoc_lib lib =
  match String.split_on_char '_' lib with
  | "odoc" :: s ->
      let libname = Fpath.(v (String.concat "_" s)) in
      Some Fpath.(v "src" // libname)
  | _ -> None

let source_files_of_odoc_module lib module_ =
  let filename =
    let module_ =
      match Astring.String.cut ~rev:true ~sep:"__" module_ with
      | None -> module_
      | Some (_, "") -> module_
      | Some (_, module_) -> module_
    in
    (* ML.ml should not be renamed *)
    if String.for_all (fun c -> Char.equal (Char.uppercase_ascii c) c) module_
    then module_
    else String.uncapitalize_ascii module_
  in
  match source_dir_of_odoc_lib lib with
  | None -> None
  | Some relpath ->
      let add_filename path ext =
        Fpath.( / ) path filename |> Fpath.add_ext ext
      in
      let find_by_extension path exts =
        exts
        |> List.map (fun ext -> add_filename path ext)
        |> List.find_opt (fun f -> Bos.OS.File.exists (relativize f) |> get_ok)
      in
      find_by_extension relpath [ "pp.ml"; "ml" ]

let compile_source_tree units =
  let sources =
    List.filter_map (fun (_, _, _, file) -> Option.map Fpath.to_string file) units
  in
  let source_map = Fpath.v "source.map" in
  let () = Bos.OS.File.write_lines source_map sources |> get_ok in
  let () = source_tree ~parent:"odoc" ~output:odoc_source_tree source_map in
  { file = odoc_source_tree ; ignore_output = false ; source = None ; assets = [] }

]}

Let's now put together a list of all possible modules. We'll keep track of
which library they're in, and whether that library is a part of [odoc] or a dependency
library.

{[
let odoc_all_unit_paths = find_units ".." |> get_ok

let odoc_units =
  List.map
    (fun lib ->
      Fpath.Set.fold
        (fun p acc ->
          if Astring.String.is_infix ~affix:lib (Fpath.to_string p) then
            let impl =
              let module_ = Fpath.basename p in
              source_files_of_odoc_module lib module_
            in
            ("odoc", lib, p, impl) :: acc
          else acc)
        odoc_all_unit_paths [])
    odoc_libraries
]}

{[
let all_units =
  let lib_units =
    List.map
      (fun (lib, p) ->
        Fpath.Set.fold
          (fun p acc -> ("deps", lib, p, None) :: acc)
          (find_units p |> get_ok)
          [])
      lib_paths in
  odoc_units @ lib_units |> List.flatten
]}

Generate the {!api_reference} page to list Odoc's libraries:

{[
let update_api_reference_page () =
  let libs =
    List.sort String.compare odoc_libraries
    |> List.map String.capitalize_ascii
  in
  OS.File.with_oc (Fpath.v "api_reference.mld") (fun oc () ->
      let pf = Printf.fprintf in
      pf oc "{0 API Reference}\n\n";
      List.iter (pf oc "- {!%s}\n") libs;
      Ok ()
    ) ()
  |> get_ok
  |> get_ok
]}

Now we'll compile all of the parent [.mld] files. To ensure that the parents are compiled before the children, we start with [odoc.mld], then [deps.mld], and so on. The result of this file is a list of the resulting [odoc] files.

{[
let search_file = "index.js"

let compile_mlds () =
  update_api_reference_page ();
  let mkpage x = "page-\"" ^ x ^ "\"" in
  let mkmod x = "module-" ^ String.capitalize_ascii x in
  let mkmld x = Fpath.(add_ext "mld" (v x)) in
  ignore
    (compile (mkmld "odoc")
       ("src-source" :: "page-deps" :: List.map mkpage (odoc_libraries @ extra_docs)));
  ignore (compile (mkmld "deps") ~parent:"odoc" (List.map mkpage dep_libraries));
  let extra_odocs =
    List.map
      (fun p ->
        ignore (compile (mkmld p) ~parent:"odoc" []);
        "page-" ^ p ^ ".odoc")
      extra_docs
  in
  let odocs =
    List.map
      (fun library ->
        let parent = List.assoc library parents in
        let children =
          List.filter_map
            (fun (parent, lib, child, _) ->
              if lib = library then Some (Fpath.basename child |> mkmod)
              else None)
            all_units
        in
        ignore (compile (mkmld ("library_mlds/"^library)) ~parent children);
        "page-" ^ library ^ ".odoc")
      all_libraries
  in
  { file = Fpath.v "page-odoc.odoc" ; ignore_output = false ; source = None ; assets = [] } ::
  List.map
    (fun f -> { file = Fpath.v f ; ignore_output = false ; source = None; assets = [] })
    ( "page-deps.odoc" :: odocs @ extra_odocs)
]}

Now we get to the compilation phase. For each unit, we query its dependencies, then recursively call to compile these dependencies. Once this is done we compile the unit itself. If the unit has already been compiled we don't do anything. Note that we aren't checking the hashes of the dependencies which a build system should do to ensure that the module being compiled is the correct one. Again we benefit from the fact that we're creating the docs for one leaf package and that there must be no module name clashes in its dependencies. The result of this function is a list of the resulting [odoc] files.

{[
let compile_all () =
  let mld_odocs = compile_mlds () in
  let source_tree = compile_source_tree all_units in
  let source_args =
    Option.map (fun source_relpath -> (source_relpath, odoc_source_tree))
  in
  let rec rec_compile ?impl parent lib file =
    let output = Fpath.(base (set_ext "odoc" file)) in
    if OS.File.exists output |> get_ok then []
    else
      let deps = compile_deps file |> get_ok in
      let files =
        List.fold_left
          (fun acc (dep_name, digest) ->
            match
              List.find_opt
                (fun (_, _, f, _) ->
                  Fpath.basename f |> String.capitalize_ascii = dep_name)
                all_units
            with
            | None -> acc
            | Some (parent, lib, dep_path, impl) ->
                let file = best_file dep_path in
                rec_compile ?impl parent lib file @ acc)
          [] deps.deps
      in
      let ignore_output = parent = "deps" in
      let source_args = source_args impl in
      compile file ~parent:lib ?source_args ~ignore_output [];
      { file = output ; ignore_output ; source = impl; assets = [] } :: files
  in
  source_tree
  :: List.fold_left
    (fun acc (parent, lib, dep, impl) ->
      acc @ rec_compile ?impl parent lib (best_file dep))
      [] all_units
   @ mld_odocs
]}

Linking is now straightforward. We link all [odoc] files.

{[
let link_all odoc_files =
  List.map
    (fun ({ file = odoc_file ; ignore_output ; _ } as unit) ->
      ignore (link ~ignore_output odoc_file);
      { unit with file = Fpath.set_ext "odocl" odoc_file })
    odoc_files
]}

Now we simply run [odoc html-generate] over all of the resulting [odocl] files.
This will generate sources, as well as documentation for non-hidden units.
We notify the generator that the javascript file to use for search is [index.js].

{[
let generate_all odocl_files =
  let relativize_opt = function None -> None | Some file -> Some (relativize file) in
  let search_uris = [Fpath.v "minisearch.js"; Fpath.v "index.js"] in
  List.iter
     (fun ({file = f ; ignore_output = _ ; source ; assets}) ->
     ignore(html_generate ~assets ~search_uris f (relativize_opt source)))
     odocl_files;
  support_files ()
]}


Finally, we generate an index of all values, types, ... This index is meant to be consumed by search engines, to create their own index. It consists of a JSON array, containing entries with the name, full name, associated comment, link and anchor, and kind. Generating the index is done via [odoc compile-index], which create a json file.

Search engines written in OCaml can also call the [Odoc_model.Fold.unit] and  [Odoc_model.Fold.page] function, in conjunction with [Odoc_search.Entry.entry_of_item] in order to get an OCaml value of each element to be indexed.

{[
let index_generate ?(ignore_output = false) () =
  let open Cmd in
  let files =
    OS.Dir.contents (Fpath.v ".")
    |> get_ok
    |> List.filter (Fpath.has_ext "odocl")
    |> List.filter (fun p -> not (String.equal "src-source.odocl" (Fpath.filename p)))
    |> List.filter (fun p -> not (is_hidden p))
    |> List.map Fpath.to_string
  in
  let index_map = Fpath.v "index.map" in
  let () = Bos.OS.File.write_lines index_map files |> get_ok in
  let cmd =
    odoc % "compile-index" % "-o" % "html/index.json" % "--file-list"
    % p index_map
  in
  let lines = run cmd in
  if not ignore_output then
    add_prefixed_output cmd generate_output "index compilation" lines
]}

We turn the JSON index into a javascript file. In order to never block the UI, this file will be used as a web worker by [odoc], to perform searches:

- The search query will be sent as a plain string to the web worker, using the standard mechanism of message passing
- The web worker has to sent back the result as a message to the main thread, containing the list of result. Each entry of this list must have the same form as it had in the original JSON file.
- The file must be given to the [odoc-support] URI.

In this driver, we use the minisearch javascript library. For more involved application, we could use [index.js] to call a server-side search engine via an API call.

{[
  let js_index () =
    let index = Bos.OS.File.read Fpath.(v "html" / "index.json") |> get_ok in
 Bos.OS.File.writef (Fpath.v search_file) {|
 let documents =
   %s
 ;

 let miniSearch = new MiniSearch({
  fields: ['id', 'doc', 'entry_id'], // fields to index for full-text search
   storeFields: ['display'], // fields to return with search results
   idField: 'entry_id',
   extractField: (document, fieldName) => {
     if (fieldName === 'id') {
       return document.id.map(e => e.kind + "-" + e.name).join('.')
     }
     return document[fieldName]
   }
 })


 // Use a unique id since some entries' id are not unique (type extension or
 // standalone doc comments for instance)
 documents.forEach((entry,i) => entry.entry_id = i)
 miniSearch.addAll(documents);

 onmessage = (m) => {
   let query = m.data;
   let result = miniSearch.search(query);
   postMessage(result.slice(0,200).map(a => a.display));
 }
 |} index |> get_ok ;
    Bos.OS.Cmd.run Bos.Cmd.(v "cp" % search_file % "html/") |> get_ok;
    Bos.OS.Cmd.run Bos.Cmd.(v "cp" % "minisearch.js" % "html/") |> get_ok;
]}



The following code executes all of the above, and we're done!

{[
let compiled = compile_all () in
let linked = link_all compiled in
let () = index_generate () in
let _ = js_index () in
generate_all linked
]}

Let's see if there was any output from the [odoc] invocations:

{[
# !compile_output;;
- : string list = [""]
# (* Not showing output from 'odoc link' as it is unstable. !link_output *);;
# !source_tree_output;;
- : string list = [""]
# !generate_output;;
- : string list = [""]
]}

We can have a look at the produced hierarchy of files, which matches the desired output. Note that source files with a [.ml.html] extension are generated for modules compiled with the [--source] option.

{@sh[
$ ls html/odoc
api_reference.html
deps
driver.html
dune.html
features.html
fonts
highlight.pack.js
index.html
interface.html
katex.min.css
katex.min.js
ocamlary
ocamldoc_differences.html
odoc.css
odoc_document
odoc_examples
odoc_for_authors.html
odoc_html
odoc_html_frontend
odoc_html_support_files
odoc_json_index
odoc_latex
odoc_loader
odoc_manpage
odoc_model
odoc_model_desc
odoc_odoc
odoc_parser
odoc_search
odoc_search.js
odoc_xref2
odoc_xref_test
parent_child_spec.html
source
$ find html/odoc/odoc_html | sort
html/odoc/odoc_html
html/odoc/odoc_html/index.html
html/odoc/odoc_html/Odoc_html
html/odoc/odoc_html/Odoc_html/Config
html/odoc/odoc_html/Odoc_html/Config/index.html
html/odoc/odoc_html/Odoc_html_frontend
html/odoc/odoc_html/Odoc_html_frontend/index.html
html/odoc/odoc_html/Odoc_html/Generator
html/odoc/odoc_html/Odoc_html/Generator/index.html
html/odoc/odoc_html/Odoc_html/Html_fragment_json
html/odoc/odoc_html/Odoc_html/Html_fragment_json/index.html
html/odoc/odoc_html/Odoc_html/Html_page
html/odoc/odoc_html/Odoc_html/Html_page/index.html
html/odoc/odoc_html/Odoc_html/index.html
html/odoc/odoc_html/Odoc_html/Json
html/odoc/odoc_html/Odoc_html/Json/index.html
html/odoc/odoc_html/Odoc_html/Link
html/odoc/odoc_html/Odoc_html/Link/index.html
html/odoc/odoc_html/Odoc_html/Link/Path
html/odoc/odoc_html/Odoc_html/Link/Path/index.html
html/odoc/odoc_html/Odoc_html/Types
html/odoc/odoc_html/Odoc_html/Types/index.html
]}

Some code to analyze the list of executed commands:

{[
(** Return the list of executed commands where the first argument was [cmd]. *)
let filter_commands cmd =
  match
    List.filter
      (fun c ->
        match Bos.Cmd.to_list c.cmd with
        | _ :: cmd' :: _ -> cmd = cmd'
        | _ -> false)
      !commands
  with
  | [] -> failwith ("No commands run for " ^ cmd)
  | (_ :: _) as cmds -> cmds

(** Returns the [k] commands that took the most time for a given subcommand. *)
let k_longest_commands cmd k =
  filter_commands cmd |>
  List.sort (fun a b -> Float.compare b.time a.time) |>
  List.filteri (fun i _ -> i < k)

(** Print an executed command and its time. *)
let print_cmd c =
  Printf.printf "[%4.2f] $ %s\n" c.time (Cmd.to_string c.cmd)
]}

If needed, the list of commands executed so far can be shown by de-commenting this block:

{[
# (* List.iter print_cmd (List.rev !commands);; *)
]}

If needed, the list of the slowest commands for each subcommands can be shown by de-commenting this block:
(for the record, these commands are run from directory `_build/default/doc`)

{[
# (* List.iter print_cmd (k_longest_commands "compile" 5) *)
# (* List.iter print_cmd (k_longest_commands "link" 5) *)
# (* List.iter print_cmd (k_longest_commands "html-generate" 5) *)
]}

This last block analyze the running times so that they can be submitted to
{{:https://github.com/ocurrent/current-bench}current-bench}.

{[
(* *)
#require "yojson" ;;

let rec compute_min_max_avg min_ max_ total count = function
  | [] -> (min_, max_, total /. float count, count)
  | hd :: tl ->
      compute_min_max_avg (min min_ hd) (max max_ hd) (total +. hd) (count + 1)
        tl

let compute_min_max_avg = function
  | [] -> assert false
  | hd :: tl -> compute_min_max_avg hd hd hd 1 tl

let compute_metric_int prefix suffix description values =
  let min, max, avg, count = compute_min_max_avg values in
  let min = int_of_float min in
  let max = int_of_float max in
  let avg = int_of_float avg in
  [
    `Assoc
      [
        ("name", `String (prefix ^ "-total-" ^ suffix));
        ("value", `Int count);
        ("description", `String ("Number of " ^ description));
      ];
    `Assoc
      [
        ("name", `String (prefix ^ "-size-" ^ suffix));
        ( "value",
          `Assoc [ ("min", `Int min); ("max", `Int max); ("avg", `Int avg) ] );
        ("units", `String "b");
        ("description", `String ("Size of " ^ description));
        ("trend", `String "lower-is-better");
      ];
  ]

(** Analyze the running time of a command. *)
let compute_metric_cmd cmd =
  let cmds = filter_commands cmd in
  let times = List.map (fun c -> c.time) cmds in
  let min, max, avg, count = compute_min_max_avg times in
  [
    `Assoc
      [
        ("name", `String ("total-" ^ cmd));
        ("value", `Int count);
        ("description", `String ("Number of time 'odoc " ^ cmd ^ "' has run."));
      ];
    `Assoc
      [
        ("name", `String ("time-" ^ cmd));
        ( "value",
          `Assoc
            [ ("min", `Float min); ("max", `Float max); ("avg", `Float avg) ] );
        ("units", `String "s");
        ("description", `String ("Time taken by 'odoc " ^ cmd ^ "'"));
        ("trend", `String "lower-is-better");
      ];
  ]

(** Analyze the size of files produced by a command. *)
let compute_produced_cmd cmd =
  let output_file_size c =
    match c.output_file with
    | Some f -> (
        match Bos.OS.Path.stat f with
        | Ok st -> Some (float st.Unix.st_size)
        | Error _ -> None)
    | None -> None
  in
  let sizes = List.filter_map output_file_size (filter_commands cmd) in
  compute_metric_int "produced" cmd
    ("files produced by 'odoc " ^ cmd ^ "'")
    sizes

(** Analyze the size of files outputed to the given directory. *)
let compute_produced_tree cmd dir =
  let acc_file_sizes path acc =
    match Bos.OS.Path.stat path with
    | Ok st -> float st.Unix.st_size :: acc
    | Error _ -> acc
  in
  Bos.OS.Dir.fold_contents ~dotfiles:true ~elements:`Files acc_file_sizes []
    (Fpath.v dir)
  |> get_ok
  |> compute_metric_int "produced" cmd ("files produced by 'odoc " ^ cmd ^ "'")

(** Analyze the running time of the slowest commands. *)
let compute_longest_cmd cmd =
  let k = 5 in
  let cmds = k_longest_commands cmd k in
  let times = List.map (fun c -> c.time) cmds in
  let min, max, avg, _count = compute_min_max_avg times in
  [
    `Assoc
      [
        ("name", `String ("longest-" ^ cmd));
        ( "value",
          `Assoc
            [ ("min", `Float min); ("max", `Float max); ("avg", `Float avg) ] );
        ("units", `String "s");
        ( "description",
          `String
            (Printf.sprintf "Time taken by the %d longest calls to 'odoc %s'" k
               cmd) );
        ("trend", `String "lower-is-better");
      ];
  ]

let metrics =
  compute_metric_cmd "compile"
  @ compute_metric_cmd "compile-deps"
  @ compute_metric_cmd "link"
  @ compute_metric_cmd "html-generate"
  @ compute_longest_cmd "compile"
  @ compute_longest_cmd "link"
  @ compute_produced_cmd "compile"
  @ compute_produced_cmd "link"
  @ compute_produced_tree "html-generate" "html/"

let bench_results =
  `Assoc
    [
      ("name", `String "odoc");
      ( "results",
        `List
          [
            `Assoc
              [ ("name", `String "driver.mld"); ("metrics", `List metrics) ];
          ] );
    ]

(* Save the result in a file. This file won't be promoted into the
   documentation. *)
let () = Yojson.to_file "driver-benchmarks.json" bench_results
]}
