The Front End, the Help Browser, Notebooks, Stylesheets, Cells, Typesetting, Buttons, Boxes, and All That

READ ME

This notebook will be part of a future version of the Mathematica GuideBooks. It also uses stylesheet of the Mathematica GuideBooks (© Springer, New York, 2004, 2005). It has been written after the author of the GuideBooks got feedback asking on how to put the GuideBooks in the help browser. It is currently only distributed from http://www.MathematicaGuideBooks.org.

All outputs, graphics, and animations have been deleted from this notebook. To (re)create them, evaluate one subsection/section at a time. It is highly recommended to quit and restart the kernel after each solution is fully evaluated. The code is tailored for Mathematica Version 5.

B.0 Remarks

The Mathematica GuideBooks deal with writing Mathematica programs to solve problems from mathematics and physics and with the generation of interesting and (sometimes meaningful) graphics. The programming, numerics, symbolics, and graphics functions needed to carry out this are discussed in the 14 main chapters of the GuideBooks. Mathematica’s notebook front end allows presenting the book’s material in a conveniently readable, appropriately styled, multimedia-enhanced, easily reusable, immediately modifiable, fully indexed and hyperlinked and also printable form.

For a direct, first-time reading, the Mathematica GuideBooks are, in the opinion of their author, best enjoyed and learned from when read as individual notebooks, meaning outside of the help browser (after all Faust is typically not published embedded in a Duden and read this way—Ulysses is another story, here a dictionary comes in handy). Reading, evaluating, and experimenting in the GuideBooks is more flexible when it is possible to have various chapter notebooks open at the same time. The content of the help browser on the other side is more of reference nature. One uses it to look up things. So potentially, once one has read through the GuideBooks, one might want to add the GuideBooks to the help browser for further reference. Incorporating the chapters (and the table of contents and the index) of the GuideBooks into the help browser is one of the major topics covered in the Applications section of this appendix. And many of the earlier sections of this appendix provide the background on the programming to be used in the Applications section.

The Mathematica notebooks are themselves Mathematica expressions (meaning they are recursively built from expressions of the form head[arguments] and many of the interactively doable operations (like opening and closing notebooks, creating cells, evaluating cells, typing characters) can be done through Mathematica functions.

In the first sections of this appendix, we give an overview over various functions that are used directly or indirectly in the 14 main chapters of the GuideBooks. Because of the many options, option values, and front end-manipulating functions present in the Mathematica front end, this overview is necessarily far from complete. For a more exhaustive listing of front end functions and related topics, see the Mathematica online documentation and [1✶]. Here, in general, we concentrate on the functions, options, and notebook-constructs used in the GuideBooks and on programmatically doable manipulations rather than keyboard and mouse-driven ones.

The fact that Mathematica notebooks are Mathematica expressions that can be operated on programmatically using the Mathematica kernel is a unique, powerful feature of Mathematica. This means that notebooks can be created, deleted, modified, expanded, embedded, reordered, and restyled programmatically. This capability allows to build, maintain, and transform into different formats large collections of data, formulas, graphics, and so on. (A prime example of how these techniques are put to work is the Wolfram Functions site http://functions.wolfram.com) In the Applications section of this appendix, we will implement a fair amount of formatting and organizational changes on the GuideBooks that will allow incorporating the GuideBook files in the help browser.

For some of the inputs and programs to be evaluated in this notebook, it is assumed that this appendix has been put in the directory of the Mathematica GuideBooks as copied from the DVD accompanying a printed version of a Mathematica GuideBook.

We end this introduction with a (strong) recommendation to the reader:

The operations to be carried out on the GuideBooks notebooks for the help browserization, will show the importance of uniform styling.

(* no spelling warnings, set fonts for tick labels, ... *)
Get[ToFileName[ReplacePart["FileName" /. 
    NotebookInformation[EvaluationNotebook[]], "Initialization.m", 2]]];

B.1 Notebooks and Cells as Expressions

B.1.1 The Structure of Notebook

Before giving some examples of simple notebook structures, we introduce a few basic functions.

A Mathematica notebook has the following structure when viewed as a Mathematica expression:

When working with notebook, one sees vertically ordered cells. These cells have the head Cell.

Cells can be hierarchically grouped into so-called cell groups.

Inside a notebook, cell groups appear always in an additional cell, meaning as Cell[CellGroupData[...]]. Such constructions can be (and usually are in most notebooks) repeatedly nested in each other. Typically, at the lowest level we have groups of input and output cells. They, together with text cells are frequently grouped into a subsection group. Multiple subsections form a section group. And multiple sections form a chapter group.

Before giving some examples of notebooks and cells, we introduce some basic functions that deal with notebooks and cells.

The currently open (in the front end) notebooks can be determined with the help of the function Notebooks[].

Here are the currently open notebooks.

Notebooks[]

The input form of the last result shows that the head of a (handle to a) notebook is NotebookObject. The arguments are details depending on the session, the number, and kind of open notebooks, and so on (this information is under most circumstances of no direct use for manipulating notebooks programmatically).

InputForm[%]

The list returned contains objects with head NotebookObject. Like always, the skeleton notation <<…>> indicates suppressed details. Here is the FullForm.

FullForm[%]

In distinction to kernel functions, when the function Options is applied to notebooks and related objects, only the options with nondefault settings are returned (which, because of the private stylesheet used, are only a few).

Options /@ %

Using AbsoluteOptions, we can get a list of all options set for this notebook.

AbsoluteOptions[InputNotebook[], #]& /@ 
                Union[First /@ Options[Notebook]]

While only one notebook is open, the reason that two were returned above is the option setting Visible -> False of the message notebook. (Here we assume that the reader does not have other notebooks open in parallel and has not already carried out other operations before opening this notebook.) Only after errors (or warnings) occurred while evaluating input, will this notebook be visible (we will discuss message notebooks later again).

Notebooks appear in many different flavors. The working notebook, message notebooks, help browser notebooks, palettes are notebooks, and so on. Here is a list of all the named notebooks that one can refer to. These functions end with Notebook.

DeleteCases[Names["System`*Notebook"], 
            "Notebook" | _?(StringMatchQ[#, "Set*"]&)]

And here are some of their definitions.

Information[Evaluate @ ToString[#],
            LongForm -> False]& /@ Names["System`*Notebook"];

Here are the most important functions to open and close notebook.

A notebook that is open in the front end (visible or not) can be converted into a kernel expression using NotebookGet.

Saving a notebook can be programmatically accomplished with the function NotebookSave.

To refer to a particular notebook, we can us either its explicit notebook object, or one of the following functions. (There are some more functions of this kind, we will discuss them later.)

The reason to differentiate between the last three functions comes from objects like palettes—which are ultimately notebooks—and do not accept typing, but could be selected.

There is, of course, only one notebook in which a certain input is currently evaluated (in each kernel). This means an expression with head NotebookObject (not List) is returned from the next input.

EvaluationNotebook[]

Here is the Mathematica expression representing this notebook.

thisNotebookExpression = NotebookGet[EvaluationNotebook[]];
Short[thisNotebookExpression, 18]

The first element contains the actual content; the remaining elements are the option settings. The next input analyzes the expression by counting the occurring Mathematica expressions in the content part. Many of the nonstring elements of the content are lists (which occur around and within cells and boxes), cells with various boxes, and options. The directly visible information is mostly stored in the form of strings and is not reflected in the result of the next input.

allNotebookPart = Sort[{First[#], Length[#]}& /@ Split[Sort[
Cases[Level[thisNotebookExpression[[1]], {-1}, Heads -> True], _Symbol]]],
                       #1[[2]] > #2[[2]]&]

Here is a simple notebook. The argument size determines the window size of the notebook.

sampleNotebookExpression[size_] = 
Notebook[{
 (* a cell group *)
 Cell[CellGroupData[{
  Cell["A small notebook", "Title"],
  (* with a section sub-cell group *)
  Cell[CellGroupData[{
   Cell["1. First Section", "Section"],
   Cell["1. First Subsection of the first section", "Subsection"],
  (* with a subsection sub-sub-cell group *) 
   Cell[CellGroupData[{
    Cell["2. Second Subsection of the first section", "Subsection"],
    (* a text cell *)
    Cell[TextData[{"A text cell followed by a simple identity in ",
                   StyleBox["StandardForm", "Input", FontWeight -> "Plain"],
                   " and ", StyleBox["TraditionalForm", "Input",
                                     FontWeight -> "Plain"], "." }], "Text"],
    (* a input cell with a famous identity in StandardForm *)
    Cell[BoxData[RowBox[{SuperscriptBox["E", 
                         RowBox[{"I", " ", "π"}]], "==", RowBox[{"-", "1"}]}]], 
         "Input"],
    (* a input cell with a famous identity in TraditionalForm *)
    Cell[BoxData[FormBox[RowBox[{SuperscriptBox["E", 
                         RowBox[{"I", " ", "π"}]], "==", 
                         RowBox[{"-", "1"}]}], TraditionalForm]], 
        "Input"],
    (* a text cell with an inline formula *)
    Cell[TextData[{"Here is a text cell with an inline formula: ",
                   Cell[BoxData[FormBox[RowBox[{RowBox[{"exp", "(", 
                               RowBox[{"2", " ", "π", " ", "I"}], ")"}], 
                               "==", "1"}], TraditionalForm]]], "."}], 
        "Text"]}, Open  ]]}, Open  ]],
 (* start of another section *)
 Cell["2. Second Section", "Section"]}, Open  ]]},
 WindowSize -> {size, size}, 
 WindowMargins -> {{245, Automatic}, {76, Automatic}}];

The next input creates six different-sized copies of this notebook in the front end.

sampleNotebooks = Table[NotebookPut[sampleNotebookExpression[size]], 
                        {size, 600, 100, -100}]

We close five of the six notebooks and keep one. NotebookClose returns Null after successfully closing a notebook.

NotebookClose /@ Rest[sampleNotebooks]

The closed notebooks exist no longer.

NotebookGet @ Rest[sampleNotebooks]

We now save the last of the sample notebooks in case an identically named notebook does not already exist on the file system.

(* all file names currently in the directory of this notebook *)
currentFileNamesInTheDirectoryOfThisNotebook = 
FileNames["*", ToFileName @ 
   ToFileName[ReplacePart["FileName" /. 
      NotebookInformation[EvaluationNotebook[]], {}, 2]]]

(* file name of the notebook to be saved *)
sampleNotebookFileName = ToFileName[ReplacePart["FileName" /. 
     NotebookInformation[EvaluationNotebook[]], "sampleNotebookSaved.nb", 2]]

(* save the notebook *)
If[FreeQ[currentFileNamesInTheDirectoryOfThisNotebook, sampleNotebookFileName],
   NotebookSave[First[sampleNotebooks], sampleNotebookFileName],
   Print["File already exists!"]]

Now the newly saved notebook exists in the directory where this notebook resides.

FileNames["*sampleNotebookSaved*", ToFileName @ 
   ToFileName[ReplacePart["FileName" /. 
      NotebookInformation[EvaluationNotebook[]], {}, 2]]]

Next, we read in the saved file literally as a string (not a notebook expression which would not contain comments and loose line formattings). In addition to the actual notebook expression Notebook[…], we see comments at the beginning and the end. The comments at the beginning are self-explanatory, the comments at the end are an outline of the notebook. This outline allows the front end to read in the notebook efficiently (especially in the presence of closed cell groups).

readNotebookString = 
ReadList[sampleNotebookFileName, Record, RecordSeparators -> {},
         WordSeparators -> {}][[1]];
         
Short[readNotebookString, 24]

We now convert the just read-in string of the literal file content into the notebook expression. Because of the presence of comments, we first convert the string to a stream and then read one expression after another (most being Null, the result of evaluating a comment) until we reach the end of the file.

stream = StringToStream[readNotebookString]

Rest[NestWhileList[Read[stream, Expression]&, {}, ( # =!= EndOfFile)&]]

We recovered the original notebook expression.

Cases[%, _Notebook, Infinity][[1, 1]] === sampleNotebookExpression[600][[1]]

We delete the just-created file.

DeleteFile[sampleNotebookFileName]

Notebooks have about 200 possible options.

Options[Notebook] // Length

Most of the options are more or less self-explanatory from their names. Here are all the notebook options.

Sort[First /@ Options[Notebook]]

The options of an entire notebook can be changed interactively through the option inspector or programmatically using the SetOptions function. The next input changes periodically the background color, font color, and width and height of the last remaining sample notebook.

Do[Pause[0.25];
   SetOptions[First[sampleNotebooks], 
              WindowSize -> (Round[400 Abs[Sin[2 Pi ω]]] + 200) {1, 1},
              Background -> Hue[1.0 ω], FontColor -> Hue[ω + 0.5]],
   {ω, 0, 2, 2/60}]

Next, we make the sample notebook invisible for a few seconds.

SetOptions[First[sampleNotebooks], Visible -> False];
Pause[3];
SetOptions[First[sampleNotebooks], Visible -> True];

We close the last of the sample notebooks.

NotebookClose[nb]

We can get convenient access to an explanation of all options through the help browser or by opening the corresponding help browser file directly. (The help browser does nothing other than display parts of notebooks in a hierarchical manner. We will discuss this subject in more detail later).

(* all files in the Mathematica directory *)
OptionInspectorHelpBrowserNb = 
   Select[FileNames["*", $InstallationDirectory, Infinity],  
          StringMatchQ[#, "*Documentation*OptionInspector.nb"]&];

nb  = NotebookOpen[OptionInspectorHelpBrowserNb[[1]]]

We quickly scroll through this notebook.

SelectionMove[nb, All, Notebook]
Pause[1];
(* walk through *)
FrontEndExecute[{FrontEndToken[nb, "SelectionOpenAllGroups"]}]
Do[Pause[1]; FrontEndExecute[{FrontEndToken[nb, "ScrollPageNext"]}],
   {100}]

We close the notebook again.

NotebookClose[nb]

We now read this file into the kernel to analyze it. It is a relatively large file that contains mainly text.

nbOIExpression = Get[OptionInspectorHelpBrowserNb[[1]]];
ByteCount[nbOIExpression]

This notebook contains the description of more than 500 descriptions of options.

usageCells = Cases[nbOIExpression, Cell[_, "Usage", ___], Infinity];
Length[usageCells]

Here is the distribution of the options for the front end, notebooks, cells, buttons, and characters. Doing the math, we see that many options can be given at different levels.

Function[optionForString, 
{optionForString,
  Count[usageCells, _?(MemberQ[#, s_String?(
        StringMatchQ[#, "*" <> optionForString <> "*"]&), 
      {0, Infinity}]&), Infinity]}] /@ 
  {"is a global option", "is an option for notebooks",
   "is an option for cells", "is an option for characters",
   "is an option for buttons"} // TableForm

Here are the 200+ option names themselves.

optionNames = Cases[nbOIExpression, Cell[_, "ObjectName", ___], 
                    Infinity] // (First /@ #&);
Length[optionNames]

About 15% of the option names are strings.

Count[optionNames, _?(Head[ToExpression[#]] === String&)]

Here are the five shortest and the five longest option names.

{Take[#, 5], "…", Take[#, -5]}& @ 
Sort[StringReplace[optionNames, {" " -> ""}],
     (StringLength[#1] < StringLength[#2])&]

Carrying out front end operations programmatically is a convenient notebook-manipulating tool. But compared with kernel operations, it is comparatively slow. Typical front end operations realized through the kernel usually take in the order of milliseconds. Next, we create a simple notebook.

nb = NotebookPut @ Notebook[{Cell["I am constantly changing!", Text]}, 
  WindowSize -> {600., 160.}, WindowMargins -> 
   {{100, Automatic}, {Automatic, 60}}, FontSize -> 20., 
  FontColor -> GrayLevel[0], Background -> RGBColor[1, 1, 1]]

The next input changes the appearance and its position of the just-created notebook 1000 times.

time1 = AbsoluteTime[];
Do[SetOptions[nb, 
       (* various random settings *)
       Background -> (RGBColor @@ Table[Random[], {3}]),
       FontColor -> (RGBColor @@ Table[Random[], {3}]),
       FontSize -> Random[Integer, {12, 24}],
       WindowSize -> {Random[Integer, {400, 600}], 
                      Random[Integer, {200, 400}]}, 
       WindowMargins -> 
         {{Random[Integer, {50, 500}], Automatic}, 
          {Automatic, Random[Integer, {50, 500}]}}], 
   {1000}];
(* average time per change *)   
(AbsoluteTime[] - time1)/1000 "Second"

(* close the crazy notebook again *)
NotebookClose[nb]

Opening and closing notebooks are typically operations that take considerably longer because they are carried out with the help of the windowing (operating) system. Next, we create and close 100 notebooks. Each opening-closing operation takes a few ten microseconds.

time1 = AbsoluteTime[]; 
nbT = Table[NotebookPut @ Notebook[
       {Cell["Notebook number " <> ToString[k], 
            "Text", FontSize -> 16]},
           WindowSize -> {500, 500} - 2k], {k, 100}];
NotebookClose /@ nbT;
(AbsoluteTime[] - time1)/100 "Second"

B.1.2 The Appearance of Cells

After the quick introduction into manipulating notebooks in the last subsection, we continue with the cells of the notebook content—the main structural elements that occur in notebooks. In addition to the two-argument Cell structure Cell[content, style] (with or without potential options); we mentioned already that cells can appear in the one-argument form Cell[CellGroupData[…]]. In addition, Cell can have one argument as a subpart of a two-argument Cell.

The most important heads of the content content of cells of the form Cell[content, style] are String, TextData, BoxData, StyleData, or RawData.

The possible styles styles of a cell follow from the style definitions given in the currently applied stylesheet. The most frequently used cell styles include "Text", "Input", "Output", "Message", "Print", "Subsection", and "Section".

A convenient way to display two-argument cells (with or without potential options) is available as an expression in the kernel is CellPrint.

Now, we have a look at the various *Data that can appear inside cells. For StyleData, we have a quick look at all the styles defined in the stylesheet of the GuideBooks.

(* read in the stylesheet of the GuideBooks as an expression *)
guideBooksStylesheet = 
Get[ToFileName[ReplacePart["FileName" /. 
 NotebookInformation[EvaluationNotebook[]], "GuideBooksStylesheet.nb", 2]]];

(* extract all style data *)
allGuideBooksStyleData = Cases[guideBooksStylesheet, _StyleData, Infinity];

(* all style names *)
Union[Last /@ allGuideBooksStyleData]

The optional second argument of StyleData is the environments. Three are defined for the GuideBooks: "Reading" for leisurely reading, "AllBrackets" for convenient editing (all cell brackets are shown in this style) and "Printout" for printouts (no cell brackets are shown). (In the next subsection, we will discuss the structure and contents of stylesheets in more detail.)

Union[Flatten[If[Length[#] == 2, Last[#], {}]& /@ allGuideBooksStyleData]]

Cells have nearly as many options as a notebook.

Options[Cell] // Length

Here are the currently available option names.

Sort[First /@ Options[Cell]]

Again, we will not discuss all possible cell options in detail. To obtain an impression of the cell options used in the GuideBooks, we now read in the notebook of the first chapter of the Symbolics volume.

firstSymbolicsChapterFileName = 
ToFileName[ReplacePart["FileName" /. 
    NotebookInformation[EvaluationNotebook[]], "4_Symbolics_1.nb", 2]]

firstSymbolicsChapterNotebook = Get[firstSymbolicsChapterFileName];

Very few options are set explicitly at the notebook level. The ShowCellLabel option for instance, makes sure that all In/Out numbers are shown. (To preserve cell labels in notebooks, the option CellLabelAutoDelete must be set to True.)

Rest[firstSymbolicsChapterNotebook]

The first element that contains the content has 17 elements.

Length[First[firstSymbolicsChapterNotebook]]

The following output shows that the first few cells are title-like cells and the remaining cells are cell groups (the sections, the exercises, the solutions, and the references).

If[Length[#] == 1, CellGroupData, #[[2]]]& /@ 
                   First[firstSymbolicsChapterNotebook]

We obtain the “physical” cells (meaning the ones that appear vertically ordered in a displayed notebook) by flattening out the cell groups. The notebook has more than 13000 cells.

firstSymbolicsChapterPhysicalCells = 
 Flatten[firstSymbolicsChapterNotebook[[1]] //. 
            Cell[CellGroupData[l_, ___]] :> l];
 
Length[firstSymbolicsChapterPhysicalCells]

Now, let us analyze these cells and their properties a bit. We start by counting the various cell types. The most present cell types are, not unexpectedly for a Mathematica-explaining book, Mathematica inputs.

cellTypesAndCounts = (* sort and count cell types *)
Sort[{First[#], Length[#]}& /@ Split[Sort[#[[2]]& /@ 
                   firstSymbolicsChapterPhysicalCells]],
     #1[[2]] > #2[[2]]&]

Next, let us look at the first elements. Pure InputForm input cells and pure text (without formulas) are represented as strings. Text cells containing inline formulas are represented through TextData. Outputs in StandardForm and explanatory formulas in TraditionalForm are represented through BoxData.

Sort[{First[#], Length[#]}& /@ Split[Sort[Head[First[#]]& /@
                   firstSymbolicsChapterPhysicalCells]],
     #1[[2]] > #2[[2]]&]

We have a look at the size distributions of these strings, text data, and box data. The TextData lists have up to a few hundred elements; the BoxData are typically considerably smaller.

Show[GraphicsArray[ListPlot[#, PlotRange -> All, 
                            DisplayFunction -> Identity]& /@ 
{(* length of strings *)
 Sort[StringLength /@ Cases[First /@ 
                      firstSymbolicsChapterPhysicalCells,
      _String]],
 (* length of TextData *)
 Sort[Length[First[#]]& /@ Cases[First /@ 
                      firstSymbolicsChapterPhysicalCells,
      _TextData]], 
 (* length of BoxData *)
 Sort[Length[First[#]]& /@ Cases[First /@ 
                           firstSymbolicsChapterPhysicalCells,
      _BoxData]]}]];

Here are the heads that occur in the arguments of the TextData. They are mainly strings (pieces of text), styled text, and inline cells. The styles text occurs either as Mathematica function names in a monospaced [typically Courier-like] font or as italic function arguments realized through a StyleBox[content, options].

Sort[{First[#], Length[#]}& /@ Split[Sort[Head /@ 
    Flatten[First /@ Cases[First /@ firstSymbolicsChapterPhysicalCells,
            _TextData]]]], #1[[2]] > #2[[2]]&]

Here are the heads that occur in the arguments of the BoxData. They are mainly boxes (2D mathematical typeset expressions) and strings.

Sort[{First[#], Length[#]}& /@ Split[Sort[Head /@ 
    Flatten[First /@ Cases[First /@ firstSymbolicsChapterPhysicalCells,
            _BoxData]]]], #1[[2]] > #2[[2]]&]

After these examples of various *Data, we look at the explicitly set cell options. The option GeneratedCell stems from the programmatic notebook processing carried out on this notebook. The CellTags option is used for the destinations of the table of contents hyperlinks and the index entry hyperlinks. (In the Applications section of this notebook, we will make heavy use of cell tags).

Sort[{First[#], Length[#]}& /@ Split[Sort[
  Flatten[(First /@ (List @@ Take[#, {3, Length[#]}]))& /@ 
                    firstSymbolicsChapterPhysicalCells]]],
     #1[[2]] > #2[[2]]&]

We sidestep for a moment and have a quick view on all cells within a notebook. For large notebooks, it is sometimes inconvenient to load the whole notebook into the kernel. In such cases, reading in the notebook outline from the saved notebook gives often all necessary information to later work directly on specific cell groups or cells of a notebook. Here we do this with the first chapter of the Symbolics volume. We scan the file until we find the word NotebookFileOutline. After this, we read in the expression corresponding to the notebook outline.

fscStream = OpenRead[firstSymbolicsChapterFileName]

Find[fscStream, "NotebookFileOutline"]

nbOutline = Read[fscStream, Expression];

Here are the size, length, and depth of the notebook.

{ByteCount[nbOutline], Length[nbOutline[[1]]], Depth[nbOutline]}

By replacing cells with points, we obtain a graphical outline of the notebook’s structure.

(* the type of an abbreviated cell from the cell outline *)
cellType[Cell[__Integer, s_String, ___]] := s
(* associate randomly chosen color with each cell type *)
(cellTypeColor[First[#]] = Hue[Random[]])& /@ cellTypesAndCounts;
(* the type of an abbreviated cell from the cell outline *)
MapIndexed[(cellTypeDepth[First[#]] = #2[[1]])&, cellTypesAndCounts];

(* linearly ordered cells, their types, and at which depth they occur 
   of the notebook *)
orderedCellTypes = Flatten @ (List @@
MapIndexed[If[Head[#] === Cell, Cell[cellType[#], Length[#2]], {}]&,
            nbOutline //. Cell[CellGroupData[cells_, ___], ___] :> {{cells}} /.
            Cell[is__Integer, s_String, ___] :> Cell[is, s],
           {-2}]);

Show[Graphics3D[
MapIndexed[{cellTypeColor[#[[1]]], 
            (* x-value is vertical position, y-value is cell type,
               z-value is depth at which the cell occurs *)
            Point[{#2[[1]], cellTypeDepth[#[[1]]], #[[2]]}]}&, 
           orderedCellTypes]], PlotRange -> All, Axes -> True,
           BoxRatios -> {3, 1, 1}]

These are the colors that are used for coloring the text of the various cell types.

StyleForm[#, FontColor -> cellTypeColor[#]]& /@ 
                   (First /@ cellTypesAndCounts)

Next, we analyze all notebooks that come with Mathematica. To find all notebooks, we locate all files in the Mathematica installation directory that have the extension .nb.

allNotebooksFilenamesThatComeWithMathematica = 
  Select[FileNames["*", $InstallationDirectory, Infinity],
         StringMatchQ[#, "*.nb"]&];
              
Length[allNotebooksFilenamesThatComeWithMathematica]

The next function analyzeNotebook[fileName] reads in the notebook fileName, and then determines the depth of the content part (ignoring the options set for the whole notebook). Then the function counts the cell groups, the TextData-objects, the StyleBox-objects, and the number of physical cells (meaning the ones having an innermost cell bracket). In addition, for later use, we collect the cell options that were used.

analyzeNotebook[fileName_] := 
Module[{nb = Get[fileName], depth, cellGroups, textData,
        styleBoxes, physicalCells}, 
 (* depth of the content part *)       
 depth = Depth[nb[[1]]];
 (* count cell groups, text data, style boxes, and cells *)
 cellGroups = Count[nb, _CellGroupData, Infinity];
 textData = Count[nb, _TextData, Infinity];
 styleBoxes = Count[nb, _StyleBox, Infinity];
 physicalCells = Length[ Flatten[nb[[1]] //. 
           Cell[CellGroupData[cells_, ___], ___] :> cells]];
 (* extract all used cell options *)
 usedCellOptions = 
    Union[Flatten[(First /@ (List @@ Drop[#, 2])& /@ 
       Cases[nb, Cell[_, _, __], Infinity])]];       
 (* return results *)
 {depth, cellGroups, textData, styleBoxes, physicalCells,
  usedCellOptions}]

notebookAnalyzingResults = {#, analyzeNotebook[#]}& /@ 
    Take[allNotebooksFilenamesThatComeWithMathematica, All];

Overall, all notebooks together have more than 200,000 cells.

Plus @@ (Take[#, {2, -2}]& /@ Last /@ notebookAnalyzingResults)

And here are the notebooks having the deepest structure, and the most cells, and the most boxes. The XMLUserGuide.nb has the deepest nesting, the index of the The Mathematica Book has the most TextData, and the reference guide of the The Mathematica Book has the most cell groups, cells, and style boxes.

MapIndexed[Function[{max, pos}, 
            Select[notebookAnalyzingResults, (#[[2, pos[[1]]]] == max)&]],
           Max /@ Transpose[Most /@ Last /@ notebookAnalyzingResults]] /.
   (* file name relative to the Mathematica installation directory *)        
   s_String :> StringDrop[s, StringPosition[s, $InstallationDirectory][[1]]]

Nearly 90% of all available options have been actually used in the one or another notebook.

{Options[Cell] // Length, 
 Union[Flatten[#[[2, -1]]& /@ notebookAnalyzingResults]] // Length}

The individual elements of a BoxData or TextData expression can be styled using a StyleBox with a second argument. The second argument is a style name that has to be a string. (The various types of mathematical typesetting-related boxes will be discussed in detail in the next section. A StyleBox can be used in text, as well as in mathematical typesetting.)

Here is a simple example.

(* the characters of a sentence *)
chars = Characters[
"A text cell that has rainbow colors.\n
This is done using StyleBoxes."];

(* number of characters *)
λ = Length[chars];

(* create and display a notebook with the 
   consecutively colored characters *)
NotebookPut @ Notebook[{Cell[TextData[
MapIndexed[StyleBox[#1, 
     FontColor -> ToColor[Hue[#2[[1]]/λ], RGBColor],
     Background -> ToColor[Hue[1/2 + #2[[1]]/λ], RGBColor],
     FontSize -> 12 + #2[[1]]]&, 
     chars]], "Text"]}, WindowSize -> {1000., 300.}]

We close the previously created notebook.

(Pause[5]; NotebookClose[%];)

Here is a slightly more complicated example. We extract all cells from this Appendix B notebook that contain the word “Mathematica”. We color all occurrences of the word.

mathematicaContainingCells =
 Select[Flatten[NotebookGet[EvaluationNotebook[]][[1]] //. 
                  Cell[CellGroupData[l_, ___], ___] :> l],
        MemberQ[#, _String?(StringMatchQ[#, "*Mathematica*"]&), {-1}]&];

Here is a way to extract these cells using front end functions.

(* find the cells containing the word "Mathematica" *)
NotebookFind[EvaluationNotebook[], "Mathematica", All, Cell];

(* extract these cells *)
mathematicaContainingCells = NotebookRead[EvaluationNotebook[]];

(* read in the stylesheet of the GuideBooks as an expression *)
guideBooksStylesheet = 
Get[ToFileName[ReplacePart["FileName" /. 
 NotebookInformation[EvaluationNotebook[]], 
                   "GuideBooksStylesheet.nb", 2]]];
                   
NotebookPut[Notebook[mathematicaContainingCells /.
                     StyleBox["Mathematica", s__] :>
                     StyleBox["Mathematica", s, 
                                FontColor -> RGBColor[1, 0, 0],
                                Background -> RGBColor[0, 0, 1]],
                     (* attach the stylesheet *)
                     StyleDefinitions -> guideBooksStylesheet]]

Now, let use investigate some displayed cells. To find these cells programmatically, we have given them cell tags. The following four cells have cell tags of the form "SampleCell[cellContentType]".

1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9

This is a text cell with a small, but famous identity: (e^(i π))==-1.

sin((π/16))=(1/2) (Sqrt[2-(Sqrt[2+(Sqrt[2])])])

The next cell is an unformatted cell.

Cell["This cell is unformatted.", "Text", 
     CellTags -> "SampleCell[RawData]"]

Cases[NotebookGet[EvaluationNotebook[]],
      Cell[_, _, ___, 
           CellTags -> _String?(StringMatchQ[#, "*SampleCell*"]&),
           ___], Infinity]

The options of cells in a notebook can be programmatically manipulated through the function SetOptions. Here is a notebook with arithmetic identities.

sumString[n_] := StringJoin @ Join[
Most[Flatten[Transpose[{Table[ToString[k], {k, n}], 
                        Table["+", {k, n}]}]]],
                       {"=", ToString[InputForm[n (n + 1)/2]]}]

nb = NotebookPut[Notebook[Table[Cell[sumString[n], "Text"], {n, 2, 100}]]];

We position the cursor at the beginning of the notebook and step through the notebook cell by cell (we will discuss the function SelectionMove in a moment). We style each selected cell by adding background color, colored text, and so on.

SelectionMove[nb, Before, Notebook]

Do[(* pause for a moment; then move to the next cell *)
   Pause[0.25]; SelectionMove[nb, Next, Cell]; 
   (* set a variety of options for the currently selected cell *)
   SetOptions[NotebookSelection[nb], 
           CellFrame -> Random[Integer, {1, 4}],
           CellDingbat -> {"■","☺", "★"}[[Random[Integer, {1, 3}]]],
           FontSize -> Random[Integer, {8, 32}],
           FontFamily -> {"Courier", "Times", 
                          "Helvetica"}[[Random[Integer, {1, 3}]]],
           CellMargins -> Table[Random[Integer, {10, 30}], {2}, {2}],
           FontColor -> (RGBColor @@ Table[Random[], {3}]),
           Background -> (RGBColor @@ Table[Random[], {3}])],
           {100}]

To remove the options, we set the values of the just-set options to Inherited (the default values).

Do[(* pause for a small moment; then remove options *) 
    Pause[0.1]; 
   SetOptions[NotebookSelection[nb], 
           CellFrame -> Inherited, CellDingbat -> Inherited,
           FontSize -> Inherited, FontFamily -> Inherited,
           CellMargins -> Inherited, FontColor -> Inherited,
           Background -> Inherited];
    (* move to the previous cell *) 
    SelectionMove[nb, Previous, Cell], {100}]

NotebookClose[nb]

As mentioned, the basic building blocks of a notebook are cells. Cell[content, style, potentialOptions] represents one cell in a notebook. But in addition, to this standard use, expressions with the head Cell can occur in two further ways in a notebook. Keeping these two possible occurrences of cells in mind is important for programmatic work with notebooks.
The second possibility of the occurrence of an expression with head Cell is within cells of a given style when we have a text cell with a typeset formula. Then the content has the form TextData[{strings, Cell[boxes], moreStrings}]. Similarly in the third possibility of the occurrence of an expression with head Cell, is as text within a typeset formula. For instance, if one wants pure text (no autoitalication of single letters, normal text-like spacing between words), one can embed a cell (head Cell; one argument) within the (mathematical typesetting) boxes of BoxData[{boxes, Cell[string], moreBoxes}]. The last Cell[string] could be replaced by a more complicated expression, such as Cell[TextData[{stringsAndStyleBoxes}]]. Here is an example.

NotebookPut @
Notebook[{
 (* the cell itself is an input cell *)
 Cell[BoxData[
    RowBox[{"2", "+", (* with embedded styled text *)
     Cell[TextData[{StyleBox["text in a formula! ", "Text",
        FontFamily -> "Times", FontWeight -> "Plain", 
        FontColor -> RGBColor[1, 0, 0]],
        (* and inside an embedded styled power *)
        Cell[BoxData[StyleBox[SuperscriptBox["2", "2"],
             FontColor -> RGBColor[0, 0, 1]]]]}]], "+", "2"}]], "Input",
  CellLabel -> "I have mixed content!"]},
         WindowSize -> {620, 320}, Magnification -> 2]

Pause[5]; NotebookClose[%]

B.1.3 Stylesheets

The appearance of notebooks (on screen and/or printed) is determined by the style definitions used. Styled elements in notebooks (text, headers, inputs and outputs, and so on) can either be explicit through style boxes, or font and cell appearance options. To avoid the (potentially thousands of times occurring) repetition of such style definitions, frequently occurring groups of such option settings are named and are collected in so-called stylesheet. A stylesheet is basically notebook (typically either called *Stylesheet.nb or/and located in a directory called Stylesheets) that has cells of the form Cell[StyleData[environmentOrAll, styleName], explicitSetOptions]. Stylesheets (meaning the actual Notebook expression) can also appear literally in the right-hand side of the option setting StyleDefinitions of a notebook.

The appearance of an element in a notebook depends on the options set: Options can be set 1) at the global level (for any notebook open within the front end), 2) through a stylesheet or/and at the notebook level, 3) at the cell level, or 4) at parts of a cell. (Options cannot currently set at the cell group level.) When not explicitly set, options are inherited from the next higher level. Most options can be set at any of the listed levels. Stylesheets contain typically many settings for the cell level and a few at the notebook level.

Here are the settings used in the stylesheet used for the GuideBooks.

(* read in the stylesheet of the GuideBooks as an expression *)
guideBooksStylesheet = 
Get[ToFileName[ReplacePart["FileName" /. 
 NotebookInformation[EvaluationNotebook[]], "GuideBooksStylesheet.nb", 2]]];

(* show some of 100+ options with the used option settings *)
Short[#, 40]& @ 
({#[[1, 1]], Alternatives @@ Union[Last /@ #]}& /@ 
 Split[Sort[Cases[guideBooksStylesheet[[1]], _Rule, Infinity]] /.
                  (ButtonFunction :> _) -> ((ButtonFunction :> Identity)),
       #1[[1]] === #2[[1]]&])

We will now modify the appearance of a copy of this notebook by creating a stylesheet that has random color, font, font size, and other modifications. The functions randomStylesheetModification carries out random modifications on the stylesheet sS.

randomStylesheetModification[sS_] := sS //. 
{(Color -> _) :> (* different color *)
  (Hold[Color] -> (RGBColor @@ Table[Random[], {3}])),
 (Thickness -> _) :> (* different thickness *)
  (Hold[Thickness] -> Random[Integer, {2, 5}]),
 (Underline -> _) :> (* some underlining *)
  (Hold[Underline] -> ({True, False}[[Random[Integer, {1, 2}]]])),
 (Background -> _) :> (* different background color *)
  (Hold[Background]  -> (RGBColor @@ Table[Random[], {3}])),
 (CellMinHeight -> _) :> (* new minimal cell height *)
  (Hold[CellMinHeight] -> Random[Integer, {2, 5}]),
 (CellFrame -> _) :> (* cell frame *)
  (Hold[CellFrame] -> Table[Random[Integer, {0, 4}], {2}, {2}]),
 (CellFrameMargins -> _) :> (* different cell frame margins *)
  (Hold[CellFrameMargins] -> Table[Random[Integer, {0, 24}], {2}, {2}]),
 (CellMargins -> _) :> (* different cell margins *)
  (Hold[CellMargins] -> Table[Random[Integer, {0, 24}], {2}, {2}]),
 (FontFamily -> _) :> (* different fonts *)
  (Hold[FontFamily] -> ({"Courier", "Helvetica", "Times"}[[
                                           Random[Integer, {1, 3}]]])),
 (FontSize -> _) :> (* different font sizes *)
  (Hold[FontSize] -> Random[Integer, {8, 18}]),
 (LineSpacing -> _) :> (* different line spacing *)
  (Hold[LineSpacing] -> Table[Random[Integer, {0, 3}], {2}]),
 (ParagraphIndent -> _) :> (* different paragraph indentation *)
  (Hold[ParagraphIndent] -> Random[Integer, {-40, 0}])} //.
 (* release held settings to get different values *) Hold[x_] :> x /.
  (* add some color to any cell type in case it is not already there *)
   Cell[sd_StyleData_, rest___] :> 
   Cell[sd, rest, FontColor -> (RGBColor @@ Table[Random[], {3}]),
                  FontSize -> Random[Integer, {8, 18}],
                  Background -> (RGBColor @@ Table[Random[], {3}])]

The next input generates a random modification of the stylesheet used in this notebook, embeds this newly created stylesheet in a copy of this notebook, and displays the resulting notebook. Then, we quickly scroll through and close the notebook again (due to the random modifications the notebook will almost surely look not so nice that it would be worth keeping). We repeat this process for three randomly generated stylesheets.

newStylesheet = randomStylesheetModification[guideBooksStylesheet];

Do[
 (* make new stylesheet *)
 newStylesheet = randomStylesheetModification[guideBooksStylesheet]; 
(* save the new stylesheet *)
 (* nSNb = NotebookPut[newStylesheet];
    nSNbFileName = ToFileName[ReplacePart[
               "FileName" /. NotebookInformation[EvaluationNotebook[]],
               "GuideBooksStylesheetModified.nb", 2]];
   NotebookSave[nSNb, nSNbFileName]
   NotebookClose[gbsM] *)
 (* style-modified version of this notebook with embedded stylesheet *)
 nbM = NotebookGet[SelectedNotebook[]] /. 
          {(StyleDefinitions -> _) ->  
           (StyleDefinitions -> newStylesheet (* gbsMFileName *)),
           (WindowSize -> _) :> (WindowSize -> {600, 600}),
           (WindowMargins -> _) :> 
            (WindowMargins -> {{Automatic, 10}, {Automatic, 10}})} //. 
           Closed -> Open;
 (* bring the style-modified notebook up *)
 uglyNb = NotebookPut[nbM];
 (* quickly scroll through the new notebook *)
 numberOfCells = Length @
   Flatten[nbM[[1]] //. Cell[CellGroupData[l_, ___], ___] :> l];
 (* some time to have a look at the notebook *) Pause[1];
 Do[Pause[0.1]; SelectionMove[uglyNb, Next, Cell], {numberOfCells - 1}];
 (* close the notebook *)
 NotebookClose[uglyNb];
 (* remove the new stylesheet *)
 (* DeleteFile[nSNbFileName] *), {3}]

The style definitions in stylesheets can contain a wealth of details about the appearance of notebooks. We already discussed some of the box options governing the appearance inside cells. Above, we discussed that the cell groups in a notebook are represented as (potentially nested instances of) Cell[CellGroupData[{headCell, childCell1, childCell2, …}, openOrClosed]]. What actually govern the grouping of cells remains to be discussed. Cells can be grouped either manually through menus, or automatically through the hierarchy of the cell styles that are defined in a stylesheet. In the style data cells of the stylesheet, the option CellGroupingRules determines the grouping. The most used settings of this option are "InputGrouping" (to groups output and graphics cells under an input cell), "GraphicsGrouping" (to groups consecutive graphics cells under the first one), and {"SectionGrouping", order} (to group text and input units). The larger the integer order is, the weaker is the grouping power of cells of this style.

In the next inputs, we programmatically generate a notebook with many nested section-like cells of different subsection levels.

(* add m sub-cells to a cell *)
makeChildSectionCellRule[k_, m_] :=
 c:Cell[TextData[{number_String, subsection_String}], 
        Section[k]] :>
  {c, Sequence @@
   Table[Cell[TextData[{number <> ToString[j] <> ".", 
              "Sub\[Hyphen]" <> subsection}], Section[k + 1]], {j, m}]}

(* a long list of sub\[Hyphen]…\[Hyphen]section cells *)
manySectionCells = 
 Fold[Flatten[1# /. makeChildSectionCellRule[#2, 3]]&, 
              {Cell[TextData[{"1.", "Section"}], Section[1]]}, 
         Range[8]] /. Section[k_] :> "Section" <> ToString[k];

We associate increasing values of order to the grouping properties of the subsub…subsections.

styleData[k_] := 
Cell[StyleData["Section" <> ToString[k]],
     (* smaller margins for lower grouping power *)
     CellMargins -> {{30 - k, 20 - k }, {20 - k, 30 - k}},
     CellBracketOptions -> 
               {"Color" -> ToColor[Hue[N[k/20]], RGBColor],
                "Thickness" -> 10 - k},
     FontFamily -> "Helvetica", FontSize -> 70 - 2 k,
     FontWeight -> "Bold", FontSlant -> "Italic",
     FontColor -> ToColor[Hue[N[k/20 + 1/2]], RGBColor], 
     CellGroupingRules -> {"SectionGrouping", k},
     CellHorizontalScrolling -> True]

Here is the resulting notebook. We embed the style information through the StyleDefinitions option at the notebook level.

nb = Notebook[manySectionCells, 
              StyleDefinitions -> Notebook[Table[styleData[k], {k, 9}]]];
              
nbH = NotebookPut[nb]

The notebook created in the Mathematica kernel had as its first argument just flat list of cells. Only when displayed in a front end, do the groupings into effect. We read the displayed notebook back into the kernel. The nested Cell[CellGroupData[…]] structures are now visible. The notebook now has 3280 CellGroupData-objects.

Count[NotebookGet[nbH], _CellGroupData, Infinity]

We quickly scroll through the notebook and then close all open cell groups.

SelectionMove[nbH, Before, Notebook]

Do[FrontEndExecute[{FrontEndToken[nbH, "ScrollLineDown"]}],
   {25000}]

Do[FrontEndExecute[{FrontEndToken[nbH, "ExpandSelection"]}],
   {12}]

FrontEndExecute[{FrontEndToken[nbH, "SelectionCloseAllGroups"]}]

We enjoy the notebook for a few more seconds and then recycle it.

Pause[5];
NotebookClose[nb]

One more remark is in order about stylesheets. Not every style actually declares all the details of the appearance of the corresponding item. Through the FormatTypes, the “style of styles” can be influenced. We load the default stylesheet that comes with Mathematica into the kernel.

(* directory of stylesheets *)
styleSheetDirectory = 
Select[FileNames["*Stylesheets*", $InstallationDirectory, Infinity],
       Not[StringMatchQ[#, "*.nb"]]&][[1]]

(* the default sheets *)
defaultStyleSheet = FileNames["*Default*", styleSheetDirectory][[1]]

(* load the default stylesheet *)
dss = Get[defaultStyleSheet];

Here are all the named styles making use of a FormatType and these format type themselves.

{#[[1]], FormatType /. (List @@ Rest[#])}& /@ 
  Cases[dss, Cell[_StyleData, ___, FormatType -> _, ___], Infinity]

And these are theFormatTypes defined in the stylesheets.

(declaredFormatTypes =  Union[ToString /@ Last /@ %]) // InputForm

And here are the styles of the declared default styles.

Cases[dss, Cell[StyleData[#], ___], Infinity]& /@ declaredFormatTypes

The stylesheet directory has many more stylesheets.

otherStyleSheets =
  DeleteCases[FileNames["*", styleSheetDirectory], defaultStyleSheet];
  
Length[otherStyleSheets]

Most of them do not declare format types themselves, but rather resort to the format types declared in the default stylesheet. Only the help browser and the demo stylesheets declare their own format types.

DeleteCases[Function[ss, {ss, Cases[Get[ss], Cell[StyleData[#], ___], 
              Infinity]& /@ declaredFormatTypes}] /@ otherStyleSheets,
              {_, {{}, {}}}]

We demonstrate the indirect influence of the format types on the style appearance. Here is a notebook with a text cell having an inline cell and an input cell in InputForm.

nbData = (* a simple notebook wit a few cells *)
      Notebook[{Cell[TextData[{"Here comes a simple calculation: ",
       Cell[BoxData[RowBox[{"sin", RowBox[{"(", SuperscriptBox[ 
         RowBox[{"(", RowBox[{RowBox[{SqrtBox["4"], "x"}], "+", 
         RowBox[{"2", "x"}]}], ")"}], RowBox[{"1", "/", "2"}]], 
         ")"}]}]]], ":"}], "Text"],
       Cell["Sin[Sqrt[Sqrt[4] x + 2 x]]", "Input"]},
                  (* set notebook options *)
                  WindowSize -> {800, 240}, Magnification -> 2];

Displaying the evaluated notebook yields the expected appearance: The inline cell is formatted as traditional mathematics and the output is formatted in StandardForm.

nb1 = NotebookPut[nbData]; Pause[1];
FrontEndTokenExecute[nb1, "SelectAll"];
FrontEndTokenExecute[nb1, "Evaluate"];

Next, we change the format types. We make the default format type of the inline cell StandardForm and the default format type of the output TraditionalForm. Now the evaluated notebook looks quite different from the above, usual one.

nb2 = NotebookPut[Insert[Insert[nbData,
  CommonDefaultFormatTypes -> 
      {"Output" -> TraditionalForm, 
       "TextInline" -> StandardForm}, 2],
       WindowTitle -> "Modified FormatTypes", 2]]; 
       
(* wait a moment and then evaluate cells *)
Pause[1];
FrontEndTokenExecute[nb2, "SelectAll"];
FrontEndTokenExecute[nb2, "Evaluate"];

Pause[5]; NotebookClose /@ {nb1, nb2};

An easy way to change the overall look-and-feel of notebooks are the so-called environments. Here is a simple notebook with the screen style environment set to sse.

putNb[sse_, δ_] := NotebookPut[
Notebook[{Cell[CellGroupData[{
   Cell["1. Text in notebooks", "Section"], 
      Cell["This is a text cell.", "Text"]}, Open]], 
   Cell[CellGroupData[{Cell["2. Inputs in notebooks", "Section"], 
    Cell[BoxData[RowBox[{"Exp", "[", 
    SuperscriptBox[RowBox[{"(", FractionBox[RowBox[{"2", "+", 
              SqrtBox["16"]}], "32"], ")"}], "4"], "]"}]], "Input"]},  Open]], 
   Cell[CellGroupData[{Cell["3. Formulas in notebooks", "Section"], 
       Cell[BoxData[
    FormBox[SuperscriptBox["E", SuperscriptBox[
          RowBox[{"(", RowBox[{FractionBox["1", "32"], " ", 
              RowBox[{"(", RowBox[{"2", "+", SqrtBox["16"]}], ")"}]}], 
              ")"}], "4"]], TraditionalForm]], "DisplayFormula"]}, Open]]}, 
   WindowSize -> {450, 500}, WindowTitle -> sse, 
   ScreenStyleEnvironment -> sse, (* window margins depend on δ *)
   WindowMargins -> {{100. + δ, Automatic}, {Automatic, 100. + + δ}}]]

And here are the notebooks in the four typically available (for most stylesheets) environments. The "Working" environment is for interactive work; the "Presentation" environment uses a larger font and is typically used for projector-based talks; the "Condensed" environment is similar to the "Working" environment, but more space-efficient; and the "Printout" environment finally is for making printed versions of a notebook. This means it uses still smaller font and true type fonts which results in a suboptimal appearance on screen. Here is the just-defined function putNb used to display the sample notebook in the four different environment styles (all at 100% magnification).

putNb @@@ {{"Working", 0}, {"Presentation", 100}, 
           {"Condensed", 200}, {"Printout", 300}};

B.1.4 Selected Cell Options

In this subsection we will collect a few cell options that are useful from time to time, but fall outside the categories discussed so far and which are sometimes useful for programmatic work.

From a front end structural point of view, the two options that can only appear in cells are CellLabel and CellTag. The CellLabel option we typically use for the In/Out numbering and the CellTags option we use for tagging cells (for hyperlinks say).

Complement[Union[First /@ Options[Cell]], 
           First /@ Options[Notebook]]

We start with options related to showing animations. What are animations in the front end from a notebook-structure point of view? Animations are nothing else then a group of graphics cells, whose cells have some additional options that specify the details of the animation.

Using the last option and the setting 0 for the first and -1 for the last graphic of an animation group results in a cyclic forward and then backward run of the animation.

Using front end functions, we will now animate a sequence of graphics. The used front end functions allow to do operations that typically are carried out by using the menus and mouse clicks in a programmatic manner. We create a new graphic that contains just one input cell.

nbH = NotebookPut[Notebook[
{Cell["Do[Plot[Sin[x + α], {x, -2Pi, 2Pi}, 
       Frame -> True, Axes -> False,
 PlotRange -> {{-2Pi, 2Pi}, {-1.1, 1.1}},
PlotStyle -> {Hue[0]}], {α, 0, 2Pi, 2Pi/60}]", "Input"]},
WindowSize -> {700, 500}, WindowTitle -> "Animation experiments"]]

We select this cell, evaluate it, remove the original input cell, and end up with a sequence of 61 graphics that will form our sample animation. (The various FrontEndTokenExecute used in the following, we will discuss in the next section.)

FrontEndTokenExecute[nbH, "SelectAll"];

FrontEndTokenExecute[nbH, "Evaluate"];

FrontEndTokenExecute[nbH, "SelectAll"];

FrontEndTokenExecute[nbH, "SelectionCloseAllGroups"]

SelectionMove[nbH, Before, Notebook]

SelectionMove[nbH, Next, Cell]

FrontEndTokenExecute[nbH, "Cut"]

We select all graphics cells and animate them.

FrontEndTokenExecute[nbH, "SelectAll"];

FrontEndTokenExecute[nbH, "SelectionAnimate"];

Now, we go on and modify the animation. Interactively this could be done through the option inspector. Here we do it programmatically. We step through all of the 61 graphics cells and set random values of the two animation options AnimationDisplayTime and AnimationCycleOffset.

FrontEndTokenExecute[nbH, "SelectAll"];

FrontEndTokenExecute[nbH, "SelectionOpenAllGroups"]

SelectionMove[nbH, Before, Notebook]

Do[SelectionMove[nbH, Next, Cell];
   (* set the AnimationDisplayTime and 
      AnimationCycleOffset options to have random values *)
   SetOptions[NotebookSelection[nbH], 
     AnimationDisplayTime -> Random[Real, {0, 0.1}], 
     AnimationCycleOffset -> Random[Integer, {-60, 60}]],
     {61}]

Carrying out the animation again yields no longer a smoothly moving sine wave.

FrontEndTokenExecute[nbH, "SelectAll"];

FrontEndTokenExecute[nbH, "SelectionCloseAllGroups"]

FrontEndTokenExecute[nbH, "SelectionAnimate"];

Pause[5]; NotebookClose

The next cell option we want to mention here is CellAutoOverwrite. It is typically never manually or even programmatically set option is CellAutoOverwrite.

The next input prints a cell that has the style "ChapterNumber".

StylePrint[Power @@ (ToExpression /@ Table[FromCharacterCode[k], 
            {k, 945, 969}]), "ChapterNumber"]

We read this cell into the kernel and display its content (we will discuss the function SelectionMove in a moment).

(SelectionMove[EvaluationNotebook[], Previous, Cell, 3];
 NotebookRead[EvaluationNotebook[]] // FullForm)

We see the option CellAutoOverwrite in the last cell. This option is present for the following reason: The cell was generated as the “output/print” of an input cell. But actually, the cell had the style "ChapterNumber". Cells of this style should in general (when manually put there by a human) not be replaced by the output of an input cell just above them. But in the case of an automatic generation, they very probably should be replaced because a reevaluation of the input cell will generate an identical cell. Thee option CellAutoOverwrite -> True forces the original printed cell to be overwritten by a new one.

We remove the printed cell.

(SelectionMove[EvaluationNotebook[], Previous, Cell, 4];
 FrontEndExecute[{FrontEndToken[FrontEnd`EvaluationNotebook[], "Cut"]}])

A more globally acting group of options are the ones that are set for the at the front end level and apply to all notebooks open in the front end. (Of course, through options set at the notebook, cell, or cell content level, one can override these global option settings.) The users’ personal global setting can be found in the private front end initialization package. We read this package into the kernel.

feInitFile = 
Select[FileNames["*", $UserBaseDirectory, ∞],
       StringMatchQ[#, "*FrontEnd\\init.m"]&][[1]]

We carefully convert the package to a set of held expression (to avoid that the options are explicitly set in case they were changed in the current session).

initOptionsSet = 
ReleaseHold[ToExpression[
       ReadList[feInitFile, Record, 
         RecordSeparators -> {}, WordSeparators -> {}][[1]],
         StandardForm, Hold] /. 
            HoldPattern[SetOptions[$FrontEnd, rest__]] :> {rest}];

Here are the options names having explicit settings.

Union[First /@ initOptionsSet]

The values of these options can be changed either interactively through the option inspector or programmatically using SetOptions function.

B.2 Front End Functions and Operations

B.2.1 Navigating and Manipulating Notebook

For automated demonstrations (and/or for automated processing of notebooks which are too large to easily be read into the kernel), operations such as opening and closing cells, as well as moving up and down in a notebook are essential.

The current selection in a notebook (the whole notebook, a cell group, some cells, a single cell, or part of a cell, a formula, or a part of a formula) can be programmatically accessed through NotebookSelection.

We can get the expression underlying the current selection into the kernel (for analysis and modification) using the function NotebookRead.

We can write new cells or parts of cells using the function NotebookWrite.

Sometimes one wants to write expressions in a notebook and might want currently selected parts of the notebook to embed into the to be printed expression. This can be done with the function NotebookApply.

Next, we create a notebook.

nb = NotebookPut @ 
         Notebook[{Cell["A notebook with a single text cell", "Text"]}];

The current selection point is at the beginning of the notebook. So writing some characters into the notebook means writing them in a newly created InputForm cell that is above the already present text cell. (The style of a newly created cell is determined by the stylesheet. The default stylesheet creates by default an input form cell.)

Do[Pause[0.2]; NotebookWrite[nb, If[OddQ[k], (k - 1)/2, "+"]], 
   {k, 25}]

We can remove the current selection with NotebookDelete.

To move around in a notebook, we use the function SelectionMove.

The possible directions are Next, Previous, After, Before, and All. Possible step units are Character, Word, Expression, CellContents, Cell, CellGroup, and Notebook.

The next input selects the just-created input cell.

SelectionMove[nb, All, Cell]

Reading now the current selection into the kernel gives the cell expression of the input cell.

NotebookRead[nb]

We close the notebook.

NotebookClose[nb]

The next input generates an output cell with its content styled as cells of type "Section".

StyleForm[Expand[(x + y)^3], "Section"]

By reading the cell into the kernel, we can see its underlying structure. The boxes representing the typeset powers (see below for a discussion of the boxes involved) are wrapped in a StyleBox of style "Section".

((* select last output cell *)
 SelectionMove[EvaluationNotebook[], Previous, Cell, 4];
 (* read the last output cell into the kernel *)
 NotebookRead[EvaluationNotebook[]] // FullForm)

In the next example, we start with a notebook having one text cell which in turn has the integers 1 to 1000 in it. We position the cursor in front of the text cell and then select the integers, one after the other, and color them randomly and change their font size.

(* make and display notebook with the integers 1 to 1000 *)
numberNb = NotebookPut @ Notebook[
 {Cell[StringJoin @ Table[ToString[k] <> " ", {k, 1000}], "Text"]}];

(* move to the top of the whole notebook *)
 SelectionMove[numberNb, Before, Notebook]

(* move to the text cell *)
SelectionMove[numberNb, Next, Cell]

(* move in front of the content of the text cell *)
SelectionMove[numberNb, Before, CellContents]

(* move from integer to integer and style it *)
Do[SelectionMove[numberNb, All, Word];
   SetOptions[NotebookSelection[numberNb], 
           FontColor -> RGBColor @@ (Table[Random[], {3}]),
           FontSize -> Random[Integer, {12, 28}]];
   SelectionMove[numberNb, After, Word], {1000}]

We close this psychedelic notebook.

Pause[5];
NotebookClose[numberNb]

Next, we create a simpler notebook. It has just one StandardForm cell and we programmatically write "1 2 + 3 4" in this cell. Then, we select the whole cell and read it in the kernel. We see the nested row box structure that was automatically created by the front end.

(* create an empty notebook in the front end *)
nb = NotebookPut[Notebook[]]; Pause[1]; 

(* create empty StandardForm cell *)
NotebookWrite[nb, Cell[BoxData[""], "Input"]];

(* move current selection in front of cell *)
SelectionMove[nb, Before, Notebook];
SelectionMove[nb, Next, Cell];
SelectionMove[nb, Before, CellContents];

(* "type" "1 2 + 3 4" *)
(NotebookWrite[nb, #]; Pause[0.5])& /@ Characters["1 2 + 3 4"];

(* elect typed cell *)
SelectionMove[nb, All, Notebook];

(* read cell into kernel *)
NotebookRead[nb]

We close the notebook and continue.

Pause[5]; NotebookClose[nb]

To go to a specified point in a notebook, we can use the function NotebookLocate.

A more general function to locate specific content in a notebook is NotebookFind.

A potential fourth argument to NotebookFind can restrict the elements where to search in. Possible values are CellContents, CellStyle, CellLabel, and CellTags. By default, the content of cells is searched.

We will use NotebookFind to determine the total running time of the evaluated units sections, subsections, exercises, subexercises) of the GuideBooks. To do this, we find all the "PrintText" cells from the TMGBs`PrintSessionSummary[] inputs and add the data for each chapter.

chapterEvaluationData[chapter_] := 
Module[{openNotebooks, nbH, cpuTimeCells},
 (* the currently open notebooks *)
 openNotebooks = Notebooks[];
 (* open the chapter notebook *)
 nbH = NotebookOpen[ToFileName[ReplacePart["FileName" /. 
                    NotebookInformation[EvaluationNotebook[]], chapter, 2]]];
 (* find all summary cells *)
 NotebookFind[nbH, "Wall clock time elapsed:", All];
 (* read summary cells *)
 summaryCells = NotebookRead[nbH];    
 If[FreeQ[openNotebooks, nbH], NotebookClose[nbH]]; 
 (* extract runtime data *)
 Block[{s = 1, MB = 1}, 
 data  = Transpose[Map[ToExpression, Map[Last, First /@ Cases[summaryCells,
                   _GridBox, Infinity], {2}] /. Cell -> List, {-1}]]];
 (* the four data extracted *)
 Transpose[{{"Inputs evaluated:", "CPU time used:", 
             "Wall clock time elapsed:", "Max. memory used:"}, 
 Flatten[{Total[data[[1]]], Total[data[[2]]], 
          Total[data[[3]]], Max[data[[4]]]}{1, "s", "s", "MB"}]}]]

ceData = chapterEvaluationData /@ guideBooksChapterFileNames

And here is a table showing all the evaluation times and memory usages.

TableForm[
  Append[#, {Plus @@ #1, Plus @@ #2, Plus @@ #3, Max[#4]}& @@ 
  Transpose[#]]&[Map[Last, ceData, {2}]] /. {"s" -> 1/3600`3, "MB" -> 1},
  TableHeadings -> Map[StyleForm[#, FontWeight -> "Bold"]&, 
    {{"P_1", "P_2", "P_3", "P_4", "P_5", "P_6", 
      "G_1", "G_2", "G_3", "N_1", "N_2", "S_1", "S_2", "S_3", "All"},
     {"Inputs evaluated", "CPU time used\n(hours)", 
      "Wall clock time\nelapsed (hours)", "Max. memory used\n(MB)"}}, {-1}]]

This cell has the cell tag "FindMe". The option ShowCellTags -> True is set for this cell to easily see this cell tag.

The next input selects the last cell.

NotebookLocate["FindMe"]

We will now create a small notebook and move around in it using SelectionMove. The function notebookWithNumbers creates a notebook with J sections, each having K subsections, and each of the subsections having M text cells.

(* make a string of numbers to uniquely identify the cells" *)
orderString[l__] := StringJoin[Most[Flatten[
                     Transpose[{ToString /@ {l}, 
                                Table["—", {Length[{l}]}]}]]]]

notebookWithNumbers[J_, K_, M_] :=
 Notebook[Table[(* the nested cells of type 
                   "Section", "Subsection", and "Text" *)
   Cell[CellGroupData[{Cell[orderString[j], "Section"],
          Sequence @@
            Table[Cell[CellGroupData[{Cell[orderString[j, k],
                                           "Subsection"],
                      Sequence @@ Table[Cell[orderString[j, k, m], 
                                             "Text"], {m, M}]},
                    Open]], {k, K}]}, Open]], {j, J}],
         WindowSize -> {300, 600}] /. (* add celltags *)
   Cell[content_, type_] :> Cell[content, type, CellTags -> content]

Here is the new notebook.

exampleNbObject = NotebookPut @ notebookWithNumbers[4, 4, 4]

The next input puts the current selection at the top of the notebook.

SelectionMove[exampleNbObject, Before, Notebook]

Now, we move through all cells and color them as we go along.

Do[(* move to next cell *)
   SelectionMove[exampleNbObject, Next, Cell];
   (* set options for cell *)
   SetOptions[NotebookSelection[exampleNbObject],
              Background -> RGBColor[Random[], Random[], Random[]],
              FontSize -> 24, FontWeight -> "Bold", 
              CellFrame -> True];
   (* wait a moment *)
   Pause[1];
   (* reset options *)
   SetOptions[NotebookSelection[exampleNbObject],
              Background -> Inherited,
              FontSize -> Inherited, FontWeight -> Inherited,
              CellFrame -> Inherited], {12}];

We close the notebook.

NotebookClose[exampleNbObject]

A function that works on Mathematica kernel functions, such as Plot, and Integrate, as well on various notebook parts, is SetOptions. Here is notebook with an important text (cell).

wisdomNb = NotebookPut @ 
Notebook[{Cell["Meine Sätze erläutern dadurch, daß sie der, 
welcher mich versteht, am End als unsinnig erkennt, wenn er durch 
sie—auf ihnen—über sie hinausgestigen ist (Er muß sozusagen 
die Leiter wegwerfen, nachdem er auf ihr hinaufgesteigen ist.)\n 
Er muß diese Sätze überwinden, dann sieht er die Welt richtig.\n           L. Wittgenstein  Tractatus logico\[Hyphen]philosophicus  6.54", 
               "Text"]}, WindowSize -> {820, 760}, 
                         WindowTitle -> "Deep Wisdom"];

SelectionMove[wisdomNb, Next, Cell];

Do[SetOptions[wisdomNb, FontColor -> ToColor[Hue[Sin[x]], RGBColor],
                        FontSize -> Round[36 Sin[x] + 12]],
   {x, 0., 1.Pi, Pi/360}]

We close the notebook.

Pause[5];
NotebookClose[wisdomNb]

Sometimes one wants to carry out menu-like operations purely through the front end. This can be achieved by using various functions in the front end contexts, such as FrontEnd`NotebookApply and FrontEnd`SelectedNotebook instead of System`NotebookApply and System`SelectedNotebook. This is of importance, for instance for button functions defined in stylesheets.

We end this subsection with quickly flipping through The Mathematica Book. Each version of Mathematica comes with a complete set of notebooks containing all of the content of The Mathematica Book. The files are located in the Mathematica program directory in the directory called MainBook. In the next input, we determine the names of the notebooks that form the main book chapters (we do not include the reference guide; we will deal with it later).

(* all files in the Mathematica directory *)
allMathematicaBookFiles = 
   Select[FileNames["*", $InstallationDirectory, Infinity],  
          (StringMatchQ[#, "*MainBook*"] && 
           MatchQ[StringTake[#, {-7, -6}], "0_" | "1_" | "2_" | "3_"])&];
           
(* number of files *)
λ = Length[allMathematicaBookFiles]

Now, we do the actual flipping-through the notebooks. At the core of this process is the function SelectionMove.

startTime = AbsoluteTime[];

Do[Pause[0.5];
   (* open a notebook *)
   nbBook[k] = NotebookOpen[allMathematicaBookFiles[[k]], 
                  WindowMargins -> {{10 + 5 k, Automatic}, 
                                    {Automatic, 10 + 5 k}}];
   (* find number of cells of the notebook *)
   cλ = Length @ Flatten[NotebookGet[nbBook[k]][[1]] //. 
                   Cell[CellGroupData[l_, ___], ___] :> l];
   (* position cursor on top of the notebook *)
    SelectionMove[nbBook[k], Before, Notebook];
    (* scroll down cell by cell *)
    Do[Pause[0.01]; SelectionMove[nbBook[k], Next, Cell], {cλ}],  
   {k, λ}];

(* close all notebooks, last ones first *)
Do[Pause[0.2]; NotebookClose[nbBook[k]], {k, λ, 1, -1}]

And here is how long it took to flip once through the whole online Mathematica Book.

(AbsoluteTime[] - startTime) "Second"

B.2.2 Performing Menu and Keyboard Operations Programmatically

The majority of operations that can be carried out interactively through menu commands (and/or and their keyboard shortcuts) can be carried out programmatically. The individual actions happen though so-called tokens. These tokens are strings and are put into programmatic action using the function FrontEndTokenExecute.

As there are many menu items in Mathematica, so there are many tokens. One can find a complete list in the two files MenuSetup.tr and KeyEventTranslations.tr, which are in the Mathematica installation directory. We read in these two files.

(* all files in the Mathematica directory *)
allMathematicaProgramFiles = FileNames["*", $InstallationDirectory, Infinity];
Length[allMathematicaProgramFiles]

pureFileName[fileNameWithDir_] := 
    StringDrop[fileNameWithDir, 
               StringLength[DirectoryName[fileNameWithDir]]]

(* the two files of interest *)
textResourceFilesWithTokens = Reverse @ 
Select[allMathematicaProgramFiles,
       MatchQ[pureFileName[#],  
              "MenuSetup.tr" | "KeyEventTranslations.tr"]&]

Here are the first few lines of these files.

Short[ReadList[#, Record, RecordSeparators -> {},
         WordSeparators -> {}], 24]& /@ textResourceFilesWithTokens

We see tokens such as "Cut" and "Paste" that have obvious meanings.

We also have access to a notebook with explanations of the tokens. The notebook is located in the help browser subdirectory of the Mathematica directory.

(* all files in the Mathematica directory *)
FETokenHelpBrowserNb = 
   Select[FileNames["*", $InstallationDirectory, Infinity],  
          StringMatchQ[#, "*Documentation*FrontEndTokens.nb"]&];

We open the notebook and quickly flip though a few pages (using the token "ScrollPageNext").

nb  = NotebookOpen[FETokenHelpBrowserNb[[1]]]

Do[Pause[1]; FrontEndExecute[{FrontEndToken[nb, "ScrollPageNext"]}],
   {50}]

We create a new notebook and carry out some token operations. Here is a "random" notebook.

nb = NotebookPut @ Notebook[
       Table[Cell[ToString[Random[Integer, {1, 10^10}]],
                  {"Section", "Subsection", "Text", "Input",
                   "Output"}[[Random[Integer, {1, 5}]]]], {100}]];

The next input selects all cells and 100 times closes and opens all cells.

FrontEndTokenExecute[nb, "SelectAll"];
Do[FrontEndTokenExecute[nb, "SelectionCloseAllGroups"];
   FrontEndTokenExecute[nb, "SelectionOpenAllGroups"], {100}]

NotebookClose[nb]

Next, we start with a new empty notebook.

nb = NotebookCreate[]

We copy an input cell with the content Expand[(x + y)^12] into the empty notebook.

NotebookWrite[nb, Cell["Expand[(x + y)^12]", "Input"]]

Next, we use the tokens "SelectAll" and "Evaluate" to select the cell programmatically and evaluate the cell.

FrontEndTokenExecute[nb, "SelectAll"];
FrontEndTokenExecute[nb, "Evaluate"];

Next, we select the last cell and cut the just-created output cell.

SelectionMove[nb, Previous, Cell];
FrontEndTokenExecute[nb, "Cut"];

Until now, we were carrying out named tokens by using the function FrontEndTokenExecute. A more general function is FrontEndExecute.

To carry out a token using the function FrontEndExecute, we use the construction FrontEndExecute[{FrontEndToken[nb, token]}]. Here the head FrontEndToken can also be used from the context FrontEnd`.

As said in the beginning of this subsection, basically every menu command has a programmatic equivalent that can be used in FrontEndTokenExecute. Here is a simple “typing” example that uses the functions NotebookWrite, SelectionMove, and FrontEndExecute a few times. The concrete tokens used represent key strokes for numbers, operators, characters, fraction and superscripts, and copy and paste. We create a new notebook, type a simple expression in a cell, and evaluate the cell.

𝓅 := Pause[0.5];

(* create empty notebook *)
nb = NotebookPut[Notebook[{}, WindowSize -> {500., 250.},
                 WindowMargins -> {{50., Automatic}, {Automatic, 20.}},
                 WindowTitle -> "Typing by program"]]; 𝓅; 𝓅;
(* create empty StandardForm cell *)
NotebookWrite[nb, Cell[BoxData[""], "Input"]];
(* move current selection in front of cell *)
SelectionMove[nb, Before, Notebook];
SelectionMove[nb, Next, Cell];
SelectionMove[nb, Before, CellContents];
(* "type" "1+2+3+4" *)
(NotebookWrite[nb, #]; 𝓅)& /@ Characters["1+2+3+4"];
(* select the last "3+4" *)
Do[FrontEndExecute[{FrontEndToken[nb, "SelectPrevious"]}]; 𝓅, {3}]
(* create a fraction *)
FrontEndExecute[{FrontEndToken[nb, "Fraction"]}];
(* create a square root in the denominator *)
FrontEndExecute[{FrontEndToken[nb, "Radical"]}];
(* the content of the radical *)
NotebookWrite[nb, 4]; 𝓅;
(* select the whole fraction *)
Do[FrontEndExecute[{FrontEndToken[nb, "ExpandSelection"]}], {3}]; 𝓅;
(* raise the fraction to power 2 *)
FrontEndExecute[{FrontEndToken[nb, "Superscript"]}];      
NotebookWrite[nb, 2]; 𝓅;       
(* move current selection off the power *)
FrontEndExecute[{FrontEndToken[nb, "MoveNextWord"]}];  
(* select the whole square *)
Do[FrontEndExecute[{FrontEndToken[nb, "SelectPrevious"]}]; 𝓅, {1}];
(* copy the square *)
FrontEndExecute[{FrontEndToken[nb, "Copy"]}]; 𝓅;    
(* make a fraction with the square as numerator *)
FrontEndExecute[{FrontEndToken[nb, "Fraction"]}]; 𝓅;
(* "type" "1+2+3+" in the denominator *)
(NotebookWrite[nb, #]; 𝓅)& /@ Characters["1+2+3+"];
(* paste the square *)
FrontEndExecute[{FrontEndToken[nb, "Paste"]}]; 𝓅; 
(* move current selection off the fraction *)
FrontEndExecute[{FrontEndToken[nb, "MoveNextWord"]}]; 𝓅;  
(* "type" the postfix operator " // Together" *)
(NotebookWrite[nb, #]; Pause[0.5];)& /@ Characters[" // Together"];
(* evaluate the expression just\[Hyphen]"typed" *)
FrontEndExecute[{FrontEndToken[nb, "EvaluateCells"]}]; 𝓅; 
(* write a cell into the notebook to close it *)
SelectionMove[nb, After, Notebook]; 𝓅; 
NotebookWrite[nb, Cell["Pause[10]; NotebookClose[EvaluationNotebook[]];", 
                       "Input"]]; 𝓅; 
SelectionMove[nb, Previous, Cell]; 𝓅; 
FrontEndExecute[{FrontEndToken[nb, "EvaluateCells"]}];

We will encounter many more examples of programmatic notebook manipulations through front end tokens in the following sections.

B.3 Typesetting and Boxes

B.3.1 Two-Dimensional Formatting

To display mathematical equations in a manner similarly to books, one has to position symbols, characters, and numbers on a 2D plane. Mathematica InputForm is a strictly sequential form of specifying an expression, OutputForm mimics traditional mathematics on a square grid, and StandardForm and TraditionalForm finally allow to positions expressions in 2D with custom sizing and spacing. Boxes are the underlying expressions in a notebook that allow for 2D positioning of characters and symbols. To show the formatting of boxes, we will in the following either use DisplayForm or CellPrint[Cell[BoxData[…]]]. The first is very convenient, but has the drawback of not using the font of the default output format type. The second form is a bit longer to type, but allows for a more flexible styling and display.

There are quite a few boxes in the System` and other contexts.

Names["*Box"]

Select[Names["*`*Box"], (Context[#] != "System`")&]

We now discuss the boxes most relevant for mathematical typesetting. We start with the typesetting of fractions.

Here is a simple example.

FractionBox["1", "2"] // DisplayForm

The arguments of a FractionBox can themselves be fraction boxes. So, we can build nested fractions.

FoldList[FractionBox, "1", ToString /@ Range[2, 6]] // DisplayForm

FractionBox has two options (most of the following boxes have these two options).

Options[FractionBox]

The option FractionLine governs the appearance of the fraction bar and the option MultilineFunction what happens when the fraction to be formatted is longer than one line. By default, no fraction bar is used in this case.

FractionBox["1", ToString[10^200]] // DisplayForm

With the option setting None, the current line width is ignored. This means that one has to scroll horizontally to see the whole fraction. And in a printed version, parts of the fraction might be missing because they are cut off at the right end of the page.

FractionBox["1", ToString[10^(10^6)], MultilineFunction -> None] // 
                                                     DisplayForm

Using the option setting LineWrapParts uses a fraction bar, numerator, and denominator are individually line wrapped to fit.

FractionBox[ToString[10^200], ToString[10^200 + 1], 
            MultilineFunction -> LineWrapParts] // DisplayForm

Here is again a simple example.

SqrtBox["2"] // DisplayForm

The argument of a SqrtBox can again be a SqrtBox. So, we can build nested square roots.

NestList[SqrtBox, "2", 6] // DisplayForm

Of course, we can also make a fraction with square root boxes as numerator and denominator.

FractionBox[SqrtBox["2"], SqrtBox["3"]] // DisplayForm

And here is a square root box with a fraction inside.

SqrtBox[FractionBox["2", "3"]] // DisplayForm

For an nth root, we can use the two argument RadicalBox.

Here is again a simple example.

RadicalBox["a", "r"] // DisplayForm

The arguments of a RadicalBox can also be of type RadicalBox. So, we can build nested radicals.

FoldList[RadicalBox, "π", ToString /@ Range[2, 6]] // DisplayForm

As mentioned above, the option MultiLineFunction regulates the behavior when too long formatted lines are encountered. So in the following output, no radical sign is used.

Expand[(α + β)^12]^(1/3)

Setting MultilineFunction -> None forces the use of a radical sign and results in a base longer than the current line width.

((* move to the ast output and read it into the kernel *)
 SelectionMove[InputNotebook[], Previous, Cell, 3];
 DisplayForm[
  RadicalBox[#, "3", MultilineFunction -> None]&  @@
    Cases[NotebookRead[EvaluationNotebook[]][[1, 1]],
          RowBox[_?(Length[#] > 10&)], Infinity]])

Here is a nested radical box in TraditionalForm.

CellPrint @ Cell[BoxData[FormBox[
                 Fold[RadicalBox, "0", ToString /@ Range[16]],
                                 TraditionalForm]], "Output"]

Powers are represented through superscripts, and typeset using a SuperscriptBox.

Here is another simple example.

SuperscriptBox["b", "e"] // DisplayForm

The arguments of a SuperscriptBox can again be of any type. This allows building nested powers.

FoldList[SuperscriptBox, "π", ToString /@ Range[2, 6]] // DisplayForm

We will now use the boxes introduced so far to construct some random mathematical typeset expressions by nesting them recursively. rIS[] generates a string of a randomly chosen number between 1 and 9 and randomBox[] generates randomly either a fraction box, a radical box, or a superscript box.

rIS[] := ToString @ Random[Integer, {1, 9}]

randomBox[] := 
Module[{r = Random[], α = 1/3},
       Which[r <= 1 α, FractionBox[rIS[], rIS[]],
             r <= 2 α, RadicalBox[rIS[], rIS[]],
             r <= 3 α, SuperscriptBox[rIS[], rIS[]]
             (* continuable *)]]

We nest the construction by iteratively replacing subparts.

randomNestedExpression[n_] := 
Nest[ReplacePart[randomBox[], #, If[Random[Integer] === 0, 1, 2]]&, 
      randomBox[], n]

SeedRandom[3]
Table[randomNestedExpression[12], {4}] // DisplayForm

Here is a larger collection of expressions generated by randomNestedExpression shown in a Graphics-object.

Show[Graphics[
   Table[{Hue[Random[]],
          Text[StyleForm[DisplayForm @ randomNestedExpression[6],
                         FontSize -> 8], 
                            {Random[], Random[]}]},
          {48}]], PlotRange -> {{-1, 2}, {-1, 2}}]

Using ToExpression, we can convert the last boxes to numerical expressions.

ToExpression[%]

We generate a few hundred of such expressions and calculate their numerical values.

randomNumericalBoxValue[] :=
TimeConstrained[N[#, 4]& @ ToExpression[randomNestedExpression[10]], 1]

SeedRandom[1];
data = Table[randomNumericalBoxValue[], {1000}];

Their first digits frequencies show the order one would expect from Benford’s rule.

{First[#], Length[#]}& /@ 
Split[Sort[RealDigits[#][[1, 1]]& /@ 
      Select[data, (NumberQ[#] && Head[#] === Real && 
                    # =!= Overflow[] && 
                    # =!= Underflow[] && # != 0)&]]]

We continue with the next typeset box. Indices on expressions are typeset using subscripts.

Here is again a simple example.

SubscriptBox["b", "s"] // DisplayForm

The mathematical expression corresponding to a SubscriptBox is Subscript. No direct mathematical meaning (like roots, radicals, or powers) is associated with subscripts.

ToExpression[%] // FullForm

Subscripts and superscripts can be attached simultaneously to a base.

The nested version of an inner subscript box and an outer superscript box is mathematically equivalent to a subsuperscript box, but not the other way around.

{SubscriptBox[SuperscriptBox["b", "e"], "s"],
 SuperscriptBox[SubscriptBox["b", "s"], "e"],
 SubsuperscriptBox["b", "s", "e"]} // DisplayForm

ToExpression[%] // FullForm

Sums and products, and sometimes also integrals are typically written with their limits above and below the sum, product, and integral signs. The boxes corresponding to SuperscriptBox, SubscriptBox, and SubsuperscriptBox with the scripts above and below instead of in the right upper and right lower corner are OverscriptBox, UnderscriptBox, and UnderoverscriptBox.

Here are three examples:

{UnderscriptBox["∑", 
  (* better use a RowBox *) "k>0"],
 OverscriptBox["∫","X"],
 UnderoverscriptBox["∫", SubscriptBox["x", "l"],
                         SubscriptBox["x", "u"]]} // DisplayForm

The simplest, but most versatile and most frequently used box is a RowBox.

Explicitly written sums (head Plus) and products (head Times) are formatted as row boxes. The summands and the operators are both present. For multiplication, the string " " can be used.

RowBox[{"1", "+", "2", "+", "3"}] // DisplayForm

RowBox[{"1", " ", "2", " ", "3"}] // DisplayForm

Function application is also realized through a RowBox.

RowBox[{"f", "[", "x", "]"}] // DisplayForm

Lists are also typeset as row boxes. Each level and each list brace has its own head RowBox.

RowBox[{"{", 
      RowBox[{
        RowBox[{"{", RowBox[{"1", ",", "2", ",", "3"}], "}"}], ",", 
        RowBox[{"{", RowBox[{"4", ",", "5", ",", "6"}], "}"}], ",", 
        RowBox[{"{", RowBox[{"7", ",", "8", ",", "9"}], "}"}]}], "}"}] // 
                                                               DisplayForm

We end our little excursion into the world of Mathematica boxes with grid boxes. Matrices can be formatted as 2D arrays using the box GridBox.

Here is a simple 3×3 array of numbers.

GridBox[{{"1", "2", "3"},
         {"4", "5", "6"},
         {"7", "8", "9"}}] //  DisplayForm

Here is a nested GridBox. The four inner grid boxes have separating lines between the rows and the columns and a frame around the whole GridBox.

Map[GridBox[Table[#, {ToExpression[#]}, {ToExpression[#]}],
            RowLines -> True, ColumnLines -> True, GridFrame -> True]&, 
       GridBox[{{"1", "2"}, {"3", "4"}},
       RowAlignments -> Bottom, ColumnAlignments -> Left], {3}] //  
                                                         DisplayForm

In the last section, we typed a simple expression programmatically. For typing larger expression, we surely do not want to place every single character manually. By knowing which boxes correspond to an expression, we can automate the programmatic typing. (Of course, we could use CellPrint[Cell[BoxData[boxes],…]] to write the typeset expression at once into a notebook, but if you consider the movement as the main action, then typing is more fun.) Here are three simple rules to type expressions of type RowBox, SqrtBox, and FractionBox. The nontrivial issue in typing larger expressions is to select the appropriate part of already typed input to make it, say, the numerator of a fraction by adding the fraction bar and the empty denominator. We could analyze the level at which the various boxes occur to achieve this. A simpler method is to just expand the current selection, and analyze if the current selection has the form of say the numerator that we already typed. (Both methods rely on no manual change of the selection while the program is running.)

Here is a nested fraction containing square roots.

nestedFractions = 
 (Fold[(Sqrt[#]/Sqrt[# /. x -> #2] - Sqrt[# /. x -> #2]/Sqrt[#])^2/
       (Sqrt[#]/Sqrt[# /. x -> #2] + Sqrt[# /. x -> #2]/Sqrt[#])^2&, 
                      1 + x, {y, z (* for the brave , w *)}] + 1)^2

The boxes corresponding to the last expression have 72 square root boxes, 45 fraction boxes, 19 superscript boxes, and 106 row boxes (for the next level, the numbers would be 584, 365, 247, and 839).

(nestedFractionBoxes = MakeBoxes[#, StandardForm]& @ nestedFractions) // 
                                             FullForm // Short[#, 8]&

{#, Count[nestedFractionBoxes, #, Infinity, Heads -> True]}& /@ 
          {SqrtBox, FractionBox, SuperscriptBox, RowBox}

As mentioned, we will use NotebookRead to read in the current selections to see if they correspond to the needed selection. We remove the unneeded form boxes and box data containers.

stripNBRead[boxes_] := boxes //. StyleBox[c_, _] :> c //. 
                       FormBox[x_, _] :> x //. BoxData[b_] :> b

We will type a superscript box (representing a power) by first typing the base (which could contain boxes itself), then selecting the base, go to the raised exponent position and then typing the exponent itself.

type[SuperscriptBox[base_, exp_]] :=  
((* type the base *) type[base]; 
 (* expand selection until the base is selected *)
 While[read = stripNBRead @ NotebookRead[nb]; read =!= base,
       FrontEndExecute[{FrontEndToken[nb, "ExpandSelection"]}]; 𝓅];
 (* move to exponent position and type exponent *)
 FrontEndExecute[{FrontEndToken[nb, "Superscript"]}]; 𝓅;
 type[exp]; 𝓅;
 (* move out *)
 FrontEndExecute[{FrontEndToken[nb, "MoveNext"]}]; 𝓅)

We will type a square root box (representing a square root), by first creating the square root sign and then typing the base (which could contain boxes itself).

type[SqrtBox[base_]] :=  
(FrontEndExecute[{FrontEndToken[nb, "Radical"]}]; 𝓅;
 (* now type the base *) type[base];
 FrontEndExecute[{FrontEndToken[nb, "MoveNext"]}]; 𝓅;)

We will type a fraction box (representing a fraction), by first typing the numerator (which could contain boxes itself), then selecting the numerator and creating the fraction with empty denominator. Then we will be typing the denominator and move out of the denominator.

type[FractionBox[num_, den_]] := 
((* type the numerator *) type[num]; 
 (* expand selection until the numerator is selected *)
 While[read = stripNBRead @ NotebookRead[nb];
       read =!= num,
       FrontEndExecute[{FrontEndToken[nb, "ExpandSelection"]}]; 𝓅];
 (* move to denominator position *)
 FrontEndExecute[{FrontEndToken[nb, "Fraction"]}]; 𝓅;
 (* type the denominator *) type[den]; 𝓅;
  (* move out *)
 FrontEndExecute[{FrontEndToken[nb, "MoveNext"]}]; 𝓅)

We will type a row box (representing a sum or a fraction of a function application), by typing the elements consecutively.

type[RowBox[l_List]] := type /@ l

We type a single character finally, by writing it into the current selection using the function NotebookWrite. For a more colorful presentation, we write the characters in different colors.

type[c_String] := (𝓅; NotebookWrite[nb, 
       StyleBox[c, FontColor -> ToColor[Hue[N[(λ++)/Λ]], RGBColor]]])

We create a notebook in which we will type the nested fractions. While notebooks have the Editable option, editing means both manual editing and programmatic editing. Because typing (or even clicking into the notebook nb) would change the current selection (and as a result the read expressions are not what they should be), we prominently warn not to type into the to-be-created notebook.

𝓅 := Pause[0.2];

nb = NotebookPut @ (* the notebook into which we will programmatically type *)
    Notebook[{Cell["Do not type (or even click) in this notebook!", "Section"]}, 
             WindowSize -> {960, 860}, Magnification -> 2.,
             WindowMargins -> {{30, Automatic}, {Automatic, 20}},
             WindowTitle -> "Magic colorful typing",
             CommonDefaultFormatTypes -> {"Input" -> StandardForm}, 
             FormatType -> DefaultInputFormatType, 
             AutomaticCommonDefaultFormatTypes -> 
             {"Input" -> StandardForm}]; 𝓅; 𝓅;

We create an empty input cell in which we will type the nested fraction boxes.

SelectionMove[nb, After, Notebook]
NotebookWrite[nb, Cell[BoxData[""], "Input",  FontSize -> 12]];
SelectionMove[nb, Previous, CellContents]

Next, we do the programmatic typing.

Λ = Count[nestedFractionBoxes, _String, {-1}]/5; λ = 0; 
type[nestedFractionBoxes];

We type one more expression programmatically. This time, we use a subsuperscript box. This is the expression to be typed.

nestedSubSuper = Nest[Subsuperscript[#, #, #]&, Δ, 6];

And these are the boxes to be typed. We remove the (anyway invisible) interpretation boxes.

(nestedSubSuperBoxes = MakeBoxes[#, StandardForm]& @ nestedSubSuper //.
                             InterpretationBox[boxes_, __] :> boxes) // 
                                              FullForm // Short[#, 14]&

The rule for typing a subsuperscript box is a generalization of the above rule for a superscript box.

type[SubsuperscriptBox[base_, sub_, sup_]] :=  
((* type the base *) type[base]; 
 (* expand selection until the base is selected *)
 While[read = stripNBRead @ NotebookRead[nb]; read =!= base,
       FrontEndExecute[{FrontEndToken[nb, "ExpandSelection"]}]; 𝓅];
 FrontEndExecute[{FrontEndToken[nb, "Subscript"]}]; 𝓅;
 (* type the subscript *) type[sub]; 𝓅;
 (* expand selection until the subscript is selected *)
 While[read = stripNBRead @ NotebookRead[nb]; read =!= sub,
       FrontEndExecute[{FrontEndToken[nb, "ExpandSelection"]}]; 𝓅];
 FrontEndExecute[{FrontEndToken[nb, "MoveNext"]}];
 (* move to superscript position *)
 FrontEndExecute[{FrontEndToken[nb, "Otherscript"]}]; 𝓅;
 (* type the superscript *) type[sup]; 𝓅;
 (* expand selection until the superscript is selected *)
 While[read = stripNBRead @ NotebookRead[nb]; read =!= sup,
       FrontEndExecute[{FrontEndToken[nb, "ExpandSelection"]}]; 𝓅];
 FrontEndExecute[{FrontEndToken[nb, "ExpandSelection"]}];
 FrontEndExecute[{FrontEndToken[nb, "MoveNext"]}]; 𝓅)

This time, we will not color the characters to be typed.

type[c_String] := (𝓅; NotebookWrite[nb, c])

We prepare a new cell for typing and type the expression as fast as possible.

SelectionMove[nb, After, Notebook]
NotebookWrite[nb, Cell[BoxData[""], "Input", 
                       ScriptSizeMultipliers -> 1, FontSize -> 10]]
SelectionMove[nb, Previous, CellContents]

time1 = AbsoluteTime[];
𝓅 := Pause[0];
type[nestedSubSuperBoxes];
time2 = AbsoluteTime[];

We summarize how much was typed for the last expression and slowly close the notebook.

SelectionMove[nb, After, Notebook]

NotebookWrite[nb,  (* box counts and time needed *)
Cell["Done!\nTyped " <> 
  ToString[Count[nestedSubSuperBoxes, _SubsuperscriptBox, Infinity]] <>
  " SubsuperscriptBoxes with " <>
   ToString[Count[nestedSubSuperBoxes, "Δ", Infinity]] <> 
   " in only " <> ToString[N[time2 - time1, 3]] <>
  " seconds.\nTry to do it faster!", "Text", 
    FontSize -> 12, FontWeight -> "Bold"]]

SelectionMove[nb, After, Notebook]
NotebookWrite[nb, Cell["Now you can type in this notebook!", "Section"]];
Pause[6];
NotebookWrite[nb, Cell["… if you are fast enough …", "Section"]];
Pause[1];
NotebookClose[nb]

B.3.2 Tweaking Formula Appearances

When one typesets a mathematical expression, for the majority of cases, Mathematica’s default choices for options will give a good-looking result. But like with everything, there are always exceptions. For instance, one would like to typeset the letter standing for a matrix in bold nonitalic versus the plain italic version used by default for a single-letter symbol (of course, Mathematica does not know in which cases a symbol represents which kind of expression in typesetting). Fortunately, Mathematica provides a long list of possible options and option settings to tweak the appearance of typeset formulas to realize a wide variety of visual results. In this subsection, we discuss some settings frequently used in typesetting higher mathematics.

We start by having a look at the options that were used in the display formulas (meaning typeset expressions appearing within cells of type "DisplayFormula") of the GuideBooks.

(* the directory of the GuideBooks *)
guideBooksDirectory = ToFileName[Delete["FileName" /. 
           NotebookInformation[EvaluationNotebook[]], 2]];
           
(* the file names of the GuideBooks chapters *)           
guideBooksChapterFiles = Flatten[FileNames[#, guideBooksDirectory]& /@ 
         {"*Programming*", "*Graphics*", "*Numerics*", "*Symbolics*"}];
         
(* the rules used in the display formulas *)
allDisplayFormulaModifications = Union[Flatten[
   Union[Cases[First /@ Cases[Get[#],
                             Cell[_, "DisplayFormula", ___], Infinity],
              _Rule, ∞]]& /@ guideBooksChapterFiles]];
              
(* the used option names and the used settings *)
{#[[1, 1]] -> (If[Length[#] > 1, Alternatives @@ #, #[[1]]]& @ 
       (Union[Last /@ #]))}& /@ 
              Split[allDisplayFormulaModifications, #1[[1]] === #2[[1]]&]

Now, let us discuss some of the most frequently used style modifications for boxes and how to apply them. We begin with the removal of the (by default) purple unmatched brackets, braces, and parenthesis. The left opening brace in the following typeset formula appears in purple. (For a direct way to see the action of an option and to easily remove the various typeset forms, we will print cells of style output showing the various characteristics. The output cells produces will be StandardForm or TraditionalForm box expressions because no nontrivial typeset modification can be carried out in InputForm because it does not have boxes at all.)

CellPrint[Cell[BoxData[FormBox[RowBox[{"{", GridBox[{
            {"x"}, {"y"}}]}], TraditionalForm]], "Output"]]

We can remove the purple coloring by setting the "UnmatchedBracketStyle" style option setting to None. Because the purple coloring happens automatically and is not explicit set through an option, the "UnmatchedBracketStyle" style option is a suboption of the AutoStyleOptions family. Here are three possibilities to remove the purple coloring: Setting the option just for the opening brace, setting the option for the whole typeset expression, and setting the option for the whole cell. Visually the three results are identical.

CellPrint[Cell[BoxData[FormBox[
   RowBox[{StyleBox["{", AutoStyleOptions -> 
                            {"UnmatchedBracketStyle" -> None}], 
    GridBox[{{"x"}, {"y"}}]}], TraditionalForm]], "Output"]]

CellPrint[Cell[BoxData[FormBox[
 StyleBox[#, AutoStyleOptions -> {"UnmatchedBracketStyle" -> None}]& @
   RowBox[{"{", GridBox[{{"x"}, {"y"}}]}], TraditionalForm]], "Output"]]

CellPrint[Cell[BoxData[FormBox[RowBox[{"{", 
       GridBox[{{"x"}, {"y"}}]}], TraditionalForm]], "Output",
            AutoStyleOptions -> {"UnmatchedBracketStyle" -> None}]]

We continue with the next option.

When displaying digits of numbers, one frequently uses a notation like π=3.14159…. If one types in the visible sequence of characters, one obtains the following.

CellPrint[Cell[BoxData[FormBox[RowBox[{"π", "=", 
                RowBox[{"3.14159", "…"}]}], TraditionalForm]], "Output"]]

But when interpreting the last expression, we obtain a product between the number 3.14159 and the ellipsis character. The ellipsis character is a letter-like character and behaves in typesetting no differently than any other letter like a or α.

ToExpression[BoxData[FormBox[RowBox[{"π", "=", 
                RowBox[{"3.14159", "…"}]}], TraditionalForm]], 
                TraditionalForm, Hold] // FullForm

While we cannot avoid the interpretation without a TagBox or InterpretationBox, the visual appearance can easily be modified using the ZeroWidthTimes option.

CellPrint[Cell[BoxData[
    FormBox[RowBox[{"π", "=", 
            StyleBox[RowBox[{"3.14159", "…"}], ZeroWidthTimes -> True]}], 
             TraditionalForm]], "Output"]]

Be aware that the following modification of the input containing just one string—the concatenated "3.14159…" is immediately converted to a product by the front end typeset code.

CellPrint[Cell[BoxData[FormBox[RowBox[{"π", "=", RowBox[{"3.14159…"}]}], 
             TraditionalForm]], "Output"]]

The most used modifications for integrals and sums are LimitsPositioning and SpanMinSize and SpanMaxSize.

Here is an example, where the LimitsPositioning option is given as a suboption of the UnderoverscriptBoxOptions, which in turn is applied through a StyleBox.

CellPrint[Cell[BoxData[MakeBoxes[1/(1 + Sum[k^z, {k, Infinity}]), 
                                 StandardForm]], "Output"]]

CellPrint[Cell[BoxData[MakeBoxes[1/(1 + Sum[k^z, {k, Infinity}]), 
                                 StandardForm]], "Output"] /. 
           HoldPattern[fb_FractionBox] :> (* apply to all parts *)
           StyleBox[fb, UnderoverscriptBoxOptions -> 
                        {LimitsPositioning -> False}]]

(* read the last cell into the kernel and show its form *)
NotebookRead[SelectionMove[InputNotebook[], Previous, Cell, 3];
             InputNotebook[]] // FullForm

The second modification of SubsuperscriptBox and UnderoverscriptBox that occurs frequently is the SpanMinSize, SpanMaxSize pair of options. Here is a cell containing a typeset expression for a multidimensional integral.

multiIntegrateCell =
Cell[BoxData[FormBox[(* a triple integral *)
      RowBox[{UnderoverscriptBox["∫", RowBox[{"-", "∞"}], "∞"], 
        RowBox[{UnderoverscriptBox["∫", RowBox[{"-", "∞"}], "∞"], 
          RowBox[{"…", RowBox[{UnderoverscriptBox["∫", 
                RowBox[{"-", "∞"}], "∞"], " ", 
              RowBox[{FractionBox["1", RowBox[{"1", "+", 
                    RowBox[{ SubscriptBox["x", "1"], 
                      SubscriptBox["x", "2"], "…", " ", 
                      SubscriptBox["x", "n"]}]}]], 
           RowBox[{"\[DifferentialD]", SubscriptBox["x", "n"]}], "…", 
           RowBox[{"\[DifferentialD]", SubscriptBox["x", "2"]}], 
           RowBox[{"\[DifferentialD]", SubscriptBox["x", "1"]}]}]}]}]}]}], 
           TraditionalForm]], "Output"];

The default display form makes the innermost integral sign too small.

CellPrint @ multiIntegrateCell

Using an explicit minimal span size yields the form one expects with all integral signs being equal size.

CellPrint @ (multiIntegrateCell /. uosb_UnderoverscriptBox :>
                                   StyleBox[uosb, SpanMinSize -> 4])

We continue with the option that determines how subscripts and superscripts shrink in size compared with their base.

Next, we discuss the ScriptSizeMultipliers option. Here is a cell showing a nested fraction.

nestedFractionCell = Cell[BoxData[
   MakeBoxes[#, StandardForm]& @ Nest[1/(1 + #)&, x, 6]], "Output"];

By default, the inner numerators and denominators get smaller until they reach a minimum size.

CellPrint @ nestedFractionCell

The minimum size of the elements is determined by the ScriptMinSize option. In the next input, we allow the inner numerators and denominators to shrink down to 1 point.

CellPrint @ Map[StyleBox[#, ScriptMinSize -> 1]&, 
                 nestedFractionCell, {2}]

By default, the size of the elements is reduced by a factor 0.71 for each nesting level. Using the option ScriptSizeMultipliers, we can give any value for the multipliers. Here are four examples. The setting ScriptSizeMultipliers -> 1 (second cell below) is typically used for continued fractions. The multipliers can be different for each level (third cell below) and the inner levels can even grow in size (fourth cell below).

Function[ssm, CellPrint @ Map[StyleBox[#, ScriptSizeMultipliers -> ssm]&, 
                              nestedFractionCell, {2}]] /@ 
                                 {0.5, 1, {0.5, 2, 0.5, 2, 0.5, 2}, 1.5};

Options[GridBox]

(SelectionMove[EvaluationNotebook[], Previous, Cell, 3];
 Do[FrontEndExecute[{FrontEndToken[FrontEnd`EvaluationNotebook[],
                    "ToggleShowExpression"]}];
    Pause[6], {2}])

So far, we discussed in this subsection options of previously discussed boxes that influence the detailed representation and rendering of the boxes. There is in addition, one box, the AdjustmentBox, that lets one modify the absolute positioning of expressions in the “display plane”.

AdjustmentBox has two options: BoxMargins and BoxBaselineShift.

Options[AdjustmentBox]

The next input generates a notebook with cells containing the expression 1 + 2. We shift the 2 up and down using the BoxBaselineShift option and change the space between the "+" and the two using the BoxMargins option.

NotebookPut[Notebook[
Table[Cell[BoxData[FormBox[
 RowBox[{"1", "+", 
 StyleBox[AdjustmentBox["2", 
   BoxMargins -> {φ {Cos[φ], Sin[φ]}, {0, 0}},
    BoxBaselineShift -> 4 Sin[φ]],
          FontColor -> RGBColor[1, 0, 0]]}], 
          TraditionalForm]], "Input", FontSize -> 24],
          {φ, 0., 3., 1/10}]]]

(Pause[5]; NotebookClose[%])

Lastly, let us mention that, of course, all the style modifications that are possible for text are also applicable for box expressions. Here is a simple formula. The multiletter symbol det is not italic and the single letter symbols A, B, and C are in italic.

CellPrint[Cell[BoxData[FormBox[
      RowBox[{"C", "=", RowBox[{"det", " ", RowBox[{"A", ".", "B"}]}]}], 
                     TraditionalForm]], "Output"]]

Now, we change det to be in italic and A, B, and C to bold, nonitalic.

CellPrint[Cell[BoxData[FormBox[
      RowBox[{StyleBox["C", FontWeight -> "Bold", 
                            FontSlant -> "Plain"], "=", 
        RowBox[{StyleBox["det", FontSlant -> "Italic"], " ", 
          RowBox[{StyleBox["A", FontWeight -> "Bold", 
                                FontSlant -> "Plain"], ".", 
            StyleBox["B", FontWeight -> "Bold",  
                          FontSlant -> "Plain"]}]}]}], 
             TraditionalForm]], "Output"]]

The next input shows the unformatted version of the last output cell for a few seconds.

B.3.3 Creating Typesetting Rules

To format expressions in a customized manner, it is possible to define custom typesetting rules. Because typeset expressions are represented as boxes, the naming of the following two functions for making and interpreting boxes is quite natural.

Here format is typically StandardForm or TraditionalForm, but other user-defined formats are possible.

A few Mathematica functions have user-visible format rules. To find them, we must first make sure that all rules are currently present (some might reside in start up packages that are only loaded when needed). So, we will evaluate function[] for all built-in functions to force the rules to be loaded. Because this will give many messages of the form function::"argrx", we will carry out this "messy" calculation in a dynamically created new notebook that we discard after evaluation. Here is the notebook that we will discard shortly.

notebookThatWillGenerateManyMessages =
Notebook[{Cell[BoxData[MakeBoxes[
 (((* print currently evaluated expression to see progress *)
   Print[StringJoin["Evaluating ", #, "[]"]]; 
         ToExpression[StringJoin[#, "[]"]]) & ) /@ 
         (* remove functions that might make "trouble" *)
   DeleteCases[Names["*"], "Abort" | "Break" | "Continue" | 
      "Dialog" | "Exit" | "Quit" | "ExitDialog" | "Edit" | 
      "EditDefinition" | "EditIn" | "Goto" | "Throw" | "On" | 
      "TraceDialog" | "FileBrowse" | "NotebookSave" | 
      "NotebookRead" | "ConsoleMessage" | "Print" | "In" | 
      "Input" | "InputString" | "NotebookCreate" | "Interrupt" | 
      "FrontEnd`NotebookPut" | "NotebookOpen" | "NotebookPut" | 
      "$Inspector" | "ConsolePrint"]; 
 (* avoid premature closing *)     
 allFunctionEvaluationsDone = True; , StandardForm]], "Input"]}];

nb = NotebookPut[notebookThatWillGenerateManyMessages]

Now, we evaluate the spun-off notebook.

FrontEndTokenExecute[nb, "SelectAll"];
FrontEndTokenExecute[nb, "Evaluate"];
Pause[5]

Because the evaluation queue of the front end will potentially evaluate the next inputs before having carried out all of the evaluation in the spun-off notebook, we check the value allFunctionEvaluationsDone that does not have the value True before the evaluations in the spun-off notebook are finished.

While[Not[TrueQ[allFunctionEvaluationsDone]]];
(* now we can discard the temporary notebook *)
NotebookClose[nb]

To access the format rules, which intentionally are hidden using the ReadProtected attribute, we remove this attribute from all non-Locked symbols.

heldFunctionsWithRules = (ToExpression[#, StandardForm, HoldForm]& /@ 
   Select[Names["*"], (MemberQ[Attributes[#], ReadProtected] &&
                       FreeQ[Attributes[#], Locked])&]);

Length[heldFunctionsWithRules]

Function[f, ClearAttributes[f, ReadProtected], {HoldAll}] @@@ 
                                       heldFunctionsWithRules;

We have a look at all added definitions. The majority of definitions present now are DownValues and UpValues.

(* which kind of values are present? *)
allAddedDefinitions = 
  Function[f, {OwnValues[f], DownValues[f], UpValues[f], 
               SubValues[f], FormatValues[f], NValues[f]},
               {HoldAll}] @@@ Take[heldFunctionsWithRules, All];

Plus @@@ Sign[Transpose[Map[Length, allAddedDefinitions, {2}]]]

These are all the contexts that supplied symbols.

Union[Context /@ allSymbols]

And here are the functions that have format rules that are user-visible.

functionsThatHaveFormatRules = 
 Select[heldFunctionsWithRules, 
        Function[f, (FormatValues @@ f) =!= {}, {HoldAll}]];
        
 Length[functionsThatHaveFormatRules]

functionsThatHaveFormatRules

For brevity, we remove the context from the definitions of the typeset rules.

(* remove the context from a symbol *)
removeContext[s_Symbol ] := 
Module[{S = ToString[s], sp, sShort},
        (* locate context markers in the string of the symbol *)
        sp = StringPosition[S, "`"]; 
        (* remove parts to the left of the context marker *)
        sShort = If[sp === {}, s, ToExpression[StringDrop[S, Max[sp]]]]; 
        sShort]

(* remove the context from all symbols of an expression *)
toContextFreeSymbols[expr_] :=
Module[{allVars = Union[Cases[expr, _Symbol, {-1}, Heads -> True]], 
        newVars, rr}, 
      allVars = Union[Cases[allVars, _Symbol, {-1}]]; 
      newVars = removeContext /@ Union[Cases[allVars, _Symbol, {-1}]];
      rr = Rule @@@ Transpose[{allVars, newVars}];
      expr //. rr]

Here are the actual typeset definitions.

formatDefinitions = 
((Function[f, FormatValues[f], {HoldAll}] @@@ 
                functionsThatHaveFormatRules) // toContextFreeSymbols)

We see many definitions of the form HoldPattern[MakeBoxes[..., format]] :> … that contain on the right-hand side a TagBox or a recursive call to MakeBoxes.

{#, Count[formatDefinitions, #, Infinity, Heads -> True]}& /@
     {TagBox, InterpretationBox, MakeBoxes}

Some of the built-in traditional form typeset rules produce quite complicated box structures. Here is an example using the Meijer G function from Chapter 3 of the Symbolics volume. We see a grid box, and many TagBox-structures. Because the order specification (Subscript[G, 4,6]^(3,2)) is derived from the actual argument structure, the subsuperscripts box part is not editable.

MakeBoxes[MeijerG[{{a, b}, {c, d}}, {{a, b, c}, {c, d, e}}, z],
          TraditionalForm] // FullForm

MakeExpression returns it result wrapped in HoldAllComplete. This avoids that any rule (downvalues, upvalues, and so on) ever fires and evaluates the expression. Here is a simple example.

MakeExpression[SqrtBox[RowBox[{"1", "+", "1"}]]]

Because MakeBoxes has the HoldAllComplete attribute, we cannot just call MakeBoxes[%] to convert the last output back to typeset boxes. We must make sure that the argument of MakeBoxes is evaluated. By applying (meaning using the function Apply), the function MakeBoxes to the Sqrt[1 + 1] expression and the HoldAllComplete gets effectively replaced by MakeBoxes and the argument has now time to evaluate. The result is again the box structure SqrtBox[RowBox[{"1", "+", "1"}]].

MakeBoxes @@ %

We give some examples of typeset rules. Assume we want to write the result of FactorInteger in a nice form. The result should be a product of prime powers. The following function implements this formatting. While we could construct the arguments to the various boxes (SuperscriptsBox and RowBox) ourselves, we prefer to write the factorization in a standard form and then use the built-in rules for formatting powers and products.

MakeBoxes[primeDecompositionForm[l_],
           StandardForm | TraditionalForm] :=
 MakeBoxes @@ If[Length[l] >= 2, (Times @@@ Apply[Power, Hold[l], {-2}]),
                 DeleteCases[Apply[Power, Hold[l], {-2}], List, 
                              {0, Infinity}, Heads -> True]]

Here are two an example.

(* a prime number *)
FactorInteger[7] // primeDecompositionForm

(* a composite number *)
FactorInteger[2^4 3^3 5^2 7] // primeDecompositionForm

The interpretation of boxes can be influenced by the second argument of MakeExpression (the format). Using for instance boxes that represent traditional mathematics and interpret them using the StandardForm rules gives a result that in most cases is not desired. Here is an example where function application turns into multiplication.

MakeBoxes[Sin[x], TraditionalForm]

MakeExpression[%, StandardForm]

FullForm[%]

We will give a slightly more complicated example for a user-defined typeset rule. The function RootSum, discussed in Chapter 1 of the Symbolics volume, represents a sum of expressions over the roots of a univariate polynomial. So, we will define a format rule that will contain a sum with the polynomial as an underscript.

MakeBoxes[RootSum[poly_, f_], TraditionalForm] := 
Block[{x},
      RowBox[{UnderscriptBox["∑", 
                             MakeBoxes[#, TraditionalForm]& @ poly[x]], 
              MakeBoxes[#, TraditionalForm]& @ f[x]}]]

Here is an indefinite integral which is found in terms of a RootSum-object.

int = Integrate[(z^5 + 2z - 4)/(z^7 + 4 z - 1), z]

Applying the just-defined typeset rule gives the following result.

TraditionalForm[int]

Implementing the corresponding MakeExpression definition is straightforward too.

MakeExpression[RowBox[{UnderscriptBox["∑", polyBoxes_], 
                       fBoxes_}], TraditionalForm] := 
Module[{p = Function[x, #]& @@ MakeExpression[polyBoxes, TraditionalForm],
        f = Function[x, #]& @@ MakeExpression[fBoxes, TraditionalForm]},
        RootSum[p, f]]

(* interprete boxes as a RootSum *)
MakeExpression[MakeBoxes[#, TraditionalForm]& @ int,
               TraditionalForm]

A possibility to embed format information in the boxes itself is FormBox.

In the next input, we add the FormBox with a TraditionalFormat box. Now, we get the desired result Sin[x].

MakeExpression[FormBox[MakeBoxes[Sin[x], TraditionalForm], 
                       TraditionalForm]]

Form boxes can be embedded within other boxes. The next input shows this situation.

MakeExpression[
  BoxData[RowBox[{RowBox[{"Sin", "[", "x", "]"}], " ", "+", " ", 
          FormBox[RowBox[{"cos", "(", "x", ")"}], "TraditionalForm"]}]]]

In the last subsection, we discussed the boxes that result in 2D expression formatting. But we did not discuss all boxes. Two important boxes, that regulate the interpretation of typeset expressions rather than their appearance, are TagBox and InterpretationBox. (This is the reason these two boxes are discussed in this subsection.)

As a rule, in StandardForm the meaning of an expression is derivable from its formatting. As with every rule, there are a few exceptions to this rule. The most notable one being the fact the string quotes are not displayed in output. Another set of exceptions are (unevaluated) derivatives. For instance, the fourth derivative of an abstract function ζ[z] displays as Derivative[4][ζ][z]. When this expression is input in a StandardForm input cell as a plain superscript box (which will look identically), it will be interpreted as the fourth power of the function ζ[z] (the parents around the four do not force a special interpretation of the underlying superscripts box. But when the expression Derivative[4][ζ][z] is evaluated and the output is edited manually (in the automatically copied down input form version of the cell), for instance by replacing the 4 by a 6 and the resulting cell is evaluated, then Mathematica correctly interpreted this as the sixth derivative of the function ζ[z]. The reason that such a change is possible is the fact that the superscripts box is the first argument of a TagBox and the second argument of this tag box is Derivative.

Here are the functions MakeBoxes and MakeExpression applied to the example at hand.

MakeBoxes[ζ''''[z], StandardForm]

MakeExpression[RowBox[{SuperscriptBox["ζ", 
                   TagBox[RowBox[{"(", "4", ")"}], 
                          Derivative]], "[", "z", "]"}]]

In TraditionalForm formatting, for many expressions, there is a larger discrepancy between the form of an expression and the underlying Mathematica meaning. To allow formulas in TraditionalForm to be edited and correctly interpreted, many special functions and other expressions have TagBoxes in their formatted version. Here is a simple example, the golden ratio.

MakeBoxes[GoldenRatio, TraditionalForm]

By default, the second argument of TagBox is applied to the parsed first argument. Here is a simple example.

MakeExpression[BoxData[TagBox[FractionBox["x", "y"], myHead]], 
               StandardForm]

The box TagBox has a few options.

Options[TagBox]

We will not discuss them here; see the help browser for details. We just want to mention the Editable option. Because the meaning of a tag box is different from its displayed form, editing the displayed form might result in an expression where the formatting and the intended meaning deviate substantially. To avoid that one ever does edit a “meaning-sensitive” structure, the option Editable is sometimes set to have the value False. Here is an example, the formatted version of a binomial coefficient in TraditionalForm. One can edit the elements m and n, but not the whole binomial coefficient. The following TagBox uses an interpretation template instead of a tag.

MakeBoxes[Binomial[m, n], TraditionalForm]

The next input defines a rule to make an expression (give a Mathematica meaning) to a Kronecker symbol.

MakeExpression[TagBox[RowBox[{"(", FractionBox["a", "b"], ")"}], 
                      KroneckerSymbol], TraditionalForm] := 
               KroneckerSymbol[MakeExpression[a, TraditionalForm], 
                               MakeExpression[b, TraditionalForm]]

Formatted it looks like a fraction with parentheses.

ks = TagBox[RowBox[{"(", FractionBox["2", "3"], ")"}], KroneckerSymbol];
ks // DisplayForm

But interpreted it has the head KroneckerSymbol.

MakeExpression[ks]

Here is the corresponding formatting the Kronecker symbol. We model it after the formatting rule predefined for JacobiSymbol.

Attributes[JacobiSymbol] = {};
??JacobiSymbol

MakeBoxes[KroneckerSymbol[a_, b_], TraditionalForm] := 
    TagBox[RowBox[{"(", FractionBox[MakeBoxes[a, TraditionalForm], 
                                    MakeBoxes[b, TraditionalForm]], ")"}],
           KroneckerSymbol]

Here is an example for abstract a and b.

KroneckerSymbol[a, b] // TraditionalForm

We see the hidden TagBox by reading the last cell by reading it into the kernel.

(SelectionMove[EvaluationNotebook[], Previous, Cell, 3];
 NotebookRead[EvaluationNotebook[]])

In a tag box, the meaning and display form of an expression can be different. But they are typically still related to each other (like the arguments are the same), and the head is just interpreted as another function. In an InterpretationBox, no relation between the meaning and the display form typically exists. The following shows that the typeset version of the harmonic numbers is a InterpretationBox (again, to avoid interactive editing, the option Editable is set to False).

MakeBoxes[HarmonicNumber[n], TraditionalForm]

Here is an extreme example. The sum 1 + 1 is interpreted as 4.

MakeExpression[InterpretationBox[RowBox[{"1", "+", "1"}], 4]]

Being at the end of this subsection, let us mention the Notation package by J. Harris. This quite large and elaborate package allows defining complicated typeset formats and their interpretation in a convenient interactive manner. Using the Action -> PrintNotationRules option of the function Notation allows to easily see the MakeBoxes and MakeExpression rules that one has to create to use more complicated notations

B.4 Buttons, Hyperlinks, and Palettes

B.4.1 General Buttons

Buttons are (inter)active front end elements that can be implemented and customized by users (unlike the predefined menu items). They allow carrying out many functions and operations with a simple mouse click (of the button). In a notebook (or a palette, which is nothing else than a notebook with appropriately set options), a button is represented as ButtonBox. (A menu item is not a button box.)

By default, a button is displayed with a background to make it easily recognizable.

ButtonBox["I am a button"] // DisplayForm

Clicking the last button has no effect (yet). There are two reasons for this: a) we have not specified any action to be performed by the button and b) buttons, by default, are set to be inactive.

Here are the options that can be specified for a ButtonBox.

Options[ButtonBox]

The options ButtonMinHeight, ButtonExpandable, ButtonMargins, ButtonFrame, and ButtonNote specify the appearance of a button. Here is a more stylish button than the last one.

(* color the letters of a string in rainbow manner *)
toRainbowColor[s_String] := 
Module[{λ = StringLength[s]},
       MapIndexed[StyleBox[#1, "Subsection", FontColor -> 
                                ToColor[Hue[N[#2[[1]]/λ]], RGBColor]]&, 
                  Characters[s]]]

Buttons are represented through boxes. And boxes imply the typeset environment. This means, inside a ButtonBox we expect boxes, for instance a RowBox.

ButtonBox[RowBox[toRainbowColor["I  am  a  stylish  button !"]], 
          ButtonMinHeight -> 2, ButtonFrame -> "DialogBox",
          ButtonMargins -> 4, ButtonNote -> "Cursor is over me!", 
          Active -> True] // DisplayForm

The functions responsible for the actions of a button are ButtonFunction, ButtonData, and ButtonStyle. We will discuss these options of ButtonBox in the following.

The simplest way to specify a button action is through the ButtonStyle option.

There are a number of predefined button styles with obvious actions. Here are some examples.

The "Paste" style represents a button that, when clicked, pastes its content (boxes) into the current selection. Here is an example button of the "Paste" style.

ButtonBox[SuperscriptBox["2", SuperscriptBox["2", "2"]], 
          ButtonStyle -> "Paste", Active -> True] //  DisplayForm

The "Evaluate" style represents a button that, when clicked, pastes its boxes into the current selection and then evaluates the pasted expression. Here is an example button of the "Evaluate" style.

ButtonBox[SuperscriptBox["2", SuperscriptBox["2", "2"]], 
          ButtonStyle -> "Evaluate", Active -> True] //  DisplayForm

Special meaning has the character ■ (\[SelectionPlaceholder,]). If a button containing this character is clicked, then the current selection is substituted in its place. (This is similar to the above-discussed function NotebookApply.) Here is an example. If a selection is made in a StandardForm cell, and the next button is clicked, then the selection is raised to power 2 and the resulting expression is evaluated in place. If a nonparsable subexpression (for example, 1+ instead of 1 + 1 is selected, then the raised power stays and a Syntax::"sntxf" is generated from an attempt to evaluate the subexpression).

ButtonBox[SuperscriptBox["2", "\[SelectionPlaceholder]"], ButtonStyle -> "Evaluate", 
          Active -> True] //  DisplayForm

Because after the evaluation, the result of the evaluation is the current selection again, one can easily repeatedly click this button.

Unfortunately, there is no front end function that, once a button is selected would emulate the action that a manual clicking of the button would have. We will emulate (with some programming) the action of the last "Evaluate" button at the end of this subsection.

Now, we come to the most flexible possibility to specify a button operations, the ButtonFunction option.

Basically, the button function can be any Mathematica function or user-defined program. The following button, when clicked, will create a new, empty notebook. This is an easy operation; no arguments from the notebook that contains the button are needed for the generation of an empty new notebook.

ButtonBox[StyleBox["Make a new notebook", 
                   FontFamily -> "Helevetica", FontWeight -> "Bold"], 
          (* make a new notebook when clicked *)         
          ButtonFunction :> NotebookCreate[], Active -> True] //  
                                                         DisplayForm

The last button only used the front end to carry out its action. Some button functions might require a Mathematica kernel. Because the default option setting of the ButtonEvaluator option is None, we will now specify the possibility to use the default kernel carry out the required operation. So clicking the following button prints the numerical value of π to 100 digits.

ButtonBox[StyleBox["Print numerical value of π", 
                   FontFamily -> "Helevetica", FontWeight -> "Bold"], 
          (* print a 100-digit approximation of π when clicked *)
          ButtonFunction :> Print[N[Pi, 100]], Active -> True,
          ButtonEvaluator -> Automatic] //  DisplayForm

The next button prints the filename of the notebook where the button resides in. Because the button uses the function Developer`NotebookInformation, evaluating the button function also needs a kernel.

ButtonBox[StyleBox["File name of this notebook", 
                   FontFamily -> "Helevetica", FontWeight -> "Bold"], 
          (* print the filename of the notebook that 
             contains the button when clicked *)
          ButtonFunction :> Print[ToFileName["FileName" /. 
          Developer`NotebookInformation[ButtonNotebook[]]]], 
          Active -> True, ButtonEvaluator -> Automatic] // 
                                                         DisplayForm

We give one more example. Clicking the next button produces a copy of the current notebook and displays it for 3 seconds. Then the copy will be destroyed again.

cloneCell = Cell[TextData[StyleBox["I am a clone!",
                 FontSize -> 100, FontWeight -> "Bold",
                 FontSlant -> "Italic", FontColor -> RGBColor[1, 0, 0],
                 Background -> GrayLevel[0.333333]]], "Text"];

ButtonBox[StyleBox["Clone this notebook for a moment!", 
                   FontFamily -> "Helevetica", FontWeight -> "Bold"], 
          (* make a temporary copy of the notebook in which 
             the button resides when clicked *)         
          ButtonFunction :> 
           Module[{nb = NotebookGet[ButtonNotebook[]]},
                  nbNew = NotebookPut[
                  Insert[#, WindowTitle -> "I am a clone!", 2]& @ 
                  MapAt[Prepend[#, cloneCell]&, 
                  nb /. (StyleDefinitions -> x_) ->
                  (StyleDefinitions -> (ToFileName[Evaluate //@ 
       MapAt[(StyleDefinitions /. Options[ButtonNotebook[]])&, "FileName" /.   
        Developer`NotebookInformation[ButtonNotebook[]], {2}]])), 1]];
                   Pause[3]; NotebookClose[nbNew]], 
           Active -> True,  ButtonEvaluator -> Automatic] //  DisplayForm

The next button extracts all current selections from within this notebook and creates a new notebook with the selected parts.

CellPrint @ 
 Cell[BoxData[ButtonBox[Cell[TextData[
               StyleBox["Make Notebook of Selection", "Text",
                        FontFamily -> "Helvetica"]]],
              Active -> True, ButtonEvaluator -> Automatic,
              ButtonFunction :> (NotebookPut[Notebook[
              Flatten[{NotebookRead[ButtonNotebook[]]}]
              (* potentially add stylesheet again *)]]&)]], 
      "Output"]

The last two ButtonBox options to discuss are ButtonSource and ButtonData. Until now, we have given an expression to be carried out as the value of the ButtonFunction option that had either no reference to any other expression or used ButtonNotebook[] to refer to the whole notebook where the button was residing. But as the name of the option implies, a (pure) function can be supplied as the option setting to ButtonFunction. And a pure function needs arguments. Two arguments can be given to the ButtonFunction option value, the first one being specified by the option value of ButtonSource and the second one being specified by the option value of the ButtonData option value.

The difference between the two arguments (the option values of ButtonSource and ButtonData) supplied is that the first argument is a location whereas the second is an explicit expression.

Here are two identical-looking buttons. The first pastes the content of the cell in which it resides and the second the whole cell when clicked.

ButtonBox[StyleBox["Clone this cell", 
                   FontFamily -> "Helevetica", FontWeight -> "Bold"], 
          ButtonStyle -> Paste, ButtonSource -> CellContents, 
          Active -> True] //  DisplayForm

ButtonBox[StyleBox["Clone this cell", 
                   FontFamily -> "Helevetica", FontWeight -> "Bold"], 
          ButtonStyle -> Paste, ButtonSource -> (* the whole cell *) Cell, 
          Active -> True] //  DisplayForm

Sources of buttons for further study are the stylesheets that come with Mathematica. There are hyperlinks such as "RefGuideLinkText" implemented. Such links are mainly implemented through specific front end tokens, such as the following. The following input will open the ButtonBox entry in the help browser.

FrontEndExecute[{FrontEnd`HelpBrowserLookup[ "RefGuide", ButtonBox]}]

And the next input will open the Section 3.2.10 of the Mathematica Book.

FrontEndExecute[{FrontEnd`HelpBrowserLookup[ "MainBookLink", "3.2.10"]}]

These are the concrete settings for the various hyperlinks in the stylesheets that come with Mathematica.

styleSheetDirectory = 
Select[FileNames["*Stylesheets*", $InstallationDirectory, Infinity],
       Not[StringMatchQ[#, "*.nb"]]&][[1]]

styleSheets = FileNames["*", styleSheetDirectory];
Length[styleSheets]

styleNamesAndButtonBoxOptions = 
({#[[1, 1]], ButtonBoxOptions /. (List @@ Rest[#])}& /@ 
     Cases[Get[#],  Cell[StyleData[_], ___, 
                         ButtonBoxOptions -> _, ___], 
                         Infinity])& /@ styleSheets;

Length /@ styleNamesAndButtonBoxOptions

allStyleNamesAndButtonBoxOptions = 
        Union[Flatten[styleNamesAndButtonBoxOptions, 1]];
allStyleNamesAndButtonBoxOptions // Length

allStyleNamesAndButtonBoxOptions

The last input shows that the predefined button styles like "Evaluate", "Paste", "CopyEvaluate", … are all defined through front end functions, especially though the use of FrontEnd`NotebookApply. We use the actual button function of the "Evaluate" to emulate the action of the button. Here is a new notebook with a single cell.

newNotebook = Notebook[{Cell[BoxData[RowBox[{"1"}]], "Input"]},
                       WindowSize -> {300, 150}];
         
newNotebookH = NotebookPut[newNotebook]

The following emulates the clicking of a button with content 1+■.

click[] := 
(FrontEndExecute[{FrontEnd`NotebookApply[
                  newNotebookH, RowBox[{"\[SelectionPlaceholder]", "+", "1"}], All]}];
 SelectionEvaluate[newNotebookH, All])

We select the cell of the just-created notebook and click the button once. We see the 1 replace by 1 + 1 and then, after the evaluation, replaced by 2. (Without assigning another kernel to the notebook newNotebook, we cannot apply click repeatedly within a Do-loop because the kernel would carry out the Do-loop first and then the SelectionEvaluate would only evaluate the whole queued up sum of ones.)

FrontEndTokenExecute[newNotebookH, "SelectAll"];

click[]

We close the last notebook and end this subsection.

NotebookClose[newNotebookH]

B.4.2 Hyperlinks

One important clickable element, that appears many thousand times in the GuideBooks and that we have not discussed yet, are hyperlinks.

The most frequently occurring hyperlinks are the ones from the main text (or exercises or solutions) to a reference. These hyperlinks have the ButtonStyle option setting "Hyperlink". The button style "Hyperlink" is defined in the stylesheet of the GuideBooks. We load the notebook expression corresponding to the stylesheet into the kernel.

stylesheetNotebook = 
Get[ToFileName[Evaluate //@ MapAt["GuideBooksStylesheet.nb"&,
      "FileName" /. Developer`NotebookInformation[EvaluationNotebook[]], 2]]];

The button function is nothing else than a simple NotebookLocate (to be exact the front end version FrontEnd`NotebookLocate to allow for a speedier and kernel-independent execution of the link). The argument supplied to the (pure) button function is the value of the button data option.

ButtonBoxOptions /. 
List @@ Rest[Cases[stylesheetNotebook, 
                   Cell[StyleData["Hyperlink"], ___], Infinity][[1]]]

The ButtonData option value is set in each individual button to be an abbreviated form of the reference (typically in the form firstAuthorLastNameLastTwoDigitsOfPublicationYear.

Here is an example: The following preprint was entirely authored with the Mathematica front end [3✶]. Here is the input form of this cell. (This cell has the cell tag "HyperlinkDemonstrationCell".)

(SelectionMove[InputNotebook[], Previous, Cell, 3]; 
 NotebookRead[SelectedNotebook[]]) // InputForm

The next input brings the actual reference cell into the visible part of the notebook and then goes back to the last text cell.

NotebookLocate["Trott00"]; 
Pause[3]; 
NotebookLocate["HyperlinkDemonstrationCell"]

A convenient way to go back from a clicked hyperlink is the button function setting FrontEndExecute[{FrontEnd`FrontEndToken["HyperlinkGoBack"]}]. This button function is used in the star on the left-hand side of each reference entry.

Hyperlinks that go outside of Mathematica can be given in the form URL[url] in the ButtonData option setting. We will do this now. Using the button style "Hyperlink" from the stylesheet gives the typical underlined appearance of hyperlinks.

ButtonBox[StyleBox["Visit the Wolfram Functions Site now!", 
                    FontFamily -> "Helevetica", FontWeight -> "Bold"], 
      ButtonStyle -> "Hyperlink", 
      ButtonData -> {URL["http://functions.wolfram.com"], None},
      Active -> True] // DisplayForm

The underlining in the last cell has some gaps, because the first argument of the ButtonBox contains boxes. And the sentence Visit the Wolfram Functions Site now! was straightforwardly interpreted and typeset by the front end as Mathematica input (in a StandardForm-type cell). As a result, the single string was split up into its words (with implicit multiplication and factorial interpretation).

(SelectionMove[InputNotebook[], Previous, Cell, 4]; 
 NotebookRead[SelectedNotebook[]]) // FullForm

By escaping the typeset mode and going into text mode (using an inlined Cell), we obtain a continuous underlining.

ButtonBox[RowBox[{Cell["Visit the Wolfram Functions Site now!"
                       (* potential style *)]}], 
          ButtonData -> {URL["http://functions.wolfram.com"], None},
          ButtonStyle -> "Hyperlink", Active -> True] //  DisplayForm

Here we use an inlined (text-style) cell inside the ButtonBox to enforce the use of a standard text-like appearance of the text.

CellPrint[
Cell[BoxData[
ButtonBox[RowBox[{Cell[TextData[
          {StyleBox["Visit the Wolfram Functions Site now!",
                    FontFamily -> "Helvetica"]}]]}],
           ButtonData -> {URL["http://functions.wolfram.com"], None},
           ButtonStyle -> "Hyperlink", Active -> True]], "Input"]]

B.4.3 Palettes

Frequently it is convenient to collect various buttons in a palette. Typically, this is done in a palette. Otherwise, a palette is just a notebook like any other notebook.

Let us now have a look at the buttons of the GuideBooks navigator palette. More than 30 button functions are used in this palette.

navigatorNotebook = 
Get[ToFileName[Evaluate //@ MapAt["GuideBooksNavigator.nb"&,
      "FileName" /. Developer`NotebookInformation[EvaluationNotebook[]], 2]]];

navigatorButtons = Cases[navigatorNotebook, _ButtonBox, Infinity];

navigatorButtonFunctions = 
         Union[Cases[navigatorButtons, HoldPattern[ButtonFunction :> _], 
               Infinity]];

Length[navigatorButtonFunctions]

The most common ones are for navigating inside a notebook and use NotebookFind, and for opening a chapter notebook and these use NotebookLocate.

navigatorButtonFunctions[[4]]

navigatorButtonFunctions[[-4]]

Head[navigatorNotebook]

A notebook becomes a palette through the settings of such options as WindowFrame (to make it look alike a palette with no superfluous window elements), Active (to allow the clicking of the buttons), Editable (to prevent editing the notebook), WindowSize (to keep the frame fit the buttons snugly), and WindowClickSelect (to have the palette always on top) set. 
Here are the options set for the GuideBooks navigator palette.

Rest[navigatorNotebook]

In case a palette (or a notebook in general, or a single cell) has buttons with identical button functions, and/or button data, ..., one does not have to give the button function for each notebook explicitly. Rather it is possible to pass the ButtonBox option along to each button through the meta-option ButtonBoxOptions -> listOfButtonBoxOptionsToBePassedToAllButtonBoxesInThisStructure.

Other obvious sources of buttons are the many palettes that come with Mathematica. Here are some of the used button box options shown.

(* directory of Mathematica palettes *)
palettesDirectory = 
Select[FileNames["*Palettes*", $InstallationDirectory, Infinity],
       Not[StringMatchQ[#, "*.nb"]]&][[-1]]

palettes = Get /@ FileNames["*", palettesDirectory];
(* number of palettes *)
Length[palettes]

(* the button box options *)
Union[Cases[palettes, HoldPattern[ButtonBoxOptions -> _], Infinity]]

Next, we make a simple palette. We start with a notebook that contains one typeset cell.

nbWithOneCell = Notebook[{Cell[BoxData[
    RowBox[{"1", "+", SqrtBox[FractionBox["16", "4"]]}]], "Input"]}, 
    WindowSize -> {252., 100.}, WindowMargins -> 
    {{144., Automatic}, {Automatic, 121.}}];

littleNb = NotebookPut @ nbWithOneCell;

We define a function that takes a selection in a notebook, and evaluates it in place. The result is colored in red and while the evaluation is running, a blue message is given.

evaluateAndColorRed[nb_] :=
Module[{eval = NotebookRead[nb]},
(* insert temporary wait message *)
NotebookWrite[nb, BoxData[Cell[TextData[{
StyleBox["Now evaluating … Please wait …",  
          FontColor -> RGBColor[0, 0, 1]]}]]], All];
(* imitate a longer calculation *) Pause[5];
(* insert result in red *)
NotebookWrite[nb, StyleBox[
  MakeBoxes @@ {ToExpression[eval]},
  FontColor -> RGBColor[1, 0, 0]], After];]

We select the square root with the fraction in the notebook littleNb.

SelectionMove[littleNb, Before, Notebook];
SelectionMove[littleNb, Next, Cell];
SelectionMove[littleNb, Before, CellContents];
SelectionMove[littleNb, Next, Character, 3];
SelectionMove[littleNb, All, Expression, 1];
FrontEndExecute[{FrontEndToken[littleNb, "ExpandSelection"]}];
FrontEndExecute[{FrontEndToken[littleNb, "ExpandSelection"]}];

Now, we evaluate the square root. It evaluates to 2.

evaluateAndColorRed[littleNb];
Pause[5];
NotebookClose[littleNb]

Now, we will make a palette that has a button that evaluates in place the selection and replaces it with the result in red. A convenient way to produce buttons and palettes is through the kernel and a front end object producing function such as CellPrint or NotebookPut. (While in principle one can write a button in an unformatted cell or even in a text editor, this is a more cumbersome method. The reason is that the parser of the front end does not understand all of Mathematica’s input form, but only the most common ones such as List, Rule, and RuleDelayed. Even semicolons for a CompoundExpression are not understood.) So in the next input, we generate a notebook that is a palette that has a button doing the above-described operation.

miniPalette =
NotebookPut @ Notebook[
    {Cell[BoxData[ButtonBox[Cell[TextData[{
          StyleBox["Evaluate and color red", "Text", 
            Rule[FontColor, RGBColor[1, 0, 0]], 
            Rule[FontFamily, "Helvetica"]]
          }]], ButtonFunction :> 
         evaluateAndColorRed[FrontEnd`InputNotebook[ ]], 
       ButtonEvaluator -> Automatic, Active -> True]]]},
 (* set notebook options appropriate for a palette *)       
 Editable -> False, PageWidth -> 340, WindowSize -> {Fit, Fit},
 WindowFrame -> "Palette", WindowElements -> {},
 WindowFrameElements -> "CloseBox", WindowClickSelect -> False,
 WindowTitle -> "Red Evaluator", CellMargins -> {{0, 0}, {Inherited, 0}},
 Active -> True, ShowCellTags -> False, ShowCellBracket -> False];

Using again the square root with the fraction as an example, one can now manually make use of the button. To do so one selects the part that one wants to evaluate in the non-palette notebook and then clicks the button in the palette notebook.

NotebookPut @ nbWithOneCell;

(Pause[10]; NotebokClose[%]; NotebokClose[miniPalette]);

Now continue with a slightly different, but conceptually simpler palette. Its buttons just pastes an integer.

maxiPalette[{nx_, ny_}] :=
Notebook[
(* grid of integer paste buttons *)
{Cell[BoxData[StyleBox[#, FontSize -> 14]& @ 
 GridBox[Table[StyleBox[#, FontColor -> 
                    RGBColor[Random[], Random[], Random[]]]& @ 
      ButtonBox[ToString[i ny + j], 
                ButtonStyle -> "Paste", Active -> True], 
              {i, 0, ny}, {j, 0, nx}],
          RowSpacings -> 0.1, ColumnSpacings -> 0.1,
          RowLines -> True, ColumnLines -> True,
          GridFrame -> True]]]}, 
 (* set notebook options appropriate for a palette *)       
 Editable -> False, PageWidth -> 340, WindowSize -> {Fit, Fit},
 WindowFrame -> "Palette", WindowElements -> {},
 WindowFrameElements -> "CloseBox", WindowClickSelect -> False,
 WindowTitle -> "Natural Number Paster", CellMargins -> {{0, 0}, {0, 0}},
 Active -> True, ShowCellTags -> False, ShowCellBracket -> False];

NotebookPut[maxiPalette[{40, 25}]]

We end with a palette that changes its color when the button in the notebook is clicked. The ButtonFunction reads in the whole palette (which is a pretty small notebook and can be done quite quickly), then changes the color from red to blue or blue to red respectively and then replaces the original palette with a new one. Because the read-in palette notebook contains information about its positioning on the screen, the new notebook will appear on the same position (and in the same size) and will be visually indistinguishable from the original one, except that it has a changed color.

redBlueButtonH = NotebookPut @ Notebook[
{Cell[BoxData[ButtonBox["Press!", 
    ButtonFunction :> 
     Module[{bnb = ButtonNotebook[], nb, newnb},
            nb = NotebookGet[bnb];
            newnb = nb /. {RGBColor[1., 0., 0.] -> RGBColor[0., 0., 1.],
                           RGBColor[0., 0., 1.] -> RGBColor[1., 0., 0.]};
            NotebookClose[bnb]; NotebookPut[newnb]], 
            Active -> True, ButtonEvaluator -> Automatic,
            Background -> RGBColor[1, 0, 0]]]]}, 
 (* set notebook options appropriate for a palette *)       
 Editable -> False, PageWidth -> 200, WindowSize -> {Fit, Fit},
 WindowFrame -> "Palette", WindowElements -> {},
 WindowFrameElements -> "CloseBox", WindowClickSelect -> False,
 WindowTitle -> "Red-Blue", CellMargins -> {{0, 0}, {0, 0}},
 Active -> True, ShowCellTags -> False, ShowCellBracket -> False];

Pause[10];
NotebookClose[redBlueButtonH];

B.5 Dynamic Boxes

B.5.1 Automatic Numbering

The references in the chapters of the GuideBooks are (manually) sorted alphabetically by the last name of the author. A reference has the following structure:

Cell[TextData[{
  ButtonBox["✶", ButtonStyle -> "HyperlinkReturn"],
  "referenceNumber  author(s). ", italicJournalOrBookTitle,
  volumeAndPageNumbers ,buttonsToReturnToMentioning}], "BibliographyItem",
  CellTags -> "lastNameOfTheFirstAuthorAndYear"]

The reference numbers are consecutively. While in general not lazy, the author has not numbered them manually and did not insert the corresponding numbers in the main text, the exercises, and the solutions. Where in the distributed versions of the GuideBooks we have explicit numbers, the original notebooks had a CounterBox["BibliographyCounter"] expression. And similarly, the corresponding items in the main text had constructions of the form CounterBox["BibliographyCounter", "lastNameOfTheFirstAuthorAndYear"] in the author’s source notebooks of the GuideBooks. These constructs dynamically carry out the needed consecutive numbering. In general, a counter box is the following two-argument box structure.

CounterBox-objects occur in many places in the stylesheets that come with Mathematica. They are used to number pages, formulas, and graphics. The next inputs extract the style names and the relevant cell option settings from all present stylesheets and that contain CounterBox-objects.

(* directory of all stylesheets that come with Mathematica *)
styleSheetDirectory = 
Select[FileNames["*Stylesheets*", $InstallationDirectory, Infinity],
       Not[StringMatchQ[#, "*.nb"]]&][[1]];

(* all stylesheets that come wit Mathematica *)
styleSheets = FileNames["*", styleSheetDirectory];

styleNamesThatNumber = 
(* form list of the form {styleName, styleDefinitions} *)
Union[{#[[1, 1]], 
 Select[List @@ Rest[#], MemberQ[#, _CounterBox, Infinity]&]}& /@ 
Union[Flatten[Cases[Get[#], Cell[StyleData[_], r___] /;
                        MemberQ[{r}, _CounterBox, Infinity], 
                     Infinity]& /@ styleSheets]]]

We see from the name of the styles that they are used for page numbering, numbering of equations, figures, tables, and so on.

In the next example, we use counter boxes to number identities in a notebook consecutively. Here are 582 beautiful trigonometric identities.

trigIdentities = 
Flatten[Function[trigFunction, 
    (* form identities of the type 
       trigFunction[multipleOfPi] == rationalOrRadicalExpression *)
    HoldForm[trigFunction[#]] ==
             Simplify[FunctionExpand[trigFunction[#]]]& /@
      Union[Pi Flatten[{0, Range[2 #]/#& /@ 
              {1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 17}}]]] /@
              {Cos, Sin, Tan, Cot, Sec, Csc}];
              
Length[trigIdentities]

Here are consecutively numbered cells with these identities. The numbering is realized through a CounterBox that is displayed at the left cell frame boundary.

trigIdentityCells = 
Cell[BoxData[FormBox[MakeBoxes[#, TraditionalForm] /.
           (* format ComplexInfinity nicely *)
           "ComplexInfinity" -> OverscriptBox["∞", "~"], 
                     TraditionalForm]], "DisplayFormula",
                     CellMargins -> {{40, 10}, {10, 10}},
                     CellFrame -> True,
  (* automatic consecutive numbering *)
  CellFrameLabels -> {{Cell[ 
          TextData[{"(", CounterBox[ "DisplayFormula"], ")"}], 
          FontFamily -> "Helvetica", FontSize -> 14, 
          FontColor -> GrayLevel[0.5], FontWeight -> "Bold",
          FontWeight -> "Bold"], None}, {None, None}},
   CounterIncrements->"DisplayFormula"]& /@ 
                               Take[trigIdentities, All];

Here are consecutively numbered cells with these identities displayed in a new spun-off notebook.

numberedTrigIdentityNotebook = 
  NotebookPut[Notebook[trigIdentityCells, WindowSize -> {1200., 600.}]];

We now step through this notebook cell by cell and alter the cells in two directions: we change the consecutive numbering to Roman numerals and we add a frame around all row boxes, fraction boxes, and square root boxes (to make them look more pretty).

Pause[4];
SelectionMove[numberedTrigIdentityNotebook, Before, Notebook]

Do[Pause[0.2];
   SelectionMove[numberedTrigIdentityNotebook, Next, Cell]; 
   nbr = NotebookRead[numberedTrigIdentityNotebook];
   (* make Roman numbering and add frames around boxes *)
   newCell = nbr /. CounterBox[x_] :> 
                  CounterBox[x, CounterFunction :> RomanNumeral] //.
        {RowBox -> FrameRowBox, FractionBox -> FrameFractionBox,
         SqrtBox -> FrameSqrtBox} //. 
         {FrameRowBox -> (FrameBox[RowBox[#]]&), 
          FrameFractionBox -> (FrameBox[FractionBox[##]]&), 
          FrameSqrtBox -> (FrameBox[SqrtBox[#]]&)};
    NotebookWrite[numberedTrigIdentityNotebook, newCell],
    {582}]

Pause[5];
NotebookClose[numberedTrigIdentityNotebook]

Counter boxes can also increment with respect to cell tags. This is used for the reference numbering in the GuideBooks. The next inputs realize this possibility by first having 582 text cells that contain both the number of the identities and a link to the identities. Then come another 582 identities that contain the actual identities (and links back to the first set of cells). The second set of cells has unique cell tags that identify each identity.

trigIdentityCellsSF = Flatten @ Transpose[MapIndexed[
{(* the cells that refer to the identities *)
 Cell[TextData[{"To trig identity number ", 
               CounterBox["TrigIdentityCounter", ToString[#2[[1]]]],
               StyleBox[ButtonBox[" ⇓ ", ButtonData -> ToString[#2[[1]]],
                         ButtonStyle -> "Hyperlink"],
                         FontVariations -> {"Underline"->False}]}], "Text"],
 (* the cells that contain the identities *)
 Cell[BoxData[{Cell[TextData[
 {"(", CounterBox["TrigIdentityCounter"], ") ",
  StyleBox[ButtonBox["⇑", Active -> True, 
   ButtonFunction :> FrontEndExecute[{
      FrontEnd`FrontEndToken[ "HyperlinkGoBack"]}]],
      FontColor -> RGBColor[0, 0, 1]]}]],
               FormBox[MakeBoxes[#, StandardForm], StandardForm]}], 
               "Input", FontWeight -> "Plain", Evaluatable -> False,
               (* counter increment name *)
               CounterIncrements -> "TrigIdentityCounter",
               CellFrame -> True, CellTags -> ToString[#2[[1]]]]}&,
           Take[trigIdentities, All]]];

numberedTrigIdentityNotebook2 = 
  NotebookPut[Notebook[trigIdentityCellsSF, WindowSize -> {1200., 600.}]];

In the moment, all identities in the notebook numberedTrigIdentityNotebook2 are numbered dynamically. One can convert counter boxes to explicit numbers with the front end token NotebookReleaseHold. We convert the automatic counters in the notebook with the trigonometric identities to explicit ones.

FrontEndExecute[{FrontEnd`NotebookReleaseHold[numberedTrigIdentityNotebook2]}]

Reading the notebook now back into the kernel and extracting the first cell shows that the numbering is now literal.

{(* original cell *)
 trigIdentityCellsSF[[1]],
 (* cell with counter box value made literal *)
 NotebookGet[numberedTrigIdentityNotebook2][[1, 1]]}

We close the notebook with the trigonometric identities and end this subsection.

Pause[5];
NotebookClose[numberedTrigIdentityNotebook2]

B.5.2 Displaying Values Automatically

So far, we have set options of front end items programmatically using SetOptions or we have determined option values through analyzing the underlying Mathematica expression that represents the (visible) structure. The option inspector allows to change and inspect the option values through a point and click interface. Still another possibility to infer the value of an option is the OptionValueBox.

The next input makes a list of all option names of the front end, the function Notebook, and the function Cell. These are more than 200 options.

allFrontEndNotebookCellOptions = Sort[ToString /@ 
   Union[Flatten[(First /@ Options[#])& /@ {$FrontEnd, Notebook, Cell}]]];
   
Length[allFrontEndNotebookCellOptions]

Next, we use the OptionValueBox to dynamically display their values. We do this by cell printing cells of type "PrintText". The option values displayed are the values within this cell style.

CellPrint[Cell[TextData["∘ The value of the option ", 
                         StyleBox[#, "MR"], " is currently: ", 
                         StyleBox[OptionValueBox[#], "MR"]], "PrintText"]]& /@ 
                                             allFrontEndNotebookCellOptions;

We give a more dynamic example that shows the behavior of boxes of type OptionValueBox. The next input generates a notebook that contains one cell with a nested power. The base and the exponents of this power have the head OptionValueBox.

nbH = NotebookPut[Notebook[
{Cell[BoxData[StyleBox[SuperscriptBox[OptionValueBox[FontSize], 
                StyleBox[SuperscriptBox[OptionValueBox[FontSize], 
                                        OptionValueBox[FontSize]],
                    ScriptSizeMultipliers -> 0.6, ScriptMinSize -> 5]], 
                    ScriptSizeMultipliers -> 0.6, ScriptMinSize -> 5]], 
     "Input"]}, WindowSize -> {780, 280},
     WindowTitle -> "Actual Fontsizes"]]

Now, we select the cell of the just-created notebook and change the overall font size of this cell. Because superscripts are displayed in a smaller font size (determined by the above-discussed ScriptSizeMultipliers option), the actual font size of the exponents are smaller than the set font size.

FrontEndTokenExecute[nbH, "SelectAll"];
Do[Pause[0.2]; SetOptions[NotebookSelection[nbH], FontSize -> j],
   {j, 12, 120}]

Similarly to the process of making explicit counters from counter boxes used in the last subsection, we can again use the front end function NotebookReleaseHold to convert the automatically displaying box into an explicit value.

FrontEndExecute[{FrontEnd`NotebookReleaseHold[FrontEnd`NotebookSelection[nbH]]}]

Now the displayed cell has contains in its underlying cell expression the displayed values explicitly.

NotebookGet[nbH][[1, 1]]

We close this notebook.

NotebookClose[generalAutomaticValues]

In addition to option values, there are a few variables whose values can also be displayed dynamically in the front end. To display them, we use the box ValueBox.

We interactively see a list of the possible variables from inspecting the "Create Value Display Object …" submenu of the Input menu. Programmatically, we can extract these variables from the corresponding help browser file. This is the notebook containing the list of variables.

(* all files in the Mathematica directory *)
inputMenuFile = Get @
   Select[FileNames["*", $InstallationDirectory, Infinity],  
          StringMatchQ[#, "*Documentation*InputMenu.nb"]&][[1]];

And these are 40+ variables whose values can be displayed automatically.

generalAutomaticValues = 
First /@ Cases[#, StyleBox[_, "MR"], Infinity]& @ 
Cases[inputMenuFile, 
      Cell[_?(FreeQ[#, _Cell, Infinity] && 
              MemberQ[#, "$LaunchDirectory", {-1}]&), ___], Infinity];
              
Length[generalAutomaticValues]

We again print cells of type "PrintText" to show the values of these cells.

CellPrint[Cell[TextData["∘ The value of ", StyleBox[#, "MR"], " is: ", 
                         StyleBox[ValueBox[#], "MR"]], "PrintText"]]& /@ 
                                                 generalAutomaticValues;

Let us again demonstrate on a single variable, namely "Time", the dynamic nature of the ValueBox.

nbH = NotebookPut[Notebook[]];
Do[NotebookWrite[nbH,
   Cell[TextData[{"The current time is ", ValueBox["Time"]}], "Text"]];
   Pause[1], {60}]

We close the last notebook and end here with the discussion of automatically displayed options and variable values.

NotebookClose[nbH]

B.6 Special Notebooks

B.6.1 Help Browser Notebooks

The help browser window of the Mathematica front end allows to view nicely hierarchical structured (and potentially large amounts of) information. Unfortunately, these pros come with a few cons(traints): In Mathematica Version 5.1 the hierarchy can only have three sublevels; the titles of all levels must be pure ASCII characters and cannot contain typesetting. Here we will discuss a few essentials of how the help browser works from the underlying notebook structure to later incorporate the GuideBooks into the help browser. A detailed discussion of the help browser structure can be found in [2✶].

The help browser structure is determined by the content of the BrowserCategory.m files, located within the Documentation directory of the Mathematica installation directory. We find the topmost of these browser categories by going downwards the directory tree.

subDirectoryLevel = 0;
While[(bc = Select[FileNames["*", $InstallationDirectory, subDirectoryLevel],
                   StringMatchQ[#, "*BrowserCategories.m"]&]) === {},
      subDirectoryLevel++]

bc[[1]]

The two main functions occurring in the browser category files are BrowserCategory and HelpDirectoryListing.

The contents of this file shows the main categories visible in the help browser: Built-in functions ("RefGuide"), Add-ons & Links ("AddOns"), The Mathematica Book ("MainBook"), Front End ("OtherInformation"), Getting Started ("GettingStarted"), Tour ("Tour"), Demos ("Demos"), and the Master Index ("Master Index"). The head of an expression in a browser category file is BrowserCategory and the subentries HelpDirectoryListing indicate that more details about the hierarchical content are found at the next levels.

Get[%] // InputForm

Traversing the directory of the help browser directory further shows more browser category files.

nextLevelBCsFileNames = 
Select[FileNames["*", $InstallationDirectory, ++subDirectoryLevel],
       StringMatchQ[#, "*BrowserCategories.m"]&]

At the next level we find a dozens entries listed for the browser categories of the next levels, a few hundred browser categories, and a few thousand actual items.

Function[fileContents,
{#, Count[fileContents, Blank[#], Infinity]}& /@ 
{BrowserCategory, HelpDirectoryListing, Item}][
                   nextLevslBCs = Get /@ nextLevelBCsFileNames]

In the last result, we saw Item-objects.

Now let us have a look at which options are used in the Item-objects.

Union[First /@ Flatten[(List @@ Drop[#, 2])& /@ 
        Cases[nextLevslBCs, Item[_, __], Infinity]]]

The first two options are of relevance for users, the last is exclusively used for the reference guide section of The Mathematica Book.

This means that to be visible in the help browser, each cell must have a CopyTag option setting.

B.6.2 The Message Notebook

From the beginning of each Mathematica session that uses a front end, in addition to the visible working notebook(s), there is one additional notebook present: the message notebook.

messageNotebook = Complement[Notebooks[], {InputNotebook[]}][[1]]

As long as no message has been generated, it is invisible (meaning the option Visible is set to False).

Options /@ messageNotebook

Now, we generate a notebook having three cells.

nb = NotebookPut[Notebook[
       {Cell["CellPrint[Cell[{a, b, c, d]]", "Input"],
        Cell["CellPrint[Cell[{RowBox[1, 2, 3]}]]", "Input"],
        Cell["CellPrint[Cell[{a, b, c, d}]]", "Input"]},
        WindowTitle -> "Front end message generating inputs",
        WindowSize -> {600, 400}]]

Evaluating these three cells gives some error/warning messages in the front end.

FrontEndTokenExecute[nb, "SelectAll"];
FrontEndTokenExecute[nb, "Evaluate"]

The messages appeared in the (now visible) message notebook.

Options @ messageNotebook

The message notebook is now visible. And here are the three messages that were generated by the three inputs.

First[NotebookGet[messageNotebook]]

We close the message-generating notebook and make the message notebook again invisible.

NotebookClose[nb]

SetOptions[messageNotebook, Visible -> False];

Note that carrying out evaluations in the message notebook is not recommended.

B.7 MathLink-Related Operations

In this relatively small section, we will discuss a few functions that are tightly related to MathLink. In general, MathLink connects the Mathematica kernel with other programs. Within the context of the GuideBooks, we are not connecting to other programs like C, but it is sometimes useful to connect a front end to multiple kernels (the connection between a front end and a kernel is also realized through MathLink) and also between two kernels directly.

Within the topics discussed in this appendix, the most important function is LinkWrite. To programmatically carry out operations in the front end, we have used until know either named kernel functions (such as NotebookSave) or the function FrontEndExecute. All of the members of these two types of functions are realized through constructs of the form LinkWrite[$ParentLink, FrontEnd`function[arguments]].

$ParentLink gives the link object that is used for input to the Mathematica kernel. Currently this link object is the front end, and here is it.

$ParentLink // FullForm

The direct use of the LinkWrite[…] input versus the named kernel function or FrontEndExecute is useful when either no named kernel version exists or if the execution speed is critical (the LinkWrite[…] versions are typically faster.)

One application of linking a kernel to a front end that is useful for carrying out “dangerous” calculations is the creation of a spin-off kernel (by “dangerous” we are meaning calculations that potentially run out of memory and crash the Mathematica kernel).

This creates the connection to a freshly started Mathematica kernel.

link1 = LinkLaunch["math -mathlink"]

We read the package waiting. It indicates that the new kernel is ready to accept input.

LinkRead[link1]

Now, we define a function evaluateInputInLinkedKernel that sends the expression expr in the kernel linkKernel. To do this, we create a string version of the input and send it for evaluation. Then we read the data sent back from the second kernel (both potential messages and any results) and returns the send back data within this notebook.

SetAttributes[evaluateInputInLinkedKernel, HoldAll];

evaluateInputInLinkedKernel[expr_, linkKernel_] :=
Module[{inputString, nwl}, 
 (* make string of the unevaluated input *)
 inputString = StringTake[ToString[InputForm[Hold[expr]]], {6, -2}];
 (* send string for evaluation *)
 lw = LinkWrite[linkKernel, EnterTextPacket[inputString]];
 (* read potential messages and result *)
 nwl = NestWhileList[LinkRead[linkKernel]&, Null, 
                      (Head[#] =!= InputNamePacket &&
                      (* crash case *) # =!= $Failed)&];
 (* return messages and result *) Rest[nwl]]

For the following nice input, this gives an OutputNamePacket and the result inside the ReturnTextPacket.

evaluateInputInLinkedKernel[Sqrt[2 + 2], link1] // InputForm

The expression 0/0 generates messages. So, from the next input, we also get a TextPacket send back.

evaluateInputInLinkedKernel[Sqrt[0/0], link1]

And the next input will surely make the second kernel run out of memory (we cannot create a dense matrix with (10^15) elements). We obtain a SuspendPacket now.

evaluateInputInLinkedKernel[
   Table[1, Evaluate[Sequence @@ Table[{10^9}, {10^6}]]], link1]

But the kernel guiding the evaluation is still intact.

(3 3 3)^(1/3)

We could now use the function evaluateInputInLinkedKernel to test say Integrate.

trigFunctions = {Cos, Sin, Tan, Cot, Sec, Csc, 
                 Cosh, Sinh, Tanh, Coth, Sech, Csch, 
                 ArcCos, ArcSin, ArcTan, ArcCot, ArcSec, ArcCsc, 
                 ArcCosh, ArcSinh, ArcTanh, ArcCoth, ArcSech, ArcCsch};

Module[{o = 100, i, j, k, i1, j1, k1, linkT, res}, 
Do[{i, j, k} = Table[Random[Integer, {1, Length[trigFunctions]}], {3}];
   {i1, j1, k1} = Table[Random[Integer, {-12, 12}], {3}];
   (* start new kernel to avoid internal caching influence *)
   linkT = LinkLaunch["math -mathlink"]; LinkRead[linkT];
   𝒾𝓃𝓉 = trigFunctions[[k]][trigFunctions[[i]][x]^i1/
                           trigFunctions[[j]][x]^j2]^k1;
   res = Function[int, evaluateInputInLinkedKernel[
                 TimeConstrained[Integrate[int, x], 2], linkT]] @ 𝒾𝓃𝓉;
   If[Last[res] === $Failed, Print["Problem with integrating ", 𝒾𝓃𝓉]];                  
   Pause[1]; LinkClose[linkT], {o}]]

Some of the front end functions are not available as kernel functions. If front end functions perform actions in the front end, they can be called through FrontEnd`FrontEndExecute. But if such functions return values, they must be used through direct MathLink calls.

A very useful function that can (only) be used in this manner is CellInformation.

The next input selects all cell of this notebook and then obtains the cell information provided by CellInformation. To obtain all cell informations, we have to explicitly read the data from the front end.

(SelectionMove[EvaluationNotebook[], All, Notebook];
 LinkWrite[$ParentLink, CellInformation[EvaluationNotebook[]]];
 allCellInfo = LinkRead[$ParentLink]);

This notebook has currently more than 1600 cells.

allCellInfo // Length

Here is a quick look at the information returned for the first cell. We see details about the cell type, its content and so on.

allCellInfo[[1]]

Here are all different details that occur in the cell informations of this notebook.

{#, Union[# /. allCellInfo]}& /@ (First /@ First[allCellInfo])

While for this relatively small notebook we could have carried out NotebookGet[EvaluationNotebook[]] and could have analyzed the notebook completely in the kernel, for notebooks with very large graphics this would be a very time-consuming and memory-intensive process. The use of CellInformation[EvaluationNotebook[]] would be much faster and virtually memoryless in such circumstances.

One MathLink related function which we will use later is Forward. It is relevant within the context of having more than one kernel is set up in to be called from the front end. From within a notebook, LinkWrite[$ParentLink, Forward[evaluator, expression] forwards expression to the evaluator (kernel) evaluator. This is a very useful feature that allows two kernels that are running independently to communicate via a front end.

B.8 Three Applications

B.8.0 Remarks

In the following three subsections, we will put many of the above-discussed functions and notebook-, cell group-, and cell-structures to work. The first subsection is of passive nature: we will analyze the electronic version of The Mathematica Book; no new notebooks will be created, existing ones are taken apart and syntactically, as well as semantically, investigated. The second subsection is of more active nature and will carry out a variety of operations to incorporate the GuideBooks into the help browser. All of the main GuideBooks chapters will be taken apart, reorganized and reindexed and then reassembled into the structure of the help browser. And the third subsection will show how to evaluate a GuideBook chapter programmatically as a whole with kernel restarts at each subsection, section, solution, and subsolution.

B.8.1 Analyzing the Notebook Version of The Mathematica Book

In this subsection, we will analyze the help browser version of The Mathematica Book. We will analyze its structure and also carry out some semantic investigations.

We start by loading the entire reference guide of The Mathematica Book into the kernel.

refGuideFile = 
Select[FileNames["*RefGuide.nb", $InstallationDirectory, Infinity],
       Not[StringMatchQ[#, "*AddOns*"]]&][[1]]

It is a relatively large notebook (taking into account that it does not contain many graphics).

refGuideNotebook = Get[refGuideFile];
{ByteCount[#], LeafCount[#], Depth[#]}&[refGuideNotebook]

More than 1200 functions (including option names, and attributes, and values) are referenced.

allDocumentedFunctions = #[[1, 1, 1, 1]]& /@ Take[refGuideNotebook[[1]], All];
Length[allDocumentedFunctions]

The whole reference guide is formatted by using only a few styles.

DeleteCases[Union[Last /@ Cases[refGuideNotebook,  _StyleBox, Infinity]], _Rule]

But the reference guide makes a lot of use of the boxes to format the mathematical expressions.

DeleteCases[Sort[
{#, Count[refGuideNotebook, #, {0, Infinity}, Heads -> True]}& /@ 
   ToExpression /@ Names["System`*Box*"], #1[[2]] > #2[[2]]&],
   {_, 0}] // TableForm

Because of the nice uniform structure of all entries, it is straightforward to analyze a variety of characteristics.

Here is the cellgroup structure for the first function shown.

refGuideNotebook[[1, 1]] // InputForm

By analyzing the first English word after a functions (and its argument), we can classify the functions according to their principal objective.

firstWord[textString_] := 
StringTake[textString, 
           {#1[[2]] + 1, #2[[1]] - 1}& @@ StringPosition[textString, " "]]

mainDoingWord[entry_] :=
Module[{usageCell, descriptionText},
 usageCell = Cases[entry, Cell[_, "Usage", ___], Infinity];
 If[usageCell =!= {}, 
 descriptionText = Cases[DeleteCases[usageCell[[1, 1, 1]], 
                                     (* some abbreviations *) " or "], 
                        _String?(Or @@ (LetterQ /@ Characters[#])&), {1}, 1];
 If[descriptionText === {}, $Failed,
    firstWord @ descriptionText[[1]]], {}]]

actionAndFunctions = DeleteCases[#, {"…", _}]& @ 
Sort[{#[[1, 2]], First /@ #}& /@ 
Split[Sort[{#[[1, 1, 1, 1]], mainDoingWord[#]}& /@ 
                       Take[refGuideNotebook[[1]], All],
        OrderedQ[{#1[[2]], #2[[2]]}]&], #1[[2]] === #2[[2]]&],
     Length[#1[[2]]] > Length[#2[[2]]]&];

Most built-in symbols “are” something. Nearly as many names “give” something. The list is followed by names that “represent” something. Then come names that “print”, “yield”, and “generate” something. Overall more than 120 different verbs are used to describe the action of the built-in names.

{#1, ToString[Length[#2]] <> " functions"}& @@@ actionAndFunctions

For the first few actions, we display a few of the functions that carry out these actions.

{#1, Take[#2, 6]}& @@@ Take[actionAndFunctions, 12]

The distribution of the size of the descriptions for each function varies between one kB and more than one MB. Here is the distribution shown on a logarithmic scale.

ListPlot[Sort[Log[10, #]& /@ ByteCount /@ refGuideNotebook[[1]]], PlotRange -> All]

Here is the version development of the number of functions references as a function of the version numbers. We extract this information by explicitly looking for occurrences of the string "New in Version".

Check[{#[[1, 2]], Length @ #}, Sequence @@ {}]& /@ (ca = 
Split[Sort[Check[{#[[1, 1, 1, 1]],
      Cases[#, _String?(StringMatchQ[#, "*New in Version*"]&), 
            Infinity]}& /@ Take[refGuideNotebook[[1]], All], 
            Sequence @@ {}], OrderedQ[{#1[[2]], #2[[2]]}]&],
     #1[[2]] === #2[[2]]&])  // TableForm

How many functions refer to how many other functions in the "See also" section? Here is the resulting numbers.

seeAlsoFunctions[group_] := 
If[# === {}, {}, First /@ #]&[
    Cases[group, ButtonBox[_, ButtonStyle -> "RefGuideLink"], Infinity]]

seeAlsos = {#[[1, 1, 1, 1]], seeAlsoFunctions[#]}& /@ 
                                     Take[refGuideNotebook[[1]], All];

{First[#], Length[#]}& /@ Split[Sort[Length[#2]& @@@ seeAlsos]]

Here is the function that refers to the most other functions.

Sort[seeAlsos, Length[#1[[2]]] > Length[#2[[2]]]&][[1]]

Let us quickly have a look how the number of the “see also” functions growth as we repeatedly follow all mentioned “see also” links.

(* make quick hash-based look-up for see also functions *)
(seeAlsoEntries[#1] = #2)& @@@ seeAlsos;
seeAlsoEntries[_] := {}

(* one step in looking at all see also functions *)
seeAlsoStep[{lastSeeAlsoFunctions_, earlierSeeAlsoFunctions_}] := 
Module[{newSeeAlsoFunctions = 
              Union[Flatten[seeAlsoEntries /@ lastSeeAlsoFunctions]]},
       {newSeeAlsoFunctions, Union[Join[newSeeAlsoFunctions, 
                                        earlierSeeAlsoFunctions]]}]

(* growth in the number of the see also functions *)
seeAlsoTreeData = 
Table[Length[#[[2]]]& /@ FixedPointList[seeAlsoStep, 
                           {{allDocumentedFunctions[[k]]}, {}}, Infinity,
                           SameTest -> (#1[[2]] === #2[[2]]&)], 
      {k, Length[allDocumentedFunctions]}];

Here are the resulting cumulative number of “see also” functions as a function of the number of recursive see also links. After about 10 recursive links, we typically have reached more than 600 to 1100 functions.

Show[Graphics[{Thickness[0.002], 
  Table[{Hue[Random[]], 
         Line[MapIndexed[{#2[[1]] - 1, #1}&, Most[seeAlsoTreeData[[k]]]]]},
         {k, Length[seeAlsoTreeData]}]}], Frame -> True, PlotRange -> All]

And here are the functions from the listing occurring in the text of the reference guide, which do not have an individual reference guide entry. Some have common reference guide entries, such as JacobiPQ and many are options to specialized functions, such as DoubleExponential and NProductExtraFactors.

allMRStyleBoxes = Union[Cases[refGuideNotebook, 
                              StyleBox[_String, "MR"], Infinity]] ;

Complement[Intersection[First /@ allMRStyleBoxes, Names["System`*"]],
           allDocumentedFunctions]

Many Mathematica functions can be called with more than one fixed number of arguments. We will use the reference guide to automatically extract some information about this. (By using the functions Attributes and Options one can easily determine the options and attributes of functions programmatically, but it is not so straightforward to determine the number of allowed arguments programmatically. The only [not very elegant] possibility is to call the functions with a variable number of arguments and see of messages of the form function::argb, function::argx, … are issued.) The cell of style "Usage" contains a listing of all officially supported invocations of a function. A quick look at the cell structure of the usage cells shows that their lines start with a filled square (◼)and that the underlying box structure for the supported function calls is of the form … "function[", arguments, "]", …. So, we first look for occurrences of a filled square to extract individual lines and the find the first closing bracket. Then we extract the arguments, remove not needed formatting boxes and options and then assemble the box structure corresponding to the StandardForm of the expressions.

These are the 1200+ pairs of the built-in names and the corresponding usage cells.

makeNameAndUsageCellPair[cg_] := 
Module[{cg1 = Flatten[cg //. 
         Cell[CellGroupData[l_List, ___], ___] :> l], t},
      t = Cases[cg1, Cell[_, #, ___]]& /@ {"ObjectName", "Usage"};
      If[Length[Flatten[t]] === 2,   
         {t[[1, 1, 1]], t[[2, 1]]}, Sequence @@ {}]]

namesAndUsageCells = makeNameAndUsageCellPair /@ refGuideNotebook[[1]];

Length[namesAndUsageCells]

Here is the first of these pairs.

namesAndUsageCells[[1]]

The function makeRowBox makes the row boxes of the StandardForm form from the individual lines of the usage cell.

makeRowBox[sL_] := 
Module[{(* position of closing ']' *)
        pos = Position[sL, StyleBox["]", "MR"]]},
 If[pos =!= {},
    (* take all elements between opening and closing brackets *)
    RowBox[{StringDrop[sL[[1, 1]], -1], "[", Sequence @@ 
            Take[sL, {2, pos[[1, 1]]}]}], {}] //.
     (* remove formatting *) StyleBox[c_, ___] :> c //.
     Cell[BoxData[FormBox[c_, TraditionalForm]], 
          "InlineFormula"] :> c]

The function makeFunctionCalls takes a pair of the function name and the corresponding usage cell lists and returns the supported calls to the function.

makeFunctionCalls[{function_, usageCell_}] := 
Module[{textDataContent, posis, lines, functionCallLines, 
        rowBoxes, expressions}, 
 (* the list of elements of the content of the usage cell *)
 textDataContent = usageCell[[1, 1]];
 (* positions of the ◼ that indicate a begin of a new line *)
 posis = Union[Flatten[{1, Position[textDataContent, 
                                    _String?(StringMatchQ[#, "*◼ "]&)], 
                        Length[textDataContent]}]];
 (* the individual lines of the usage cell *)
 lines = (Rest @ Take[textDataContent, #])& /@ Partition[posis, 2, 1];
 (* the function call containing  lines of the usage cell *)
 functionCallLines = Select[lines, 
   MatchQ[#,  {StyleBox[_String?(StringMatchQ[#, "*["]&), "MR"],
           ___}]&];
 (* the row boxes of the StandardForm expression *)
 rowBoxes = Flatten[makeRowBox /@ functionCallLines];
 (* the actual expressions *)
 expressions = Cases[MakeExpression /@ rowBoxes, HoldComplete[_]]]

Here is the held version of the function call for the first function.

makeFunctionCalls @ namesAndUsageCells[[1]]

The function Partition can be called with up to six arguments. Here are all the possibilities.

makeFunctionCalls @ 
            Cases[namesAndUsageCells, {"Partition", _}][[1]]

Now, we carry out the conversion to Mathematica inputs for all expressions. More than 600 out of the 1200+ built-in names that are listed in the reference guide are “functions” that take zero or more arguments.

allowedFunctionInvocations = 
 DeleteCases[{#1, makeFunctionCalls[{##}]}& @@@ namesAndUsageCells,
             {_, {}}];
             
Length[allowedFunctionInvocations]

The list allowedFunctionArgumentCounts contains pairs of the form {functionName, listOfSupportedArguments}.

allowedFunctionArgumentCounts = {#1, Union[#2]}& @@@ 
        (allowedFunctionInvocations /.
                 HoldComplete[x_] :> Length[Unevaluated[x]]);

Now, we have all data about the number of arguments that can be given to the built-in functions. We will now analyze the results. We begin by counting how many functions can take k elements. We see that one argument is the most frequently occurring case, followed by two arguments.

{First[#], Length[#]}& /@ 
Split[Sort[Flatten[Last /@ allowedFunctionArgumentCounts]]]

Here is the number of functions that take exactly k arguments. We see that most functions take just one argument.

{#, Count[allowedFunctionArgumentCounts, 
                    {_, {#}}]}& /@ {1, 2, 3, 4, 5, 6, 7, 8}

Here are the functions that have the largest number of different argument counts. We make lists of the function name and the number of different argument counts.

Take[#, 3]& @ 
Split[Sort[{#1, Length[#2]}& @@@ allowedFunctionArgumentCounts,
      #1[[2]] > #2[[2]]&], #1[[2]] === #2[[2]]&]

Some Mathematica functions take an arbitrary number of arguments. In their descriptions, we recognize these functions on the presence of ellipses among the arguments. The functions Max for instance allows for an arbitrary number of arguments.

Cases[allowedFunctionInvocations, {"Max", _}]

Here are all functions whose description contains ellipses as one of their arguments (some arguments are iterators).

First /@ 
  Select[allowedFunctionInvocations, MemberQ[Level[#[[2]], {3}], …]&]

We continue with the main part of The Mathematica Book. The variable mathematicaBookSections contains the Notebook expressions with the actual book content.

mathematicaBookSectionFileNames = 
Select[FileNames["*", $InstallationDirectory, Infinity],
      (StringMatchQ[#, "*MainBook*"] && 
       NumberQ[ToExpression[StringTake[#, {-4, -4}]]])&];

mathematicaBookSections = Get /@ mathematicaBookSectionFileNames;

The notebooks contain hundreds of individual Mathematica sessions, hundreds of graphics, nearly a thousand boxes with important Mathematica definitions, thousands of Mathematica inputs, and tens of thousands of inline cells in the style of Mathematica functions.

{#, Count[mathematicaBookSections, #, Infinity]}& /@ 
      {CellLabel -> "In[1]:= ", _GraphicsData, "DefinitionBox", 
       "Input", StyleBox[_, "MR"]}

The word “Mathematica” appears more than 3000 times in the notebooks.

Flatten[StringPosition[#, "Mathematica"]& /@
    Take[Cases[mathematicaBookSections, _String, {-1}], All], 1] // 
                                                             Length

Next, we extract all input cells and convert them to expressions

allInputCells = Cases[mathematicaBookSections, Cell[_, "Input", ___], 
                      Infinity];

The function makeExpression convert the input cells to their corresponding Mathematica expressions.

makeExpression[Cell[c_, "Input", ___]] :=
Module[{trySF, tryIF, tryTF},
       If[(* ignore shell calls *) 
          Head[c] === String && StringTake[c, 1] === "!", $Failed, 
          (* try various forms *)
          trySF = ToExpression[c, StandardForm, Hold];
          If[trySF =!= $Failed && Head[trySF] =!= ToExpression, trySF,
             tryIF = ToExpression[c, InputForm, Hold]; 
             If[tryIF =!= $Failed && Head[tryIF] =!= ToExpression, tryIF,
                tryTF = ToExpression[c, TraditionalForm, Hold];
                If[tryTF =!= $Failed && Head[tryTF] =!= ToExpression, 
                   tryTF, $Failed]]]]]

allInputs = Internal`DeactivateMessages[
                     DeleteCases[makeExpression /@ Take[allInputCells, All],  
                                 $Failed]];

Now that we have all input as held Mathematica expressions, it is easy to analyze it. Here, we will only look at the elementary ingredients of the inputs. We extract the atoms of the inputs and sort them according to the number of times they occur.

SetAttributes[hold, HoldAll];

inputIngredients = 
Sort[{HoldForm @@ First[#], Length[#]}& /@ Split[Sort[
DeleteCases[Level[Map[hold, hold @@@ allInputs, {-1}, Heads -> True], 
                  {-2}, Heads -> True],
            hold[hold]]], #1 == #2&], #1[[2]] > #2[[2]]&];

List is by far the most-used built-in function of the inputs of The Mathematica Book. And x is the most popular user-variable.

Take[inputIngredients, 12]

And the following built-in functions appear just once in all inputs.

Sort[First /@ 
Select[Cases[inputIngredients, {_, 1}], 
       (MatchQ[#[[1]], HoldForm[_Symbol]] && (Context @@ #[[1]]) === "System`")&]]

Length[%]

B.8.2 Incorporating the GuideBooks into the Help Browser

In this subsection, we will repurpose the notebooks of the GuideBooks and incorporate them into the help browser. To accommodate the reference character of the help browser, we will make some reorganizational and linking changes:
• The individual exercises will be joined with the corresponding solution.
• The references will be split and displayed with each subsection, section, and exercise/solution pair. And we will number the references consecutively within each such unit.
• We remove the Overview section and link the Mathematica function mentioned directly to the reference guide part of the help browser.
• We replace the notebook navigator with the corresponding detailed listing of the chapters, sections, and subsection in the categories of the help browser.

We will not incorporate the animations into the help browser (window), because the animation notebooks have all different sizes. Consequently, the animation notebooks will open outside the help browser.

Technically, we keep one notebook per chapter. We add and change the groupings and orders, and add plenty of tags that allow for the corresponding IndexTag and CopyTag taggings that are needed within the help browser.

To incorporate the Mathematica GuideBooks into the help browser, we have to carry out the following steps:

1. Copy the files of the GuideBooks into a subdirectory of (English) Documentation subdirectory of the Mathematica installation directory.

2. Make a variety of changes to make the GuideBooks fit better in character into the help browser. (This includes making bitmaps of all graphics cells to have a smaller file size, reorganization and retagging of cells and cell groups and corresponding changes in the table of contents).

3. Make the BrowserIndex.m notebook.

4. Extract the chapter, section, and subsection titles to create a BrowserCategories.m file.

5. Add the index entries into the master index.

We will number the 14 chapter notebooks from 1 to 14 instead of the default 1-Programming-1,…, 4-Symbolics-3 numbering. This has a couple of programming advantages—we have a natural order for all chapters, as well as display advantages within the help browser. The help browser allows for categories to be nested up to three times. The uppermost category within the AddOn unit of the help browser is the set of GuideBooks itself, the second category level contains the 14 chapters, the third contains the sections, and the fourth contains the subsection and problems.

We start with step one, which can be carried out in just one line. To be independent of the OS used, we define a function pathNameSeparator. It allows to go back and forth from absolute pathnames and Mathematica list versions of path names.

pathNameSeparator = 
StringDrop[StringDrop[ToFileName[{"dir"}, "fileName"], 3], -8]

Alternatively, we could use the undocumented function $PathnameSeparator.

The function toRelativePathName converts a pathname string to a list of subdirectories.

toRelativePathName[dirName_] := DeleteCases[#, ""]& @ 
(StringTake[dirName, # + {1, -1}]& /@ (Partition[#, 2]& @
 Flatten[{0, StringPosition[dirName, pathNameSeparator], 
            StringLength[dirName] +
             If[StringTake[dirName, {-1, -1}] =!= pathNameSeparator, 1, 0]}]))
             
(* or, shorter in Version >= 5.1:      
   toRelativePathName[dirName_] := StringSplit[dirName, $PathnameSeparator]
*)

We will add the GuideBooks to the AddOns directory of the help browser.

helpBrowserAddOnDirectory = (* un uni*ces, add / *)
If[pathNameSeparator === "/", Prepend[#, pathNameSeparator], #]& @
(toRelativePathName @
  Select[FileNames["*", $InstallationDirectory, Infinity],
          StringMatchQ[#, "*Documentation*English*AddOns"]&][[1]])

We assume that this appendix is in the directory containing the GuideBook notebooks. (If this is not the case, the following input has to be changed accordingly.)

(* assuming this notebook is in the GuideBooks directory;
   else use
   guideBooksDirectory = toRelativePathName @ GuideBooksDirectory *)
guideBooksDirectory = 
First[("FileName" /. Developer`NotebookInformation[InputNotebook[]])]

We will name the subdirectory "Mathematica GuideBooks" within the help browser. (White space within a directory name should not create any problem as long as all the following operations are carried out purely programmatically.)

helpBrowserGuideBooksDirectory = 
    Append[helpBrowserAddOnDirectory, "Mathematica GuideBooks"]

Now, we copy the whole set of GuideBooks files over. This will take a few minutes.

(* remove potentially already existing version of the GuideBooks 
   in the help browser (in case this subsection is evaluated twice) *)
If[FileNames["*", ToFileName[helpBrowserGuideBooksDirectory]] =!= {},
   DeleteDirectory[ToFileName[helpBrowserGuideBooksDirectory], 
                   DeleteContents -> True]]

(* the time consuming step *)
CopyDirectory[ToFileName[guideBooksDirectory], 
              ToFileName[helpBrowserGuideBooksDirectory]]

Overall, we will keep the look-and-feel of the GuideBooks by reusing its stylesheet. But we will slightly modify the stylesheet by adding a "ProblemHeading" style that groups the exercise text and the corresponding solution together. (We assume the reader has gone through the exercises already and now we can make it easier to go to the solution after reading the exercise text.)

styleSheet = Get[ToFileName[guideBooksDirectory, "GuideBooksStylesheet.nb"]];

modifiedStylesheet = styleSheet /. c:Cell[StyleData["ExerciseHeading"], ___] :>
 Sequence[(* add a "ProblemHeading" style *)
    Cell[StyleData["ProblemHeading"], CellMargins -> {{12, 12}, {4, 16}},
         CellBracketOptions -> {"Color" -> RGBColor[0, 0.3, 0.7]},
         CellElementSpacings -> {"ClosedGroupTopMargin" -> 6},
         CellGroupingRules -> {"SectionGrouping", 20}, ParagraphIndent -> -30,
         FontFamily -> "Helvetica", FontSize -> 14, FontWeight -> "Bold",
         FontColor -> RGBColor[0, 0.4, 0.2]], 
         (* original "ExerciseHeading" style *) c];

ssH = NotebookPut[modifiedStylesheet];
NotebookSave[ssH, ToFileName[helpBrowserGuideBooksDirectory, 
                             "GuideBooksStylesheet.nb"]];
NotebookClose[ssH];

When the front end opens an existing notebook (meaning a notebook that was produced by and saved by the Mathematica front end), it typically reads only the immediately-needed information for displaying the currently visible part of the notebook. In addition, for graphics it loads only the bitmap caches instead of the whole underlying graphic (the full PostScript version of the graphics is only needed for resizing). As a result, opening large (hundreds of MB large or even more than 1 GB large notebooks) happens quite quickly within the front end. But in the following, we want to manipulate notebooks in the kernel, where the whole notebook would have to be present and stored at once. (While in principle we could do all operations using front end tokens, this would be considerably more cumbersome; the use of the Mathematica kernel allows to carry out all operations much more elegantly and with less coding.) Depending on the RAM of the reader’s computer, some of the chapter notebooks might be too large to be read into the kernel. In this case, we could quickly transform all graphics into bitmaps using the following front end functions. For the help browser version of the GuideBook notebooks this should not be a serious limitation, one rarely interactively evaluates, manipulates, and changes notebooks inside the help browser.

makeBitmapGraphics[file_] :=
Module[{nbH = NotebookOpen[ToFileName[helpBrowserGuideBooksDirectory, file]]},
       (* open all cells and force page breaks to load graphics *)
       FrontEndTokenExecute[nbH, #]& /@ 
            {"SelectAll", "SelectionOpenAllGroups", 
             "ShowPageBreaks", "ShowPageBreaks"};
       NotebookFind[nbH, "Graphics", All, CellStyle];
       FrontEndTokenExecute[nbH, "SelectionConvert", "Bitmap"];
       NotebookSave[nbH]; NotebookClose[nbH]]

chapterNotebooks = 
 {"0_Introduction.nb", 
  "1_Programming_1.nb", "1_Programming_2.nb", "1_Programming_3.nb", 
  "1_Programming_4.nb", "1_Programming_5.nb", "1_Programming_6.nb", 
  "2_Graphics_1.nb", "2_Graphics_2.nb", "2_Graphics_3.nb", 
  "3_Numerics_1.nb", "3_Numerics_2.nb", 
  "4_Symbolics_1.nb", "4_Symbolics_2.nb", "4_Symbolics_3.nb", 
  (* renamed appendix to accommodate for this second appendix *)
  If[Select[FileNames["*", ToFileName[guideBooksDirectory]],
            StringMatchQ[#, "*Appendix_A.nb"]&] =!= {},
     "Appendix_A.nb", "Appendix.nb"]}

(* carry out to avoid processing large files *)
makeBitmapGraphics /@ Take[chapterNotebooks, {2, -2}]

At this point, we start caring about the index entries that we want to use for the GuideBooks in the help browser. In the GuideBooks notebooks, index entries are present as cell tags that have the form "Index[consecutivelyNumberedChapter, entry]".

The index entries have the integers corresponding to consecutively numbered chapters. We convert to the volume type (Programming or Graphics or Numerics or Symbolics) and the chapter number within each volume.

(indexAbbreviation[#1] = #2)& @@@ Transpose[{chapterNotebooks,
  {"In.", "P.1.", "P.2.", "P.3.", "P.4.", "P.5.", "P.6.", 
   "G.1.", "G.2.", "G.3.", "N.1.", "N.2.", 
   "S.1.", "S.2.", "S.3.", "Ap."}}];

Next, we extract the section or subsection number from a cell group.

sectionAndSubsectionNumber[chapterFileName_, cg_] := 
If[chapterFileName === "Preface.nb", "", 
Module[{headCell, title},
 (* a Section group *)
 headCell = Cases[cg, Cell[_, "Section", ___], Infinity]; 
 (* a Subsection group *)
 If[headCell === {}, headCell = Cases[cg, Cell[_, "Subsection", ___], Infinity]];
 title = headCell[[1, 1]]; 
 (* string or TextData *)
 titleString = If[Head[title] === String, title, title[[1, 1]]];
 StringTake[titleString, {3, StringPosition[titleString, " "][[1, 1]] - 1}]]]

The index entries come in four main flavors: a single textual entry, a built-in Mathematica function (indicated by the tag "BuiltInFunction"), a Mathematica function defined in the GuideBooks (indicated by the tag "GuideBooksFunction"), and subentries (indicated by a list of the form {mainEntry, subEntry}).

preFormatIndexEntry[s_String] :=
Module[{p, lcbp},
(* last closing bracket position--needed for versions a, b, … *)
lcbp[st_String] := StringPosition[st, "]"][[-1, 2]];
(* choose appropriate formatting *)
Function[h, Check[h, Print["Formatting failure in ", s]; $Failed], 
                                                       {HoldAll}] @
Which[(* a function from the GuideBooks with GB *)
      StringMatchQ[s, "*M[*GB]]*"], {{StringTake[s, {13, lcbp[s] - 6}]}, 
                             "GuideBooksFunction"},
      (* a built\[Hyphen]in function with M[…] *)
      StringMatchQ[s, "*M[*]]*"], {{StringTake[s, {13, lcbp[s] - 2}]}, 
                              "BuiltInFunction"},
      (* a potential level 2 entry *)
      StringMatchQ[s, "*{*}]*"],  
         p = StringPosition[s, ","]; 
         If[Length[p] === 1, {{StringTake[s, {12, lcbp[s] - 3}]}, "Text"},
             {{StringTake[s, {12, p[[2, 1]] - 1}],  
               StringTake[s, {p[[2, 1]] + 2, lcbp[s] - 2}]}, "Text"}],
      (* level 1 entry *)
      True,  {{StringTake[s, {11, lcbp[s] - 1}]}, "Text"}]]

Here are examples of the four types of entries.

preFormatIndexEntry /@ 
 {"Index[02, M[Infinity]]", "Index[01, M[GrayRhombusesPartition, GB]]b", 
  "Index[07, {Apollonius circles}]", "Index[08, mobile]", 
  "Index[06, {matrices, Pauli ~}]"} // InputForm

We now convert all index entries to a standard form convenient for further manipulation: IndexEntry[{{mainEntry, optionalDetails}, type}, {occurringPosition, fileName, tag}]

indexTags[cg_] := 
 Select[Flatten[Last /@ Cases[cg, HoldPattern[CellTags -> _], Infinity]],
        StringMatchQ[#, "Index*"]&]

extractClassifiedIndexEntries[chapterFileName_] :=
Module[{chapter = Get[ToFileName[helpBrowserGuideBooksDirectory, 
        chapterFileName]]}, 
 (* subsection *)
 subSectionGroups = 
 Cases[chapter, CellGroupData[{Cell[___, "Subsection", ___], ___}, ___], 
       Infinity];
 (* section without subsections *) 
 sectionGroups = 
 Cases[chapter, cg:CellGroupData[{Cell[___, "Section", ___], ___}, ___] /;
                   FreeQ[cg, Cell[___, "Subsection", ___], Infinity], 
       Infinity];
 (* exercise and solutions sections *)
 {exercisesSection, solutionsSection} = 
 Cases[chapter, CellGroupData[{Cell[___, #, ___], ___}, ___],  Infinity]& /@
                                     {"ExercisesSection", "SolutionsSection"};   
 (* individual exercises and solutions *)
 {exercisesGroups, solutionsGroups} = 
   Cases[#1, CellGroupData[{Cell[___, #2, ___], ___}, ___], Infinity]& @@@
   {{exercisesSection, "ExerciseHeading"}, 
    {solutionsSection, "SolutionHeading"}};   
 (* extract and annotate index entries *)
 Flatten @                  
 Join[Function[cg, ssn = sectionAndSubsectionNumber[chapterFileName, cg];            
              IndexEntry[preFormatIndexEntry[#],
                         {indexAbbreviation[chapterFileName] <> ssn,
                          {chapterFileName, #}}]& /@ indexTags[cg]] /@ 
                                  Join[sectionGroups, subSectionGroups],
      Function[{es, ES}, (* count exercises and solutions *)
               MapIndexed[Function[{ℯ𝓈, p}, 
                          IndexEntry[preFormatIndexEntry[#],
                          {indexAbbreviation[chapterFileName] <> 
                                     ES <> ToString[p[[1]]], 
                                     {chapterFileName, #}}]& /@ 
                                                indexTags[ℯ𝓈]], es]] @@@
               {{exercisesGroups, "Ex."}, {solutionsGroups, "Sol."}}]]

In addition, we will capitalize the first letter of each entry.

capitalizeEntry[IndexEntry[{sL_List, "Text"}, r_]] :=
 With[{sL1 = StringReplace[#, {"{" -> "", "}" -> ""}]& /@ sL},
      IndexEntry[{MapAt[StringJoin[MapAt[ToUpperCase, 
                        Characters[#], 1]]&, sL1, 1], "Text"}, r]]

capitalizeEntry[e_] := e

We have extracted more than 11,000 index entries.

indexEntries1 = capitalizeEntry /@ Flatten[Table[
                extractClassifiedIndexEntries[chapterNotebooks[[k]]], 
                              {k, Length[chapterNotebooks]}]];
     
Length[indexEntries1]

Here are a few from the middle shown.

Take[indexEntries1, {4998, 5002}] // InputForm

Next, we will order the entries. Because entries can contain nonletters, we must program the ordering ourselves and cannot just resort to a call to Sort.

Here are the characters in the order we want them to appear: first numbers, then symbols, and the letters.

sortedCharacters = 
{"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", 
 "ε", "η", "Κ", "Λ", "λ", "Π", "π", "φ", "ζ", "σ", "τ", 
 "℘",
 "A", "a", "á", "à", "ä", "B", "ℬ", "b", "C", "c", "D", "d", 
 "E", "e", "é", "F", "f", "G", "g",
 "H", "h", "I", "i", "í", "J", "j", "K", "k", "L", "l", "M", 
 "m", "N", "n", 
 "O", "o", "ó", "ô", "ö", "P", "p", "𝒫", "Q", "q", "R", "r", 
 "S", "s", "T", "𝒯", "t", 
 "U", "u", "ü", "V", "v", "W", "w", "X", "x", "Y", "y", "Z", "z",
 "$", (* to keep nested entries together *) ",", " "};

We will ignore various characters for the ordering.

ignoredCharacters = 
{"(", ")", "[", "]", "}", "-", "+", "`", ".", "\"", "–", "\[Hyphen]", "~",
 "”", "’", "“", "≠", "^", "=", "|", "\\", ">", "'", "/",
 (* invisible space *) ""};

Here is a quick check that all characters appearing in the index entries either have a defined position in the ordered list or are ignored.

Complement[Union[Flatten[Characters /@ Flatten[#[[2, 1]]& /@ indexEntries1]]],
           Join[sortedCharacters, ignoredCharacters]] // InputForm

The function sortedQ returns True when its first argument comes before its second when ordered. In case of a tie, we compare the next characters.

(* is a string a single character? *)
characterQ[c_String] := StringLength[c] === 1

(* the first character of a string *)
firstCharacter[s_String] := 
      If[StringLength[s] > 0, StringTake[s, {1, 1}], $Failed]

(* are two single pairs sorted? *)
sortedQ[{string1a_String, string1b___String}, 
        {string2a_String, string2b___String}] := 
sortedQ[string1a <> string1b, string2a <> string2b]

(* are two single strings sorted? *)
sortedQ[string1_String, string2_String] := 
Module[{pos1, pos2},
   Which[(* ignore certain characters *)
         MemberQ[ignoredCharacters, firstCharacter[string1]], 
         sortedQ[StringDrop[string1, 1], string2],
         MemberQ[ignoredCharacters, firstCharacter[string2]], 
         sortedQ[string1, StringDrop[string2, 1]],
         (* one of the strings is empty *)
         StringLength[string1] === 0 && StringLength[string2] > 0,
         True, 
         StringLength[string1] > 0 && StringLength[string2] === 0,
         False, 
         (* both strings have characters left *)
         StringLength[string1] > 0 && StringLength[string2] > 0,
         {pos1, pos2} = Position[sortedCharacters, 
                          firstCharacter[#]][[1, 1]]& /@ {string1, string2};
         pos1 = Position[sortedCharacters, firstCharacter[string1]][[1, 1]];
         pos2 = Position[sortedCharacters, firstCharacter[string2]][[1, 1]];
         Which[pos1 < pos2, True, 
               pos2 < pos1, False,
               pos1 === pos2, 
               sortedQ[StringDrop[string1, 1], StringDrop[string2, 1]]],
         True, True]]

Here is a simple example: "L’Hôpital’s rule" will appear before "Limits, simple ~".

sortedQ["L’Hôpital’s rule", "Limits, simple ~"]

All entries can be sorted in a few minutes. Approximately 100000 strings will be split in characters and then compared character by character in this process.

(indexEntries2 = Sort[indexEntries1, 
                      Check[sortedQ[#1[[1, 1]], #2[[1, 1]]],
                            Print[{#1[[1, 1]], #2[[1, 1]]} // 
                       InputForm]]&]); // Timing

Here are the first four and the last four entries after sorting.

Take[indexEntries2, +4]

Take[indexEntries2, -4]

We unite the entries having identical content but having different links.

indexEntries3 = Split[indexEntries2, (#1[[1]] === #2[[1]])&];

{First[#], Length[#]}& /@ Split[Sort[Length /@ indexEntries3]]

indexEntries4 = IndexEntry[#1[[1, 1]], List @@ (Last /@ #)]& /@ indexEntries3;

For example, the first entry has two links.

indexEntries4[[1]]

Here is the distribution of the number of links per entry.

{First[#], Length[#]}& /@ Split[Sort[Length[Last[#]]& /@ indexEntries4]]

ListPlot[{#1, Log[#2]}& @@@ %]

The sorted list of index entries indexEntries4 could be used to generate the index of the GuideBooks. Some further styling (like single letter italics for mathematical symbol entries) would be needed. But here we are interested in creating the browser index notebook for the GuideBooks. Because a strictly linear ordering of the 14 main chapter notebooks has some advantages within the help browser setting, we revert our numbering to such a scheme.

For the browser index, we need entries that have button boxes with button data of the form {indexTagOfTheVisiblePart, scrollTag}. So, we implement a few functions that carry out the transformations needed to build these buttons.

(* preface all tags with 'GuideBooks' *)
toCopyTagScrollTagPair[{nb_, tag_}] :=  
         {containingCopyTag[tag], "GuideBooks:" <> tag}

(* convert four-volume tags to 14-chater tags *)
fourVolumeToInteger[string_String] :=
With[{sm = StringMatchQ[string, #]&}, 
Which[sm @ "P.1*",  "1", sm @ "P.2*",  "2", sm @ "P.3*",  "3", 
      sm @ "P.4*",  "4", sm @ "P.5*",  "5", sm @ "P.6*",  "6", 
      sm @ "G.1*",  "7", sm @ "G.2*",  "8", sm @ "G.3*",  "9", 
      sm @ "N.1*", "10", sm @ "N.2*", "11", 
      sm @ "S.1*", "12", sm @ "S.2*", "13", sm @ "S.3*", "14",
      sm @ "Pr*", "Preface", sm @ "In*", "0", sm @ "Ap*", "A"]]

(* the number of an exercise/solution pair *)
exerciseSolutionNumber[s_String] := 
         StringTake[s, {StringPosition[s, "."][[-1, 1]] + 1, -1}]

(* the number of a subsection or section *)
sectionSubsectionNumber[s_String] := 
Module[{rest = Which[MemberQ[{"P", "G", "N", "S"}, StringTake[s, 1]],  
                     StringDrop[s, 4],
                     StringTake[s, 1] === "I",  StringDrop[s, 3],
                     StringTake[s, 1] === "A",  StringDrop[s, 3]]}, 
       {If[Length[StringPosition[rest, "."]] === 0, 
           "Section", "Subsection"], rest}]

(* identity where tag refers to *)
containerCopyTag[abbr_] :=
Which[(* exercise or solution *) StringMatchQ[abbr, "*Ex*"] || 
      StringMatchQ[abbr, "*Sol*"], 
      "GuideBooks:Problem:" <> fourVolumeToInteger[abbr] <> "." <> 
                                      exerciseSolutionNumber[abbr],
      (* section or subsection *)  True,
      {ss, sn} = sectionSubsectionNumber[abbr];
      "GuideBooks:" <> ss <> ":" <> fourVolumeToInteger[abbr] <> 
                "." <> sn]

(* make a single hyperlink button *)
makeHyperLink[{abbr_, {nb_, tag_}}] :=  ButtonBox[abbr, 
    RuleDelayed @@ {ButtonData,  {containerCopyTag[abbr], 
                                 "GuideBooks:" <> tag}},
    ButtonStyle -> "AddOnsLink"]

(* make a sequence of hyperlink buttons *)
makeHyperLinkSequence[l_] := Sequence @@ Flatten[
     Transpose[{Table[", ", {Length[l]}], makeHyperLink /@ l}]]

It remains to make the cells of style "Index". Depending on the content of the cells, we will use different formatting (Mathematica functions in a Courier plain style and text in a Times-like style).

makeBrowserIndexCell[IndexEntry[{{s_String}, "Text"}, links_]] :=
 Cell[TextData[{s, (makeHyperLinkSequence @ links)}], 
      "IndexEntry", CellTags -> s];

makeBrowserIndexCell[IndexEntry[{{s1_String, s2_String}, "Text"}, links_]] :=
 Cell[TextData[{s1, ", ", s2, (makeHyperLinkSequence @ links)}], 
      "Index", CellTags -> (s1 <> ", " <> s2)];

makeBrowserIndexCell[IndexEntry[{{s_String}, 
                     "BuiltInFunction" | "GuideBooksFunction"}, links_]] :=
 Cell[TextData[{StyleBox[s, "MR"], (makeHyperLinkSequence @ links)}], 
      "Index", CellTags -> s];

And here is the browser index notebook.

browserIndexNotebook = Notebook[
 Join[{Cell[TextData[{StyleBox["Mathematica GuideBooks",  
                                FontSlant -> "Italic"]}], "IndexSection",
             CellTags -> "MasterIndexHeading"]}, 
      makeBrowserIndexCell /@ Take[indexEntries4, All]],
      StyleDefinitions -> "HelpBrowser.nb"];

We display the notebook, have a quick glance at it, save it, and then close it.

browserIndexNotebookH = NotebookPut[browserIndexNotebook];

Pause[3];

Do[Pause[Random[Real, {0.06, 0.18}]]; 
   FrontEndExecute[{FrontEndToken[browserIndexNotebookH, "ScrollPageNext"]}],
   {222}]
   
Pause[3];   

NotebookSave[browserIndexNotebookH, 
              ToFileName[helpBrowserGuideBooksDirectory, "BrowserIndex.nb"]]

NotebookClose[browserIndexNotebookH]

For the help browser to act in its usual way, we have to carry out a few modifications on the GuideBooks. The cells that should be displayed under a given topic should all have the same value of the CopyTag option. And the first cell in the part to be displayed should have an IndexTag option setting.

An important restriction of the help browser is that all CellTags must be unique within the corresponding directories.

Because the IndexTag values are displayed in the help browser text field, we rename the current index names from the current abbreviations to fully spelled out phrases.

(* rename all tags within a notebook *)
renameCellTags[nb_Notebook, nbTag_] :=
Module[{bibliographyTags, renamedBibliographyTags, dpr, nbL1, nbL2},
       (* rename the reference tags *)
       bibliographyTags = Last /@ Flatten[
                    Cases[Cases[nb, Cell[_, "BibliographyItem", __], Infinity],
                          HoldPattern[CellTags -> _], Infinity]];
       renamedBibliographyTags = 
        ("GuideBooks:" <> nbTag <> ":Reference:"  <> #)& /@ bibliographyTags;  
       dpr = Dispatch[Rule @@@ Transpose[{bibliographyTags, 
                                          renamedBibliographyTags}]];
       nbL1 = nb //. (* do the renaming *)
           c:Cell[_, "BibliographyItem", __] :> (c //. dpr) //.
           cb:CounterBox["BibliographyCounter", _] :> (cb //. dpr) /.
           bb:ButtonBox["✶", r1___, ButtonData :> t_, r2___] :>
           ButtonBox["✶", ButtonData -> (t //. dpr), r1, r2, ButtonNote -> t];
       (* rename all other tags *)
       nbL2 = nbL1 /. (CellTags -> tags_) :> 
                      (CellTags -> Flatten[{renameCellTags[tags]}])];

(* rename all tags in the right-hand side values of CellTags *)
renameCellTags[l_List] := renameCellTags /@ l

(* rename an individual cell tag *)
renameCellTags[s_String] := 
Which[StringMatchQ[s, "T[S[*"], 
      "GuideBooks:Section:" <> StringTake[s, {5, -3}],
      StringMatchQ[s, "T[SS[*"], 
      "GuideBooks:Subsection:" <> StringTake[s, {6, -3}],
      StringMatchQ[s, "T[D[*"],
      "GuideBooks:Topic:" <> StringTake[s, {5, -3}],
      StringMatchQ[s, "Index*"], "GuideBooks:" <> s,
      True, s]

The function addCellTags adds cell tags to a cell. We need to add the CopyTag option to all cells that should be visible in the help browser.

addCellTags[Cell[c_, s_], tags_] := Cell[c, s, CellTags -> Flatten[{tags}]];

addCellTags[Cell[c_?(FreeQ[c, Cell[__], Infinity]&), s_, options___], 
            tags_] := 
If[FreeQ[{options}, CellTags -> _], 
   Cell[c, s, options, CellTags -> Flatten[{tags}]], 
   Cell[c, s, Sequence @@ ({options} /. 
              (CellTags -> ct_) :> (CellTags -> Flatten[{tags, ct}]))]]
                                        
addCellTags[c_Cell, {}] := c;

To extract the tags that indicate headers (containing substrings like "GuideBooks:Section"), we use the function extractStringMatchedTag. For a given cell, it extracts all cell tags that match the given substring tag.

Clear[extractStringMatchedTags, extractStringMatchedTag]

(* extract all tags *)
extractStringMatchedTags[Cell[_, _, options___], tag_String] :=
 Select[Flatten[{CellTags /. {options}}], 
                (StringMatchQ[#, "GuideBooks*" <> tag <> "*"] && 
                 Not[StringMatchQ[#, "*CopyTag"]])&]
               
extractStringMatchedTags[c_Cell, tags_Alternatives] := 
       Flatten[extractStringMatchedTag[c, #]& /@ (List @@ tags)]

(* extract only one tag *)
extractStringMatchedTag[c_Cell, tagOrTags_] := 
 If[# === {}, {}, First[#]]& @ extractStringMatchedTags[c, tagOrTags]

We define an auxiliary function flattenCellGroup that removes all outer Cell and CellGroupData heads of a cell group and returns a flat list of cells.

(* flatten out cells of an individual cell group *)
flattenCellGroup[c_] := Flatten[c //. Cell[CellGroupData[l_List, ___], ___] :> l]

(* flatten out all cells of a notebook *)
flattenCellGroup[nb_Notebook] := 
   MapAt[Flatten,  Flatten[nb //. Cell[CellGroupData[l_List, ___], ___] :> l], 1]

We also have to add index tags to the exercises and solutions, so that they will be displayed individually in the help browser.

addTaggedProblemSection[nb_, nbTag_] :=
Module[{exercises, solutions, problems, indexTag, copyTag, nbL1, nbL2}, 
 (* the exercises *)
 exercises = Cases[nb, Cell[CellGroupData[{Cell[_, "ExerciseHeading", ___], 
                             ___}, ___], ___], Infinity];
 (* the solutions *)
 solutions = Cases[nb, Cell[CellGroupData[{Cell[_, "SolutionHeading", ___], 
                             ___}, ___], ___], Infinity];
 (* make problems by uniting exercises with solutions *)
 problems = Cell[CellGroupData[Flatten @
       {Cell["Problems", "ExercisesSection"],
        (* add index tags and copy tags *)
        MapIndexed[(indexTag = "GuideBooks:Problem:" <> nbTag <>
                                          "." <> ToString[#2[[1]]];
                    copyTag = indexTag <> "<=CopyTag";                      
        Cell[CellGroupData[ 
         Function[c, addCellTags[c, copyTag]] /@ 
         Flatten[{Cell["Problem " <> ToString[#2[[1]]], "ProblemHeading",
                       CellTags -> {indexTag}], #}]]])&,
                flattenCellGroup /@ Transpose[{exercises, solutions}]]}]] /.
        (* use generic titles to avoid long titles in the help browser *)             
        {Cell[_, eh:"ExerciseHeading", o___] :> Cell["Exercise", eh, o],
         Cell[_, sh:"SolutionHeading", o___] :> Cell["Solution", sh, o]};
 (* the delete no longer needed solution section *)
 nbL1 = DeleteCases[nb, Cell[CellGroupData[{Cell[_, "SolutionsSection", ___], 
                              ___}, ___], ___], Infinity];
 (* exercise section now contains the questions and the answers *)
 nbL2 = nbL1 /. Cell[CellGroupData[{Cell[_, "ExercisesSection", ___], 
                             ___}, ___], ___] -> problems ]

Because the help browser displays parts of a notebook as individual units, we have to add appropriate initialization cells to each unit.

newInitializationCell[{cellTag___}] := 
(* read file relative to help browser notebook *)
Cell[TextData[{
 StyleBox["(* no spelling warnings, set fonts for tick labels, ... *)", 
          "CodeComment"],
  "\nGet[ToFileName[ReplacePart[\"FileName\" /. " <>
  "\n      NotebookInformation[HelpBrowserNotebook[]], " <>
  "\"Initialization.m\", 2]]];"}], "Input", 
  InitializationCell -> True, (* add the copy tag *)
  If[{cellTag} === {}, Sequence @@ {}, CellTags -> First[{cellTag}]]];

addInitizalizationCells[hhUnit_] := 
Module[{hhUnit1 = DeleteCases[hhUnit, 
                        Cell[_, "Input", ___, InitializationCell -> True, ___], 
                              Infinity],
        flatPhysicalCells, inputCells, firstInputCellPosition, copyTags},
       (* insert an initialization cell before the first input cell *)
       If[FreeQ[hhUnit1, Cell[_, "Input", ___], Infinity], hhUnit1,
          flatPhysicalCells = Flatten[hhUnit1 /. 
                                      Cell[CellGroupData[l_List, ___], ___] :> l];
          inputCells = Cases[flatPhysicalCells, Cell[_, "Input", ___]];         
          If[inputCells === {}, hhUnit1, 
             (* add corresponding copytag *)
             firstInputCellPosition = Position[hhUnit1, inputCells[[1]]][[1]];
             copyTags = Cases[Cases[inputCells[[1]], HoldPattern[CellTags -> _], 
                                   Infinity], 
                              _String?(StringMatchQ[#, "*<=CopyTag"]&), {-1}];
             MapAt[(Sequence @@ {newInitializationCell[copyTags], #1})&,
                   hhUnit1, firstInputCellPosition]]]]

The function addSectionCopyTags and addReferenceCopyTags adds CopyTags to all individual units that we want to be displayed.

addSectionCopyTags[nb_] := nb /.
{C:Cell[CellGroupData[{c:Cell[_, "Subsection", ___], ___}, ___], ___] :>
 With[{copyTag = extractStringMatchedTag[c, "Subsection"] <> "<=CopyTag"}, 
       Cell[CellGroupData[addCellTags[#, copyTag]& /@ flattenCellGroup[C]]]],
 C:Cell[CellGroupData[{c:Cell[_, "Section", ___], ___}, ___], ___] :>
 With[{copyTag = extractStringMatchedTag[c, "Section"] <> "<=CopyTag"}, 
       Cell[CellGroupData[addCellTags[#, copyTag]& /@ flattenCellGroup[C]]]] /;
                                FreeQ[C, Cell[_, "Subsection", ___], Infinity]}

addReferenceCopyTags[nb_] := 
Module[{citation, oldReferenceSection, theReferences, referenceTags, 
        hhUnits, occurringReferences, unAccountedReferences, 
        hhUnitsIndexTags, newhhUnitsPre, newhhUnits, refsAndCopyTags, 
        theNewReferences, newReferenceSectionTrimmed, nbL1}, 
 (* the original section with the references *)
 oldReferenceSection = Cases[nb, 
 Cell[CellGroupData[{Cell[_, "ReferencesSection", ___], ___}, ___], ___], 
                          Infinity][[1]];
 (* the references *)
 theReferences =  
 DeleteCases[Cases[oldReferenceSection, Cell[_,  "BibliographyItem", ___], 
                   Infinity],
             Cell[_BoxData?(MemberQ[#, "ReferenceInstance"]&) |
             _String?(StringMatchQ[#, "*_CopyTag"]&), {0, Infinity}]];
 (* hash the individual references *)
 (citation[Flatten[CellTags /. (List @@ Drop[#, 2])][[1]]] = #)& /@ 
                                                     theReferences;       
 (* the (cell) tags of the references *)
 referenceTags = Flatten @ (CellTags /. ((List @@ Drop[#, 2])& /@
                                                   theReferences));
 (* sections, subsections, and problem sections *)
 hhUnits = 
 Cases[nb,  Cell[CellGroupData[{Cell[_, "Section" | "Subsection" | 
                                         "ProblemHeading",
                                      ___], r___}, ___], ___] /; 
                          FreeQ[{r}, Cell[_, "Subsection", ___], 
                          Infinity], Infinity];
 (* the references occurring in the sections, subsections, 
    and problem sections *)
 occurringReferences = Intersection[#, referenceTags]& /@
                (Cases[#, HoldPattern[ButtonData -> _], Infinity]& /@ 
                                   hhUnits //. (ButtonData -> x_) :> x); 
 (* in case of unaccounted references *)
 unAccountedReferences = Complement[Flatten[occurringReferences], 
                                    theReferences];
 occurringReferences = DeleteCases[occurringReferences, 
                                  Alternatives @@ unAccountedReferences];
 (* the index tags of the sections, subsections, and problem sections *)
 hhUnitsIndexTags = extractStringMatchedTag[#[[1, 1, 1]], 
                               "Section" | "Subsection" | "Problem"]& /@ 
                                                                hhUnits; 
 (* the sections, subsections, and problem sections with the 
    local reference tags *)
 newhhUnitsPre = 
 Table[If[occurringReferences[[k]] === {}, hhUnits[[k]],
          {hhUnits[[k]], 
           localCopyTag = Cases[hhUnits[[k]],
                                _String?(StringMatchQ[#, "*<=CopyTag"]&), 
                                {-1}][[1]];
           (* references subheader *)
           Cell["References", "ReferencesSection", 
                              CellGroupingRules -> {"SectionGrouping", 80},
                              CellTags -> localCopyTag]}],
                    {k, Length[hhUnits]}] //. 
                    (* neutralize patterns in interpretation boxes *)
                    {Blank -> blank, Pattern -> pattern}; 
 newhhUnits = addInitizalizationCells /@ newhhUnitsPre;                   
 (* references and the corresponding copytags  *)
 refsAndCopyTags = {#[[1, 1]], (# <> "<=CopyTag")& /@ Last /@ #}& /@ 
  Split[Sort[Flatten[Function[{refs, tag}, {#, tag}& /@ refs] @@@
        Transpose[{occurringReferences, hhUnitsIndexTags}], 1]],
           #1[[1]] === #2[[1]]&];
 (* the copy-tagged local references of the units  *)
 theNewReferences = addCellTags[citation[#1], #2]& @@@ refsAndCopyTags;  
 (* the new references section  *)
 newReferenceSection = oldReferenceSection /. 
   Dispatch[Rule @@@ Transpose[{theReferences, theNewReferences}]];
 (* remove reference instance links  *)
 newReferenceSectionTrimmed = DeleteCases[newReferenceSection, 
              Cell[_BoxData?(MemberQ[#, "ReferenceInstance"]&), Infinity]];
 (* the new notebook  *)
 nbL1 = Join[Notebook[Flatten[
 nb[[1]] /. oldReferenceSection -> newReferenceSectionTrimmed  /.                            
             (Rule @@@ Transpose[{hhUnits, newhhUnits}]) //.
              Cell[CellGroupData[l_, ___], ___] :> l]], Rest[nb]] //.
                                {blank -> Blank, pattern -> Pattern}]

We could now continue to carry out various cosmetic operations on the notebook. Because we can safely skip the Overview section in the help browser (due to the presence of the reference guide), we will in exchange link all occurrences of built-in functions in the text cells to the corresponding reference entries.

refGuideFile = 
   Select[FileNames["*RefGuide.nb", $InstallationDirectory, Infinity],
          Not[StringMatchQ[#, "*AddOns*"]]&][[1]];

refGuideNotebook = Get[refGuideFile];

allDocumentedFunctions = #[[1, 1, 1, 1]]& /@ Take[refGuideNotebook[[1]], All];

(* is a function listed in the reference guide? *)
(refGuideFunctionQ[#] := True)& /@ allDocumentedFunctions;
refGuideFunctionQ[x_] := False

addRefGuideHyperlinks[nb_, fileName_] := 
Module[{nbL1},
 nbL1 = nb //. {StyleBox[f_?refGuideFunctionQ, "MR"] :> 
                ButtonBox[f, ButtonStyle -> "RefGuideLink"],
                sb:StyleBox[_String, "MR"] :> 
                   findRefGuideFunctionsInAString[sb]};
 (* print number of added reference guide links *)
 Print["Number of links to the reference guide added in " <> fileName <> ": ", 
       Count[nbL1, "RefGuideLink", {-1}]];
 nbL1]

findRefGuideFunctionsInAString[StyleBox[s_String, "MR"]] :=
Module[{dividersPos, dividers, potentialsPos, potentials, 
        hyperlinkedPotentials, hyperLinkedStringParts}, 
 (* position of function-dividing non-letter characters *)
 dividersPos = Union[Flatten[{StringPosition[s, #]& /@ 
               {" ", ",", "_", "[", "]", ";", "*", "/", "'", "(", ")",
                "{", "}", "^", "="}}]];
 (* dividing non-letter caracter sequences *)
 dividers = {{#, #}, StringTake[s, {#, #}]}& /@ dividersPos;               
 (* position of candidate functions *)
 potentialsPos = (# + {1, -1})& /@ 
                 Partition[Flatten[{0, dividersPos, StringLength[s] + 1}], 2, 1];
 (* candidate functions *)
 potentials = {#, StringTake[s, #]}& /@ potentialsPos;
 (* in case the function occurs in the reference guide, add hyperlink *)
 hyperlinkedPotentials = {#1, If[refGuideFunctionQ[#2], 
    ButtonBox[#2, ButtonStyle -> "RefGuideLink"], #2]}& @@@ potentials;           
 (* function-dividing non-letter characters and 
    potentially hyperlinked functions (in sorted order *)
 hyperLinkedStringParts = Last /@ Sort[Join[dividers, hyperlinkedPotentials]];
 (* format the sequence of styles characters and hyperlinked functions *)
 If[FreeQ[hyperLinkedStringParts, _ButtonBox], StyleBox[s, "MR"], Sequence @@
    (((If[Head[#] === ButtonBox, #, StyleBox[#, "MR"]]& /@ 
     hyperLinkedStringParts)) //. 
     {a___, StyleBox[s1_String, "MR"], StyleBox[s2_String, "MR"], b___} :>
     {a, StyleBox[s1 <> s2, "MR"], b})]]

In the original 14 main chapters of the GuideBooks, all references occur together at the end and are sorted alphabetically by the last name of the first author. Because within the help browser we are displaying only a small fraction of each chapter at once, it is preferable to only display the explicitly cited references in alphabetical order. The following functions remove the hard-coded reference numbers and add dynamic counters instead.

removeExplicitCounter[s_String] :=
NestWhile[StringDrop[#, 1]&, s, IntegerQ[ToExpression[StringTake[#, 1]]]&]

removeExplicitReferenceNumber[s_String] :=
NestWhile[StringDrop[#, -1]&, s, StringTake[#, -1] =!= "["&]

addDynamicReferenceCounting[nb_] := 
Module[{references, referenceTags, isReferenceTagQ, nbL1},
 (* the references *)
 references = Cases[nb,  Cell[_, "BibliographyItem", ___], Infinity];
 (* the tags of the references *)
 referenceTags = Select[Flatten[Last /@ Flatten[
          Cases[references, HoldPattern[CellTags -> _], Infinity]]],
          StringMatchQ[#, "*Reference*"]&];
 (* define reference identifying function *)
 (isReferenceTagQ[#] = True)& /@ referenceTags;
 nbL1 = nb //.
   (* remove explicit counter of references *)
   Cell[TextData[{bb_ButtonBox, s_String, rest___}],
                 "BibliographyItem", os___] :>
   Cell[TextData[{bb, CounterBox["BibliographyCounter"],
                  removeExplicitCounter[s], rest}],
                 "BibliographyItem", os] //.
   (* replace explicit counter by dynamic counter *)
   TextData[{a___, s1_String, 
             bb:ButtonBox[___, ButtonData -> t_?isReferenceTagQ, ___], b___}] :> 
   TextData[{a, removeExplicitReferenceNumber @ s1, 
             CounterBox["BibliographyCounter", t], bb,  b}]]

We end the manipulations to be carried out on the notebooks by adding the open-close triangles on the left hand side for the exercises, solutions, and the local reference sections.

addOpenCloseTriangles[nb_ ] :=
nb /. c:Cell[_,  "ExerciseHeading" | "SolutionHeading" | "ReferencesSection",
             ___] :> Append[c, ShowGroupOpenCloseIcon -> True]

fixVariousProblems[nb_, fileName_] := 
 Which[fileName === "4_Symbolics_1.nb",
       DeleteCases[nb, "GuideBooks:Index[12, M[DSolve]]" |
                       "GuideBooks:Index[12, {Riccati, differential equations}]", 
                   Infinity], 
       True, nb]

variousTweaks[nb_, fileName_] := 
    fixVariousProblems[#, fileName]& @ 
       addDynamicReferenceCounting @ 
          addOpenCloseTriangles @ 
             addRefGuideHyperlinks[#, fileName]& @ nb

We will number the notebooks of the four GuideBook volumes consecutively from 1 to 14 to emphasize the ordering of the volumes.

removeComponents[nb_] := 
DeleteCases[nb, Cell[CellGroupData[{Cell[_, "OverviewSection", ___], ___}, ___], ___] |
                        Cell[BoxData[FormBox[ButtonBox[__, 
                        ButtonStyle -> "ReferenceInstance"], __]]], Infinity];

Now, we have all functions together to define the final notebook-transforming function helpBrowserizeGuideBookNotebook that reads in the chapter fileName from the AddOns directory, carries out various transformations on the notebook, renders the transformed notebook in the front end and saves out the transformed version. (For larger notebook, the step to display the notebook in the front end instead of directly writing a file from the kernel expression of the notebook is essential because the front end generates the cached data that allows for a much faster retrieval of parts of the notebook for display.)

The function guideBooksNotebookSave opens a programmatically constructed notebook and saves it. When the stylesheet needed for opening the notebook is not a globally visible stylesheet, the opening process might produce dialog boxes. To avoid this, we temporarily embed the stylesheet and then, after the notebook is displayed, switch the stylesheet definition.

guideBooksNotebookSave[nb_, fileName_] := 
Module[{nbSs, nbH},
  (* just for opening, add styledefinition in embedded form *)
  nbSs = nb /. (StyleDefinitions -> _) :> 
               (StyleDefinitions -> ToFileName[helpBrowserGuideBooksDirectory, 
                                              "GuideBooksStylesheet.nb"]);
  nbH = NotebookPut[nbSs];
  (* save notebook *)
  NotebookSave[nbH, ToFileName[helpBrowserGuideBooksDirectory, fileName]];
  (* refer to local stylesheet *)
  SetOptions[nbH, StyleDefinitions -> "GuideBooksStylesheet.nb",
                  ScreenStyleEnvironment -> "Reading"];
  NotebookSave[nbH]; NotebookClose[nbH]]

helpBrowserizeGuideBookNotebook[fileName_, chapterTag_] :=
Module[{nb0, nb1, nb2, nb3, nb4, nb5, nb6},
 (* read in original notebook *)
 nb0 =  Get[ToFileName[helpBrowserGuideBooksDirectory, fileName]]; 
 (* remove components not needed for the help browser version *)
 nb1 = removeComponents[nb0];
 (* rename cell tags *)
 nb2 = renameCellTags[nb1, chapterTag]; 
 (* add exercises and solutions as problems *)
 nb3 = addTaggedProblemSection[nb2, chapterTag];
 (* add copy tags to all units *)
 nb4 = addSectionCopyTags[nb3]; 
 (* add reference copy tags to the references that belong to units *)
 nb5 = addReferenceCopyTags[nb4]; 
 (* add reference guide links, and other little niceties *)
 nb6 = variousTweaks[nb5, fileName]; 
 guideBooksNotebookSave[nb6, fileName]]

chapterNotebooksAndChapterTags =
       MapIndexed[{#1, If[#2[[1]] =!= 16, ToString[#2[[1]] - 1], "Ap"]}&, 
                  chapterNotebooks];

Overall, more than 12,000 links to the reference guide will be added this way.

Do[helpBrowserizeGuideBookNotebook @@ 
       chapterNotebooksAndChapterTags[[k]], {k, 16}]

Now, we quickly process the preface (which we display at once) and make a notebook from the copyright file.

prefaceNb =  Get[ToFileName[helpBrowserGuideBooksDirectory, "Preface.nb"]];

(* carry out some minor changes on the preface *)
prefaceNb1 = DeleteCases[
  prefaceNb /. c:Cell[__] /; FreeQ[c, _CellGroupData, Infinity] :> 
                             addCellTags[c, "GuideBooks:Preface<=CopyTag"] /.
               c:Cell[_, "ChapterNumber"] :> 
                          addCellTags[c, "GuideBooks:Preface.Start"],
                         Cell[_BoxData?(MemberQ[#, "ReferenceInstance", 
                              Infinity]&)], Infinity];

(* save modified preface *)
guideBooksNotebookSave[prefaceNb1, "Preface.nb"]

(* text of the copyright text file *)
copyrightText = ReadList[ToFileName[helpBrowserGuideBooksDirectory, 
                         "Copyright.txt"],
                         Record, RecordSeparators -> {}][[1]];

(* split text in paragraphs *)
paragraphs = 
StringReplace[StringTake[copyrightText, # + {1, -1}], "\n" -> " "]& /@
Partition[Flatten[{0, StringPosition[copyrightText, "\n\n"], 
                       StringLength[copyrightText] + 1}], 2];

(* make notebook from paragraphs *)
copyrightNb = 
Notebook[{Cell["Copyright", "Section", 
               CellTags -> "GuideBooks:Copyright.Start"],
          Sequence @@ (Cell[#, "Text"]& /@ paragraphs)},
          ScreenStyleEnvironment -> "Reading"] /.
          c_Cell :> addCellTags[c, "GuideBooks:Copyright<=CopyTag"];

(* save notebook of copyright statement *)
guideBooksNotebookSave[copyrightNb, "Copyright.nb"]

Now we will deal with the table of content. We will not change its format, but we have to change the hyperlinks.

toc = Get[ToFileName[helpBrowserGuideBooksDirectory, "TableOfContents.nb"]];

We remove some items that we do not need for the versions of the GuideBooks in the help browser.

toc1 = DeleteCases[toc, 
     Cell[_, "ChapterNumber", ___] |
     Cell[_?(MemberQ[#, _String?(StringMatchQ[#, "*PREFACE*"]&), Infinity]&),
          "Section", ___] |
     Cell[_?(MemberQ[#, _String?((StringMatchQ[#, " Overview"] ||
                                  StringMatchQ[#, " References"])&),
                     Infinity]&),
     "ContentsItem", ___], Infinity] //. 
     ButtonBox[c_, ___, 
               ButtonData :> {_, _String?(StringMatchQ[#, "*T[CH[*"]&)}, ___] :> c;

We also have to change the hyperlinks because we have renamed all cell tags.

toHelpBrowserHeaderLink[
 ButtonBox[c_, ___, ButtonData :> {nb_, tag_}, ___,
                    ButtonStyle -> "ContentsLink", ___]] := 
 ButtonBox[c, ButtonData -> renameCellTags @ tag, 
              ButtonStyle -> "ContentsLink", ButtonNote -> None,
              ButtonFunction :> (FrontEndExecute[ {
                                   FrontEnd`HelpBrowserLookup["AddOns", #]}]&)]

toc2 = toc1 /. 
        c:Cell[_, "ContentsItem" | "ContentsSubItem", ___] :>
        (c /. bb_ButtonBox :> (toHelpBrowserHeaderLink[bb])) /;
                                  FreeQ[c, "AddOns", Infinity];

toc3 = toc2 /. 
        Cell["Contents", "Chapter", r___] :>
        Cell["Contents", "Chapter", CellTags -> "GuideBooks:TOC:Start"] /.
        Cell[c_, t:("ContentsItem" |  "ContentsSubItem" |
                    "ContentsVolumeLabel" | "Section" | 
                    "ContentsLinkList"), ___] :>
        Cell[c, t, CellTags -> "GuideBooks:TOC<=CopyTag"];

contentItemLists = Cases[toc3, 
       Cell[CellGroupData[{Cell[_, "ContentsItem", ___], ___}, ___], ___], 
                         Infinity];
 
flattenedContentItemLists = flattenCellGroup /@ contentItemLists;

Next, we have to rename the over 2000 links to detailed topic indicators. Because these topics are discussed within a section, subsection, or problem, we must use the two-element list version of the FrontEnd`HelpBrowserLookup. The following list topicRules contains a list of rules how to convert from the notebook numbers to the corresponding index tag numbers.

topicRules = 
Module[{contentsLinkListPositions, indexTag, 
        oldButtonData, newButtonData},
Flatten @ Table[
contentsLinkListPositions = 
  Flatten[Position[flattenedContentItemLists[[j]], 
                   Cell[_, "ContentsLinkList", ___], Infinity]];
  Table[indexTag = Cases[flattenedContentItemLists[[j]][[
                           contentsLinkListPositions[[k]] - 1]],
                 HoldPattern[ButtonData -> _], Infinity][[1, 2]];
        (* original button data of the topic links *)
        oldButtonData = Last /@ Cases[flattenedContentItemLists[[j]][[
            contentsLinkListPositions[[k]]]], 
            HoldPattern[ButtonData :> _], Infinity];
        (* construct new button data *)
        newButtonData = {indexTag, renameCellTags[#2]}& @@@ oldButtonData;          
        Rule @@@ Transpose[{oldButtonData, newButtonData}],         
         {k, Length[contentsLinkListPositions]}],
       {j, Length[flattenedContentItemLists]}]];

topicRules // Length

toc4 = toc3 //. Dispatch[topicRules] //. bb_ButtonBox :> 
 Append[bb,  ButtonFunction :> (FrontEndExecute[
                   FrontEnd`HelpBrowserLookup["AddOns", #]]&)] /;
                 FreeQ[bb, "AddOns", Infinity];

It remains to deal with the topic links into exercises and solutions. Because we united the exercises and the solutions to problems, we cannot infer the index tag for the problems for the existing table of contents. We have to resort to the new chapter notebooks. We read in once more the chapter notebooks, extract the individual problems, and extract their index tag as well as the cell tags for the topics of the form "GuideBooks:Topic:...".

indexTagsTopicTagListForProblems = Flatten[#, 1]& @ 
Table[
 (* load chapter into kernel *)
 nb = Get[ToFileName[helpBrowserGuideBooksDirectory, chapterNotebooks[[k]]]];
 (* extract problem units *)
 problemSections = Cases[nb, 
        Cell[CellGroupData[{Cell[_, "ProblemHeading", ___], ___}, ___], ___], 
                         Infinity];
 (* extract index tag and topic tags *)
 Flatten[Table[problemHeadCell = problemSections[[j]][[1, 1, 1]];
               indexTag = extractStringMatchedTag[problemHeadCell, "Problem"];
               (* make pairs {indexTag, topicCelltag} *)
               {indexTag, #}& /@  Flatten[extractStringMatchedTags[#, "Topic"]& /@ 
                                          flattenCellGroup[problemSections[[j]]]], 
                {j, Length[problemSections]}], 1],
       {k, Length[chapterNotebooks]}];

About 700 topic links have to be changed.

Length[indexTagsTopicTagListForProblems]

These are the original hyperlink specifications (the button data) from the table of contents notebook.

originalTopicButtonDataAll = 
    Last /@ Cases[toc4, HoldPattern[ButtonData :> _], Infinity];

originalTopicButtonData = 
 Select[originalTopicButtonDataAll,
        (Head[#[[1]]] === String &&
        (StringMatchQ[#[[1]], "T[Ex*"] || StringMatchQ[#[[1]], "T[Sol*"]))&];

We find the associated new links by matching the topic title and in case of multiple choices the chapter numbers.

mapDataGroups = 
(Function[t, {t, Select[indexTagsTopicTagListForProblems, #[[2]] === t[[2]]&]}] /@
                                              Take[originalTopicButtonData, All]);

extractIndexTagTopicPair[{{tExSol_, topicTag_}, matches_}] :=
Module[{(* from the old table of content *)
        chapterNumber = 
          StringTake[tExSol, {StringPosition[tExSol, "["][[-1, -1]] + 1,
                              StringPosition[tExSol, "]"][[+1, +1]] - 1}],
        chapterNumberToo},
        (* from the actual cell tags of the individual problems *)
        chapterNumberToo[s_String] := 
                         StringTake[s, {StringPosition[s, ":"][[-1, -1]] + 1,
                                        StringPosition[s, "."][[-1, -1]] - 1}];
  Select[matches, (chapterNumber === chapterNumberToo[#[[1]]])&][[1]]]

(* the list of replacement rules of the button data *)
mapDataRules = (#[[1]] -> extractIndexTagTopicPair[#])& /@ 
                          DeleteCases[mapDataGroups, {_, {}}];

toc5 = toc4 //. Dispatch[mapDataRules];

We save the table of contents with the new links.

guideBooksNotebookSave[toc5, "TableOfContents.nb"]

Now, we must deal with the various cells that we want to act as headers and that should appear in the help browser.

headerCellTypes = "Section"| "Subsection" | "ExercisesSection" | "ProblemHeading"

We start by extracting the data corresponding to an outlines of the chapter. Concretely this means we keep the section cells, the subsection cells, and the problem headings and their relative nesting properties.

makeChapterOutline[file_] := 
Module[{nb, nbL1, nbL2, nbL3, nbL4}, 
 (* read in notebook *)
 nb = Get[ToFileName[helpBrowserGuideBooksDirectory, file]];
 (* remove all no-head cells *)
 nbL1 = DeleteCases[nb[[1]], Cell[_, _?(Not[MatchQ[#, headerCellTypes]]&), 
                                 ___], Infinity];
 (* remove empty groups *)
 nbL2 = DeleteCases[nbL1, Cell[CellGroupData[{} , ___], ___], Infinity];
 (* keep only first cell of cell groups {keep nesting unchanged) *)
 nbL3 = nbL2 //. Cell[CellGroupData[cells_, ___], ___] :> cells;
 (* remove empty items iteratively *)
 nbL4 = FixedPoint[DeleteCases[#, {}, Infinity]&, nbL3]]

Here is, for instance, the outline of the first chapter of the Programming volume.

mco = makeChapterOutline["1_Programming_1.nb"]

The notebook section and subsection titles use occasionally non-ASCII characters (like a hyphen) and some small mathematically typeset formulas. For the help browser listing, we must translate these items into ASCII-only text.

removeTrailingWhiteSpace[s_String] :=
Module[{s1 = s},
       While[StringTake[s1, {-1, -1}] === " ", s1 = StringDrop[s1, -1]]; s1]

(* transform a special character (greek symbols, integrals) into ASCII *)
specialCharacterToASCII[s_] := 
Module[{chars = Characters[StringTake[ToString[FullForm[s]], {4, -3}]]},
       StringJoin @ 
        Which[(* special ASCII equivalents *)
              MatchQ[chars, {"C", "a", "p", "i", "t", "a", "l", ___}],
              Drop[chars, 7], 
              MatchQ[chars, {"H", "y", "p", "h", "e", "n"}], "-",
              MatchQ[chars, {"C", "l", "o", "s", "e", "C", "u", "r", "l", "y", 
                             "Q", "u", "o", "t", "e"}], "'",
              MatchQ[chars, {___, "G", "r", "a", "v", "e"}],
              (* use Mathematica longname *)
              MapAt[ToLowerCase, Drop[chars, -5], 1] <> "'", 
              True, MapAt[ToLowerCase, chars, 1]]]

(* transform a string into ASCII *)
toASCIIString[s_String] := removeTrailingWhiteSpace[
   StringJoin[(If[StringLength[ToString[FullForm[#1]]] === 3, #, 
     specialCharacterToASCII[#]]&) /@ 
      Characters[FixedPoint[StringReplace[#, {"\n" -> " ", "  " -> " "}]&, s]]]]

(* transform a TextData-object into ASCII *)
toASCIIString[TextData[l_List]] := 
toASCIIString @ StringJoin[Flatten[ (* ascii-tex-fy typeset boxes *)
  l //. {Cell[s_] :> s, BoxData[s_] :> s, FormBox[s_, _] :> s, 
         StyleBox[s_, __] :> s, SuperscriptBox[b_, e_] :> {b, "^", e},
         FractionBox[a_, b_] :> {a, "/", b},
         SubscriptBox[b_, s_] :> {b, "_", s}, RowBox[s_] :> s,
         SqrtBox[b_] :> {"sqrt(", b, ")"}, 
         RadicalBox[b_, e_] :> {b, "^(1/", e, ")"}}]]

removeLeadingNumbers[s_String] := 
Module[{s1 = s},
       While[StringTake[s1, {1, 1}] =!= " ", s1 = StringDrop[s1, 1]]; 
       StringDrop[s1, 1]]

Here is an example of the ASCII-fied version of a TextData-object containing mathematical typesetting.

removeLeadingNumbers @ toASCIIString @ TextData[{"1.9.2 ",
  Cell[BoxData[FormBox[RowBox[{"cos", "(", FractionBox[
            RowBox[{"2", "π"}], "257"], ")"}], TraditionalForm]]], 
       " à la Gauss "}]

The functions toPlainTextTitle makes a plain (ASCII only) text version of a header cell.

toPlainTextTitle[Cell[t_, "Section" | "Subsection", ___]] := 
removeLeadingNumbers @ toASCIIString @ t

toPlainTextTitle[Cell[t_, "ExercisesSection" |
                          "SolutionsSection", ___]] := t

Now, we must transform the nested list of section, subsection, and problem cells into the corresponding browser categories. To do this, we have to ASCII-fy and shorten titles and extract cell tags. The function makeBrowserCategoryItem carries out these operations on a whole Notebook-expression recursively.

(* thread over lists *)
makeBrowserCategoryItem[nb_, l_List] := makeBrowserCategoryItem[nb, #]& /@ l

(* items for sections without subsections *)
makeBrowserCategoryItem[nb_, {c:Cell[title_, "Section", options___]}] := 
 With[{tag = extractStringMatchedTag[c, "Section"]}, 
  Item[toPlainTextTitle[c], nb, IndexTag -> tag, 
                                CopyTag -> (tag <> "<=CopyTag")]]

(* recursive browser categories for sections with subsection *)
makeBrowserCategoryItem[nb_, {c:Cell[title_, "Section", options___],
                               subsections__}] := 
BrowserCategory[toPlainTextTitle[c], None, 
                  makeBrowserCategoryItem[nb, #]& /@ {subsections}]

(* items for subsections *)
makeBrowserCategoryItem[nb_, {c:Cell[title_, "Subsection", options___]}] := 
 With[{tag = extractStringMatchedTag[c, "Subsection"]},  
  Item[toPlainTextTitle[c], nb, IndexTag -> tag, 
                                CopyTag -> (tag <> "<=CopyTag")]]

(* recursive browser categories for exercises and solutions section *)
makeBrowserCategoryItem[nb_, 
   {c:Cell[title_, "ExercisesSection", options___], problems__}] := 
  BrowserCategory[toPlainTextTitle[c], None, 
   (makeBrowserCategoryItem[nb, #1] & ) /@ {problems}]

(* items for problems *)
makeBrowserCategoryItem[nb_, 
                        {c:Cell[title_, "ProblemHeading", options___]}] := 
  With[{tag = extractStringMatchedTag[c, "Problem"]},
       Item[title, nb, IndexTag -> tag, CopyTag -> (tag <> "<=CopyTag")]]

(* use chapter outlines to make browser categories for each chapter *)
makeBrowserCategoryOfChapter[fileName_] := 
    makeBrowserCategoryItem[fileName, makeChapterOutline[fileName]]

Here is an example, the browser categories from the first chapter of the Programming volume.

makeBrowserCategoryOfChapter["1_Programming_1.nb"]

Now, we build the complete browser category of the GuideBooks. To better differentiate between the four volumes, we add delimiters and titles at the topmost category level.

guideBooksBrowserCategories = 
BrowserCategory["Mathematica GuideBooks", None, 
   {(* table of contents *)
    Item["Table of Contents", "TableOfContents.nb", 
         IndexTag ->  "GuideBooks:TOC:Start",
         CopyTag -> "GuideBooks:TOC<=CopyTag"], 
    Item[Delimiter], Item[Delimiter], 
    (* preface *)
    Item["Preface", "Preface.nb", IndexTag-> "GuideBooks:Preface.Start",
                                  CopyTag -> "GuideBooks:Preface<=CopyTag"], 
    Item[Delimiter], 
    (* introduction *)
    BrowserCategory["Introduction and Orientation", None, 
                     makeBrowserCategoryOfChapter["0_Introduction.nb"]], 
    Item[Delimiter], Item[Delimiter], 
    (* six chapters of the Programing volume *)
    Item["PROGRAMMING VOLUME:", None], 
    BrowserCategory["Ch.1: Introduction to Mathematica", 
                    None, makeBrowserCategoryOfChapter["1_Programming_1.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.2: Structure of Mathematica Expressions", 
                    None, makeBrowserCategoryOfChapter["1_Programming_2.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.3: Definitions and Properties of Functions", 
                    None, makeBrowserCategoryOfChapter["1_Programming_3.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.4: Meta-Mathematica", 
                    None, makeBrowserCategoryOfChapter["1_Programming_4.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.5: Restricted Patterns and Replacement Rules", 
                    None, makeBrowserCategoryOfChapter["1_Programming_5.nb"]],          
     Item[Delimiter],
    BrowserCategory["Ch.6: Operations on Lists, and Linear Algebra ", 
                    None, makeBrowserCategoryOfChapter["1_Programming_6.nb"]],                                   
    Item[Delimiter], Item[Delimiter], 
    (* three chapters of the Graphics volume *)
    Item["GRAPHICS VOLUME:", None],               
    BrowserCategory["Ch.1: Two-Dimensional Graphics", 
                    None, makeBrowserCategoryOfChapter["2_Graphics_1.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.2: Three-Dimensional Graphics", 
                    None, makeBrowserCategoryOfChapter["2_Graphics_2.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.3: Contour and Density Plots", 
                    None, makeBrowserCategoryOfChapter["2_Graphics_3.nb"]],
    Item[Delimiter], Item[Delimiter], 
    (* two chapters of the Numerics volume *)
    Item["NUMERICS VOLUME:", None], 
    BrowserCategory["Ch.1: Numerical Computations", 
                    None, makeBrowserCategoryOfChapter["3_Numerics_1.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.2: Computations with Exact Numbers", 
                    None, makeBrowserCategoryOfChapter["3_Numerics_2.nb"]],
    Item[Delimiter], Item[Delimiter], 
    (* three chapters of the Symbolics volume *)
    Item["SYMBOLICS VOLUME:", None],
    BrowserCategory["Ch.1: Symbolic Computations", 
                    None, makeBrowserCategoryOfChapter["4_Symbolics_1.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.2: Classical Orthogonal Polynomials", 
                    None, makeBrowserCategoryOfChapter["4_Symbolics_2.nb"]],
    Item[Delimiter],
    BrowserCategory["Ch.3: Classical Special Functions", 
                    None, makeBrowserCategoryOfChapter["4_Symbolics_3.nb"]],
    Item[Delimiter], Item[Delimiter],
    (* appendix *)
    BrowserCategory["Appendix A", 
                    None, makeBrowserCategoryOfChapter[chapterNotebooks[[-1]]]],
    Item[Delimiter], Item[Delimiter],
    (* copyright statement *)
    Item["Copyright Statement", "Copyright.nb", 
         IndexTag -> "GuideBooks:Copyright.Start",
         CopyTag -> "GuideBooks:Copyright<=CopyTag"]}] /.
                                    _makeBrowserCategoryItem :> Sequence @@ {};

About 500 displayable units occur in the GuideBooks.

λ = Cases[guideBooksBrowserCategories, Item[_, _, _, _], Infinity] // Length

We save the just-extracted browser categories as the BrowserCategories.m file for the GuideBooks.

Put[guideBooksBrowserCategories, 
    ToFileName[helpBrowserGuideBooksDirectory, "BrowserCategories.m"]]

Now, we add the GuideBooks to the listing inside the add-on directory listing. We do this by explicitly modifying the BrowserCategories.m file of this directory. This allows to make sure that the GuideBooks appear early in the listing. (Another possibility would be to put the notebooks and browser files into the subdirectory "Wolfram Research"->"Mathematica"->"Version"->"AddOns"->"Applications"->"Mathematica GuideBooks"->"Documentation"->"English". This would allow Mathematica to automatically detect the presence of the corresponding browser category file.)

addOnBrowserCategories =
     Get[ToFileName[helpBrowserAddOnDirectory, "BrowserCategories.m"]];

(* list GuideBooks after the first item *)
firstDelimiterPosition = Position[addOnBrowserCategories, 
                                  Item[Delimiter], {2}, 1]

(* the new browser categories package *)
addOnBrowserCategoriesWithGuideBooks = 
MapAt[(Sequence @@ {Item[Delimiter], 
                HelpDirectoryListing[{FrontEnd`FileName[
                              {"AddOns", "Mathematica GuideBooks"}]}, 
                                     False, False],
                Item[Delimiter]})&,
      addOnBrowserCategories, firstDelimiterPosition];
      
Short[addOnBrowserCategoriesWithGuideBooks, 28]

(* keep a copy of the original browser categories *)
RenameFile[ToFileName[helpBrowserAddOnDirectory, "BrowserCategories.m"],
           ToFileName[helpBrowserAddOnDirectory, 
                      "BrowserCategories_BeforeAddingTheGuideBooks.m"]]

(* save the new browser categories package *)
If[(* avoid second copy in case this subsection is evaluated twice *)
   FreeQ[Cases[addOnBrowserCategories, _String, {-1}],
        "Mathematica GuideBooks"],
   Put[addOnBrowserCategoriesWithGuideBooks, 
       ToFileName[helpBrowserAddOnDirectory, "BrowserCategories.m"]]]

Now, we rebuild the help browser. In this process, the front end creates the actual entries in the help browser listing and the look-up entries of the master index.

FrontEndExecute[{FrontEndToken["RebuildHelpIndex"]}]

Now the process of incorporating the GuideBook notebooks into the help browser is finished.

NotebookClose @ HelpBrowserNotebook[]

Now, we are ready to use the GuideBooks in the help browser. We can do this either manually or programmatically. We can directly link to an item in the help browser version of the GuideBooks. The next input opens the help browser and the table of contents of the GuideBooks.

FrontEndExecute[{
      FrontEnd`HelpBrowserLookup["AddOns", "GuideBooks:TOC:Start"]}]

The next input opens the section “Brillouin Zones of Cubic Lattices” of the second chapter of the graphics volume.

Pause[3]; FrontEndExecute[{
          FrontEnd`HelpBrowserLookup["AddOns", "GuideBooks:Section:8.4"]}]

We can also quickly run through all individually displayed units—the sections, subsection, and problems.

allIndexTags = Last /@ 
    Cases[guideBooksBrowserCategories, HoldPattern[IndexTag -> _], Infinity];

(Pause[0.25]; FrontEndExecute[{FrontEnd`HelpBrowserLookup["AddOns", #]}])& /@
                               Take[allIndexTags, All];

We could also run through the items in a randomized manner.

randomPermutation[n_] :=
  Module[{l = Range[n], tmp1, tmp2, j},
         Do[tmp1 = l[[i]]; j = Random[Integer, {i, n}]; tmp2 = l[[j]];
         l[[i]] = tmp2; l[[j]] = tmp1, {i, Length[l]}]; l]

(Pause[0.5]; FrontEndExecute[{FrontEnd`HelpBrowserLookup["AddOns", #]}])& /@
                               Take[allIndexTags[[randomPermutation[λ]]], All];

Using a two-element list in the second argument of FrontEnd`HelpBrowserLookup allows to scroll to a certain position within a help browser unit. The next input opens Subsection 11.2 of the Numerics first chapter and scrolls to the graphic of a Riemann surface of a septic.

FrontEndExecute[{FrontEnd`HelpBrowserLookup["AddOns", 
                {"GuideBooks:Subsection:10.12.2",
                 "GuideBooks:Index[10, {polynomials, septic ~}]"}]}]

We could now, for the mouse-lazy, fast-reading reader, implement a function that scrolls through the displayed notebooks.

scrollThroughCurrentHelpBrowserNotebook[cellScrollTime_, opts___] :=
Module[{hbNb = HelpBrowserNotebook[], visibleNotebookPart, 
        visibleCells, cellNumber},
 (* number of cells in the currently displayed help browser notebook *)
 visibleNotebookPart = NotebookGet[hbNb][[1]];
 visibleCells = flattenCellGroup[visibleNotebookPart];
 cellNumber = Length[visibleCells];
 (* move selection on top of notebook *)
 animationButtonCellNumbers = Flatten @ Position[visibleCells,
     Cell[_?(MemberQ[#, "ShowAnimationButton", Infinity]&), 
          "ProgramLabel", ___]];
showAnimationsQ = showAnimations /. {opts} /. showAnimations -> False;     
 SelectionMove[hbNb, Before, Notebook];
(* scroll through all cells, highlight each cell *) 
 Do[Pause[cellScrollTime/3]; 
     (* move to next cell *)
     SelectionMove[hbNb, Next, Cell];
     (* highlight currently selected cell *)
     SetOptions[NotebookSelection[hbNb], CellFrame -> True,
               Background -> RGBColor[0.04, 0.78, 0.18],
               FontColor -> RGBColor[0.74, 0.074, 0.26]];
     Pause[2 cellScrollTime/3];     
     (* restore default look of the currently selected cell *)
     SetOptions[NotebookSelection[hbNb], Background -> Inherited,
               CellFrame -> Inherited, FontColor -> Inherited];
     Pause[cellScrollTime/3];
     (* show animation *)
     If[showAnimationsQ, 
     If[MemberQ[animationButtonCellNumbers, κ], 
        animationNotebookRelativeFileName = Cases[visibleCells[[κ]],
                                          _FrontEnd`FileName, Infinity];
        hbNbFileName = ("FileName" /. Developer`NotebookInformation[hbNb])[[1]];
        animationNotebookFileName = 
                 {Append[hbNbFileName, animationNotebookRelativeFileName[[1]]],
                  animationNotebookRelativeFileName[[2]]};                               
        anNb = NotebookOpen[animationNotebookFileName];         
        FrontEndTokenExecute[anNb, "SelectAll"];
        FrontEndTokenExecute[anNb, "SelectionAnimate"]; 
        Pause[20 cellScrollTime];
        NotebookClose[anNb]]], 
    {κ, cellNumber}]]

The next input opens the problem number two of the second chapter of the Graphics volume and scrolls programmatically through all cells.

FrontEndExecute[{
     FrontEnd`HelpBrowserLookup["AddOns", "GuideBooks:Problem:8.1" ]}]
scrollThroughCurrentHelpBrowserNotebook[0.5]

We close the help browser.

NotebookClose @ HelpBrowserNotebook[]

We end here the process of incorporating the GuideBooks into the help browser. From a notebook-interface point of view, all necessary changes have been made. We leave it to the reader to make minor adjustment to some of the actual inputs to accommodate for the new home of the GuideBooks in the help browser. (For instance, inputs that read in the current evaluation notebook should use different relative file names.)

B.8.3 Evaluating a Complete GuideBooks Chapter Programmatically

The evaluatable 14 chapters of the four parts of the GuideBooks (six chapters of the Programming volume, three chapters of the Graphics volume, two chapters of the Numerics volume, and three chapters of the Symbolics volume) are subdivided into sections, subsections, solutions, and subparts of solutions (called evaluation units in the following). It is strongly recommended to evaluate each of these units with a freshly started kernel. This reduces the overall memory usage, avoids potential symbol name collisions, and allows for a more coherent and uniform In/Out- numbering scheme. But evaluating the tens or sometimes even hundreds of independent evaluation units interactively is a bit cumbersome. Because, for instance, with a newly released version of Mathematica, or after making extensive additions to customize the GuideBooks chapters, the reader might want to reevaluate all evaluation units of a chapter, in this subsection, we will develop a few tools that allow for a programmatic evaluation of a whole GuideBook chapter. (Just adding a Quit[] at the end of each unit does not help because the front end removes all cells queued up for evaluation whenever the kernel quits.)

In the following, we will use two kernels: one for evaluating the evaluation units of the chapter notebooks, and one for guiding the process of evaluating one unit after the other. The program developed below copies and evaluates a unit from the original (source) notebook to the evaluation notebook. Then the program quits the evaluation kernel, copies the next evaluation unit, evaluates it, and so on.

We start by creating a new directory for the evaluated notebooks. We assume that the new directory will be at the same level as the one containing this appendix. So, we can programmatically create the new directory with the following code. The name of the new directory will have _FreshlyEvaluated appended to the name of the current directory where this notebook resides.

evaluatedDirectoryNameExtension = "_FreshlyEvaluated";

Here are the hierarchical and the absolute paths to the directory containing this notebook.

(* the directory containing this file *)
thisDirectoryName = DeleteCases[Delete["FileName" /.   
       NotebookInformation[EvaluationNotebook[]], 2],
       _Rule, Infinity]

thisAbsoluteDirectoryName = ToFileName[thisDirectoryName]

Similarly, here are the hierarchical and the absolute paths to the directory that will contain the evaluated chapter.

newDirectoryName = Evaluate //@ 
   MapAt[StringJoin[#, evaluatedDirectoryNameExtension]&, 
         thisDirectoryName, {1, -1}]

newAbsoluteDirectoryName = ToFileName[newDirectoryName];
evaluatedChaptersDirectory = newAbsoluteDirectoryName

If not already existing, we now create the new directory for the to-be-evaluated chapter.

(* create the new directory if it not already exists *)
If[FileNames["*", newAbsoluteDirectoryName, ∞] =!= {},
   Print["Directory: ", newAbsoluteDirectoryName, " exists already."],
   Print["Creating the directory: ", newAbsoluteDirectoryName];
   CreateDirectory[newAbsoluteDirectoryName]];

With the exception of the 14 main chapter files, we copy all other GuideBooks files into the new directory.

filesToBeCopiedRelativeNames = Cases[DeleteCases[
     StringDrop[#, StringLength[thisAbsoluteDirectoryName]]& /@ 
         FileNames["*", thisAbsoluteDirectoryName],
         (* not the 14 Chapter files *)
       _?(StringMatchQ[#, "*1_Programming*"] ||
          StringMatchQ[#, "*2_Graphics*"] ||
          StringMatchQ[#, "*3_Numerics*"] ||
          StringMatchQ[#, "*4_Symbolics*"]&)],
       (* not the animation folder *)
       _?(StringMatchQ[#, "*.nb"] || StringMatchQ[#, "*.m"] ||
          StringMatchQ[#, "*.txt"]&)]

(* copy the files to the new directory; 
   do not touch potentially existing files *)
If[(* check if file exists *)
   FreeQ[FileNames["*", ToFileName[newDirectoryName]],
         ToFileName[Append[newDirectoryName, #]]],
   (* copy file *)
   CopyFile[ToFileName[Append[thisDirectoryName, #]], 
            ToFileName[Append[newDirectoryName, #]]]]& /@ 
                                 filesToBeCopiedRelativeNames;

Here is a list of the files that are currently in the newly created directory.

StringDrop[#, StringLength[evaluatedChaptersDirectory]]& /@   
                 FileNames["*", evaluatedChaptersDirectory]

To carry out the automatic evaluation of a whole chapter with kernel restarts, we need two kernels. Manually, one can create two kernels under the `Kernel'⟶`Kernel configuration options' menu. The guiding kernel must be local on the computer having access to the two directories (the one containing this notebook and the one containing the to-be-evaluated chapter). We assume that the current evaluator will be the guiding kernel. This is the name of the currently used kernel (in the following, we will refer to this kernel through the variable guidingKernel).

guidingKernel = Evaluator /. Options[EvaluationNotebook[]]

The second kernel will be the evaluation kernel for the evaluation units. We assume it is called "GuideBooksEvaluationKernel". and has the same configuration settings as the current kernel of this notebook. In the following, we will refer to this kernel through the variable evaluationKernel.

evaluationKernel = "GuideBooksEvaluationKernel";

The following input sets up this evaluation kernel on the same computer as guidingKernel. (The evaluation kernel could run on a remote computer; in this case, it must be set up in a customized manner by the reader.)

Module[{ofe = Options[$FrontEnd, EvaluatorNames], pos},
 If[FreeQ[ofe, evaluationKernel],
    pos = Position[ofe, Options[EvaluationNotebook[], 
                                Evaluator][[1, 2]]]; 
    (* add a new kernel *)
    SetOptions[$FrontEnd, Sequence @@
    MapAt[Append[#, If[pos =!= {}, ofe[[1, 2, pos[[1, 3]]]]] /. 
                       guidingKernel -> evaluationKernel]&, ofe, {1, 2}]],
    Print["Kernel " <> evaluationKernel <> " exists already."]]];

Here are the currently defined kernels.

First /@ (EvaluatorNames /. Options[$FrontEnd, EvaluatorNames])

And these are the nondefault suboptions specified for the evaluation kernel.

evaluationKernel /. 
          (EvaluatorNames /. Options[$FrontEnd, EvaluatorNames])

Before evaluating the notebooks, we have to change some of the default options at the global level in the front end. This allows the initialization cells to-be-evaluated for each kernel start without interactively answering the brought up dialog if they should be evaluated.

SetOptions[$FrontEnd, 
           InitializationCellEvaluation -> Automatic, 
           InitializationCellWarning -> False,
           PrivateNotebookOptions -> 
                 {"BackgroundPostScriptRendering" -> True, 
                  "ShortBoxForm" -> False}]

For concreteness, we will evaluate the second chapter of the Programming volume. It is a chapter that needs quite a few kernel restarts, but the whole evaluation time of the chapter is below an hour. This makes it a good demonstration chapter.

nbName = "1_Programming_2";
(* takes much longer:
   nbName = "4_Symbolics_1";
*)

We skip the evaluation of the Make Input units (there are the Program cells that in most cases generate spin-off notebooks or animations).

evaluateMakeInputCellsQ = False;
evaluateMakeInputCellAnimationsQ = False;

As mentioned, we assume that the notebook to be evaluated is present in the same directory as this Appendix B. We construct the new filename of the to be evaluated notebook.

originalNotebookFileName = thisAbsoluteDirectoryName <> nbName <> ".nb"

newNotebookFileName = newAbsoluteDirectoryName <> nbName <> ".nb"

If such a file already exists, we stop here and ask for interactive confirmation to avoid overwriting an existing (potentially already evaluated) notebook.

If[MemberQ[FileNames["*", evaluatedChaptersDirectory], newNotebook],
   shouldEvaluationContinue = 
   InputString["The file already exists.
                Continuing will delete the existing file. 
                \n Continue? 'Yes' or 'No'"];
   If[StringMatchQ[shouldEvaluationContinue, "*No*"], 
      (* emergency stop *) Quit[]]];

Next, we have to prepare the notebook to be evaluated. Because we are interested in all outputs, prints, and graphics newly evaluated, we remove all currently existing cells of this type. In this step, we also set the Evaluator option in this notebook. For some of the larger notebooks, before carrying out this step, which needs the whole notebook at once present as an expression in the Mathematica kernel, the reader might want to remove some of the graphics or convert them to bitmaps (as we have done in the last subsection).

originalNb = Get[originalNotebookFileName];
ByteCount[originalNb]/10.^6 "MB"

removeOldOutputs[nba_]:=
DeleteCases[nba, (* remove existing output cells, 
                    generated graphics, and prints *)
                  (c:Cell[___,"Output", ___] /; 
                         FreeQ[c,  Deletable -> False, ∞]) | 
                   Cell[___, "Message" | "Print" | 
                             "PrintText" | "SaveAndQuit", ___] |
                   Cell[___, "Graphics", sss___] /; 
                        (* keep explanatory graphics *)
                        FreeQ[{sss}, Deletable -> False], ∞] /.
                   (* remove old In[…] cell labels *)   
                   Cell[in_, "Input", a___, CellLabel -> _, b___] -> 
                                        Cell[in, "Input", a, b] /.
                   (* potentially inactivate long running inputs: 
                      Cell[in_, "Input", a__] -> 
                       Cell[in, "Input", a, Evaluatable -> False] /; 
                            MemberQ[{a}, "LongRunningInput", {-1}] /.  *)                     
                   (* close all cell groups *)              
                   CellGroupData[data_, _] -> CellGroupData[data, Closed] /.
(* set the options of the to-be-evaluated notebook *)
Notebook[c_, l___] :> 
   Notebook[c, (* most important! *) Evaluator -> evaluationKernel,
        (* some more option settings *)
        InitializationCellEvaluation -> True, AutoGeneratedPackage -> None,
        ScreenStyleEnvironment -> "AllBrackets", WindowSize -> {800, 640},
        ShowCellTags -> False, ShowCellLabel -> True,
        WindowMargins -> {{1., Automatic}, {Automatic, 3.}},
        PrivateNotebookOptions -> {"ShortBoxForm" -> False},
        ScreenStyleEnvironment -> "Working" (* or Reading *)
        (*, Magnification -> 1.25, WindowTitle -> "FullFileName" *)]

preparedNb = removeOldOutputs[originalNb];

Potentially we could make the Program cells and animations evaluatable too.

preparedNb = 
If[evaluateMakeInputCellsQ, preparedNb //. 
   {a___, c1:Cell[pl_?(FreeQ[#, "Animation", {-1}]&), 
                  "ProgramLabel", ___], 
    cp:Cell[p_, "Program", ___],
    c2:Cell[_, "ProgramBottom"], b___} :> 
                  {a, c1, cp /. "Program" -> "Input", c2, b}, 
    preparedNb];

preparedNb = 
If[evaluateMakeInputCellAnimationsQ, preparedNb //. 
   {a___, c1:Cell[pl_?(MemberQ[#, "Animation", {-1}]&), 
             "ProgramLabel", ___], cp:Cell[p_, "Program", ___],
    c2:Cell[_, "ProgramBottom"], b___} :> 
                  {a, c1, cp /. "Program" -> "Input", c2, b}, 
    preparedNb];

These are the individual units of the uppermost level that have to be processed. (The units are three chapter title related cells, four sections, one overview section, one exercise section, one solutions section, and one reference section.)

nbUnits = preparedNb[[1]] /.
 Cell[contents_, style__, opts___?OptionQ] -> 
        Cell[contents, style, GeneratedCell -> False, opts];

nbUnits // Length

Now, we create an empty notebook in the evaluation directory, and attach the stylesheet.

newNb = NotebookCreate @@ 
        Join[(* just use the options *) Rest[preparedNb],
             Notebook[AutoGeneratedPackage -> None,
                      InitializationCellEvaluation -> Automatic,
                      InitializationCellWarning -> False]];
                      
NotebookSave[newNb, newNotebookFileName]

SetOptions[newNb, StyleDefinitions -> "GuideBooksStylesheet.nb"];
NotebookSave[newNb]

In the just-created notebook, we will now, unit-by-unit copy an individual evaluation unit and evaluate it. To do so, we must recursively define what kind of actions should be performed with the various parts of a notebook. The five evaluatable types of units (sections without subsections, subsections, solutions without subsolutions, and subsolutions, and the overview section) should be evaluated individually. Sections with subsections and solutions with subsolutions must be split and evaluated. All other cells, like the exercises, do not need evaluation (they could be evaluated, but avoiding to prepare, start, and quit the kernel saves some time). The function evaluationAction implements the just-described operations. From a notebook-structural point of view, we should in addition take the inner CellGroupData structures into account. The following definitions assume that the reader has only added cells of type ReaderInput and ReaderAnnotation within evaluation units, but has not created now types of grouping cells.

(* various header\[Hyphen]type cells that need no evaluation *)
evaluationAction[c:Cell[_, "VolumeLabel", ___]] := 
          NotebookWrite[newNb, c, GeneratedCell -> False]
          
evaluationAction[c:Cell[_, "ChapterNumber", ___]] := 
          NotebookWrite[newNb, c, GeneratedCell -> False]
          
evaluationAction[c:Cell[_, "Chapter", ___]] := 
          NotebookWrite[newNb, c, GeneratedCell -> False]
          
evaluationAction[c:Cell[_, "Section", ___]] := 
           NotebookWrite[newNb, c, GeneratedCell -> False]
           
evaluationAction[c:Cell[_, "ExerciseHeading", ___]] := 
          NotebookWrite[newNb, c, GeneratedCell -> False]
          
evaluationAction[c:Cell[_, "SolutionsSection", ___]] := 
           NotebookWrite[newNb, c, GeneratedCell -> False]

evaluationAction[c:Cell[_, "SolutionHeading", ___]] := 
           NotebookWrite[newNb, c, GeneratedCell -> False]

(* sections without subsection can be evaluated *)
evaluationAction[cg:Cell[CellGroupData[{
          Cell[_, "Section", ___], ___}, ___], ___]] := 
          If[FreeQ[cg, Cell[_, "Subsection", ___], Infinity],
            evaluateUnit[cg], evaluationAction /@ cg[[1, 1]]]

(* evaluate the overview section *)
evaluationAction[cg:Cell[CellGroupData[{
         Cell[_, "OverviewSection", ___],  ___}, ___], ___]] := 
          evaluateUnit[cg]

(* exercises need no evaluation *)
evaluationAction[cg:Cell[CellGroupData[{Cell[_,  
                 "ExercisesSection", ___],  ___}, ___], ___]] := 
           NotebookWrite[newNb, cg, GeneratedCell -> False]

(* references need no evaluation *)
evaluationAction[cg:Cell[CellGroupData[{Cell[_,  
                 "ReferencesSection", ___], ___}, ___], ___]] := 
           NotebookWrite[newNb, cg, GeneratedCell -> False]

(* evaluate the individual solutions *)
evaluationAction[cg:Cell[CellGroupData[{Cell[_, "SolutionsSection", ___],
                     ___}, ___], ___]] := 
           evaluationAction /@ cg[[1, 1]]

(* evaluate the subsolutions *)
evaluationAction[cg:Cell[CellGroupData[{Cell[_, "SolutionHeading", ___],
                     ___}, ___], ___]] := 
              If[FreeQ[cg, Cell[_, "SolutionSubgroup", ___], Infinity],
                 evaluateUnit[cg], evaluationAction /@ cg[[1, 1]]]

(*
(* for the WebsiteAdditions.nb notebook *)

evaluationAction[cg:Cell[CellGroupData[{
          Cell[_, "Section", ___], ___}, ___], ___]] := 
          evaluationAction /@ cg[[1, 1]]
          
evaluationAction[cg:Cell[CellGroupData[{
          Cell[_, "ExerciseHeading", ___], ___}, ___], ___]] := 
          evaluationAction[cg]
*)

(* evaluate anything else (it will not hurt in any case) *)
evaluationAction[cg_] := evaluateUnit[cg]

The function evaluateUnit used on the right-hand side of some of the evaluationAction definitions, has to copy an evaluation unit into the evaluation notebook and evaluate the unit. After the evaluation has finished, the guiding kernel must be informed that the unit is completely evaluated and that the guiding kernel can proceed with the evaluation action appropriate for the next unit. Here are the steps carried out by the function evaluationAction: First, the cells to be evaluated are written into the evaluation notebook (using NotebookWrite). In addition, one special input cell is written at the end of the notebook. This special input cell does three operations: 1) it saves the evaluation notebook after the unit is evaluated, 2) it informes the guiding kernel that the evaluation is finished (by using the MathLink function Forward to forward a list of three elements (the string "evaluationCompleted", the number of Mathematica inputs evaluated, and the CPU time use) to the guiding kernel) and 3) shuts down the evaluation kernel. After the guiding kernel has been informed that the evaluation of a unit is finished, the special cell is deleted and the group of evaluated cells will be closed. To avoid any possible desynchronization between the two kernels and the front end, we use some Pause commands. The function evaluationAction in addition counts the number of kernel restarts performed and monitors the CPU-time used for the evaluations.

(* send back to guiding kernel information about the session *)
completionAction = "{\"evaluationCompleted\", $Line, TimeUsed[]}";

evaluateUnit[cg_] := 
Module[{doneQ},
 Pause[1];
 (* write cells to notebook *)
 NotebookWrite[newNb, 
    {cg, (* cell that saves the evaluation notebook,
            send back a list to the guiding kernel,  
            and quits the evaluation kernel *)
        Cell[TextData[StyleBox[#, FontColor -> RGBColor[0, 0, 1]]& @ 
        StringJoin[
        "(NotebookSave[EvaluationNotebook[]]; \n",
        " LinkWrite[$ParentLink, Forward[", 
         "\"" <> guidingKernel <> "\"", ", ", 
        completionAction, "]]; \n Pause[3]; \n Quit[])"]],
        "Input", GeneratedCell -> False,
                 CellLabel -> "This cell is only temporary!",
                 (* a tag to locate the additional cell *)
                 CellTags -> "DELETE ME!"]}, All]; 
 Pause[3];        
 (* evaluate notebook *)
 FrontEndExecute[FrontEndToken[newNb, "EvaluateCells"]];
 (* wait until evaluation is finished; 
    check permanently if it is already finished *)
 doneQ = False;
 While[Not[doneQ],
       While[Pause[1]; Not[LinkReadyQ[$ParentLink]]];
             readLinkData = LinkRead[$ParentLink];
             If[Length[readLinkData] === 3 &&
                readLinkData[[1]] === "evaluationCompleted", 
             doneQ = True]]; 
 (* save number of inputs and CPU time used *)    
 If[(* were cells other than initialization cells and
       last cell evaluated? *) readLinkData[[2]] > 3, 
    AppendTo[dataList, Take[readLinkData, -2] - {3, 0}]];        
 (* time to quite the kernel *)    
 Pause[6];
 (* delete the last cell with the additional action *)
 NotebookLocate[{newNb, "DELETE ME!"}];
 NotebookDelete[newNb]; 
 (* time to let the front end delete the cells *)    
 Pause[1];
 (* clean up and close all cells *)
 FrontEndTokenExecute[newNb, "SelectAll"];
 FrontEndTokenExecute[newNb,"SelectionCloseAllGroups"];
 SelectionMove[newNb, After, Notebook];
 kernelRestartCounter]

We have all functions together to evaluate the Structure of Mathematica Expressions chapter at once. Because we copy the reference section as the last section into the new notebook, in intermediate stages, the references hyperlinks will not work properly until the evaluation of the whole notebook is completed.

dataList = {};
Do[evaluationAction[nbUnits[[k]]], {k, 1, Length[nbUnits]}]; // 
                                                         Timing

We save the fully evaluated notebook.

SetOptions[newNb, InitializationCellEvaluation -> Automatic];
NotebookSave[newNb]

We enjoy a view on the freshly evaluated notebook for a few seconds and close it.

Pause[10]
NotebookClose[newNb]

Evaluating this chapter took about less than a second CPU time of the guiding kernel. The evaluation kernel had to work harder: The list dataList contains a list of pairs with the number of inputs evaluated and the CPU time used for each individual kernel session. Here is the number of kernel restarts, the total number of inputs evaluated (not counting the inputs arising form the additional cell), and the total CPU time used to evaluate this chapter.

analyseDataList[dataList_] :=
 Print[Length[dataList] " independent units evaluated.\n",
       Plus @@ (First /@ dataList) " inputs evaluated.\n",
       Round[Plus @@ (Last /@ dataList)] " seconds CPU time total."]

analyseDataList[dataList]

The above-implemented procedure for evaluating a whole GuideBooks chapter (or parts of it) at once will work as long as the front end has enough memory available to store the resulting notebook (the intermediate explicit calls to NotebookSave[] do not remove the corresponding information from the front end, so the memory consumption of the front end is strictly increasing. If not manually or programmatically interrupted and/or brought out of sync (for instance by tinkering about with message passing, or evaluating functions that quit the evaluation kernel), and with enough memory available, the automatic chapter evaluation should evaluate each of the 14 GuideBooks notebooks as a whole.