LaTeX (pronounced "laytech") is the most widely used system for typesetting mathematical and complex (cross referencing, bibliographical citations and indexes) documents. Like HTML, it is a markup language: you describe what you want in a plain text file, and then use a program to convert this description to PostScript or PDF.
You must edit LaTeX source files using a text editor. A particularly good choice is Emacs, which includes support (AUCTeX) for editing LaTeX files.
To learn how to use LaTeX, begin by consulting LaTeX: from Quick and Dirty to Style and Finesse, by Tony Roberts. More information can be found at LaTeX Project.
The standard way to use LaTeX is via the shell (command interpreter), which is possible on Linux, macOS, and Windows/Cygwin. But it is also possible to use a special-purpose application, such as TeXShop or MiKTeX.
On Linux, macOS or Windows/Cygwin, if LaTeX is not already installed, you can install the TeX Live distribution. On macOS, TeX Live may be installed via MacTeX.
The Perl script Latexmk ("mk" stands for "make") automates the process of running LaTeX enough times to resolve cross references, bibliographic citations and indexing. It can also use supporting programs (see below) to turn LaTeX's output (a DVI file) into PostScript or PDF.
On macOS, I recommend viewing PDF files using the open source application Skim, which is capable redisplaying a PDF document as the file changes.
A moderately involved example of what can be achieved using LaTeX can be found in the file example.tex, which uses the files macros.tex (some general purpose settings and macros that I've found useful) and commands.tex (some macros for typesetting formal language theory).
To turn example.tex into PDF on Linux, macOS or Windows/Cygwin using latex, run the following commands (after copying all three files into a new directory):
latex example.tex latex example.tex dvips -Ppdf example.dvi ps2pdf example.ps
(latex had to be run again, because there were undefined cross-references after the first run.)
To turn example.tex into PDF on Linux, macOS or Windows/Cygwin using pdflatex, run the following commands (after copying all three files into a new directory):
pdflatex example.tex pdflatex example.tex
(pdflatex had to be run again, because there were undefined cross-references after the first run.)
On Linux and macOS, you can use the program smlnjtrans to generate LaTeX transcripts of SML/NJ sessions.