I should have at least rudimentary syntax highlighting when editing QL

(User Story, Investigate , Priority: Critical, Test Status: No automated tests yet , Reported By Justin du Coeur, )
Summary: This story is the beginning of the "QL IDE". The point here isn't to build everything we want, but just to get something better than we have now.
Here's a sketch of the plan:
  • Replace the current textarea with a div that uses contentEditable. This initially shouldn't be in any way different from the current code, but should immediately give us lots more options. Note that this will need to do things like translate line endings between br and newline; the saved code should not include any HTML. Also, any HTML that gets typed in needs to be escaped as it is typed, and de-escaped when it gets stringified.
  • Choose two fonts, one for QL and one for QText. The QL one should be monospace; not sure about the QText one.
  • Figure out whether we can apply classes obscenely, not respecting other HTML structures. That is, start a class in one block, and end it in another. I suspect this will work, but I'm not sure.
  • Write the phase-one parser, which simply takes a block of QText and finds the switch points between QL and QText. It should probably produce a tree data structure, where each node has the start and end. Note that the input and output need to be HTML; we need to think about how to translate between that and the parsing.
  • When the editor is rendered, and after a suitable debounce whenever it is edited, run that parser and apply the correct fonts in the right places.
That's phase one -- at that point, we should have something that is already much better than what we have now. Continuing on (these should all become separate stories, some of them well down the road):
  • Start paying attention to which mode the cursor is currently inside of, and have event handlers for that mode.
  • Highlight the segment that the cursor is current inside of. But if I am in a QL segment, do not highlight QText inside it, and vice-versa. The idea is to provide subtle visual cues (through highlighting, lowlighting, boldface, etc) of which "part" I am currently in.
  • When I type double-open-square-brackets inside QText, insert the close-brackets, and position the cursor in the middle. Similarly, whenever I type double-double-quotes in QL, add the close quotes. Ideally, when I am in a mode and I type the end-delimiter of that mode, position me at the end. (This may require a debounce, and may have some nasty edge cases; we'll see if it works.)
  • Write a simplistic QL parser that deals with errors by breaking things on arrows. Provide syntax highlighting for the QL bits, using colors and maybe fonts. These subsidiary highlights should run asynchronously, in small chunks, to avoid blocking the user experience.
  • Provide match highlighting of bound names -- when the cursor is on an instance of a name, highlight both that and the other usages and definitions.
  • Write a simplistic QText parser, and do GitHub-style syntax highlighting of that. (Eg, boldface bold bits, italicize italic bits, indent bullets, etc.) This doesn't need to be perfect, but even a little will be helpful. This is itself many small stories.
  • Asynchronously, parse out the symbols in QL expressions, and check with the server whether they are defined. If a symbol is unknown, warning-highlight that. (It's not strictly illegal -- it could be a Tag -- but that's unusual in QL.) Cache the results of this, for efficiency; ideally, cache in the client rather than the Page, so we really don't need to redo too often. Ideally, keep track of what state version number we are fetching this info against; we can recheck periodically, and the server can reply with a simple "no change". If the symbol is known, highlight it subtly as such.
  • When doing these checks, fetch any documentation of the symbol in question. When I hover on a symbol, show the documentation basics.
  • When doing these checks, fetch the signatures of internal functions, and check those signatures against the provided parameters, with warning displays if it doesn't match.
  • In the soi-distant future, do live type-checking as I type.