Rebuild Search along more-modern lines
Summary: While v2 of the search engine is a little better than the first version, it bears no resemblance to a real search engine. We can do better.
What we're doing now is a full text search of the entire Space each time. That's good from a memory POV, but otherwise idiotic.
A better approach -- certainly a more orthodox one -- would be to pre-index the Space, probably doing a Huffman coding of all of the words in the Space, where each leaf points to a Thing/Property pair. When we are searching, we follow that through for each search term, to gather up all of the candidates quickly, and then just evaluate those -- that would likely be orders of magnitude faster, after the initial indexing.
Ideally, we would be smart about this:
- For the initial indexing, we use a mutable version of the Huffman tree, to build up the full index (possibly single-threaded), and then convert that to immutable.
- We don't store this on the Space itself, but in a sub-Actor of the troupe under the main SpaceRouter. This only initializes itself when we receive a Search request, at which point it builds the Huffman tree and responds.
- This registers itself with the
SpaceChangeManager.thingChanges() mechanism, in order to listen to all changes happening in the Space. See the DeriveNameModule for an example of this mechanism. (Or maybe using updateStateCache, although it looks like that is currently somewhat broken.)
- After it is initialized, it begins to keep the Huffman tree up to date based on that. (This likely happens as messages routed to the Search Actor, so that the process doesn't hold up the main processing of changes.)
This is a bit elaborate, but if we can make it work, it should be adequately space-efficient (since Search will only be initialized once requested), and lightning-fast (since searches are pretty well optimized and we are keeping the search tree up to date). That said, it will definitely require significant memory overhead in the Troupe when it is active, so it is possible that we want to time out the Space sub-Actor.
