Invocation should be based on the State Monad

Summary: This is to fix the "elem" problem -- the fact that you have to manually remember, after calling something like contextBundlesAndElements, to use the elem later in the for comprehension. If you goof, you get multiplicative horror. We need this to be automatic.

This needs sanity-checking, and more reading into the State Monad, but I think it makes sense conceptually. The problem is currently because Invocation contains a Context. Instead, we should be passing a State[Context, InvocationValue] down the for comprehension, and all of Invocation's functions should be wrapped in State. Initially, the State gets set to the State from the Invocation itself. Each function potentially uses or modifies the State.

This raises the question: can we continue the horror, and wrap State into InvocationValue? After all, nearly all of the functions are already returning InvocationValue, and Context is already its usual "output". Could / should we rewrite InvocationValue to be a variation of State? It arguably already is a variation of State, really, with a Future[IVData[T]] as its specialized S type. So the question is, could we add the current Context to the IVData, and rewrite all of the functions on Invocation to be functions that take an IVData as a second, curried parameter, and produce a new IVData? Or something like that?

The potential for improvement here is considerable, but man -- it's going to break the whole world if we do this. This is at least a week's work, starting with a one-day research spike to see if it's even plausible.

Although, on second thought, it’s just barely possible that we might be able to make that change without actually changing any of the calls to Invocation, if we’re sufficiently clever. Hmm.

The theory is that IVData would contain an Option[QLContext], and that would be passed as the second, curried, parameter list to each Invocation function. It would get passed into the functions inside some sort of wrapper (let's call it InvState) that encapsulates both that and the Invocation itself, and which uses the appropriate QLContext as necessary. The functions would never be allowed to call their own Invocation's context; instead, they should always call that. Indeed, the exposed Invocation.context member would want to be replaced by a call to that, and the actual context value inside Invocation changed to something private that is almost never referenced.

So the signature of all functions in Invocation would stop returning InvocationValue[T], and would instead return InvFunc[T], which itself is a function InvState => (InvState, InvocationValue[T]). Actually, that would probably be the signature of InvFuncImpl[T] -- InvFunc itself would simply be => InvocationValue[T], and the fact that that is always produced via currying is completely under the hood.

A typical function becomes something like this:

  def contextTypeAs[T : ClassTag]:InvocationValue[T] = InvFuncImpl { state =>
    val clazz = implicitly[ClassTag[T]].runtimeClass
    if (clazz.isInstance(state.context.value.pType))
      (state, InvocationValueImpl(Some(state.context.value.pType.asInstanceOf[T])))
    else
      error("Func.wrongType", displayName, state.context.value.pType.displayName)
  }

InvFuncImpl is a case class with one parameter, the implementing function. The need to pass state around explicitly like that is annoying, but probably can't be avoided -- we need to be able to access and set the new state. (Although "state" might just be "context".) The really annoying thing is that this would actually be much easier to implement in QL than in Scala!

The implication is that the entire for comprehension would wind up being of type InvFunc. So we probably couldn't quite get away with no changes -- the whole thing would have to be wrapped in a call that takes the Invocation, creates the initial InvState from it, calls the comprehension, and then extracts the final QFut from the final InvocationValue. I don't see an obvious way to do that without tweaks, unfortunately, but I do find myself wondering if we could simplify / standardize the qlApply() signature boilerplate along the way.

Yeah, actually -- that's probably the best way to deal with this. If we have to touch the entire world anyway, we could do it by having InternalMethod define an abstract process() method. This wouldn't even be a function per se, but a val of type InvFunc(). The standard qlApply() would do both the initialization and the final conversion from InvocationValue to QFut. In the rare cases where we really do want to override qlApply, we would have to stub that, but it's a short stub.

So basically, the usual for comprehensions wouldn't change at all -- the function that they are defined in would. But we would wind up with less boilerplate overall.

This should allow us to then get rid of the "elem" parameter in functions like processAs() -- this handling becomes automatic. Indeed, the interesting question then becomes, are there any functions that depend on not working this way?

Also, this probably means that, eg, contextBundlesAndContexts simply goes away, because contextAllBundles becomes effectively a synonym for it.

Of course, if we're going to touch the world anyway, we could get even more radical. One of the great annoyances of the current system is the need to keep saying "inv." at the start of every function call. That's because the functions are defined on the Invocation object. But maybe that's completely the wrong way to do it.

Consider: we could instead turn Invocation into a simple data object. In this new design, it's being passed everywhere in the InvState anyway. Since all the functions take InvState as a curried parameter, we could rip all of the functions out of Invocation itself, into a separate trait, and then mix that trait into MethodDefs. At that point, all of those "inv."s go away, the need for inv as an explicit parameter goes away, and the code gets simpler.

This is probably worth implementing, possibly very soon. But I'm going to have to seek ways to break this down into manageable chunks. Moving the Invocation functions almost has to happen in a second pass, because otherwise we're going to break the world to the point where the misery is almost impossible to recover from.

How can we add in InvState best? That's going to be the really nasty step: I don't see a good way to add InvState and InvFunc piecemeal. So we're looking at three broad steps, I think: