KAsync overhaul
Closed, ResolvedPublic

Description

This is a design draft for a first overhaul of KAsync based on the things we learned.

This is intends to address the following issues:

  • We currently have poor support for handling errors (only a synchronous function)
    • It's not possible to reconcile jobs after errors
    • It's not easily possible to do cleanup work in error handlers
  • Errorhandling breaks composability (because we have the error handler in the continuation)
  • Job types break composability
  • The current design is currently in parts overly flexible/complex where it shouldn't (It's easy to provide solutions for that externally)
    • Multiple input values
    • KJob support
    • Member function support

Overview

Job<O, I> represents a potentially asynchronous operation with input I and output O.

The following basic operations exist:

  • start<O, I>(Worker, ErrorHandler) creates a Job<O, I> with a worker that will be executed, and an error handler that is executed if the Worker reports an error.
  • null() creates a Job<void, void> that does nothing.
  • value<O>(v) creates a Job<O, void> that returns the passed in value v of type O.
  • error(e) creates a Job<void, void> that triggers the passed in error e.

Worker

The worker is a function with one of the forms:

  • void(): Same as a function that returns null()
  • Job<O,I>(I'): A function that has an optional input from the previous job, and returns a job itself
  • void(I', Future): Handle based approach

A worker can return a job, which means the current job will only complete once the returned job completes.
This allows for nested jobs.

ErrorHandler

The error-handler is a function with one of the forms:

  • void(Error): Same as a function that returns error()
  • Job<O,I>(Error): A function that receives the error and returns a job.

Returning jobs from error handlers has the following meaning:

  • null/value: Ignore the error and thus reconcile. Folloing jobs will behave as if no error occurred.
  • error: Rethrow the error to the next handler.

This design allows the error handler to execute further work in order to handle the error.

Compositions

  • .then(Job): A continuation. Establishes a dependency of the job inside the then clause on the outer job.
  • .serialMap(Job): A serial map function that can be applied to containers. Subjobs spawned will execute serially. One job failing doesn't fail the others, but results in an error at the end.
  • .parallelMap(Job): A parallel map function that can be applied to containers. Subjobs spawned will execute simultaneously. One job failing doesn't fail the others, but results in an error at the end.

Shorthands

  • .then(Worker, ErrorHandler): Same as .then(start(Worker, ErrorHandler))
  • .error(ErrorHandler): An error handler. Does nothing if no error occurs. Same as .then(null(), ErrorHandler).
  • .final(Worker<Error>): A continuation that is executed in any case (and thus receives the optional error). Same as .then with the same code in the worker and the error handler.

Type of compositions

A composition always takes on the type of the initial input value with the last output value, so it become irrelevant what intermediate steps there are in between.

start<O, I>(...)   //Job<O,I>
.then<O', O>(...)    //-> Job<O', I>
.then<O'', O'>(...)  //-> Job<O'', I>

At all times it is possible to convert to a void return value to ignore the value:

Job<O, I> -> Job<void, I>

Design considerations

  • The worker and the error handler are kept together to not break composition. We may want to catch an error without handling the continuation (which is left to the caller).
  • Returning a Job from a worker is a much more powerful (and less error prone) approach than the handle based one, and allows the compiler to assist the programmer. We still need the handle based approach for integration into other systems (such as KJob) though.
  • A function must not accept more than one input value, otherwise composability is lost. Of course you can use any sort of tuple as input value to group values.
  • KJob support is as simple as this, which makes me doubt if we should have it directly in the Job API.
template <typename T>
static KAsync::Job<T> runJob(KJob *job, const std::function<T(KJob*)> &f)
{
    return KAsync::start<T>([job, f](KAsync::Future<T> &future) {
        QObject::connect(job, &KJob::result, [&future, f](KJob *job) {
            if (job->error()) {
                future.setError(job->error(), job->errorString());
            } else {
                future.setValue(f(job));
                future.setFinished();
            }
        });
        job->start();
    });
}
cmollekopf updated the task description. (Show Details)

This may be fuzzier than I hoped for, but I hope you can understand the general gist of where I'd like to go with this.

What I forgot is the context:
Job should have a context that is either a QSet<QVariant> or a QMap<QByteArray, QVariant> or both. The primary usecase at this point is to set smart pointers as the context and thus make sure that the external object lives for the duration of the job. This of course means that the context must survive further compositions (you should be able to set the context in a function that returns a job and have the guarantee that the context will be available during execution.

We may also want to make the context accessible to continuations somehow, but I'm not sure that is really necessary. (The usecase would be to pass stuff from one continuation to the other without having to incorporate it into the return value of the previous job(s).)

If the handle would become a smart pointer then we could detect if the implementation looses the handle without calling setFinished or setError, and could thus automatically fail the job. Doesn't guard against all failures (if the smart pointer is captured in a lambda that never get's called), but could help with some.

I have implemented most of this (though it changed a bit) in the dev/kasync2 branch and already ported Sink to it.

It's overall a clear improvement in Sink.

please let me know what you think about the overhaul, and whether you see any problems with the approach. cheers

cmollekopf closed this task as Resolved.Dec 2 2016, 10:20 AM

Merged to master.