Better support for subjobs
Closed, ResolvedPublic

Description

A pattern that is often used goes something like this:

.each<void, QByteArray>([](const QByteArray &arg, KAsync::Future<void> &future) {
    .....
    if (...) {
        auto subjob = foo()
        subjob.template then<void>([&future](){future.setFinished();}).exec();
    } else if (...) {
        future.setFinished();
    } else {
        future.setError(....);
    }
}).exec();

So we have a bunch of decision logic, that depends on the evaluation of some arguments, that will then execute one or the other subjob, or result in an error etc.
This code ends up being both overly verbose, and especially error prone. We have to guarantee that future.setFinished is always called, always only once, which quickly becomes non trivial if the logic ends up being a bit more complex.

The verbosity of

subjob.template then<void>([&future](){future.setFinished();}).exec();

couly easily be fixed by allowing

subjob.template then<void>(future).exec();

However, the IMO better solution would be to allow:

.each<void, QByteArray>([](const QByteArray &arg) -> Async::Job<void> {
    .....
    if (...) {
        return foo();
    } else if (...) {
        return KAsync::null();
    }
     return KAsync::error(....);
}).exec();

This solution is less error prone, because the compiler can ensure that all codepaths contain a valid return value in form of a job, and thus also always only one return value. It still allows the logic to depend on the result of the previous job (arg), and is considerably less verbose than the first solution.
Also, this way error propagation doesn't have to be done manually and is covered by kasync.

cmollekopf updated the task description. (Show Details)
cmollekopf raised the priority of this task from to Needs Triage.
cmollekopf added a project: KASync.
cmollekopf added a subscriber: cmollekopf.

That looks sensible, yes. I'll take a look at it asap.

I implemented this for start and then in dev/returnjob.

Not sure it actually makes sense for start,,,
but we'll need it for each.

The general approach looks good to me. I tried a rather complex but more generic approach with having an internal conversion from KAsync::Job<T> to T, but failed on implicit conversion and compilers not being smart enough to deduce all the types :)

I first wanted to do custom executors but that seemed to complicated, so I just went with the nestedJobWrapper... I finished the implementation with each now and the branch is ready to merge with your ok.

I ran into a problem with returning Job<void> where start(SyncThenTask.... is found as ambiguous overload. So far I failed to disambiguate them properly.
As a workaround I introduced an additional template parameter on the NestedThenTask variant that works (see branch), but is a bit ugly.

I also tried to disambiguate it like so:

template<typename Out, typename ... In, typename ContOut = Out>
inline typename std::enable_if<!std::is_same<ContOut, Job<Out>>::value, Job<Out, In ...>>::type
start(SyncThenTask<ContOut, In ...> func, ErrorHandler errorFunc = ErrorHandler());

But for some reason the compiler still thinks it's a viable alternative:

/work/source/kasync/autotests/asynctest.cpp:314:16: error: call to 'start' is ambiguous
    auto job = KAsync::start<void>(
               ^~~~~~~~~~~~~~~~~~~
/work/source/kasync/autotests/../src/async.h:314:1: note: candidate function [with Out = void, In = <>, ContOut = void]
start(SyncThenTask<ContOut, In ...> func, ErrorHandler errorFunc = ErrorHandler());
^
/work/source/kasync/autotests/../src/async.h:708:1: note: candidate function [with Out = void, ContOut = KAsync::Job<void>]
start(NestedThenTask<void> func, ErrorHandler error)

The compiler should now that ContOut isn't void (since I'm returning a Job<void>), but I guess it doesn't look at the arguments at this point, and thus gets confused. I haven't found the trick yet how to get it to look at the arguments, so I guess I'm going with the ugly extra template argument for now.

Note that the above problem only applies to returning Job<void>, returning Job<int> works just fine without any additional measures (and I'm not sure why that is either).

The worst thing about my current solution is that if you forget to supply the additional template parameter like so:

.then<void, KAsync::Job<void> >([&innerDone2, &innerDone3]() -> KAsync::Job<void> {
    return KAsync::start<void>([&innerDone2]() {
        innerDone2 = true;
    })
    .then<void>([&innerDone3]() {
        innerDone3 = true;
    });
});

Then the SyncThenTask variant will be silently chosen, meaning the inner job will never execute and you'll have no idea why.
Perhaps we should get rid of SyncThenTask as it is, and always require wrapping Sync return values in something like a Async::Result. At least I would hope that this gives the compiler enough information to distinguish the two variants well enough.

cmollekopf moved this task from Backlog to In Progress on the KASync board.Jan 18 2016, 10:55 AM

I also tried a tag dispatch like so

template<class R, class... Args>
struct function_traits<R(*)(Args...)> : public function_traits<R(Args...)>
{};

template<class R, class... Args>
struct function_traits<R(Args...)>
{
    using return_type = R;

    static constexpr std::size_t arity = sizeof...(Args);

    template <std::size_t N>
    struct argument
    {
        static_assert(N < arity, "error: invalid parameter index.");
        using type = typename std::tuple_element<N,std::tuple<Args...>>::type;
    };
};


template<typename OutOther, typename F, typename ... InOther>
Job<OutOther, InOther ...>
foo(F func, ErrorHandler errorFunc = ErrorHandler())
{
    typedef function_traits<decltype(func)> traits;
    return foo<OutOther, InOther ...>(func, errorFunc, std::is_same<typename F::result_type, Job<OutOther>>{});
}

template<typename OutOther, typename ... InOther>
Job<OutOther, InOther ...>
foo(SyncThenTask<OutOther, InOther ...> func, ErrorHandler errorFunc, std::false_type)
{
    qWarning() << "regular foo";
    return Job<OutOther, InOther ...>(Private::ExecutorBasePtr(
        new Private::SyncThenExecutor<OutOther, InOther ...>(func, errorFunc, mExecutor)));
}

template<typename OutOther, typename ... InOther>
Job<OutOther, InOther ...>
foo(std::function<Job<OutOther>(InOther ...)> func, ErrorHandler errorFunc, std::true_type)
{
    qWarning() << "irregular foo";
    return then<OutOther, InOther ...>(nestedJobWrapper<OutOther, InOther ...>(func), errorFunc);
}

(Note is implemented foo instead of then to avoid conflicts with existing implementations)

The idea is that the main implementation will catch all calls to foo, and then manually dispatch to either implementation depending on the return type of func.

This works nicely when explicitly wrapping the lambda in a std::function when calling foo:

.foo<void>(std::function<KAsync::Job<void>()>([&innerDone2, &innerDone3]() -> KAsync::Job<void> {
    return KAsync::start<void>([&innerDone2]() {
        innerDone2 = true;
    })
    .then<void>([&innerDone3]() {
        innerDone3 = true;
    });
}))

However, this is rather verbose, and without the explicit wrapping, func always ends up being a std::function<void()>. I don't fully understand why, or have found a solution to that...

Yes, the implicit cast to void :) That's where I got stuck too. Basically C++ allows that any function pointer R(*)(T ...) can be cast to void(*)(T ...), which leads to compiler picking either different overload, or throwing ambiguous overload error.

It should be possible to write a custom simplified std::function (or a std::function wrapper) that prevents the cast, but all my attempts (and StackOverflow :)) failed.

I can try to give it another shot, I got pretty depressed the last time after wasting several days on it....

I feel you, I wasted a bunch of time as well already ;-)
Well, at least we learned something. I'll probably give it another try over time, but for the time being I can also live with the extra template argument.
So from my side we can also merge it like this, and try in parallel to improve the solution over time.

Of course, if you solve it, I'll set up a bounty of 5 beers....

Sorry I did not back earlier. The code looks ok, so please merge it as it is, and hopefully we get to solve the cast problem at some point.

Merged. Should we perhaps keep this ticket open until we figure out the cast problem?

cmollekopf assigned this task to ivan.Apr 9 2016, 9:45 PM

Let's see if Ivan has any idea how to solve this.

ivan removed ivan as the assignee of this task.Nov 29 2016, 7:43 PM
ivan added a subscriber: ivan.

I'm guessing this is no longer active

cmollekopf closed this task as Resolved.Jan 12 2017, 1:23 PM
cmollekopf claimed this task.

Finally managed to resolve this issue for good (I hope).

The solution was this:

template<typename OutOther = void, typename ... InOther, typename F>
auto then(F func) const -> typename std::enable_if<std::is_base_of<JobBase, decltype(func(std::declval<Out>()))>::value,
                                                   Job<typename decltype(func(std::declval<Out>()))::OutType, In...>
                                                  >::type
{
    using ResultJob = decltype(func(std::declval<Out>())); //Job<QString, int>
    return thenImpl<typename ResultJob::OutType, Out>({func}, Private::ExecutionFlag::GoodCase);
}

With the new return type notation at the end we can use "decltype(func(std::declval<Out>())" to correctly determine the return type of the lambda, and from there on everything works as it should. The whole construct is still overly long because I failed to reuse "decltype(func(std::declval<Out>())" somehow within the return type (I think that will be easier with c++ 14 and some of it's template alias/variable constructs), but it get's the job done just as it should.

This not only removes the requirement for the ugly syncThen continuation (because the overload works now), it also removes the requirement to specify any template arguments as they will all be properly deduced, making writing continuation much less error prone and a lot more readable.