Capture still uses Remember Job Progress. When loading a sequence, ignoreJobProgress = !Options::rememberJobProgress()

Check capture.cpp starting from line 2427

Remember Job Progress is required in capture module as it does not require the scheduler in order to run when loading the sequence, it should be able to resume from where it left.

Sorry, Eric, but arc patch complains about missing commit a651bd7ee7ba7ebba7e40f5b263021d534db14b6. It is neither in your repository nor in the main one (at least I cannot find it).

Ok currently there is no way in the capture module to "Forget job progress" hence it is taken from options. So there should be a way for the user to know "Ok, I don't want to resume, I want to start from scratch now". Maybe there are other use cases where the user _always_ want to start to scratch so this has to be taken into consideration.

I agree with @TallFurryMan, Capture and Scheduler should not overlap in terms of control logics. We should keep Capture as simple as possible and leave all history stuff to Scheduler.

On the other hand, Capture should keep its state as good as possible even when it fails. Currently, when the Scheduler restarts Capture, it starts always with the first SequenceJob entry and does not continue where it failed. But this is not issue of this differential, it should be addressed later.

In D16058#339684, @mutlaqja wrote:

Ok currently there is no way in the capture module to "Forget job progress" hence it is taken from options. So there should be a way for the user to know "Ok, I don't want to resume, I want to start from scratch now". Maybe there are other use cases where the user _always_ want to start to scratch so this has to be taken into consideration.

The end-user can click the reset button to forget all progresses. This serves the use case "start from scratch now". We don't have the use case "start this particular sequence job from the beginning", and we fixed the guiding deviation resetting the count in agreement.
There was a feature to reset job progress in the Scheduler too by double-clicking a single job, but it was weird in terms of UI and I removed it when all jobs were rescheduling to attempt again processing.

In D16058#339705, @wreissenberger wrote:

On the other hand, Capture should keep its state as good as possible even when it fails. Currently, when the Scheduler restarts Capture, it starts always with the first SequenceJob entry and does not continue where it failed. But this is not issue of this differential, it should be addressed later.

Yes, Scheduler is processing the sequence bluntly from scratch whenever it starts, because it starts by loading a sequence. We could imagine looking at the sequence that is requested, and if it's the same, not reload and keep the current settings. That would bring back the need for the d-bus entry point "clearSequence" though to really clear the sequence when needed, such as when repeating a job.
I should comment on exactly what happens when "Remember Job Progress" is enabled: Scheduler orders the capture frames it sees stored with no regard for what sequence job captured those, and determines the state (completed or not) of each sequence job before execution. But let's not get carried away yet :)

Sorry for the distraction :-)
I meanwhile use the scheduler excessively and I have a lot of features in mind... But let's concentrate on making it stable as fast as possible.

I was about to push the missing history, but I'm delaying as I found an issue with the way the completed frame count is set when ignoreJobProgress is not set.
The count is taken to be sequenceID minus one, but that can't be correct if there are missing files in the series on disk.
I'll disconnect this relation, Capture should do what it is told to by the configured sequence, or rely on what required count the Scheduler is providing.

A few notes for clarity, and for you to shout if you disagree in advance:

The full history to come for this differential updates the UI to show the completed/total count of frames in the job table.
The use case "I Want To Restart Captures From Scratch" will be obtained with the button "Reset Jobs", which will reset both state and capture count.

Following the comments, I will change the resetJobs function to enable the user to reset the state and count of a single job, when that job is being edited (could be "selected" only if it's clearer for the end-user).
This will serve the use case "I Want To Restart This Particular Capture From Scratch".

And after thinking over your comments a bit more, I will rework this differential to introduce a new option, "Always Reset Sequence When Starting", and connect this to ignoreJobProgress, defaulting to No.
This will have the same (with meaning inverted) effect as "Remember Job Progress" had, except "Remember Job Progress" is now for the Scheduler module, and "Always Reset Sequence When Starting" is for the Capture module.
This will serve the use case "Anytime I Press Start, I Want All Jobs Processed".
"Remember Job Progress" will override "Always Reset Sequence When Starting" through the used of capturedFrameMap.

From a Capture perspective this sounds reasonable and very comfortable.

I am slightly skeptical about the new flag Always Reset Sequence When Starting. What does this mean from the Scheduler perspective? Is it ignored or does it implicitly mean Remember Job Progress = NO?

From I simplicity point of view, I would recommend not introducing this flag, as long as we have the reset buttons. It's meaning might be misinterpreted.

... and it's behavior is different depending on whether the sequence has been freshly loaded or not.

Thanks for these comments!

In D16058#340619, @wreissenberger wrote:

I am slightly skeptical about the new flag Always Reset Sequence When Starting. What does this mean from the Scheduler perspective? Is it ignored or does it implicitly mean Remember Job Progress = NO?

From Scheduler perspective, the flag has to be ignored by Capture because Scheduler provides a captured frame map. Scheduler overrides Capture, and not the opposite.
If Remember Job Progress is not set, then Capture honors Always Reset Sequence When Starting. But it doesn't make any difference as a new sequence list is then started.
In any case, I still need to remove the block of code initializing the count of captures from sequenceID inserted by Jasem on Sep 24, because I believe a coherent Capture should not consider storage, only what it already executed.
If Capture finds a signature matching the sequence it is processing in the frame map, it will initialise the completed count of the sequence job with the value in the map.

However I already see a slight problem in terms of UI: currently the count is initialized when the job is prepared, not when the frame map is received.
This means the X/N count will be updated from the frame map only when a sequence job starts. And it can happen that this count is lost if the end-user stops Capture.
That's probably fixable by moving the count initialization to the function receiving the map item.

In D16058#340625, @wreissenberger wrote:

From I simplicity point of view, I would recommend not introducing this flag, as long as we have the reset buttons. It's meaning might be misinterpreted.
... and it's behavior is different depending on whether the sequence has been freshly loaded or not.

I would agree, but as stated by Jasem, we would lose the use case "Anytime I Press Start, I Want All Jobs Processed" if we don't keep some sort of Remember Job Progress for the Capture module.
And we do see that the option is now different from the one used in the Scheduler.

From I simplicity point of view, I would recommend not introducing this flag, as long as we have the reset buttons. It's meaning might be misinterpreted.
... and it's behavior is different depending on whether the sequence has been freshly loaded or not.

I would agree, but as stated by Jasem, we would lose the use case "Anytime I Press Start, I Want All Jobs Processed" if we don't keep some sort of Remember Job Progress for the Capture module.
And we do see that the option is now different from the one used in the Scheduler.

OK, that's right. But in this case I would make this option directly accessible (and visible) in the Capture tab. All options in the Options tab bear the risk of being forgotten.

I tested a few use cases, and I couldn't find a situation where I myself would benefit from that option. So I added a warning in the bottom console when such reset happens. Will push soon.

There are a few options on the bottom-left too, that are taken from the sequence file. I could add the option there also. However, from the end-user perspective, I believe these fields make a mess as they overwrite the general options without warning.

Additionally, I think we should intercept all logs starting with "Warning" and duplicate them to the Ekos tab for clarity.

Jasem I also need to remove your (early) optimization which is not re-loading the sequence when looping a schedule job, as counts have to be reset by the Scheduler if the end-user manipulated the storage. Also if for some reason a repeating job was interrupted and is restarting anew, we need to send again the sequence.

Ok I would need to test these changes to see if they would make sense from the end user's perspective. A few users are already used to the current workflow. We have to cover all scenarios which needs to be documented now to reduce chances of regressions.

For sure. Right now both Scheduler and Capture source codes are very hard to read, and it's possible that many contributions aren't concluding into pull requests because of that. Any attempt at cleanup should base on such a list of use cases. How could we involve the community on this?

Having a visible option is not that good for user experience, I'll settle for the warning only.