Logs, and reworking job count update and repeated jobs estimation
ClosedPublic

Authored by TallFurryMan on Apr 21 2018, 12:53 PM.

Details

Summary

Homogeneized scheduler log strings for clarity when reviewing test results. Warning, all these logs are using i18n, so those changes obviously have an impact on translation, if they are indeed translated.

Reworked job estimation to check the number of light frames for each sequence job in the schedule, in order to consolidate the total number of frames effectively captured without dropping a sequence job by mistake.

Reworked job time estimation to properly account for progress when using repeats. Now repeats-required field is expectedly 1 by default.

Reworked capture count algorithm to rely on the state of the job to execute the search, and reuse previous results whenever possible. This reduces the directory parsing to storages of job which actually changed. Also, this does not handle the duplicate scheduler job situation, only the duplicate sequence job. But possibly we don't care about the duplicate scheduler job situation, that's completely acceptable and so much simpler for the end-user to have two duplicated jobs update to an identical status. BUT the case that is not covered is the situation where the end-user would like to capture a target multiple times, at different times. I believe the option "Remember Job Progress" might provide a solution there, but this is to be tested.

There is still work to be done as jobs won't update their capture count in all situations. This should be fixed with better control of the job's state, not in
this function.

I believe this is still quite slow, less than before but still slow. Perhaps remove all logs from getCompletedFiles.

This change also re-evaluates job in different situations, to get a more reactive and always up-to-date interface. The objective is to never use the "evaluate" button except for really resetting jobs.

Also, added mitigation in the case findNextJob can't decide which job to select.

Fixed zero-job evaluation crash, a regression induced by those changes.

This change also works around issue with job definitely abandoned although they could be rescheduled later, eventually on another night. This also mitigates the issue with altitude cutoff, which apparently can't reliably determine when the target is rising instead of falling.

Also, removed scattered return instructions to clarify the algorithm. When a capture says "Complete", changed the state to "Evaluation" so that job properly recounts its captures and properly considers the work left to be done.

Fixed a few uses of QString::arg(). One call to QString::arg will replace all "%X" in the string. Next call to QString::arg will replace all "%X+1", etc.

A few additional fixes for scheduler jobs logs to use job's date format.

FIXME on attempting to request the local server (vs. client) for the number of captures.
FIXME on refactoring the signature path, which is duplicated at several places.
FIXME on fixing an issue with filter name, which doesn't get included in the capture path when there is no filter wheel, but still gets included in the signature (workaround: use a filter wheel sim).
FIXME on finishedFramesCount variable, which is part of an unfinished mechanism and should be removed.
FIXME on files counted in signature path, because all files are counted, not just captures.
FIXME on duplicate scheduler jobs vs. duplicate sequence jobs.
FIXME on altitude cutoff, which might cause jobs to loop over themselves until altitude is properly either allowing startup, or causing effective abort (3 degrees sidereal delay).

Test Plan
  • Logs: most of scheduler logs are now homogeneized to start with "Job 'xx'", for ease of reading. However, while they help on understanding what the scheduler is doing, there are probably too many of them.
  • Counting captures: sequences containing multiple jobs with the same signature should properly count their captures ; e.g. 2xLPR, 3xR, 4xLPR with 3xLPR done should properly count 3xR+1xLPR remaining.
  • Repeated scheduler jobs: capture count should display properly, accounting for the number of repeats, even if the job is set to run to sequence completion only for instance ; also, completion should check repeats too.
  • Capture enumeration: jobs should properly count the number of captures for a complex sequence, BUT ONLY if there is a filter wheel available, else will never find any capture.
  • Duplicate scheduler jobs: while duplicate sequence jobs are now behaving properly, duplicate scheduler jobs are clones and evaluate as if they were one single job ; that's not a change/regression, it's just clearer now.
  • Job evaluation: evaluation will now occur in various situations, adding a job for instance ; evaluation is still not stable (in the sense "same user visible input" => "same output"), but better than before.
  • Aborting jobs: aborted jobs will now be often re-evaluated, so that transitory errors are discarded to retry the job ; does not work as well as it could.

Diff Detail

Repository
R321 KStars
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
TallFurryMan created this revision.Apr 21 2018, 12:53 PM
Restricted Application added a project: KDE Edu. · View Herald TranscriptApr 21 2018, 12:53 PM
Restricted Application added a subscriber: KDE Edu. · View Herald Transcript
TallFurryMan requested review of this revision.Apr 21 2018, 12:53 PM
TallFurryMan edited the summary of this revision. (Show Details)Apr 21 2018, 12:56 PM

I'm sorry if this looks quite long. I didn't foresee that testing all the commits separately would take so long (one KStars build is ~30min in my VM), so I squashed the commits as I could to make the whole change understandable and testable.

Did you use Arcanist to upload this differential?

Yes, arc diff on several commits I rebased and squashed.

Looks like I cannot land this use arc getting this error:

remote: Audit failure - Commit 966dd84344fd8c54d69d028783386c3061444aaf - Non-full name: TallFurryMan

Yes, my fault! Committer/author discrepancy vs. kde identity. I worked on some of these changes on another computer. I'm uploading a new differential.
Also, I spotted a change that is required to fix evaluateJobs() when it is called with either a zero-length job list or an all-complete job list, that I had made on a later commit. I'll reintegrate it in this changeset, because it is needed here to avoid a crash.

Fixed zero-job evaluation crash, a regression induced by those changes.

For some reason the commit log had a "/bin/bash: q: command not found" instead of the description of my additional change.

TallFurryMan edited the summary of this revision. (Show Details)Apr 21 2018, 7:50 PM
mutlaqja resigned from this revision.Apr 21 2018, 8:55 PM

Ok please submit it then another request.

Splitted commits into more understandable portions (now that I understand Arcanist better)

  • [1/9] Fixed zero-job list or full-complete job list crash when evaluating
  • [2/9] Fixup for dateTimeFormat
  • [3/9] Slight optimization when seaching for capture files
  • [4/9] Clarify job selection algorithm
  • [5/9] Capture count algorithm
  • [6/9] Job estimation
  • [7/9] Fixes for QString::arg()
  • [8/9] Job evaluation
  • [9/9] Homogeneize logs
  • [Summary] Logs, and reworking job count update and repeated jobs estimation

Hello Jasem, I splitted the differential into separate commits for you to review this more easily, now that I understand Arcanist better.
Tested with a daytime tests on a complex list of targets, and three nights of stability tests on M51 (800 captures). There are still issues, but we're going in the right direction I think.
I have another differential ready waiting, but I don't want to go too fast.

TallFurryMan added a reviewer: mutlaqja.
mutlaqja accepted this revision.Apr 22 2018, 12:31 PM

Great work Eric! A quick test here with the Messier Marathon went uneventful. I just changed platesolving to plate solving

This revision is now accepted and ready to land.Apr 22 2018, 12:31 PM
This revision was automatically updated to reflect the committed changes.

Are you sure you have the metadata right by Arcanist? it still submitted the differential with me as the the author .. maybe because I made one small change to it? I think I'll apply it manually next time since this is just more headache

Thanks! Your change should be no big deal in my history, or, huh, so I hope :) I'd say it's expected you become the author if you change something to a commit, except if it's history manipulation, in which case you are the committer. But nothing new for you here indeed.
Now on to the last batch of commits!

By the way, is Phabricator the way to go for contributors? Am I doing things the right way?

I'm not a fan of Phabricator, you can also just email me the diffs as well if you like.