this problem can be solved in a more simple way by adding

if (!m_loginDone && m_session->backend()->name() == QLatin1String("Octave"))
    loginToSession();

in Worksheet::enableHighlighting(). This fix would be similar to d2bb5bbc44e23eadfdb21b772bd94222b29e9e42 with a similar problem for the completion.

The idea behind the postponed login is described in T4760. These fixes now make the second point described in T4760 impossible. We'll need to refactor this I guess.

Independent of how the final/temporary fix will look like, we urgently need to fix this for the 18.04 release.

Yes, I have read T4760, and that's why no just added Octave activation when loading worksheet. With this fix, I also can modify Octvae worksheet, even I haven't Octave in the system. But there is one problem with my fix: if Octave don't installed and we open Octave worksheet, initialization starts (cursor changing, status bar with "initializing"), but don't end, because backend don't send loginDone signal. But I work on it.

In D11588#233690, @sirgienko wrote:

Yes, I have read T4760, and that's why no just added Octave activation when loading worksheet. With this fix, I also can modify Octvae worksheet, even I haven't Octave in the system. But there is one problem with my fix: initialization starts (cursor changing, status bar with "initializing"), but don't end, because backend don't send loginDone signal. But I work on it.

It will end after the timeout in m_process->waitForStarted() (5 minutes). The problem is that we don't evaluate at the moment the return codes of all those waitForStarted() calls and don't react accordingly. Ideally, the login()-functions shouldn't be called at all if the executables for octave, maxima, etc. are not found. The worksheet should be in a read-only mode without no login done nowhere in such cases. This needs to be implemented centrally.

As to the fix of the actual problem you mentioned here, did you check the two lines I wrote in my previous comment? Can you please validate it also solves the problem on your side?

As to the fix of the actual problem you mentioned here, did you check the two lines I wrote in my previous comment? Can you please validate it also solves the problem on your side?

No, it don't fix the problem. The problem is that we call syntaxHighlighter in the Worksheet constructor, which calls the OctaveHighliter constructor, which calls the OctaveSession::evaluateExpression, inside which a attempt is made to access the m_process method, which is initialized with nullptr (from OctaveSession constructor).
So, for Octave, we *need* call login before OctaveHighliter::constructor, or we have seg. fault.
Or, We can add new state to Session::Status, because after Session constructor and before login calling status equals Status::Done and it incorrect state, I think (because session starts evaluateExpression, if it isn't in Status::Running state, even if login calling don't done and m_process equals nullptr).

In D11588#233714, @sirgienko wrote:

As to the fix of the actual problem you mentioned here, did you check the two lines I wrote in my previous comment? Can you please validate it also solves the problem on your side?

No, it don't fix the problem. The problem is that we call syntaxHighlighter in the Worksheet constructor, which calls the OctaveHighliter constructor, which calls the OctaveSession::evaluateExpression, inside which a attempt is made to access the m_process method, which is initialized with nullptr (from OctaveSession constructor).
So, for Octave, we *need* call login before OctaveHighliter::constructor, or we have seg. fault.

Yes. And this is exactly what my code snippet does - it calls loginToSession() before the constructor of OctaveHighlither is called. My snippet has to be added before the line with m_highlighter=session()->syntaxHighlighter(this); - I didn't mentioned this, I assumed it was clear.

Sorry, my mistake.
Now, we haven't segmentation fault, but we also haven't any output from octave-cli.
It looks like this:

In D11588#233747, @sirgienko wrote:

Sorry, my mistake.
Now, we haven't segmentation fault, but we also haven't any output from octave-cli.
It looks like this:

The output from octave-cli is not empty but we somehow fail to parse it. I'm checking it already.

octavesession.cpp:258: else if (line.contains(m_prompt)) equals True all time, because m_prompt equals QRegExp(patternSyntax=0, pattern='""') (empty regex). It means, that we don't found octave prompt, because m_currentExpression never equals nullptr, for some reason (we also don't set temp directory too, for the same reason). So we never reach parseOutput call and m_resultString == "" all time.

This fix the problem and backend works correctly.

In D11588#233759, @sirgienko wrote:

octavesession.cpp:258: else if (line.contains(m_prompt)) equals True all time, because m_prompt equals QRegExp(patternSyntax=0, pattern='""') (empty regex). It means, that we don't found octave prompt, because m_currentExpression never equals nullptr, for some reason (we also don't set temp directory too, for the same reason). So we never reach parseOutput call and m_resultString == "" all time.

I pushed a fix for this in 3c0251f120a13a87a21ac6abc8b53cbcfcf820f0. Can you please also give it a try?

P.S.: I also submitted the fix to login to octave prior to the initialization of the highlighter.

I pushed a fix for this in 3c0251f120a13a87a21ac6abc8b53cbcfcf820f0. Can you please also give it a try?

Yes, it works, I confirm it.