This is my test worksheet:

Thursday, March 22 2018: KDE Applications 18.04 Freeze and Beta (18.03.80) tag & release

User visible strings are frozen: exceptions need approval from kde-i18n-doc@kde.org.

So, that you think about it?

@asemke, Could you test it? I think I have solved this problem.

In D12074#250058, @sirgienko wrote:

@asemke, Could you test it? I think I have solved this problem.

Yes, it works. But the implementation seems to be a bit complicated. Wouldn't be easier to simply split the output of lua with QString::split('\n') and QString::split('>') and to remove the input commands from the obtained QStringList?

In D12074#251640, @asemke wrote:

In D12074#250058, @sirgienko wrote:

@asemke, Could you test it? I think I have solved this problem.

Yes, it works. But the implementation seems to be a bit complicated. Wouldn't be easier to simply split the output of lua with QString::split('\n') and QString::split('>') and to remove the input commands from the obtained QStringList?

Not sure, that I have understood, that you mean. Output from lua can comes at different times and in parts, so we don't know, when lua ends, without counting input commands in output

In D12074#251667, @sirgienko wrote:

Not sure, that I have understood, that you mean. Output from lua can comes at different times and in parts, so we don't know, when lua ends, without counting input commands in output

Can you check my solution for this problem?

In D12074#251920, @asemke wrote:

In D12074#251667, @sirgienko wrote:

Not sure, that I have understood, that you mean. Output from lua can comes at different times and in parts, so we don't know, when lua ends, without counting input commands in output

Can you check my solution for this problem?

lua_multiline_output.patch947 BDownload

Unfortunately, this patch have problems:

I also run tests:
2 or 3 failed tests.

Input in outputs maybe appears like '> {input}\n' or '>> {input}\n' or '{input}\n', so this

for (const auto input : inputs)
    output.remove(input + QLatin1Char('\n'));

don't catch all input strings in output.

In D12074#251964, @sirgienko wrote:

Input in outputs maybe appears like '> {input}\n' or '>> {input}\n' or '{input}\n', so this

for (const auto input : inputs)
    output.remove(input + QLatin1Char('\n'));

don't catch all input strings in output.

The new patch takes care of lua's promts in a better way. Can you give it a try?

In D12074#252014, @asemke wrote:

In D12074#251964, @sirgienko wrote:

Input in outputs maybe appears like '> {input}\n' or '>> {input}\n' or '{input}\n', so this

for (const auto input : inputs)
    output.remove(input + QLatin1Char('\n'));

don't catch all input strings in output.

The new patch takes care of lua's promts in a better way. Can you give it a try?

lua_multiline_output_2.patch1 KBDownload

Well, it's better, but we still have problems:

I think, you also should change condition if(m_currentExpression && !m_output.isEmpty() && m_output.trimmed().endsWith(QLatin1String(">"))) in luasession.cpp, because sometimes lua prints > before ends work, and as you see, lua backend finishs work with command too early and lua output added to next entries.

In D12074#252018, @sirgienko wrote:

Well, it's better, but we still have problems:
[...]

I cannot reproduce this with lua 5.2.4. All commands in your test project file produce a correct output with my patch.

The problems seem to be either version dependent or there are some other timing issues involved here.

I think, you also should change condition if(m_currentExpression && !m_output.isEmpty() && m_output.trimmed().endsWith(QLatin1String(">"))) in luasession.cpp, because sometimes lua prints > before ends work, and as you see, lua backend finishs work with command too early and lua output added to next entries.

Did you try this? Does it work for you? Can you please provide the outputs in LuaExpression.cpp:70 for one of such multi-line commands where you still have problems with?

In D12074#252465, @asemke wrote:

In D12074#252018, @sirgienko wrote:

Well, it's better, but we still have problems:
[...]

I cannot reproduce this with lua 5.2.4. All commands in your test project file produce a correct output with my patch.

The problems seem to be either version dependent or there are some other timing issues involved here.

I have Lua 5.2.4 too. And I agree, that this is timing thing, because results time dependent.

I think, you also should change condition if(m_currentExpression && !m_output.isEmpty() && m_output.trimmed().endsWith(QLatin1String(">"))) in luasession.cpp, because sometimes lua prints > before ends work, and as you see, lua backend finishs work with command too early and lua output added to next entries.

Did you try this? Does it work for you? Can you please provide the outputs in LuaExpression.cpp:70 for one of such multi-line commands where you still have problems with?

Did you test with 'Reevaluate Entries automatically' option? Without this option all works fine, but with the option I often have something like this:

final command to be executed  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\n" 

QSizeF(1131, 157) QSizeF(1131, 126)
parsing the output  "function fib(n)\n>>   if n < 2 then return 1 end\n>>   return fib(n - 2) + fib(n - 1)\n>> end\n> \n> -- Closures and anonymous functions are ok:\n> function adder(x)\n>>   -- The returned function is created when adder is\n>>   -- called, and remembers the value of x:\n>>   return function (y) return x + y end\n>> end\n> a1 = adder(9)\n> a2 = adder(36)\n> print(a1(16))  --> 25\n25\n> print(a2(64))  --> 100\n100\n> function bar(a, b, c)\n>>   print(a, b, c)\n>>   return 4, 8, 15, 16, 23, 42\n>> end\n> \n> x, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\nzaphod\tnil\tnil\n> "
final output of the command  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"" :  "function fib(n)\nif n < 2 then return 1 end\nreturn fib(n - 2) + fib(n - 1)\n-- Closures and anonymous functions are ok:\nfunction adder(x)\n-- The returned function is created when adder is\n-- called, and remembers the value of x:\nreturn function (y) return x + y end\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\n25\nprint(a2(64))  --> 100\n100\nzaphod\tnil\tnil" 

settting result to a type  1  result
update Entry
new result
wsStatusChange 1

In D12074#252468, @sirgienko wrote:

Did you test with 'Reevaluate Entries automatically' option? Without this option all works fine, but with the option I often have something like this:

final command to be executed  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\n" 

QSizeF(1131, 157) QSizeF(1131, 126)
parsing the output  "function fib(n)\n>>   if n < 2 then return 1 end\n>>   return fib(n - 2) + fib(n - 1)\n>> end\n> \n> -- Closures and anonymous functions are ok:\n> function adder(x)\n>>   -- The returned function is created when adder is\n>>   -- called, and remembers the value of x:\n>>   return function (y) return x + y end\n>> end\n> a1 = adder(9)\n> a2 = adder(36)\n> print(a1(16))  --> 25\n25\n> print(a2(64))  --> 100\n100\n> function bar(a, b, c)\n>>   print(a, b, c)\n>>   return 4, 8, 15, 16, 23, 42\n>> end\n> \n> x, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\nzaphod\tnil\tnil\n> "
final output of the command  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"" :  "function fib(n)\nif n < 2 then return 1 end\nreturn fib(n - 2) + fib(n - 1)\n-- Closures and anonymous functions are ok:\nfunction adder(x)\n-- The returned function is created when adder is\n-- called, and remembers the value of x:\nreturn function (y) return x + y end\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\n25\nprint(a2(64))  --> 100\n100\nzaphod\tnil\tnil" 

settting result to a type  1  result
update Entry
new result
wsStatusChange 1

Yes, same problem here. I was testing without this auto eval. option all the time. With this option the results are sometimes very weird. Did you also observe this error with other backends or is this something specific to lua?

In D12074#252496, @asemke wrote:

In D12074#252468, @sirgienko wrote:

Did you test with 'Reevaluate Entries automatically' option? Without this option all works fine, but with the option I often have something like this:

final command to be executed  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\n" 

QSizeF(1131, 157) QSizeF(1131, 126)
parsing the output  "function fib(n)\n>>   if n < 2 then return 1 end\n>>   return fib(n - 2) + fib(n - 1)\n>> end\n> \n> -- Closures and anonymous functions are ok:\n> function adder(x)\n>>   -- The returned function is created when adder is\n>>   -- called, and remembers the value of x:\n>>   return function (y) return x + y end\n>> end\n> a1 = adder(9)\n> a2 = adder(36)\n> print(a1(16))  --> 25\n25\n> print(a2(64))  --> 100\n100\n> function bar(a, b, c)\n>>   print(a, b, c)\n>>   return 4, 8, 15, 16, 23, 42\n>> end\n> \n> x, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\nzaphod\tnil\tnil\n> "
final output of the command  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"" :  "function fib(n)\nif n < 2 then return 1 end\nreturn fib(n - 2) + fib(n - 1)\n-- Closures and anonymous functions are ok:\nfunction adder(x)\n-- The returned function is created when adder is\n-- called, and remembers the value of x:\nreturn function (y) return x + y end\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\n25\nprint(a2(64))  --> 100\n100\nzaphod\tnil\tnil" 

settting result to a type  1  result
update Entry
new result
wsStatusChange 1

Yes, same problem here. I was testing without this auto eval. option all the time. With this option the results are sometimes very weird. Did you also observe this error with other backends or is this something specific to lua?

Well, I have worked a lot of time with Octave, Maxima and few times with Python 2, Python 3, Lua, Julia. I have this problem only with lua.

In D12074#252497, @sirgienko wrote:

Well, I have worked a lot of time with Octave, Maxima and few times with Python 2, Python 3, Lua, Julia. I have this problem only with lua.

The problem is reproducible with these two multi-line commands:

The debug output after the first command is executed

final command to be executed  "function fib(n)\n  if n < 2 then return 1 end\n  return fib(n - 2) + fib(n - 1)\nend\n\n-- Closures and anonymous functions are ok:\nfunction adder(x)\n  -- The returned function is created when adder is\n  -- called, and remembers the value of x:\n  return function (y) return x + y end\nend\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\nprint(a2(64))  --> 100\n" 

parsing the output  "function fib(n)\n>>   if n < 2 then return 1 end\n>>   return fib(n - 2) + fib(n - 1)\n>> end\n> \n"
final output of the command  "function fib(n)\n  if n < 2 then return 1 end\n  return fib(n - 2) + fib(n - 1)\nend\n\n-- Closures and anonymous functions are ok:\nfunction adder(x)\n  -- The returned function is created when adder is\n  -- called, and remembers the value of x:\n  return function (y) return x + y end\nend\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\nprint(a2(64))  --> 100" :  ""

and after the second command is executed

final command to be executed  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\n" 

parsing the output  "> -- Closures and anonymous functions are ok:\n> function adder(x)\n>>   -- The returned function is created when adder is\n>>   -- called, and remembers the value of x:\n>>   return function (y) return x + y end\n>> end\n> a1 = adder(9)\n> a2 = adder(36)\n> print(a1(16))  --> 25\n25\n> print(a2(64))  --> 100\n100\n> "
final output of the command  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"" :  "-- Closures and anonymous functions are ok:\nfunction adder(x)\n-- The returned function is created when adder is\n-- called, and remembers the value of x:\nreturn function (y) return x + y end\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\n25\nprint(a2(64))  --> 100\n100" 

parsing the output  "function bar(a, b, c)\n>>   print(a, b, c)\n>>   return 4, 8, 15, 16, 23, 42\n>> end\n> \n> x, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\nzaphod\tnil\tnil\n> "
final output of the command  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"" :  "zaphod\tnil\tnil"

So, the expected result of the first command

25
100

comes _after_ the second command is executed. Also, there is the third output here with the desired result of the second command

zaphod nil nil

This all looks a bit strange...

In D12074#253885, @asemke wrote:

This all looks a bit strange...

The problem was in LuaSession::readOutput(). The patch

In D12074#253885, @asemke wrote:

This all looks a bit strange...

Well, in fact this is a problem of incorrect parse end condition, which produce false positive errors.
I add printing of readed line from backend whatever make it clear.

final command to be executed  "function fib(n)\n  if n < 2 then return 1 end\n  return fib(n - 2) + fib(n - 1)\nend\n\n-- Closures and anonymous functions are ok:\nfunction adder(x)\n  -- The returned function is created when adder is\n  -- called, and remembers the value of x:\n  return function (y) return x + y end\nend\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\nprint(a2(64))  --> 100\n" 

QSizeF(937, 397) QSizeF(937, 297)
newLine:  "function fib(n)\n"
newLine:  ">>   if n < 2 then return 1 end\n"
newLine:  ">>   return fib(n - 2) + fib(n - 1)\n"
newLine:  ">> end\n"
newLine:  "> \n"
newLine:  "> -- Closures and anonymous functions are ok:\n"
newLine:  "> function adder(x)\n"
newLine:  ">>   -- The returned function is created when adder is\n"
newLine:  ">>   -- called, and remembers the value of x:\n"
newLine:  ">>   return function (y) return x + y end\n"
newLine:  ">> end\n"
newLine:  "> a1 = adder(9)\n"
newLine:  "> a2 = adder(36)\n"
newLine:  "> "
parsing the output  "function fib(n)\n>>   if n < 2 then return 1 end\n>>   return fib(n - 2) + fib(n - 1)\n>> end\n> \n> -- Closures and anonymous functions are ok:\n> function adder(x)\n>>   -- The returned function is created when adder is\n>>   -- called, and remembers the value of x:\n>>   return function (y) return x + y end\n>> end\n> a1 = adder(9)\n> a2 = adder(36)\n> "
final output of the command  "function fib(n)\n  if n < 2 then return 1 end\n  return fib(n - 2) + fib(n - 1)\nend\n\n-- Closures and anonymous functions are ok:\nfunction adder(x)\n  -- The returned function is created when adder is\n  -- called, and remembers the value of x:\n  return function (y) return x + y end\nend\na1 = adder(9)\na2 = adder(36)\nprint(a1(16))  --> 25\nprint(a2(64))  --> 100" :  ""

Command a2 = adder(36) not last, but after this command we read > (lua printing lag) and it satisfy parse end condition. So, backend finish the first command and run next, and then:

QSizeF(937, 157) QSizeF(937, 126)
newLine:  "print(a1(16))  --> 25\n"
newLine:  "25\n"
newLine:  "> print(a2(64))  --> 100\n"
newLine:  "100\n"
newLine:  "> "
parsing the output  "print(a1(16))  --> 25\n25\n> print(a2(64))  --> 100\n100\n> "
final output of the command  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"" :  "print(a1(16))  --> 25\n25\nprint(a2(64))  --> 100\n100"

We set tail of previous command as result for second command.
And then:

newLine:  "function bar(a, b, c)\n"
newLine:  ">>   print(a, b, c)\n"
newLine:  ">>   return 4, 8, 15, 16, 23, 42\n"
newLine:  ">> end\n"
newLine:  "> \n"
newLine:  "> x, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\n"
newLine:  "zaphod\tnil\tnil\n"
newLine:  "> "
parsing the output  "function bar(a, b, c)\n>>   print(a, b, c)\n>>   return 4, 8, 15, 16, 23, 42\n>> end\n> \n> x, y = bar('zaphod')  --> prints \"zaphod  nil nil\"\nzaphod\tnil\tnil\n> "
final output of the command  "function bar(a, b, c)\n  print(a, b, c)\n  return 4, 8, 15, 16, 23, 42\nend\n\nx, y = bar('zaphod')  --> prints \"zaphod  nil nil\"" :  "zaphod\tnil\tnil"

Set result again.

In D12074#253921, @asemke wrote:

In D12074#253885, @asemke wrote:

This all looks a bit strange...

The problem was in LuaSession::readOutput(). The patch

lua_multiline_output_3.patch2 KBDownload

handles the outputs correctly. @sirgienko Does it also work for you?

No, output have changed, but still have problems:

In D12074#253922, @sirgienko wrote:

The problem was in LuaSession::readOutput(). The patch

lua_multiline_output_3.patch2 KBDownload

handles the outputs correctly. @sirgienko Does it also work for you?

No, output have changed, but still have problems:

But last two commands work fine.

I cannot reproduce this with my third patch. Can you paste here your debug output for this example?

In D12074#254033, @asemke wrote:

In D12074#253922, @sirgienko wrote:

The problem was in LuaSession::readOutput(). The patch

lua_multiline_output_3.patch2 KBDownload

handles the outputs correctly. @sirgienko Does it also work for you?

No, output have changed, but still have problems:

But last two commands work fine.

I cannot reproduce this with my third patch. Can you paste here your debug output for this example?

In D12074#254056, @sirgienko wrote:

I also have added printing each readed line.

I cannot exactly reproduce this problem but I see some other weird cases. Sometimes there is no output at all but the calculation is still running (status bar shows "calculating...") and cantor eats up all the available memory. Such problems I also get with your patch. These problems only occur with the auto eval options switched on.

I'd suggest to submit my lua_multiline_output_3.patch now since it produces correct and stable results without the auto evaluation and the implemented logic is clear and not very far from the final solution I think. With this Cantor's behavior for multi-line commands in lua is much better already as before. After this we can work on the remaining problems. Do you agree?

In D12074#254482, @asemke wrote:

Do you agree?

Well, I agree, maybe it's better to come back to this later. Just in case I want to notice, that your lua_multiline_output_3.patch don't handle output from commands like print('> '). I think, we should solve this problem in the final solution,

And that about lua tests? What results do you get with patch 3?

In D12074#254487, @sirgienko wrote:

In D12074#254482, @asemke wrote:

Do you agree?

Well, I agree, maybe it's better to come back to this later. Just in case I want to notice, that your lua_multiline_output_3.patch don't handle output from commands like print('> '). I think, we should solve this problem in the final solution,

I created a bug report for this problem in order not to forget about it - https://bugs.kde.org/show_bug.cgi?id=393579.
For the problem with the auto evaluation mode I created https://bugs.kde.org/show_bug.cgi?id=393580.

And that about lua tests? What results do you get with patch 3?

You skipped already the problematic tests in D11791. We'll need to check this, too.

I extended a bit the description in T3872 with "Test loading of saved projects - open a project, execute all commands and check the outputs. Test with and without "auto evaluation" option enabled.". The test project file you used here covers quite a lot of different features and we should create couple of tests with it. In LabPlot we have already tests for the project load/import - load a saved project and check its content (https://cgit.kde.org/labplot.git/tree/tests/import_export/project/ProjectImportTest.cpp). Would be great to add similar tests in Cantor too.

We can close this code review ticket. The situation with lua was greatly improved now. All the remaining open issues will be addressed in other tickets/reviews.