Julia backend: command execution
ClosedPublic

Authored by ivanlakhtanov on Jun 25 2016, 7:45 PM.

Details

Summary

Impelemented command execution with DBUS

Used Julia embedding. Features: execution, syntax errors,
exceptions, multiline input, printing of the last result.

Julia path is detected automatically.

BUGS and FIXMES:
* Workaround: julia redirect commands user pipes/sockets with
  limited buffer to store data. So on large output writing pipe
  end blocks, and julia server execution is blocked. It's
  better to reimplement this by using concurrent threads to
  run julia command and read from pipe. Now using files as they
  provide unlimited buffer.

Diff Detail

Repository
R55 Cantor
Lint
Lint Skipped
Unit
Unit Tests Skipped
ivanlakhtanov retitled this revision from to Julia backend: command execution.
ivanlakhtanov updated this object.
ivanlakhtanov edited the test plan for this revision. (Show Details)
ivanlakhtanov added a reviewer: filipesaraiva.
ivanlakhtanov set the repository for this revision to R55 Cantor.
filipesaraiva added a project: Cantor.
filipesaraiva added a subscriber: Cantor.
filipesaraiva edited edge metadata.Jun 29 2016, 3:19 PM

Nice @ivanlakhtanov, see my reviews below. Fix them and you can push it but remember: push to a feature branch like 'julia-backend'.

Please, don't push it to master ok? :)

src/backends/julia/juliabackend.cpp
28

Try to remove it, but if you need use just 'julia'.

46

Implements the description function here, like in Octave Backend.

src/backends/julia/juliabackend.h
38

Add the description function here, like in Octave Backend.

src/backends/julia/juliabackend.json
6

Change to 'Julia'.

8

Change to 'Julia'.

src/backends/julia/juliaserver/CMakeLists.txt
7

I think you don't need that ${JuliaServer_RSCS} in the end of line. It is just a mistake with the correct ${JuliaServer_SRCS}, right? :)

9–11

Try to build and test it without this commands. I think INSTALL_RPATH_USE_LINK_PATH is something related with R but I don't know.

src/backends/julia/juliaserver/juliaserver.cpp
39

Maybe you can fix it using settings widgets. Give a look at settings widgets in Octave Backend: open Cantor, go to Settings -> Settings Cantor -> Octave (maybe I am not using the correct names because my Cantor is in Portuguese). You will see the 'Path to Octave' line edit there.

Give a look in the implementation of this in octavebackend.kcfg. You can call the command configured using something like this.

You need to add something like this to the CMakeLists.txt of Julia Backend.

filipesaraiva requested changes to this revision.Jul 3 2016, 1:53 PM
filipesaraiva edited edge metadata.
This revision now requires changes to proceed.Jul 3 2016, 1:53 PM
ivanlakhtanov updated this object.
ivanlakhtanov edited edge metadata.

Fixed latest TODOs/FIXMEs

ivanlakhtanov marked 6 inline comments as done.Jul 17 2016, 12:07 PM
ivanlakhtanov marked 2 inline comments as done.
ivanlakhtanov edited edge metadata.

Reimplemented stdout/err catching by using temp files. Previous approach used linux pipes inside and was hanging when output is big enough (pipe limit reached, so process is blocking). Server is reading after the command execution, so it stays blocked.

Really, it's better to spawn a concurent process to read from another end of pipe while execution is processing. Current solution is fast hack that allows me to implement more important functionality.

Reimplemented stdout/err catching by using temp files. Previous approach used linux pipes inside and was hanging when output is big enough (pipe limit reached, so process is blocking). Server is reading after the command execution, so it stays blocked.

What do you think about just append a command to print some "flag string" in end of the command sent by user to verify if all was processed, like in Scilab backend?

For example, the user send something like 1 + 1 to Julia backend but the command to be processed will be something like 1 + 1 (newline) print('terminated-cantor-julia-command-processing'), so if the output has terminated-cantor-scilab-command-processing the input was processed. Do you see any problem with this approach?

Reimplemented stdout/err catching by using temp files. Previous approach used linux pipes inside and was hanging when output is big enough (pipe limit reached, so process is blocking). Server is reading after the command execution, so it stays blocked.

What do you think about just append a command to print some "flag string" in end of the command sent by user to verify if all was processed, like in Scilab backend?

For example, the user send something like 1 + 1 to Julia backend but the command to be processed will be something like 1 + 1 (newline) print('terminated-cantor-julia-command-processing'), so if the output has terminated-cantor-scilab-command-processing the input was processed. Do you see any problem with this approach?

This won't solve mentioned problem. Now I can clearly know when command finished as juliaserver running in synchronous mode with julia command execution. The problem is that when you have pipe in Linux, as soon as it's buffer limit is reached OS blocks writing end until a portion of data is retrieved from buffer by reading process. So good approach is to implement command execution on concurent process/thread with reading end. The signal of the end of output is same as for any Linux pipe -- EOF in the end as soon as writing end is closed and no data in buffer. By using files I go around buffer limit problem, as files are limited to disk space.

I've mentioned a marker approach in my proposal, cuz I were thinking that I will implement execution through interacting with julia REPL (where I can't undertand output end), but I'm now using C bindings.

In D2006#40686, @ivanlakhtanov wrote:
(...) The signal of the end of output is same as for any Linux pipe -- EOF (...)

Do you know if this approach will work for other OS like Windows?

In D2006#40686, @ivanlakhtanov wrote:
(...) The signal of the end of output is same as for any Linux pipe -- EOF (...)

Do you know if this approach will work for other OS like Windows?

That's all is implemented inside of Julia redirect_stdout. I don't know how they do it on Windows. Documentation mentions pipes. In stack traces I've seen epoll things, so it's possible so Julia mean socket connection under pipe (sockets as I know also have some fixed buffers).

Anyway current approach with temporary file will work.

filipesaraiva accepted this revision.Jul 25 2016, 2:15 PM
filipesaraiva edited edge metadata.

OK @ivanlakhtanov push it!
Please, remember: push to a new branch like julia-backend and use the special keyword Differential review.

This revision is now accepted and ready to land.Jul 25 2016, 2:15 PM
ivanlakhtanov updated this object.
ivanlakhtanov edited edge metadata.
This revision was automatically updated to reflect the committed changes.