diff --git a/ChangeLog b/ChangeLog index b98a52e5d..a2302df8d 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,223 +1,223 @@ -----2.6----- New features: * Import of ROOT (CERN) TH1 histograms * Import of Ngspice raw files (ASCII and binary) * Import of data in JSON format (JSON arrays and objects) * Convolution/Deconvolution of data sets (sampling interval, linear/circular, normalization, wrap, standard kernel) * Cross-/Autocorrelation of data sets (sampling interval, linear/circular, normalization) * Allow to rotate plot legends * Allow to specify the number format when exporting spreadsheet and matrix * Improved user interface for data fitting (add fit function preview, show parameters directly, make options foldable) * [spreadsheet] when filling a float column with row numbers, automatically convert its type to integer * [spreadsheet] when filling an integer column with function values, automatically convert its type to float * [spreadsheet] data manipulation: add/subtract a value from column values Bug fixes: * Fixed several problems in live data support * [spreadsheet] properly calculate function values out of integer x-values * [matrix] fix editing integer values + * Don't crash when a scaling factor equal to zero was set for axis -----2.5----- New features: * Support for reading and plotting of live-data * Improved data fitting * Automatically guess parameter of custom models * Better result presentation * Support different weight types * Consider given x- and y-error when fitting (can be switched off) * Show t statistics, P > |t| and confidence interval * Calculate p-value for chi-square and F test in nonlinear fitting * added fit models for most statistical distributions * Improved theming * Apply themes to worksheet and to all its children * Respect theme settings also in plot legends and labels * Allow to disable theming in worksheets and plots after a theme was selected * Show currently active theme in the "Apply theme" button * New application option in the settings for the default theme used for new worksheets. * Support different data types * auto detect integer and datetime data in import * support number locale and datetime formats * improved data type support in spreadsheets * Import from SQL databases (tables or custom queries) * Import Origin OPJ projects * Much better support for Windows and macOS * Syntax highlighting for LaTeX in the text label * Allow to set the background color for LaTeX labels * Support Hermite polynomials from GSL 2.4 * Support error functions and related functions from libcerf * "Used in" sub-menu in column contex menu for faster navigation to the curves consuming the column * Direct application of analysis functions (smoothing, interpolating, etc.) on the ploted data via curve's context menu * Direct application of analysis functions on the data in the spreadsheet and plotting of the results via spreadsheet's context menu * Drag columns in the project explorer and drop them on plots (either in a worksheet view or in the project explorer) to create curves * "Show last N points" and "Show first N points" data ranges in cartesian plot * Added CLI option --presenter to start LabPlot directly in the presenter mode * Added CLI parameter to directly open project files (LabPlot or Origin) * Allow drag&drop of projects files (LabPlot and Origin) on the main window to load the project * Allow drag&drop of data files on the main window to import the data * Show tooltips for the supported mathematical functions and constants in the expression text field * Automatically switch to the scientific representation for numbers bigger than 10^4 on the axis tick labels * Automatically allow the latex typesetting in the application after the latex environment was installed later without making the user to go to the settings dialog * Allow to change the color scheme for the application * Smooth and animated zooming in the worksheet view * Allow to add text labels to plots * Improved building with MSVC, Intel and PGI compiler Performance improvements: * Faster copy&paste in the spreadsheet Bug fixes: * Bug 379877 - masked rows in spreadsheet not restored in project * Calculation of fit results corrected * Axes now support values larger than FLT_MAX (~10^38) and smaller than FLT_MIN (~10^-38) * When a LabPlot project is being droped in the main window, load the project directly instead of showing the import file dialog * Correctly save and restore masked cells * Don't crash if the rc-file was not found during the startup - * Don't crash when a scaling factor equal to zero was set for axis -----2.4----- New features: * Support themes for plots * Import and editing of FITS data files * Data reduction by removing data points using multiple algorithms * Numerical differentiation and integration with several options * Many new pre-defined fit models (Gompertz, Weibull, Log-Normal, Gumbel, etc.) sorted in categories * Fit parameter now support fixed values, lower/upper limits and use Unicode * Fit model and random number distribution formulas are now rendered with LaTeX * Support user specified x range in all analysis functions * Allow to enter complete LaTeX documents in text labels * Configuration parameter to use different LaTex engines (LuaLaTex, XeLatex, pdfLaTex, LaTex) * Disable LaTeX typesetting if no LaTex installation (and other required tools) were found at runtime * Presenter mode for worksheets * Support for Mac OS * Support for Julia's vectors and tuples in CAS worksheets (requires Cantor v. 16.12 or higher) * Allow to jump directly to the data source spreadsheet via XYCurve's context menu * Select and delete multiple objects in project explorer * Improved and extended internal parser for mathematical expressions * Copy of worksheet elements as image to the clipboard via CTRL+C Bug fixes: * BUG: 361326 - Allow to select curves with overlapping bounding boxes * Correctly load worksheet sizes from saved templates * Fixed crash when removing columns in spreadsheet * Fixed crash when fitting using GSL >= 2 * List of available functions corrected * Constants are now available with full accuracy * Windows: Import of files and open recent files fixed -----2.3----- New features: * Integration of Cantor - Support for different open-source computer algebra systems * Statistics on spreadsheets and matrices * Export of spreadsheets and matrices to LaTeX tables * Interpolation of data including different splines, cosine, exponential, cubic Hermite (Catmull-Rom, cardinal, Kochanek-Bartels) and rational functions * Data smoothing using moving average (centered or lagged), percentile filter or Savitzky-Golay algorithm * Fourier filter (low pass, high pass, band pass, band reject) with ideal, Butterworth, Chebychev I+II, Legendre or Bessel-Thomson filter * Fourier transform with many window functions (Welch, Hann, Hamming, etc.) calculating magnitude, amplitude, power, phase, dB, etc. and supporting one/two sided spectrum with or without shift and x scaling to frequency, index or period * Filter and search capabilities in the drop down box for the selection of data sources * Sigmoid function as a new pre-defined fit model * Support for compiling on Microsoft Windows Performance improvements: * Faster generation of random values * Global option to enable/disable the double-buffering for faster painting of curves (enabled on default) Bug fixes: * Save and restore last used setting in RandomValuesDialog * Update axis title shape on title rotations correctly * Save and restore custom column widths in the spreadsheet * Fixed sporadic crashes during project close -----2.2----- New features: * Datapicker - tool for extracting curves and data points from imported images * Custom point on the plot with user-defined position and appearance * Accept drag&drop events * Support GSL 2.x * Import and export dialogs save and restore their settings and sizes Performance improvements: * Faster rendering of the image view of the matrix Bug fixes: * BUG: 354744 - make setting of range auto scaling in CartesianPlot undo/redo-able * Removed couple of hard coded sizes given in pixels for better user-experience on hidpi-displays * Fixes the bug with disabled undo/redo-actions in after the undo-history was cleared * Keep the information about the columns to be shown in the project explorer after project close * Fixed some bugs in painting of the selection band on the worksheet * Allow to open gz- and bz2-compressed LabPlot project files on the command line interface -----2.1----- New features: * New Matrix view for handling matrix data. * Workbook - a new container for grouping several objects of type Spreadsheet and/or Matrix. * Import of binary, image, NetCDF and HDF data into spreadsheet or matrix. * Visual HDF and NetCDF parser to view content of files and import data sets. * Preview of all supported file types in import dialog. * Transparently import compressed data files. * In xy-curve the points may not be connected by the line if there are NANs in-between. This behaviour is controlled by the parameter "skip gaps". * Multiplier of Pi format of the plot axis for periodical functions. * New operations on columns in Spreadsheet - reverse, drop values and mask values. * Formula used to generate the values in a column is stored and can be changed/adjusted in the formula dialog afterwards. * Curve filling: the area below, under, left to or right to the curve can be filled. * Support for multiple variables in "Function Values"-dialog - new data in the spreadsheet can be calculated out of multiple columns. Performance improvements: * Speeded up the creation of new columns during the import Bug fixes: * Fixed wrong behaviour when doing "zoom&select" in a plot and then deselecting the plot - it was not possible anymore to select the plot again on the worksheet. -----2.0.2----- New features: * Plot 2D-curves defined by mathematical equations in cartesian and polar coordinates or via a parametric equation. * Linear and non-linear regression analysis. Several predefined fit-models are provided. User-defined models are also possible. * Besides fixed worksheet sizes (predefined sizes like A4 etc. or user-defined), complete view size can be used. All sizes are automatically adjusted on resize. * Different axis arrow types. * "select region and zoom in", "select x-region and zoom in", "select y-region and zoom in" functions for cartesian plot. * Rounded border for several worksheet objects (plot area, legend etc.) * Hover effect for axis, curve and text label. * Added a MessageBox - ask befor deleting worksheet objects. * Added three new types for drop lines - "zero baseline", "min baseline" and "max baseline" * Fill the selection in Spreadsheet with a constant value provided by the user * Fill columns with uniform and non-uniform random numbers, several distributions are available. * Fill columns with function values * Allow custom resolutions for PNG-export * Export of the spreadsheet to a text file. * Simultaneous zooming and navigation accross multiple plots. * Implemented "Powers of 10/2/e" for the axis tick labels Bug fixes: * Don't crash when trying to create a plot in case no rc-file was installed. * Don't allow to select unwanted objects in TreeViewComboBox in ImportDialog and XYCurveDock. * Corrected painting of background images in plot area and legend. * BUG: 330723 - fixed weird selection in spreadsheet. * BUG: 330774 - fixed wrong positioning of axis on orientation changes. * Update main window title when project name is changed -----2.0.1----- Bug fix release. Solved issues: * Fixed wrong scaling of legend's text labels in pdf-export * Fixed memory corruption in CartesianPlotDock that can lead to crashes -----2.0.0----- First stable release of LabPlot2. LabPlot2 is a complete rewrite of LabPlot1 and lacks in this release a lot of features available in the predecessor. On the other hand, the GUI and the usability is more superior as compared to LabPlot1 and there are several new features that were not available in LabPlot1. Brief summary of features and capabilities of LabPlot2 implemented in the first release: * project-based management of data * created objects are organized in a tree and are visualized and accessible in the project explorer * for a better management of objects, additional folders and sub-folders can be created within a project * spreadsheet with very basic functionality is available for manual data entry * "file data source" can be used to import a file and to let LabPlot2 watch for changes in that file * external data from an ascii file can be also directly imported to a spreadsheet for further editing * worksheet is the main object where plots, labels etc. are placed on * several zooming functions for the worksheet * only cartesian plot is implemented in the first release * arbitrary number of freely positionable axes is possible * xy-curve is implemented. As the data source for the x- and y-values columns either from a spreadsheet or from a file data source can be used * several zooming and "movement" functions are available for the plots which help to navigate through data * legend for xy-plots * a lot of properties of the worksheet elements can be edited in a very easy way in the corresponding dock widgets * plots on the worksheet can be put into a horizontal, vertical or grid layouts * export of worksheet (entire worksheet or current seleciton) to pdf, eps, png and svg diff --git a/src/backend/datasources/filters/AsciiFilter.cpp b/src/backend/datasources/filters/AsciiFilter.cpp index 362d5bb4d..0d3133b7b 100644 --- a/src/backend/datasources/filters/AsciiFilter.cpp +++ b/src/backend/datasources/filters/AsciiFilter.cpp @@ -1,2960 +1,2958 @@ /*************************************************************************** File : AsciiFilter.cpp Project : LabPlot Description : ASCII I/O-filter -------------------------------------------------------------------- Copyright : (C) 2009-2018 Stefan Gerlach (stefan.gerlach@uni.kn) Copyright : (C) 2009-2017 Alexander Semke (alexander.semke@web.de) ***************************************************************************/ /*************************************************************************** * * * This program is free software; you can redistribute it and/or modify * * it under the terms of the GNU General Public License as published by * * the Free Software Foundation; either version 2 of the License, or * * (at your option) any later version. * * * * This program is distributed in the hope that it will be useful, * * but WITHOUT ANY WARRANTY; without even the implied warranty of * * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * * GNU General Public License for more details. * * * * You should have received a copy of the GNU General Public License * * along with this program; if not, write to the Free Software * * Foundation, Inc., 51 Franklin Street, Fifth Floor, * * Boston, MA 02110-1301 USA * * * ***************************************************************************/ #include "backend/datasources/LiveDataSource.h" #include "backend/core/column/Column.h" #include "backend/core/Project.h" #include "backend/datasources/filters/AsciiFilter.h" #include "backend/datasources/filters/AsciiFilterPrivate.h" #include "backend/worksheet/plots/cartesian/CartesianPlot.h" #include "backend/worksheet/plots/cartesian/XYCurve.h" #include "backend/lib/macros.h" #include "backend/lib/trace.h" #ifdef HAVE_MQTT #include "backend/datasources/MQTTClient.h" #include "backend/datasources/MQTTTopic.h" #endif #include #include #include #include #include #ifdef Q_OS_LINUX #include #endif /*! \class AsciiFilter \brief Manages the import/export of data organized as columns (vectors) from/to an ASCII-file. \ingroup datasources */ AsciiFilter::AsciiFilter() : AbstractFileFilter(), d(new AsciiFilterPrivate(this)) {} AsciiFilter::~AsciiFilter() = default; /*! reads the content of the device \c device. */ void AsciiFilter::readDataFromDevice(QIODevice& device, AbstractDataSource* dataSource, AbstractFileFilter::ImportMode importMode, int lines) { d->readDataFromDevice(device, dataSource, importMode, lines); } void AsciiFilter::readFromLiveDeviceNotFile(QIODevice &device, AbstractDataSource* dataSource) { d->readFromLiveDevice(device, dataSource); } qint64 AsciiFilter::readFromLiveDevice(QIODevice& device, AbstractDataSource* dataSource, qint64 from) { return d->readFromLiveDevice(device, dataSource, from); } #ifdef HAVE_MQTT /*! reads the content of a message received by the topic. */ void AsciiFilter::readMQTTTopic(const QString& message, const QString& topic, AbstractDataSource* dataSource) { d->readMQTTTopic(message, topic, dataSource); } /*! provides a preview of the data received by the topic. */ void AsciiFilter::MQTTPreview(QVector& list, const QString& message, const QString& topic) { d->MQTTPreview(list, message, topic); } /*! Returns the statistical data, that the MQTTTopic needs for the will message. */ QString AsciiFilter::MQTTColumnStatistics(const MQTTTopic* topic) const{ return d->MQTTColumnStatistics(topic); } /*! Returns the column mode of the last column (the value column of the MQTTTopic). */ AbstractColumn::ColumnMode AsciiFilter::MQTTColumnMode() const{ return d->MQTTColumnMode(); } /*! After the MQTTTopic is loaded, prepares the filter for reading. */ void AsciiFilter::setPreparedForMQTT(bool prepared, MQTTTopic* topic, const QString& separator) { d->setPreparedForMQTT(prepared, topic, separator); } #endif /*! returns the separator used by the filter. */ QString AsciiFilter::separator() const { return d->separator(); } /*! returns the separator used by the filter. */ int AsciiFilter::isPrepared() { return d->isPrepared(); } /*! reads the content of the file \c fileName. */ void AsciiFilter::readDataFromFile(const QString& fileName, AbstractDataSource* dataSource, AbstractFileFilter::ImportMode importMode) { d->readDataFromFile(fileName, dataSource, importMode); } QVector AsciiFilter::preview(const QString& fileName, int lines) { return d->preview(fileName, lines); } QVector AsciiFilter::preview(QIODevice &device) { return d->preview(device); } /*! reads the content of the file \c fileName to the data source \c dataSource. */ //void AsciiFilter::read(const QString& fileName, AbstractDataSource* dataSource, AbstractFileFilter::ImportMode importMode) { // d->read(fileName, dataSource, importMode); //} /*! writes the content of the data source \c dataSource to the file \c fileName. */ void AsciiFilter::write(const QString& fileName, AbstractDataSource* dataSource) { d->write(fileName, dataSource); // emit() } /*! loads the predefined filter settings for \c filterName */ void AsciiFilter::loadFilterSettings(const QString& filterName) { Q_UNUSED(filterName); } /*! saves the current settings as a new filter with the name \c filterName */ void AsciiFilter::saveFilterSettings(const QString& filterName) const { Q_UNUSED(filterName); } /*! returns the list with the names of all saved (system wide or user defined) filter settings. */ QStringList AsciiFilter::predefinedFilters() { return QStringList(); } /*! returns the list of all predefined separator characters. */ QStringList AsciiFilter::separatorCharacters() { return (QStringList() << "auto" << "TAB" << "SPACE" << "," << ";" << ":" << ",TAB" << ";TAB" << ":TAB" << ",SPACE" << ";SPACE" << ":SPACE" << "2xSPACE" << "3xSPACE" << "4xSPACE" << "2xTAB"); } /*! returns the list of all predefined comment characters. */ QStringList AsciiFilter::commentCharacters() { return (QStringList() << "#" << "!" << "//" << "+" << "c" << ":" << ";"); } /*! returns the list of all predefined data types. */ QStringList AsciiFilter::dataTypes() { const QMetaObject& mo = AbstractColumn::staticMetaObject; const QMetaEnum& me = mo.enumerator(mo.indexOfEnumerator("ColumnMode")); QStringList list; for (int i = 0; i <= 100; ++i) // me.keyCount() does not work because we have holes in enum if (me.valueToKey(i)) list << me.valueToKey(i); return list; } QString AsciiFilter::fileInfoString(const QString& fileName) { QString info(i18n("Number of columns: %1", AsciiFilter::columnNumber(fileName))); info += QLatin1String("
"); info += i18n("Number of lines: %1", AsciiFilter::lineNumber(fileName)); return info; } /*! returns the number of columns in the file \c fileName. */ int AsciiFilter::columnNumber(const QString& fileName, const QString& separator) { KFilterDev device(fileName); if (!device.open(QIODevice::ReadOnly)) { DEBUG("Could not open file " << fileName.toStdString() << " for determining number of columns"); return -1; } QString line = device.readLine(); line.remove(QRegExp("[\\n\\r]")); QStringList lineStringList; if (separator.length() > 0) lineStringList = line.split(separator); else lineStringList = line.split(QRegExp("\\s+")); DEBUG("number of columns : " << lineStringList.size()); return lineStringList.size(); } size_t AsciiFilter::lineNumber(const QString& fileName) { KFilterDev device(fileName); if (!device.open(QIODevice::ReadOnly)) { DEBUG("Could not open file " << fileName.toStdString() << " to determine number of lines"); return 0; } // if (!device.canReadLine()) // return -1; size_t lineCount = 0; #ifdef Q_OS_LINUX //on linux use wc, if available, which is much faster than counting lines in the file if (!QStandardPaths::findExecutable(QLatin1String("wc")).isEmpty()) { QProcess wc; wc.start(QLatin1String("wc"), QStringList() << QLatin1String("-l") << fileName); size_t lineCount = 0; while (wc.waitForReadyRead()) lineCount = wc.readLine().split(' ')[0].toInt(); lineCount++; // last line not counted return lineCount; } #endif while (!device.atEnd()) { device.readLine(); lineCount++; } return lineCount; } /*! returns the number of lines in the device \c device and 0 if sequential. resets the position to 0! */ size_t AsciiFilter::lineNumber(QIODevice &device) { if (device.isSequential()) return 0; // if (!device.canReadLine()) // DEBUG("WARNING in AsciiFilter::lineNumber(): device cannot 'readLine()' but using it anyway."); size_t lineCount = 0; device.seek(0); while (!device.atEnd()) { device.readLine(); lineCount++; } device.seek(0); return lineCount; } void AsciiFilter::setCommentCharacter(const QString& s) { d->commentCharacter = s; } QString AsciiFilter::commentCharacter() const { return d->commentCharacter; } void AsciiFilter::setSeparatingCharacter(const QString& s) { d->separatingCharacter = s; } QString AsciiFilter::separatingCharacter() const { return d->separatingCharacter; } void AsciiFilter::setDateTimeFormat(const QString &f) { d->dateTimeFormat = f; } QString AsciiFilter::dateTimeFormat() const { return d->dateTimeFormat; } void AsciiFilter::setNumberFormat(QLocale::Language lang) { d->numberFormat = lang; } QLocale::Language AsciiFilter::numberFormat() const { return d->numberFormat; } void AsciiFilter::setAutoModeEnabled(const bool b) { d->autoModeEnabled = b; } bool AsciiFilter::isAutoModeEnabled() const { return d->autoModeEnabled; } void AsciiFilter::setHeaderEnabled(const bool b) { d->headerEnabled = b; } bool AsciiFilter::isHeaderEnabled() const { return d->headerEnabled; } void AsciiFilter::setSkipEmptyParts(const bool b) { d->skipEmptyParts = b; } bool AsciiFilter::skipEmptyParts() const { return d->skipEmptyParts; } void AsciiFilter::setCreateIndexEnabled(bool b) { d->createIndexEnabled = b; } bool AsciiFilter::createIndexEnabled() const{ return d->createIndexEnabled; } void AsciiFilter::setSimplifyWhitespacesEnabled(bool b) { d->simplifyWhitespacesEnabled = b; } bool AsciiFilter::simplifyWhitespacesEnabled() const { return d->simplifyWhitespacesEnabled; } void AsciiFilter::setNaNValueToZero(bool b) { if (b) d->nanValue = 0; else d->nanValue = NAN; } bool AsciiFilter::NaNValueToZeroEnabled() const { if (d->nanValue == 0) return true; return false; } void AsciiFilter::setRemoveQuotesEnabled(bool b) { d->removeQuotesEnabled = b; } bool AsciiFilter::removeQuotesEnabled() const { return d->removeQuotesEnabled; } void AsciiFilter::setVectorNames(const QString& s) { d->vectorNames.clear(); if (!s.simplified().isEmpty()) d->vectorNames = s.simplified().split(' '); } QStringList AsciiFilter::vectorNames() const { return d->vectorNames; } QVector AsciiFilter::columnModes() { return d->columnModes; } void AsciiFilter::setStartRow(const int r) { d->startRow = r; } int AsciiFilter::startRow() const { return d->startRow; } void AsciiFilter::setEndRow(const int r) { d->endRow = r; } int AsciiFilter::endRow() const { return d->endRow; } void AsciiFilter::setStartColumn(const int c) { d->startColumn = c; } int AsciiFilter::startColumn() const { return d->startColumn; } void AsciiFilter::setEndColumn(const int c) { d->endColumn = c; } int AsciiFilter::endColumn() const { return d->endColumn; } //##################################################################### //################### Private implementation ########################## //##################################################################### AsciiFilterPrivate::AsciiFilterPrivate(AsciiFilter* owner) : q(owner), commentCharacter("#"), separatingCharacter("auto"), numberFormat(QLocale::C), autoModeEnabled(true), headerEnabled(true), skipEmptyParts(false), simplifyWhitespacesEnabled(true), nanValue(NAN), removeQuotesEnabled(false), createIndexEnabled(false), startRow(1), endRow(-1), startColumn(1), endColumn(-1), mqttPreviewFirstEmptyColCount (0), m_actualStartRow(1), m_actualRows(0), m_actualCols(0), m_prepared(false), m_columnOffset(0) { } /*! * get a single line from device */ QStringList AsciiFilterPrivate::getLineString(QIODevice& device) { QString line; do { // skip comment lines in data lines if (!device.canReadLine()) DEBUG("WARNING in AsciiFilterPrivate::getLineString(): device cannot 'readLine()' but using it anyway."); // line = device.readAll(); line = device.readLine(); } while (line.startsWith(commentCharacter)); line.remove(QRegExp("[\\n\\r]")); // remove any newline if (simplifyWhitespacesEnabled) line = line.simplified(); DEBUG("data line : \'" << line.toStdString() << '\''); QStringList lineStringList = line.split(m_separator, (QString::SplitBehavior)skipEmptyParts); //TODO: remove quotes here? QDEBUG("data line, parsed: " << lineStringList); return lineStringList; } /*! * returns -1 if the device couldn't be opened, 1 if the current read position in the device is at the end and 0 otherwise. */ int AsciiFilterPrivate::prepareDeviceToRead(QIODevice& device) { DEBUG("AsciiFilterPrivate::prepareDeviceToRead(): is sequential = " << device.isSequential() << ", can readLine = " << device.canReadLine()); if (!device.open(QIODevice::ReadOnly)) return -1; if (device.atEnd() && !device.isSequential()) // empty file return 1; ///////////////////////////////////////////////////////////////// // Find first data line (ignoring comment lines) DEBUG(" Skipping " << startRow - 1 << " lines"); for (int i = 0; i < startRow - 1; ++i) { QString line; if (!device.canReadLine()) DEBUG("WARNING in AsciiFilterPrivate::prepareDeviceToRead(): device cannot 'readLine()' but using it anyway."); line = device.readLine(); DEBUG(" line = " << line.toStdString()); if (device.atEnd()) { if (device.isSequential()) break; else return 1; } } // Parse the first line: // Determine the number of columns, create the columns and use (if selected) the first row to name them QString firstLine; do { // skip comment lines if (!device.canReadLine()) DEBUG("WARNING in AsciiFilterPrivate::prepareDeviceToRead(): device cannot 'readLine()' but using it anyway."); if (device.atEnd()) { DEBUG("device at end! Giving up."); if (device.isSequential()) break; else return 1; } firstLine = device.readLine(); } while (firstLine.startsWith(commentCharacter)); DEBUG(" device position after first line and comments = " << device.pos()); firstLine.remove(QRegExp("[\\n\\r]")); // remove any newline if (simplifyWhitespacesEnabled) firstLine = firstLine.simplified(); DEBUG("First line: \'" << firstLine.toStdString() << '\''); // determine separator and split first line QStringList firstLineStringList; if (separatingCharacter == "auto") { DEBUG("automatic separator"); QRegExp regExp("(\\s+)|(,\\s+)|(;\\s+)|(:\\s+)"); firstLineStringList = firstLine.split(regExp, (QString::SplitBehavior)skipEmptyParts); if (!firstLineStringList.isEmpty()) { int length1 = firstLineStringList.at(0).length(); if (firstLineStringList.size() > 1) { int pos2 = firstLine.indexOf(firstLineStringList.at(1), length1); m_separator = firstLine.mid(length1, pos2 - length1); } else { //old: separator = line.right(line.length() - length1); m_separator = ' '; } } } else { // use given separator // replace symbolic "TAB" with '\t' m_separator = separatingCharacter.replace(QLatin1String("2xTAB"), "\t\t", Qt::CaseInsensitive); m_separator = separatingCharacter.replace(QLatin1String("TAB"), "\t", Qt::CaseInsensitive); // replace symbolic "SPACE" with ' ' m_separator = m_separator.replace(QLatin1String("2xSPACE"), QLatin1String(" "), Qt::CaseInsensitive); m_separator = m_separator.replace(QLatin1String("3xSPACE"), QLatin1String(" "), Qt::CaseInsensitive); m_separator = m_separator.replace(QLatin1String("4xSPACE"), QLatin1String(" "), Qt::CaseInsensitive); m_separator = m_separator.replace(QLatin1String("SPACE"), QLatin1String(" "), Qt::CaseInsensitive); firstLineStringList = firstLine.split(m_separator, (QString::SplitBehavior)skipEmptyParts); } DEBUG("separator: \'" << m_separator.toStdString() << '\''); DEBUG("number of columns: " << firstLineStringList.size()); QDEBUG("first line: " << firstLineStringList); DEBUG("headerEnabled: " << headerEnabled); //optionally, remove potential spaces in the first line if (simplifyWhitespacesEnabled) { for (int i = 0; i < firstLineStringList.size(); ++i) firstLineStringList[i] = firstLineStringList[i].simplified(); } if (headerEnabled) { // use first line to name vectors vectorNames = firstLineStringList; QDEBUG("vector names =" << vectorNames); m_actualStartRow = startRow + 1; } else m_actualStartRow = startRow; // set range to read if (endColumn == -1) { if (headerEnabled || vectorNames.size() == 0) endColumn = firstLineStringList.size(); // last column else //number of vector names provided in the import dialog (not more than the maximal number of columns in the file) endColumn = qMin(vectorNames.size(), firstLineStringList.size()); } if (createIndexEnabled) { vectorNames.prepend(i18n("Index")); endColumn++; } m_actualCols = endColumn - startColumn + 1; //TEST: readline-seek-readline fails /* qint64 testpos = device.pos(); DEBUG("read data line @ pos " << testpos << " : " << device.readLine().toStdString()); device.seek(testpos); testpos = device.pos(); DEBUG("read data line again @ pos " << testpos << " : " << device.readLine().toStdString()); */ ///////////////////////////////////////////////////////////////// // parse first data line to determine data type for each column if (!device.isSequential()) firstLineStringList = getLineString(device); columnModes.resize(m_actualCols); int col = 0; if (createIndexEnabled) { columnModes[0] = AbstractColumn::Integer; col = 1; } for (auto& valueString : firstLineStringList) { // parse columns available in first data line if (simplifyWhitespacesEnabled) valueString = valueString.simplified(); if (removeQuotesEnabled) valueString.remove(QRegExp("[\"\']")); if (col == m_actualCols) break; columnModes[col++] = AbstractFileFilter::columnMode(valueString, dateTimeFormat, numberFormat); } // parsing more lines to better determine data types for (unsigned int i = 0; i < m_dataTypeLines; ++i) { if (device.atEnd()) // EOF reached break; firstLineStringList = getLineString(device); createIndexEnabled ? col = 1 : col = 0; for (auto& valueString : firstLineStringList) { if (simplifyWhitespacesEnabled) valueString = valueString.simplified(); if (removeQuotesEnabled) valueString.remove(QRegExp("[\"\']")); if (col == m_actualCols) break; AbstractColumn::ColumnMode mode = AbstractFileFilter::columnMode(valueString, dateTimeFormat, numberFormat); // numeric: integer -> numeric if (mode == AbstractColumn::Numeric && columnModes[col] == AbstractColumn::Integer) columnModes[col] = mode; // text: non text -> text if (mode == AbstractColumn::Text && columnModes[col] != AbstractColumn::Text) columnModes[col] = mode; col++; } } QDEBUG("column modes = " << columnModes); // ATTENTION: This resets the position in the device to 0 m_actualRows = (int)AsciiFilter::lineNumber(device); const int actualEndRow = (endRow == -1 || endRow > m_actualRows) ? m_actualRows : endRow; m_actualRows = actualEndRow - m_actualStartRow + 1; DEBUG("start/end column: " << startColumn << ' ' << endColumn); DEBUG("start/end row: " << m_actualStartRow << ' ' << actualEndRow); DEBUG("actual cols/rows (w/o header): " << m_actualCols << ' ' << m_actualRows); if (m_actualRows == 0 && !device.isSequential()) return 1; return 0; } /*! reads the content of the file \c fileName to the data source \c dataSource. Uses the settings defined in the data source. */ void AsciiFilterPrivate::readDataFromFile(const QString& fileName, AbstractDataSource* dataSource, AbstractFileFilter::ImportMode importMode) { DEBUG("AsciiFilterPrivate::readDataFromFile(): fileName = \'" << fileName.toStdString() << "\', dataSource = " << dataSource << ", mode = " << ENUM_TO_STRING(AbstractFileFilter, ImportMode, importMode)); KFilterDev device(fileName); readDataFromDevice(device, dataSource, importMode); } qint64 AsciiFilterPrivate::readFromLiveDevice(QIODevice& device, AbstractDataSource* dataSource, qint64 from) { DEBUG("AsciiFilterPrivate::readFromLiveDevice(): bytes available = " << device.bytesAvailable() << ", from = " << from); if (device.bytesAvailable() <= 0) { DEBUG(" No new data available"); return 0; } //TODO: may be also a matrix? auto* spreadsheet = dynamic_cast(dataSource); if (spreadsheet->sourceType() != LiveDataSource::SourceType::FileOrPipe) if (device.isSequential() && device.bytesAvailable() < (int)sizeof(quint16)) return 0; if (!m_prepared) { DEBUG(" Preparing .."); switch (spreadsheet->sourceType()) { case LiveDataSource::SourceType::FileOrPipe: { const int deviceError = prepareDeviceToRead(device); if (deviceError != 0) { DEBUG(" Device error = " << deviceError); return 0; } break; } case LiveDataSource::SourceType::NetworkTcpSocket: case LiveDataSource::SourceType::NetworkUdpSocket: case LiveDataSource::SourceType::LocalSocket: case LiveDataSource::SourceType::SerialPort: m_actualRows = 1; if (createIndexEnabled) { m_actualCols = 2; columnModes << AbstractColumn::Integer << AbstractColumn::Numeric; vectorNames << i18n("Index") << i18n("Value"); } else { m_actualCols = 1; columnModes << AbstractColumn::Numeric; vectorNames << i18n("Value"); } QDEBUG(" vector names = " << vectorNames); case LiveDataSource::SourceType::MQTT: break; } // prepare import for spreadsheet spreadsheet->setUndoAware(false); spreadsheet->resize(AbstractFileFilter::Replace, vectorNames, m_actualCols); DEBUG(" data source resized to col: " << m_actualCols); DEBUG(" data source rowCount: " << spreadsheet->rowCount()); //columns in a file data source don't have any manual changes. //make the available columns undo unaware and suppress the "data changed" signal. //data changes will be propagated via an explicit Column::setChanged() call once new data was read. for (int i = 0; i < spreadsheet->childCount(); i++) { spreadsheet->child(i)->setUndoAware(false); spreadsheet->child(i)->setSuppressDataChangedSignal(true); } int keepNValues = spreadsheet->keepNValues(); if (keepNValues == 0) spreadsheet->setRowCount(m_actualRows > 1 ? m_actualRows : 1); else { spreadsheet->setRowCount(keepNValues); m_actualRows = keepNValues; } m_dataContainer.resize(m_actualCols); DEBUG(" Setting data .."); for (int n = 0; n < m_actualCols; ++n) { // data() returns a void* which is a pointer to any data type (see ColumnPrivate.cpp) spreadsheet->child(n)->setColumnMode(columnModes[n]); switch (columnModes[n]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } DEBUG(" Prepared!"); } qint64 bytesread = 0; #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportTotal: "); #endif LiveDataSource::ReadingType readingType; if (!m_prepared) { readingType = LiveDataSource::ReadingType::TillEnd; } else { //we have to read all the data when reading from end //so we set readingType to TillEnd if (spreadsheet->readingType() == LiveDataSource::ReadingType::FromEnd) readingType = LiveDataSource::ReadingType::TillEnd; //if we read the whole file we just start from the beginning of it //and read till end else if (spreadsheet->readingType() == LiveDataSource::ReadingType::WholeFile) readingType = LiveDataSource::ReadingType::TillEnd; else readingType = spreadsheet->readingType(); } DEBUG(" Reading type = " << ENUM_TO_STRING(LiveDataSource, ReadingType, readingType)); //move to the last read position, from == total bytes read //since the other source types are sequential we cannot seek on them if (spreadsheet->sourceType() == LiveDataSource::SourceType::FileOrPipe) device.seek(from); //count the new lines, increase actualrows on each //now we read all the new lines, if we want to use sample rate //then here we can do it, if we have actually sample rate number of lines :-? int newLinesForSampleSizeNotTillEnd = 0; int newLinesTillEnd = 0; QVector newData; if (readingType != LiveDataSource::ReadingType::TillEnd) newData.resize(spreadsheet->sampleSize()); int newDataIdx = 0; { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportReadingFromFile: "); #endif DEBUG(" source type = " << ENUM_TO_STRING(LiveDataSource, SourceType, spreadsheet->sourceType())); while (!device.atEnd()) { if (readingType != LiveDataSource::ReadingType::TillEnd) { switch (spreadsheet->sourceType()) { // different sources need different read methods case LiveDataSource::SourceType::LocalSocket: newData[newDataIdx++] = device.readAll(); break; case LiveDataSource::SourceType::NetworkUdpSocket: newData[newDataIdx++] = device.read(device.bytesAvailable()); break; case LiveDataSource::SourceType::FileOrPipe: newData.push_back(device.readLine()); break; case LiveDataSource::SourceType::NetworkTcpSocket: //TODO: check serial port case LiveDataSource::SourceType::SerialPort: newData[newDataIdx++] = device.read(device.bytesAvailable()); case LiveDataSource::SourceType::MQTT: break; } } else { // ReadingType::TillEnd switch (spreadsheet->sourceType()) { // different sources need different read methods case LiveDataSource::SourceType::LocalSocket: newData.push_back(device.readAll()); break; case LiveDataSource::SourceType::NetworkUdpSocket: newData.push_back(device.read(device.bytesAvailable())); break; case LiveDataSource::SourceType::FileOrPipe: newData.push_back(device.readLine()); break; case LiveDataSource::SourceType::NetworkTcpSocket: //TODO: check serial port case LiveDataSource::SourceType::SerialPort: newData.push_back(device.read(device.bytesAvailable())); case LiveDataSource::SourceType::MQTT: break; } } newLinesTillEnd++; if (readingType != LiveDataSource::ReadingType::TillEnd) { newLinesForSampleSizeNotTillEnd++; //for Continuous reading and FromEnd we read sample rate number of lines if possible //here TillEnd and Whole file behave the same if (newLinesForSampleSizeNotTillEnd == spreadsheet->sampleSize()) break; } } QDEBUG(" data read: " << newData); } //now we reset the readingType if (spreadsheet->readingType() == LiveDataSource::ReadingType::FromEnd) readingType = spreadsheet->readingType(); //we had less new lines than the sample size specified if (readingType != LiveDataSource::ReadingType::TillEnd) QDEBUG(" Removed empty lines: " << newData.removeAll("")); //back to the last read position before counting when reading from files if (spreadsheet->sourceType() == LiveDataSource::SourceType::FileOrPipe) device.seek(from); const int spreadsheetRowCountBeforeResize = spreadsheet->rowCount(); int currentRow = 0; // indexes the position in the vector(column) int linesToRead = 0; int keepNValues = spreadsheet->keepNValues(); DEBUG(" Increase row count. keepNValues = " << keepNValues); if (m_prepared) { //increase row count if we don't have a fixed size //but only after the preparation step if (keepNValues == 0) { DEBUG(" keep All values"); if (readingType != LiveDataSource::ReadingType::TillEnd) m_actualRows += qMin(newData.size(), spreadsheet->sampleSize()); else { //we don't increase it if we reread the whole file, we reset it if (!(spreadsheet->readingType() == LiveDataSource::ReadingType::WholeFile)) m_actualRows += newData.size(); else m_actualRows = newData.size(); } //appending if (spreadsheet->readingType() == LiveDataSource::ReadingType::WholeFile) linesToRead = m_actualRows; else linesToRead = m_actualRows - spreadsheetRowCountBeforeResize; } else { // fixed size DEBUG(" keep " << keepNValues << " values"); if (readingType == LiveDataSource::ReadingType::TillEnd) { //we had more lines than the fixed size, so we read m_actualRows number of lines if (newLinesTillEnd > m_actualRows) { linesToRead = m_actualRows; //TODO after reading we should skip the next data lines //because it's TillEnd actually } else linesToRead = newLinesTillEnd; } else { //we read max sample size number of lines when the reading mode //is ContinuouslyFixed or FromEnd, WholeFile is disabled linesToRead = qMin(spreadsheet->sampleSize(), newLinesTillEnd); } } DEBUG(" actual rows = " << m_actualRows); if (linesToRead == 0) return 0; } else { // not prepared linesToRead = newLinesTillEnd; if (headerEnabled) --m_actualRows; } DEBUG(" lines to read = " << linesToRead); if (spreadsheet->sourceType() == LiveDataSource::SourceType::FileOrPipe || spreadsheet->sourceType() == LiveDataSource::SourceType::NetworkUdpSocket) { if (m_actualRows < linesToRead) { DEBUG(" SET lines to read to " << m_actualRows); linesToRead = m_actualRows; } } //new rows/resize columns if we don't have a fixed size //TODO if the user changes this value..m_resizedToFixedSize..setResizedToFixedSize if (keepNValues == 0) { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportResizing: "); #endif if (spreadsheet->rowCount() < m_actualRows) spreadsheet->setRowCount(m_actualRows); if (!m_prepared) currentRow = 0; else { // indexes the position in the vector(column) if (spreadsheet->readingType() == LiveDataSource::ReadingType::WholeFile) currentRow = 0; else currentRow = spreadsheetRowCountBeforeResize; } // if we have fixed size, we do this only once in preparation, here we can use // m_prepared and we need something to decide whether it has a fixed size or increasing for (int n = 0; n < m_actualCols; ++n) { // data() returns a void* which is a pointer to any data type (see ColumnPrivate.cpp) switch (columnModes[n]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } } else { // fixed size //when we have a fixed size we have to pop sampleSize number of lines if specified //here popping, setting currentRow if (!m_prepared) { if (spreadsheet->readingType() == LiveDataSource::ReadingType::WholeFile) currentRow = 0; else currentRow = m_actualRows - qMin(newLinesTillEnd, m_actualRows); } else { if (readingType == LiveDataSource::ReadingType::TillEnd) { if (newLinesTillEnd > m_actualRows) { currentRow = 0; } else { if (spreadsheet->readingType() == LiveDataSource::ReadingType::WholeFile) currentRow = 0; else currentRow = m_actualRows - newLinesTillEnd; } } else { //we read max sample size number of lines when the reading mode //is ContinuouslyFixed or FromEnd currentRow = m_actualRows - qMin(spreadsheet->sampleSize(), newLinesTillEnd); } } if (m_prepared) { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportPopping: "); #endif // enable data change signal for (int col = 0; col < m_actualCols; ++col) spreadsheet->child(col)->setSuppressDataChangedSignal(false); for (int row = 0; row < linesToRead; ++row) { for (int col = 0; col < m_actualCols; ++col) { switch (columnModes[col]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(spreadsheet->child(col)->data()); vector->pop_front(); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(spreadsheet->child(col)->data()); vector->pop_front(); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(spreadsheet->child(col)->data()); vector->pop_front(); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(spreadsheet->child(col)->data()); vector->pop_front(); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } } } } // from the last row we read the new data in the spreadsheet DEBUG(" Reading from line " << currentRow << " till end line " << newLinesTillEnd); DEBUG(" Lines to read:" << linesToRead <<", actual rows:" << m_actualRows << ", actual cols:" << m_actualCols); newDataIdx = 0; if (readingType == LiveDataSource::ReadingType::FromEnd) { if (m_prepared) { if (newData.size() > spreadsheet->sampleSize()) newDataIdx = newData.size() - spreadsheet->sampleSize(); //since we skip a couple of lines, we need to count those bytes too for (int i = 0; i < newDataIdx; ++i) bytesread += newData.at(i).size(); } } DEBUG(" newDataIdx: " << newDataIdx); static int indexColumnIdx = 1; { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportFillingContainers: "); #endif int row = 0; if (readingType == LiveDataSource::ReadingType::TillEnd || (readingType == LiveDataSource::ReadingType::ContinuousFixed)) { if (headerEnabled) { if (!m_prepared) { row = 1; bytesread += newData.at(0).size(); } } } if (spreadsheet->sourceType() == LiveDataSource::SourceType::FileOrPipe) { if (readingType == LiveDataSource::ReadingType::WholeFile) { if (headerEnabled) { row = 1; bytesread += newData.at(0).size(); } } } for (; row < linesToRead; ++row) { DEBUG("\n Reading row " << row + 1 << " of " << linesToRead); QString line; if (readingType == LiveDataSource::ReadingType::FromEnd) line = newData.at(newDataIdx++); else line = newData.at(row); //when we read the whole file we don't care about the previous position //so we don't have to count those bytes if (readingType != LiveDataSource::ReadingType::WholeFile) { if (spreadsheet->sourceType() == LiveDataSource::SourceType::FileOrPipe) { bytesread += line.size(); } } DEBUG(" Line bytes: " << line.size() << " line: " << line.toStdString()); if (simplifyWhitespacesEnabled) line = line.simplified(); if (line.isEmpty() || line.startsWith(commentCharacter)) // skip empty or commented lines continue; QLocale locale(numberFormat); QStringList lineStringList; // only FileOrPipe support multiple columns if (spreadsheet->sourceType() == LiveDataSource::SourceType::FileOrPipe) lineStringList = line.split(m_separator, (QString::SplitBehavior)skipEmptyParts); else lineStringList << line; QDEBUG(" line = " << lineStringList << ", separator = \'" << m_separator << "\'"); if (createIndexEnabled) { if (spreadsheet->keepNValues() == 0) lineStringList.prepend(QString::number(currentRow + 1)); else lineStringList.prepend(QString::number(indexColumnIdx++)); } QDEBUG(" column modes = " << columnModes); for (int n = 0; n < m_actualCols; ++n) { DEBUG(" actual col = " << n); if (n < lineStringList.size()) { QString valueString = lineStringList.at(n); DEBUG(" value string = " << valueString.toStdString()); // set value depending on data type switch (columnModes[n]) { case AbstractColumn::Numeric: { DEBUG(" Numeric"); bool isNumber; const double value = locale.toDouble(valueString, &isNumber); static_cast*>(m_dataContainer[n])->operator[](currentRow) = (isNumber ? value : nanValue); qDebug() << "dataContainer[" << n << "] size:" << static_cast*>(m_dataContainer[n])->size(); break; } case AbstractColumn::Integer: { DEBUG(" Integer"); bool isNumber; const int value = locale.toInt(valueString, &isNumber); DEBUG(" container size = " << m_dataContainer.size() << ", current row = " << currentRow); static_cast*>(m_dataContainer[n])->operator[](currentRow) = (isNumber ? value : 0); qDebug() << "dataContainer[" << n << "] size:" << static_cast*>(m_dataContainer[n])->size(); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(valueString, dateTimeFormat); static_cast*>(m_dataContainer[n])->operator[](currentRow) = valueDateTime.isValid() ? valueDateTime : QDateTime(); break; } case AbstractColumn::Text: if (removeQuotesEnabled) valueString.remove(QRegExp("[\"\']")); static_cast*>(m_dataContainer[n])->operator[](currentRow) = valueString; break; case AbstractColumn::Month: //TODO break; case AbstractColumn::Day: //TODO break; } } else { DEBUG(" missing columns in this line"); switch (columnModes[n]) { case AbstractColumn::Numeric: static_cast*>(m_dataContainer[n])->operator[](currentRow) = nanValue; break; case AbstractColumn::Integer: static_cast*>(m_dataContainer[n])->operator[](currentRow) = 0; break; case AbstractColumn::DateTime: static_cast*>(m_dataContainer[n])->operator[](currentRow) = QDateTime(); break; case AbstractColumn::Text: static_cast*>(m_dataContainer[n])->operator[](currentRow) = ""; break; case AbstractColumn::Month: //TODO break; case AbstractColumn::Day: //TODO break; } } } currentRow++; } } if (m_prepared) { //notify all affected columns and plots about the changes PERFTRACE("AsciiLiveDataImport, notify affected columns and plots"); const Project* project = spreadsheet->project(); QVector curves = project->children(AbstractAspect::Recursive); QVector plots; for (int n = 0; n < m_actualCols; ++n) { Column* column = spreadsheet->column(n); //determine the plots where the column is consumed for (const auto* curve : curves) { if (curve->xColumn() == column || curve->yColumn() == column) { auto* plot = dynamic_cast(curve->parentAspect()); if (plots.indexOf(plot) == -1) { plots << plot; plot->setSuppressDataChangedSignal(true); } } } column->setChanged(); } //loop over all affected plots and retransform them for (auto* plot : plots) { plot->setSuppressDataChangedSignal(false); plot->dataChanged(); } } m_prepared = true; DEBUG("AsciiFilterPrivate::readFromLiveDevice() DONE"); return bytesread; } /*! reads the content of device \c device to the data source \c dataSource. Uses the settings defined in the data source. */ void AsciiFilterPrivate::readDataFromDevice(QIODevice& device, AbstractDataSource* dataSource, AbstractFileFilter::ImportMode importMode, int lines) { DEBUG("AsciiFilterPrivate::readDataFromDevice(): dataSource = " << dataSource << ", mode = " << ENUM_TO_STRING(AbstractFileFilter, ImportMode, importMode) << ", lines = " << lines); if (!m_prepared) { const int deviceError = prepareDeviceToRead(device); if (deviceError != 0) { DEBUG("Device error = " << deviceError); return; } // matrix data has only one column mode if (dynamic_cast(dataSource)) { auto mode = columnModes[0]; //TODO: remove this when Matrix supports text type if (mode == AbstractColumn::Text) mode = AbstractColumn::Numeric; for (auto& c : columnModes) if (c != mode) c = mode; } m_columnOffset = dataSource->prepareImport(m_dataContainer, importMode, m_actualRows, m_actualCols, vectorNames, columnModes); m_prepared = true; } DEBUG("locale = " << QLocale::languageToString(numberFormat).toStdString()); QLocale locale(numberFormat); // Read the data int currentRow = 0; // indexes the position in the vector(column) if (lines == -1) lines = m_actualRows; //skip data lines, if required DEBUG(" Skipping " << m_actualStartRow - 1 << " lines"); for (int i = 0; i < m_actualStartRow - 1; ++i) device.readLine(); DEBUG(" Reading " << qMin(lines, m_actualRows) << " lines"); for (int i = 0; i < qMin(lines, m_actualRows); ++i) { QString line = device.readLine(); line.remove(QRegExp("[\\n\\r]")); // remove any newline if (simplifyWhitespacesEnabled) line = line.simplified(); if (line.isEmpty() || line.startsWith(commentCharacter)) // skip empty or commented lines continue; QStringList lineStringList = line.split(m_separator, (QString::SplitBehavior)skipEmptyParts); // remove left white spaces if (skipEmptyParts) { for (int n = 0; n < lineStringList.size(); ++n) { QString valueString = lineStringList.at(n); if (!QString::compare(valueString, " ")) { lineStringList.removeAt(n); n--; } } } for (int n = 0; n < m_actualCols; ++n) { // index column if required if (n == 0 && createIndexEnabled) { static_cast*>(m_dataContainer[0])->operator[](currentRow) = i + 1; continue; } if ((createIndexEnabled ? n - 1 : n) < lineStringList.size()) { QString valueString = lineStringList.at(createIndexEnabled ? n - 1 : n); // set value depending on data type switch (columnModes[n]) { case AbstractColumn::Numeric: { bool isNumber; const double value = locale.toDouble(valueString, &isNumber); static_cast*>(m_dataContainer[n])->operator[](currentRow) = (isNumber ? value : nanValue); break; } case AbstractColumn::Integer: { bool isNumber; const int value = locale.toInt(valueString, &isNumber); static_cast*>(m_dataContainer[n])->operator[](currentRow) = (isNumber ? value : 0); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(valueString, dateTimeFormat); static_cast*>(m_dataContainer[n])->operator[](currentRow) = valueDateTime.isValid() ? valueDateTime : QDateTime(); break; } case AbstractColumn::Text: { if (removeQuotesEnabled) valueString.remove(QRegExp("[\"\']")); - DEBUG(" set value (" << currentRow << ", " << n << "): " << valueString.toStdString()); - //TODO: QString size is not defined. Can not preallocate data matrix! - //QVector* colData = static_cast*>(m_dataContainer[n]); - //colData->operator[](currentRow) = valueString; + QVector* colData = static_cast*>(m_dataContainer[n]); + colData->operator[](currentRow) = valueString; break; } case AbstractColumn::Month: // never happens case AbstractColumn::Day: break; } } else { // missing columns in this line switch (columnModes[n]) { case AbstractColumn::Numeric: static_cast*>(m_dataContainer[n])->operator[](currentRow) = nanValue; break; case AbstractColumn::Integer: static_cast*>(m_dataContainer[n])->operator[](currentRow) = 0; break; case AbstractColumn::DateTime: static_cast*>(m_dataContainer[n])->operator[](currentRow) = QDateTime(); break; case AbstractColumn::Text: static_cast*>(m_dataContainer[n])->operator[](currentRow) = ""; break; case AbstractColumn::Month: // never happens case AbstractColumn::Day: break; } } } currentRow++; emit q->completed(100 * currentRow/m_actualRows); } DEBUG(" Read " << currentRow << " lines"); dataSource->finalizeImport(m_columnOffset, startColumn, endColumn, currentRow, dateTimeFormat, importMode); } /*! * preview for special devices (local/UDP/TCP socket or serial port) */ QVector AsciiFilterPrivate::preview(QIODevice &device) { DEBUG("AsciiFilterPrivate::preview(): bytesAvailable = " << device.bytesAvailable() << ", isSequential = " << device.isSequential()); QVector dataStrings; if (!(device.bytesAvailable() > 0)) { DEBUG("No new data available"); return dataStrings; } if (device.isSequential() && device.bytesAvailable() < (int)sizeof(quint16)) return dataStrings; #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportTotal: "); #endif int linesToRead = 0; QVector newData; //TODO: serial port "read(nBytes)"? while (!device.atEnd()) { if (device.canReadLine()) newData.push_back(device.readLine()); else // UDP fails otherwise newData.push_back(device.readAll()); linesToRead++; } QDEBUG(" data = " << newData); if (linesToRead == 0) return dataStrings; int col = 0; int colMax = newData.at(0).size(); if (createIndexEnabled) colMax++; columnModes.resize(colMax); if (createIndexEnabled) { columnModes[0] = AbstractColumn::ColumnMode::Integer; col = 1; vectorNames.prepend(i18n("Index")); } vectorNames.append(i18n("Value")); QDEBUG(" vector names = " << vectorNames); for (const auto& valueString : newData.at(0).split(' ', QString::SkipEmptyParts)) { if (col == colMax) break; columnModes[col++] = AbstractFileFilter::columnMode(valueString, dateTimeFormat, numberFormat); } for (int i = 0; i < linesToRead; ++i) { QString line = newData.at(i); if (simplifyWhitespacesEnabled) line = line.simplified(); if (line.isEmpty() || line.startsWith(commentCharacter)) // skip empty or commented lines continue; QLocale locale(numberFormat); QStringList lineStringList = line.split(' ', QString::SkipEmptyParts); if (createIndexEnabled) lineStringList.prepend(QString::number(i + 1)); QStringList lineString; for (int n = 0; n < lineStringList.size(); ++n) { if (n < lineStringList.size()) { QString valueString = lineStringList.at(n); switch (columnModes[n]) { case AbstractColumn::Numeric: { bool isNumber; const double value = locale.toDouble(valueString, &isNumber); lineString += QString::number(isNumber ? value : nanValue, 'g', 16); break; } case AbstractColumn::Integer: { bool isNumber; const int value = locale.toInt(valueString, &isNumber); lineString += QString::number(isNumber ? value : 0); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(valueString, dateTimeFormat); lineString += valueDateTime.isValid() ? valueDateTime.toString(dateTimeFormat) : QLatin1String(" "); break; } case AbstractColumn::Text: if (removeQuotesEnabled) valueString.remove(QRegExp("[\"\']")); lineString += valueString; break; case AbstractColumn::Month: // never happens case AbstractColumn::Day: break; } } else // missing columns in this line lineString += QLatin1String(""); } dataStrings << lineString; } return dataStrings; } /*! * generates the preview for the file \c fileName reading the provided number of \c lines. */ QVector AsciiFilterPrivate::preview(const QString& fileName, int lines) { QVector dataStrings; KFilterDev device(fileName); const int deviceError = prepareDeviceToRead(device); if (deviceError != 0) { DEBUG("Device error = " << deviceError); return dataStrings; } //number formatting DEBUG("locale = " << QLocale::languageToString(numberFormat).toStdString()); QLocale locale(numberFormat); // Read the data if (lines == -1) lines = m_actualRows; // set column names for preview if (!headerEnabled) { int start = 0; if (createIndexEnabled) start = 1; for (int i = start; i < m_actualCols; i++) vectorNames << "Column " + QString::number(i + 1); } QDEBUG(" column names = " << vectorNames); //skip data lines, if required DEBUG(" Skipping " << m_actualStartRow - 1 << " lines"); for (int i = 0; i < m_actualStartRow - 1; ++i) device.readLine(); DEBUG(" Generating preview for " << qMin(lines, m_actualRows) << " lines"); for (int i = 0; i < qMin(lines, m_actualRows); ++i) { QString line = device.readLine(); line.remove(QRegExp("[\\n\\r]")); // remove any newline if (simplifyWhitespacesEnabled) line = line.simplified(); if (line.isEmpty() || line.startsWith(commentCharacter)) // skip empty or commented lines continue; QStringList lineStringList = line.split(m_separator, (QString::SplitBehavior)skipEmptyParts); QDEBUG(" line = " << lineStringList); QStringList lineString; for (int n = 0; n < m_actualCols; ++n) { // index column if required if (n == 0 && createIndexEnabled) { lineString += QString::number(i + 1); continue; } if ((createIndexEnabled ? n - 1 : n) < lineStringList.size()) { QString valueString = lineStringList.at(createIndexEnabled ? n - 1 : n); //DEBUG(" valueString = " << valueString.toStdString()); if (skipEmptyParts && !QString::compare(valueString, " ")) // handle left white spaces continue; // set value depending on data type switch (columnModes[n]) { case AbstractColumn::Numeric: { bool isNumber; const double value = locale.toDouble(valueString, &isNumber); lineString += QString::number(isNumber ? value : nanValue, 'g', 15); break; } case AbstractColumn::Integer: { bool isNumber; const int value = locale.toInt(valueString, &isNumber); lineString += QString::number(isNumber ? value : 0); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(valueString, dateTimeFormat); lineString += valueDateTime.isValid() ? valueDateTime.toString(dateTimeFormat) : QLatin1String(" "); break; } case AbstractColumn::Text: if (removeQuotesEnabled) valueString.remove(QRegExp("[\"\']")); lineString += valueString; break; case AbstractColumn::Month: // never happens case AbstractColumn::Day: break; } } else // missing columns in this line lineString += QLatin1String(""); } dataStrings << lineString; } return dataStrings; } /*! writes the content of \c dataSource to the file \c fileName. */ void AsciiFilterPrivate::write(const QString & fileName, AbstractDataSource* dataSource) { Q_UNUSED(fileName); Q_UNUSED(dataSource); //TODO: save data to ascii file } //############################################################################## //################## Serialization/Deserialization ########################### //############################################################################## /*! Saves as XML. */ void AsciiFilter::save(QXmlStreamWriter* writer) const { writer->writeStartElement( "asciiFilter"); writer->writeAttribute( "commentCharacter", d->commentCharacter); writer->writeAttribute( "separatingCharacter", d->separatingCharacter); writer->writeAttribute( "autoMode", QString::number(d->autoModeEnabled)); writer->writeAttribute( "createIndex", QString::number(d->createIndexEnabled)); writer->writeAttribute( "header", QString::number(d->headerEnabled)); writer->writeAttribute( "vectorNames", d->vectorNames.join(' ')); writer->writeAttribute( "skipEmptyParts", QString::number(d->skipEmptyParts)); writer->writeAttribute( "simplifyWhitespaces", QString::number(d->simplifyWhitespacesEnabled)); writer->writeAttribute( "nanValue", QString::number(d->nanValue)); writer->writeAttribute( "removeQuotes", QString::number(d->removeQuotesEnabled)); writer->writeAttribute( "startRow", QString::number(d->startRow)); writer->writeAttribute( "endRow", QString::number(d->endRow)); writer->writeAttribute( "startColumn", QString::number(d->startColumn)); writer->writeAttribute( "endColumn", QString::number(d->endColumn)); writer->writeEndElement(); } /*! Loads from XML. */ bool AsciiFilter::load(XmlStreamReader* reader) { KLocalizedString attributeWarning = ki18n("Attribute '%1' missing or empty, default value is used"); QXmlStreamAttributes attribs = reader->attributes(); QString str = attribs.value("commentCharacter").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("commentCharacter").toString()); else d->commentCharacter = str; str = attribs.value("separatingCharacter").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("separatingCharacter").toString()); else d->separatingCharacter = str; str = attribs.value("createIndex").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("createIndex").toString()); else d->createIndexEnabled = str.toInt(); str = attribs.value("autoMode").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("autoMode").toString()); else d->autoModeEnabled = str.toInt(); str = attribs.value("header").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("header").toString()); else d->headerEnabled = str.toInt(); str = attribs.value("vectorNames").toString(); d->vectorNames = str.split(' '); //may be empty str = attribs.value("simplifyWhitespaces").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("simplifyWhitespaces").toString()); else d->simplifyWhitespacesEnabled = str.toInt(); str = attribs.value("nanValue").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("nanValue").toString()); else d->nanValue = str.toDouble(); str = attribs.value("removeQuotes").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("removeQuotes").toString()); else d->removeQuotesEnabled = str.toInt(); str = attribs.value("skipEmptyParts").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("skipEmptyParts").toString()); else d->skipEmptyParts = str.toInt(); str = attribs.value("startRow").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("startRow").toString()); else d->startRow = str.toInt(); str = attribs.value("endRow").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("endRow").toString()); else d->endRow = str.toInt(); str = attribs.value("startColumn").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("startColumn").toString()); else d->startColumn = str.toInt(); str = attribs.value("endColumn").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("endColumn").toString()); else d->endColumn = str.toInt(); return true; } int AsciiFilterPrivate::isPrepared() { return m_prepared; } #ifdef HAVE_MQTT /*! * \brief Offers a preview about the data received by the topic * \param list * \param message * \param topic */ void AsciiFilterPrivate::MQTTPreview(QVector& list, const QString& message, const QString& topic) { QVector dataStrings; if (!message.isEmpty()) { qDebug()<<"ascii mqtt preview for: " << topic; //check how many lines can be read int linesToRead = 0; QVector newData; QStringList newDataList = message.split(QRegExp("\n|\r\n|\r"), QString::SkipEmptyParts); for (const auto& valueString : newDataList) { const QStringList splitString = valueString.split(' ', QString::SkipEmptyParts); for (const auto& valueString2 : splitString) { if (!valueString2.isEmpty() && !valueString2.startsWith(commentCharacter) ) { linesToRead++; newData.push_back(valueString2); } } } qDebug() <<" data investigated, lines to read: "< list.size()) { linesToRead = list.size(); } } //Determine the number of columns int colSize = 0; if(list.isEmpty()) { if (createIndexEnabled) colSize = 2; else colSize = 1; } else colSize = columnModes.size() + 1; columnModes.resize(colSize); //if this is the first topic, we check if index has to be added if(list.isEmpty()) if (createIndexEnabled) { columnModes[0] = AbstractColumn::ColumnMode::Integer; vectorNames.prepend("index"); } vectorNames.append( topic); //Set the column mode for the topic columnModes[colSize - 1] = AbstractFileFilter::columnMode(newData[0], dateTimeFormat, numberFormat); //improve the column mode, based on the rest of the data for(int i = 0; i < linesToRead; i++) { QString tempLine = newData[i]; if (simplifyWhitespacesEnabled) tempLine = tempLine.simplified(); AbstractColumn::ColumnMode mode = AbstractFileFilter::columnMode(tempLine, dateTimeFormat, numberFormat); // numeric: integer -> numeric if (mode == AbstractColumn::Numeric && columnModes[colSize - 1] == AbstractColumn::Integer) { columnModes[colSize - 1] = mode; } // text: non text -> text if ( (mode == AbstractColumn::Text) && (columnModes[colSize - 1] != AbstractColumn::Text) ) { columnModes[colSize - 1] = mode; } } int forStart = 0; int forEnd = 0; if (list.isEmpty()) { forStart = 0; forEnd = linesToRead; } else { if (mqttPreviewFirstEmptyColCount == 0) { dataStrings = list; forStart = 0; forEnd = dataStrings.size(); } else { forStart = 1; forEnd = linesToRead; dataStrings = list; const QLocale locale(numberFormat); //this is the first column having values, after at least 1 empty column //until now only one row was filled with NaN values, so for starters we fill this row with the new value switch (columnModes[colSize - 1]) { case AbstractColumn::Numeric: { bool isNumber; const double value = locale.toDouble(newData[0], &isNumber); dataStrings[0] += QString::number(isNumber ? value : nanValue, 'g', 16); break; } case AbstractColumn::Integer: { bool isNumber; const int value = locale.toInt(newData[0], &isNumber); dataStrings[0] += QString::number(isNumber ? value : 0); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(newData[0], dateTimeFormat); dataStrings[0] += valueDateTime.isValid() ? valueDateTime.toString(dateTimeFormat) : QLatin1String(" "); break; } case AbstractColumn::Text: if (removeQuotesEnabled) newData[0].remove(QRegExp("[\"\']")); dataStrings[0] += newData[0]; break; case AbstractColumn::Month: // never happens case AbstractColumn::Day: break; } } } //We add every forEnd-forStart values to the list for (int i = forStart; i < forEnd; ++i) { QString line = newData[i]; if (simplifyWhitespacesEnabled) line = line.simplified(); //skip empty or commented lines if (line.isEmpty() || line.startsWith(commentCharacter)) continue; QLocale locale(numberFormat); QStringList lineString; //We don't have to do any preparation if there were already data containing topics if(!list.isEmpty() && mqttPreviewFirstEmptyColCount == 0) lineString = dataStrings[i]; //Add index if it is the case if(list.isEmpty() || (!list.isEmpty() && mqttPreviewFirstEmptyColCount > 0) ) if (createIndexEnabled) lineString += QString::number(i); //If this is the first not empty topic, add nan value to all the empty topics before this one if(!list.isEmpty() && mqttPreviewFirstEmptyColCount > 0){ for(int j = 0; j < mqttPreviewFirstEmptyColCount; j++) { lineString += QString::number(nanValue, 'g', 16); } } //Add the actual value to the topic switch (columnModes[colSize - 1]) { case AbstractColumn::Numeric: { bool isNumber; const double value = locale.toDouble(line, &isNumber); lineString += QString::number(isNumber ? value : nanValue, 'g', 16); break; } case AbstractColumn::Integer: { bool isNumber; const int value = locale.toInt(line, &isNumber); lineString += QString::number(isNumber ? value : 0); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(line, dateTimeFormat); lineString += valueDateTime.isValid() ? valueDateTime.toString(dateTimeFormat) : QLatin1String(" "); break; } case AbstractColumn::Text: if (removeQuotesEnabled) line.remove(QRegExp("[\"\']")); lineString += line; break; case AbstractColumn::Month: // never happens case AbstractColumn::Day: break; } qDebug()<<"column updated with value"; // if the list was empty, or this is the first not empty topic, we append the new line if(list.isEmpty() || (!list.isEmpty() && mqttPreviewFirstEmptyColCount > 0)) dataStrings << lineString; //Otherwise we update the already existing line if (!list.isEmpty() && mqttPreviewFirstEmptyColCount == 0) dataStrings[i] = lineString; } //The first empty topics were taken care of if(mqttPreviewFirstEmptyColCount > 0) { mqttPreviewFirstEmptyColCount = 0; } } //the message is empty and there were only empty messages before this else if (list.isEmpty() || (!list.isEmpty() && mqttPreviewFirstEmptyColCount > 0) ) { //increment the counter mqttPreviewFirstEmptyColCount ++; //determine number of columns int colSize; if (mqttPreviewFirstEmptyColCount == 1){ if (createIndexEnabled) colSize = 2; else colSize = 1; } else colSize = columnModes.size() + 1; columnModes.resize(colSize); //add index if needed if (mqttPreviewFirstEmptyColCount == 1) if (createIndexEnabled) { columnModes[0] = AbstractColumn::ColumnMode::Integer; vectorNames.prepend("index"); } //Add new column fot the mepty topic vectorNames.append( topic); columnModes[colSize-1] = AbstractColumn::ColumnMode::Numeric; //Add a NaN value to the topic's column QStringList lineString; if (mqttPreviewFirstEmptyColCount == 1) { //Add index if needed if(createIndexEnabled) lineString += QString::number(0); lineString += QString::number(nanValue, 'g', 16); //Append since this is the first empty column dataStrings << lineString; } else { //Update the already existing line dataStrings = list; dataStrings[0] += QString::number(nanValue, 'g', 16); } } //The message is empty but there already was a non empty message else if(!list.isEmpty()) { //Append vector name vectorNames.append( topic); int colSize = columnModes.size() + 1; columnModes.resize(colSize); columnModes[colSize-1] = AbstractColumn::ColumnMode::Numeric; dataStrings = list; //Add as many NaN values as many lines the list already has for (int i = 0; i < dataStrings.size(); ++i) { dataStrings[i] += QString::number(nanValue, 'g', 16); } } //update the list list = dataStrings; } /*! * \brief Returns the statistical data that is needed by the topic for its MQTTClient's will message * \param topic */ QString AsciiFilterPrivate::MQTTColumnStatistics(const MQTTTopic* topic) const{ Column* const tempColumn = topic->child(m_actualCols - 1); QString statistics; QVector willStatistics = topic->mqttClient()->willStatistics(); //Add every statistical data to the string, the flag of which is set true for(int i = 0; i <= willStatistics.size(); i++) { if(willStatistics[i]) { switch (static_cast(i) ) { case MQTTClient::WillStatistics::ArithmeticMean: statistics += QLatin1String("Arithmetic mean: ") + QString::number(tempColumn->statistics().arithmeticMean) + "\n"; break; case MQTTClient::WillStatistics::ContraharmonicMean: statistics += QLatin1String("Contraharmonic mean: ") + QString::number(tempColumn->statistics().contraharmonicMean) + "\n"; break; case MQTTClient::WillStatistics::Entropy: statistics += QLatin1String("Entropy: ") + QString::number(tempColumn->statistics().entropy) + "\n"; break; case MQTTClient::WillStatistics::GeometricMean: statistics += QLatin1String("Geometric mean: ") + QString::number(tempColumn->statistics().geometricMean) + "\n"; break; case MQTTClient::WillStatistics::HarmonicMean: statistics += QLatin1String("Harmonic mean: ") + QString::number(tempColumn->statistics().harmonicMean) + "\n"; break; case MQTTClient::WillStatistics::Kurtosis: statistics += QLatin1String("Kurtosis: ") + QString::number(tempColumn->statistics().kurtosis) + "\n"; break; case MQTTClient::WillStatistics::Maximum: statistics += QLatin1String("Maximum: ") + QString::number(tempColumn->statistics().maximum) + "\n"; break; case MQTTClient::WillStatistics::MeanDeviation: statistics += QLatin1String("Mean deviation: ") + QString::number(tempColumn->statistics().meanDeviation) + "\n"; break; case MQTTClient::WillStatistics::MeanDeviationAroundMedian: statistics += QLatin1String("Mean deviation around median: ") + QString::number(tempColumn->statistics().meanDeviationAroundMedian) + "\n"; break; case MQTTClient::WillStatistics::Median: statistics += QLatin1String("Median: ") + QString::number(tempColumn->statistics().median) + "\n"; break; case MQTTClient::WillStatistics::MedianDeviation: statistics += QLatin1String("Median deviation: ") + QString::number(tempColumn->statistics().medianDeviation) + "\n"; break; case MQTTClient::WillStatistics::Minimum: statistics += QLatin1String("Minimum: ") + QString::number(tempColumn->statistics().minimum) + "\n"; break; case MQTTClient::WillStatistics::Skewness: statistics += QLatin1String("Skewness: ") + QString::number(tempColumn->statistics().skewness) + "\n"; break; case MQTTClient::WillStatistics::StandardDeviation: statistics += QLatin1String("Standard deviation: ") + QString::number(tempColumn->statistics().standardDeviation) + "\n"; break; case MQTTClient::WillStatistics::Variance: statistics += QLatin1String("Variance: ") + QString::number(tempColumn->statistics().variance) + "\n"; break; default: break; } } } return statistics; } AbstractColumn::ColumnMode AsciiFilterPrivate::MQTTColumnMode() const{ return columnModes[m_actualCols - 1]; } /*! * \brief reads the content of a message received by the topic. * Uses the settings defined in the MQTTTopic's MQTTClient * \param message * \param topic * \param dataSource */ void AsciiFilterPrivate::readMQTTTopic(const QString& message, const QString& topic, AbstractDataSource*dataSource) { //If the message is empty, there is nothing to do if (message.isEmpty()) { DEBUG("No new data available"); return; } MQTTTopic* spreadsheet = dynamic_cast(dataSource); const int keepNValues = spreadsheet->keepNValues(); if (!m_prepared) { qDebug()<<"Start preparing filter for: " << spreadsheet->topicName(); //Prepare the filter const int mqttPrepareError = prepareMQTTTopicToRead(message, topic); if (mqttPrepareError != 0) { DEBUG("Mqtt Prepare Error = " << mqttPrepareError); qDebug()<setUndoAware(false); spreadsheet->resize(AbstractFileFilter::Replace, vectorNames, m_actualCols); qDebug() << "fds resized to col: " << m_actualCols; qDebug() << "fds rowCount: " << spreadsheet->rowCount(); //columns in a MQTTTopic don't have any manual changes. //make the available columns undo unaware and suppress the "data changed" signal. //data changes will be propagated via an explicit Column::setChanged() call once new data was read. for (int i = 0; i < spreadsheet->childCount(); i++) { spreadsheet->child(i)->setUndoAware(false); spreadsheet->child(i)->setSuppressDataChangedSignal(true); } if (keepNValues == 0) spreadsheet->setRowCount(m_actualRows > 1 ? m_actualRows : 1); else { spreadsheet->setRowCount(spreadsheet->keepNValues()); m_actualRows = spreadsheet->keepNValues(); } m_dataContainer.resize(m_actualCols); for (int n = 0; n < m_actualCols; ++n) { // data() returns a void* which is a pointer to any data type (see ColumnPrivate.cpp) spreadsheet->child(n)->setColumnMode(columnModes[n]); switch (columnModes[n]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } } #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportTotal: "); #endif MQTTClient::ReadingType readingType; if (!m_prepared) { //if filter is not prepared we read till the end readingType = MQTTClient::ReadingType::TillEnd; } else { //we have to read all the data when reading from end //so we set readingType to TillEnd if (spreadsheet->readingType() == MQTTClient::ReadingType::FromEnd) { readingType = MQTTClient::ReadingType::TillEnd; } else { readingType = static_cast(spreadsheet->readingType()); } } //count the new lines, increase actualrows on each //now we read all the new lines, if we want to use sample rate //then here we can do it, if we have actually sample rate number of lines :-? int newLinesForSampleSizeNotTillEnd = 0; int newLinesTillEnd = 0; QVector newData; if (readingType != MQTTClient::ReadingType::TillEnd) { newData.reserve(spreadsheet->sampleSize()); newData.resize(spreadsheet->sampleSize()); } int newDataIdx = 0; bool sampleSizeReached = false; { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportReadingFromFile: "); #endif QStringList newDataList = message.split(QRegExp("\n|\r\n|\r"), QString::SkipEmptyParts); for (auto& valueString : newDataList) { if(!valueString.startsWith(commentCharacter)) { const QStringList splitString = valueString.split(m_separator, static_cast(skipEmptyParts)); for (const auto& valueString2 : splitString) { if (!valueString2.isEmpty() && !valueString2.startsWith(commentCharacter)) { if (readingType != MQTTClient::ReadingType::TillEnd) newData[newDataIdx++] = valueString2; else newData.push_back(valueString2); newLinesTillEnd++; if (readingType != MQTTClient::ReadingType::TillEnd) { newLinesForSampleSizeNotTillEnd++; //for Continuous reading and FromEnd we read sample rate number of lines if possible if (newLinesForSampleSizeNotTillEnd == spreadsheet->sampleSize()) { sampleSizeReached = true; break; } } } } //if the sample size limit is reached we brake from the outer loop as well if(sampleSizeReached) break; } } } qDebug()<<"Processing message done"; //now we reset the readingType if (static_cast(spreadsheet->readingType()) == MQTTClient::ReadingType::FromEnd) readingType = static_cast(spreadsheet->readingType()); //we had less new lines than the sample rate specified if (readingType != MQTTClient::ReadingType::TillEnd) qDebug() << "Removed empty lines: " << newData.removeAll(""); qDebug()<<"Create index enabled: "<rowCount(); if(m_prepared ) { if (keepNValues == 0) m_actualRows = spreadsheetRowCountBeforeResize; else { //if the keepNValues changed since the last read we have to manage the columns accordingly if(m_actualRows != spreadsheet->keepNValues()) { if(m_actualRows < spreadsheet->keepNValues()) { spreadsheet->setRowCount(spreadsheet->keepNValues()); qDebug()<<"rowcount set to: " << spreadsheet->keepNValues(); } //Calculate the difference between the old and new keepNValues int rowDiff = 0; if(m_actualRows > spreadsheet->keepNValues()) { rowDiff = m_actualRows - spreadsheet->keepNValues(); } if(m_actualRows < spreadsheet->keepNValues()) { rowDiff =spreadsheet->keepNValues() - m_actualRows; } for (int n = 0; n < columnModes.size(); ++n) { // data() returns a void* which is a pointer to any data type (see ColumnPrivate.cpp) switch (columnModes[n]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); m_dataContainer[n] = static_cast(vector); //if the keepNValues got smaller then we move the last keepNValues count of data //in the first keepNValues places if(m_actualRows > spreadsheet->keepNValues()) { for(int i = 0; i < spreadsheet->keepNValues(); i++) { static_cast*>(m_dataContainer[n])->operator[] (i) = static_cast*>(m_dataContainer[n])->operator[](m_actualRows - spreadsheet->keepNValues() + i); } } //if the keepNValues got bigger we move the existing values to the last m_actualRows positions //then fill the remaining lines with NaN if(m_actualRows < spreadsheet->keepNValues()) { vector->reserve( spreadsheet->keepNValues()); vector->resize( spreadsheet->keepNValues()); for(int i = 1; i <= m_actualRows; i++) { static_cast*>(m_dataContainer[n])->operator[] (spreadsheet->keepNValues() - i) = static_cast*>(m_dataContainer[n])->operator[](spreadsheet->keepNValues() - i - rowDiff); } for(int i = 0; i < rowDiff; i++) { static_cast*>(m_dataContainer[n])->operator[](i) = nanValue; } } break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); m_dataContainer[n] = static_cast(vector); //if the keepNValues got smaller then we move the last keepNValues count of data //in the first keepNValues places if(m_actualRows > spreadsheet->keepNValues()) { for(int i = 0; i < spreadsheet->keepNValues(); i++) { static_cast*>(m_dataContainer[n])->operator[] (i) = static_cast*>(m_dataContainer[n])->operator[](m_actualRows - spreadsheet->keepNValues() + i); } } //if the keepNValues got bigger we move the existing values to the last m_actualRows positions //then fill the remaining lines with 0 if(m_actualRows < spreadsheet->keepNValues()) { vector->reserve( spreadsheet->keepNValues()); vector->resize( spreadsheet->keepNValues()); for(int i = 1; i <= m_actualRows; i++) { static_cast*>(m_dataContainer[n])->operator[] (spreadsheet->keepNValues() - i) = static_cast*>(m_dataContainer[n])->operator[](spreadsheet->keepNValues() - i - rowDiff); } for(int i = 0; i < rowDiff; i++){ static_cast*>(m_dataContainer[n])->operator[](i) = 0; } } break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(spreadsheet->child(n)->data()); m_dataContainer[n] = static_cast(vector); //if the keepNValues got smaller then we move the last keepNValues count of data //in the first keepNValues places if(m_actualRows > spreadsheet->keepNValues()) { for(int i = 0; i < spreadsheet->keepNValues(); i++) { static_cast*>(m_dataContainer[n])->operator[] (i) = static_cast*>(m_dataContainer[n])->operator[](m_actualRows - spreadsheet->keepNValues() + i); } } //if the keepNValues got bigger we move the existing values to the last m_actualRows positions //then fill the remaining lines with "" if(m_actualRows < spreadsheet->keepNValues()) { vector->reserve( spreadsheet->keepNValues()); vector->resize( spreadsheet->keepNValues()); for(int i = 1; i <= m_actualRows; i++) { static_cast*>(m_dataContainer[n])->operator[] (spreadsheet->keepNValues() - i) = static_cast*>(m_dataContainer[n])->operator[](spreadsheet->keepNValues() - i - rowDiff); } for(int i = 0; i < rowDiff; i++) static_cast*>(m_dataContainer[n])->operator[](i) = ""; } break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); m_dataContainer[n] = static_cast(vector); //if the keepNValues got smaller then we move the last keepNValues count of data //in the first keepNValues places if(m_actualRows > spreadsheet->keepNValues()) { for(int i = 0; i < spreadsheet->keepNValues(); i++) { static_cast*>(m_dataContainer[n])->operator[] (i) = static_cast*>(m_dataContainer[n])->operator[](m_actualRows - spreadsheet->keepNValues() + i); } } //if the keepNValues got bigger we move the existing values to the last m_actualRows positions //then fill the remaining lines with null datetime if(m_actualRows < spreadsheet->keepNValues()) { vector->reserve( spreadsheet->keepNValues()); vector->resize( spreadsheet->keepNValues()); for(int i = 1; i <= m_actualRows; i++) { static_cast*>(m_dataContainer[n])->operator[] (spreadsheet->keepNValues() - i) = static_cast*>(m_dataContainer[n])->operator[](spreadsheet->keepNValues() - i - rowDiff); } for(int i = 0; i < rowDiff; i++) static_cast*>(m_dataContainer[n])->operator[](i) = QDateTime(); } break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } //if the keepNValues got smaller resize the spreadsheet if(m_actualRows > spreadsheet->keepNValues()) spreadsheet->setRowCount(spreadsheet->keepNValues()); //set the new row count m_actualRows = spreadsheet->keepNValues(); qDebug()<<"actual rows: "<sampleSize()); else { m_actualRows += newData.size(); } } //fixed size if (keepNValues != 0) { if (readingType == MQTTClient::ReadingType::TillEnd) { //we had more lines than the fixed size, so we read m_actualRows number of lines if (newLinesTillEnd > m_actualRows) { linesToRead = m_actualRows; } else linesToRead = newLinesTillEnd; } else { //we read max sample size number of lines when the reading mode //is ContinuouslyFixed or FromEnd if(spreadsheet->sampleSize() <= spreadsheet->keepNValues()) linesToRead = qMin(spreadsheet->sampleSize(), newLinesTillEnd); else linesToRead = qMin(spreadsheet->keepNValues(), newLinesTillEnd); } } else { linesToRead = m_actualRows - spreadsheetRowCountBeforeResize; } if (linesToRead == 0) return; } else { if(keepNValues != 0) { linesToRead = newLinesTillEnd > m_actualRows ? m_actualRows : newLinesTillEnd; } else { linesToRead = newLinesTillEnd; } } qDebug()<<"linestoread = " << linesToRead; //new rows/resize columns if we don't have a fixed size if (keepNValues == 0) { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportResizing: "); #endif if (spreadsheet->rowCount() < m_actualRows) spreadsheet->setRowCount(m_actualRows); if (!m_prepared) currentRow = 0; else { // indexes the position in the vector(column) currentRow = spreadsheetRowCountBeforeResize; } // if we have fixed size, we do this only once in preparation, here we can use // m_prepared and we need something to decide whether it has a fixed size or increasing for (int n = 0; n < m_actualCols; ++n) { // data() returns a void* which is a pointer to any data type (see ColumnPrivate.cpp) switch (columnModes[n]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(spreadsheet->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } } else { //when we have a fixed size we have to pop sampleSize number of lines if specified //here popping, setting currentRow if (!m_prepared) currentRow = m_actualRows - qMin(newLinesTillEnd, m_actualRows); else { if (readingType == MQTTClient::ReadingType::TillEnd) { if (newLinesTillEnd > m_actualRows) currentRow = 0; else { currentRow = m_actualRows - newLinesTillEnd; } } else { //we read max sample rate number of lines when the reading mode //is ContinuouslyFixed or FromEnd currentRow = m_actualRows - linesToRead; } } if (m_prepared) { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportPopping: "); #endif for (int row = 0; row < linesToRead; ++row) { for (int col = 0; col < m_actualCols; ++col) { switch (columnModes[col]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(spreadsheet->child(col)->data()); vector->pop_front(); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(spreadsheet->child(col)->data()); vector->pop_front(); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(spreadsheet->child(col)->data()); vector->pop_front(); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(spreadsheet->child(col)->data()); vector->pop_front(); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[col] = static_cast(vector); break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } } } } // from the last row we read the new data in the spreadsheet qDebug() << "reading from line: " << currentRow << " lines till end: " << newLinesTillEnd; qDebug() << "Lines to read: " << linesToRead <<" actual rows: " << m_actualRows; newDataIdx = 0; //From end means that we read the last sample size amount of data if (readingType == MQTTClient::ReadingType::FromEnd) { if (m_prepared) { if (newData.size() > spreadsheet->sampleSize()) newDataIdx = newData.size() - spreadsheet->sampleSize(); } } qDebug() << "newDataIdx: " << newDataIdx; static int indexColumnIdx = 0; { #ifdef PERFTRACE_LIVE_IMPORT PERFTRACE("AsciiLiveDataImportFillingContainers: "); #endif int row = 0; for (; row < linesToRead; ++row) { QString line; if (readingType == MQTTClient::ReadingType::FromEnd) line = newData.at(newDataIdx++); else line = newData.at(row); if (simplifyWhitespacesEnabled) line = line.simplified(); if (line.isEmpty() || line.startsWith(commentCharacter)) { qDebug()<<"found empty line "<*>(m_dataContainer[0])->operator[](currentRow) = (isNumber ? value : nanValue); break; } case AbstractColumn::Integer: { bool isNumber; const int value = locale.toInt(tempIndex, &isNumber); static_cast*>(m_dataContainer[0])->operator[](currentRow) = (isNumber ? value : 0); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(tempIndex, dateTimeFormat); static_cast*>(m_dataContainer[0])->operator[](currentRow) = valueDateTime.isValid() ? valueDateTime : QDateTime(); break; } case AbstractColumn::Text: if (removeQuotesEnabled) tempIndex.remove(QRegExp("[\"\']")); static_cast*>(m_dataContainer[0])->operator[](currentRow) = tempIndex; break; case AbstractColumn::Month: //TODO break; case AbstractColumn::Day: //TODO break; } } //setting timestamp on current time static_cast*>(m_dataContainer[col])->operator[](currentRow) = QDateTime::currentDateTime(); QString valueString = line; qDebug() << "putting in " << valueString << " in line: " << currentRow; // set value depending on data type switch (columnModes[m_actualCols - 1]) { case AbstractColumn::Numeric: { bool isNumber; const double value = locale.toDouble(valueString, &isNumber); static_cast*>(m_dataContainer[m_actualCols - 1])->operator[](currentRow) = (isNumber ? value : nanValue); break; } case AbstractColumn::Integer: { bool isNumber; const int value = locale.toInt(valueString, &isNumber); static_cast*>(m_dataContainer[m_actualCols - 1])->operator[](currentRow) = (isNumber ? value : 0); break; } case AbstractColumn::DateTime: { const QDateTime valueDateTime = QDateTime::fromString(valueString, dateTimeFormat); static_cast*>(m_dataContainer[m_actualCols - 1])->operator[](currentRow) = valueDateTime.isValid() ? valueDateTime : QDateTime(); break; } case AbstractColumn::Text: if (removeQuotesEnabled) valueString.remove(QRegExp("[\"\']")); static_cast*>(m_dataContainer[m_actualCols - 1])->operator[](currentRow) = valueString; break; case AbstractColumn::Month: //TODO break; case AbstractColumn::Day: //TODO break; } currentRow++; } } if (m_prepared) { qDebug()<<"notifying plots"; //notify all affected columns and plots about the changes PERFTRACE("AsciiLiveDataImport, notify affected columns and plots"); const Project* project = spreadsheet->project(); QVector curves = project->children(AbstractAspect::Recursive); QVector plots; qDebug()<<"Uploading plots"; for (int n = 0; n < m_actualCols; ++n) { Column* column = spreadsheet->column(n); //determine the plots where the column is consumed for (const auto* curve : curves) { if (curve->xColumn() == column || curve->yColumn() == column) { CartesianPlot* plot = dynamic_cast(curve->parentAspect()); if (plots.indexOf(plot) == -1) { plots << plot; plot->setSuppressDataChangedSignal(true); } } } column->setChanged(); } //loop over all affected plots and retransform them for (auto* const plot : plots) { //TODO setting this back to true triggers again a lot of retransforms in the plot (one for each curve). // plot->setSuppressDataChangedSignal(false); plot->dataChanged(); } } //Set prepared true if needed if(!m_prepared) m_prepared = true; } /*! * \brief Prepares the filter to read from messages received by the MQTTTopic. * Returns whether the preparation was successful or not * \param message * \param topic */ int AsciiFilterPrivate::prepareMQTTTopicToRead(const QString& message, const QString& topic) { Q_UNUSED(topic) vectorNames.append("value"); if (endColumn == -1) endColumn = 1; else endColumn++; vectorNames.prepend("timestamp"); endColumn++; if (createIndexEnabled) { vectorNames.prepend("index"); endColumn++; } m_actualCols = endColumn - startColumn + 1; qDebug()<<"actual cols: "<(skipEmptyParts)); if (!firstLineStringList.isEmpty()) { int length1 = firstLineStringList.at(0).length(); if (firstLineStringList.size() > 1) { int pos2 = firstLine.indexOf(firstLineStringList.at(1), length1); m_separator = firstLine.mid(length1, pos2 - length1); } else { //old: separator = line.right(line.length() - length1); m_separator = ' '; } } } else { // use given separator // replace symbolic "TAB" with '\t' m_separator = separatingCharacter.replace(QLatin1String("2xTAB"), "\t\t", Qt::CaseInsensitive); m_separator = separatingCharacter.replace(QLatin1String("TAB"), "\t", Qt::CaseInsensitive); // replace symbolic "SPACE" with ' ' m_separator = m_separator.replace(QLatin1String("2xSPACE"), QLatin1String(" "), Qt::CaseInsensitive); m_separator = m_separator.replace(QLatin1String("3xSPACE"), QLatin1String(" "), Qt::CaseInsensitive); m_separator = m_separator.replace(QLatin1String("4xSPACE"), QLatin1String(" "), Qt::CaseInsensitive); m_separator = m_separator.replace(QLatin1String("SPACE"), QLatin1String(" "), Qt::CaseInsensitive); firstLineStringList = firstLine.split(m_separator, (QString::SplitBehavior)skipEmptyParts); } DEBUG("separator: \'" << m_separator.toStdString() << '\''); DEBUG("number of columns: " << firstLineStringList.size()); QDEBUG("first line: " << firstLineStringList); DEBUG("headerEnabled = " << headerEnabled); // parse first data line to determine data type for each column columnModes.resize(m_actualCols); int col = 0; if (createIndexEnabled) { columnModes[0] = AbstractColumn::Integer; col = 1; } //set column mode for timestamp columnModes[col] = AbstractColumn::DateTime; col++; auto firstValue = firstLineStringList.takeFirst();//use first value to identify column mode while(firstValue.isEmpty() || firstValue.startsWith(commentCharacter) )//get the first usable value firstValue = firstLineStringList.takeFirst(); if (simplifyWhitespacesEnabled) firstValue = firstValue.simplified(); columnModes[m_actualCols-1] = AbstractFileFilter::columnMode(firstValue, dateTimeFormat, numberFormat); //Improve the column mode based on the other values from the first line for (auto& valueString : firstLineStringList) { if(!valueString.isEmpty() && !valueString.startsWith(commentCharacter) ){ if (createIndexEnabled) col = 1; else col = 0; if (simplifyWhitespacesEnabled) valueString = valueString.simplified(); AbstractColumn::ColumnMode mode = AbstractFileFilter::columnMode(valueString, dateTimeFormat, numberFormat); // numeric: integer -> numeric if (mode == AbstractColumn::Numeric && columnModes[m_actualCols-1] == AbstractColumn::Integer) { columnModes[m_actualCols-1] = mode; } // text: non text -> text if (mode == AbstractColumn::Text && columnModes[m_actualCols-1] != AbstractColumn::Text) { columnModes[m_actualCols-1] = mode; } } } //Improve the column mode based on the remaining values for (const auto& valueString : lineList) { QStringList splitString = valueString.split(m_separator, (QString::SplitBehavior)skipEmptyParts); for (auto& valueString2 : splitString) { if (!valueString2.isEmpty() && !valueString2.startsWith(commentCharacter)) { if (simplifyWhitespacesEnabled) valueString2 = valueString2.simplified(); AbstractColumn::ColumnMode mode = AbstractFileFilter::columnMode(valueString2, dateTimeFormat, numberFormat); // numeric: integer -> numeric if (mode == AbstractColumn::Numeric && columnModes[m_actualCols-1] == AbstractColumn::Integer) { columnModes[m_actualCols-1] = mode; } // text: non text -> text if (mode == AbstractColumn::Text && columnModes[m_actualCols-1] != AbstractColumn::Text) { columnModes[m_actualCols-1] = mode; } } } } //determine rowcount int tempRowCount = 0; QStringList newDataList = message.split(QRegExp("\n|\r\n|\r"), QString::SkipEmptyParts); for (const auto& valueString : newDataList) { if(!valueString.startsWith(commentCharacter)) { QStringList splitString = valueString.split(m_separator, static_cast(skipEmptyParts)); for (const auto& valueString2 : splitString) { if (!valueString2.isEmpty() && !valueString2.startsWith(commentCharacter)) { tempRowCount ++; } } } } QDEBUG("column modes = " << columnModes); m_actualRows = tempRowCount; ///////////////////////////////////////////////////////////////// int actualEndRow = endRow; DEBUG("endRow = " << endRow); if (endRow == -1 || endRow > m_actualRows) actualEndRow = m_actualRows; if (m_actualRows > actualEndRow) m_actualRows = actualEndRow; DEBUG("start/end column: " << startColumn << ' ' << endColumn); DEBUG("start/end row: " << m_actualStartRow << ' ' << actualEndRow); DEBUG("actual cols/rows (w/o header incl. start rows): " << m_actualCols << ' ' << m_actualRows); return 0; } /*! * \brief After the MQTTTopic was loaded, the filter is prepared for reading * \param prepared * \param topic * \param separator */ void AsciiFilterPrivate::setPreparedForMQTT(bool prepared, MQTTTopic* topic, const QString& separator) { m_prepared = prepared; //If originally it was prepared we have to restore the settings if(prepared) { m_separator = separator; m_actualCols = endColumn - startColumn + 1; m_actualRows = topic->rowCount(); //set the column modes columnModes.resize(topic->columnCount()); for(int i = 0; i < topic->columnCount(); ++i) { columnModes[i] = topic->column(i)->columnMode(); } //set the data containers m_dataContainer.resize(m_actualCols); for (int n = 0; n < m_actualCols; ++n) { // data() returns a void* which is a pointer to any data type (see ColumnPrivate.cpp) topic->child(n)->setColumnMode(columnModes[n]); switch (columnModes[n]) { case AbstractColumn::Numeric: { QVector* vector = static_cast* >(topic->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Integer: { QVector* vector = static_cast* >(topic->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::Text: { QVector* vector = static_cast*>(topic->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } case AbstractColumn::DateTime: { QVector* vector = static_cast* >(topic->child(n)->data()); vector->reserve(m_actualRows); vector->resize(m_actualRows); m_dataContainer[n] = static_cast(vector); break; } //TODO case AbstractColumn::Month: case AbstractColumn::Day: break; } } } } #endif /*! * \brief Returns the separator used by the filter * \return */ QString AsciiFilterPrivate::separator() const { return m_separator; } diff --git a/src/kdefrontend/datasources/ImportFileWidget.cpp b/src/kdefrontend/datasources/ImportFileWidget.cpp index ddbd61348..8b6283b1e 100644 --- a/src/kdefrontend/datasources/ImportFileWidget.cpp +++ b/src/kdefrontend/datasources/ImportFileWidget.cpp @@ -1,2914 +1,2914 @@ /*************************************************************************** File : ImportFileWidget.cpp Project : LabPlot Description : import file data widget -------------------------------------------------------------------- Copyright : (C) 2009-2018 Stefan Gerlach (stefan.gerlach@uni.kn) Copyright : (C) 2009-2018 Alexander Semke (alexander.semke@web.de) Copyright : (C) 2017-2018 Fabian Kristof (fkristofszabolcs@gmail.com) Copyright : (C) 2018 Kovacs Ferencz (kferike98@gmail.com) ***************************************************************************/ /*************************************************************************** * * * This program is free software; you can redistribute it and/or modify * * it under the terms of the GNU General Public License as published by * * the Free Software Foundation; either version 2 of the License, or * * (at your option) any later version. * * * * This program is distributed in the hope that it will be useful, * * but WITHOUT ANY WARRANTY; without even the implied warranty of * * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * * GNU General Public License for more details. * * * * You should have received a copy of the GNU General Public License * * along with this program; if not, write to the Free Software * * Foundation, Inc., 51 Franklin Street, Fifth Floor, * * Boston, MA 02110-1301 USA * * * ***************************************************************************/ #include "ImportFileWidget.h" #include "FileInfoDialog.h" #include "backend/datasources/filters/AsciiFilter.h" #include "backend/datasources/filters/BinaryFilter.h" #include "backend/datasources/filters/HDF5Filter.h" #include "backend/datasources/filters/NetCDFFilter.h" #include "backend/datasources/filters/ImageFilter.h" #include "backend/datasources/filters/FITSFilter.h" #include "backend/datasources/filters/JsonFilter.h" #include "backend/datasources/filters/QJsonModel.h" #include "backend/datasources/filters/NgspiceRawAsciiFilter.h" #include "backend/datasources/filters/NgspiceRawBinaryFilter.h" #include "backend/datasources/filters/ROOTFilter.h" #include "AsciiOptionsWidget.h" #include "BinaryOptionsWidget.h" #include "HDF5OptionsWidget.h" #include "ImageOptionsWidget.h" #include "NetCDFOptionsWidget.h" #include "FITSOptionsWidget.h" #include "JsonOptionsWidget.h" #include "ROOTOptionsWidget.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_MQTT #include "kdefrontend/widgets/MQTTWillSettingsWidget.h" #include "MQTTConnectionManagerDialog.h" #include "MQTTConnectionManagerWidget.h" #include "backend/core/Project.h" #include #include #include #include #include #include #include #endif /*! \class ImportFileWidget \brief Widget for importing data from a file. \ingroup kdefrontend */ ImportFileWidget::ImportFileWidget(QWidget* parent, bool liveDataSource, const QString& fileName) : QWidget(parent), m_asciiOptionsWidget(nullptr), m_binaryOptionsWidget(nullptr), m_hdf5OptionsWidget(nullptr), m_imageOptionsWidget(nullptr), m_netcdfOptionsWidget(nullptr), m_fitsOptionsWidget(nullptr), m_jsonOptionsWidget(nullptr), m_rootOptionsWidget(nullptr), m_fileName(fileName), m_fileEmpty(false), m_liveDataSource(liveDataSource), m_suppressRefresh(false) #ifdef HAVE_MQTT , m_client(nullptr), m_searching(false), m_searchTimer(new QTimer(this)), m_connectTimeoutTimer(new QTimer(this)), m_mqttReadyForPreview (false), m_initialisingMQTT(false), m_connectionTimedOut(false) #endif { ui.setupUi(this); auto* completer = new QCompleter(this); completer->setModel(new QDirModel); ui.leFileName->setCompleter(completer); //add supported file types if (!liveDataSource) { ui.cbFileType->addItem(i18n("ASCII data"), AbstractFileFilter::Ascii); ui.cbFileType->addItem(i18n("Binary data"), AbstractFileFilter::Binary); ui.cbFileType->addItem(i18n("Image"), AbstractFileFilter::Image); #ifdef HAVE_HDF5 ui.cbFileType->addItem(i18n("Hierarchical Data Format 5 (HDF5)"), AbstractFileFilter::HDF5); #endif #ifdef HAVE_NETCDF ui.cbFileType->addItem(i18n("Network Common Data Format (NetCDF)"), AbstractFileFilter::NETCDF); #endif #ifdef HAVE_FITS ui.cbFileType->addItem(i18n("Flexible Image Transport System Data Format (FITS)"), AbstractFileFilter::FITS); #endif ui.cbFileType->addItem(i18n("JSON data"), AbstractFileFilter::JSON); #ifdef HAVE_ZIP ui.cbFileType->addItem(i18n("ROOT (CERN) Histograms"), AbstractFileFilter::ROOT); #endif ui.cbFileType->addItem(i18n("Ngspice RAW ASCII"), AbstractFileFilter::NgspiceRawAscii); ui.cbFileType->addItem(i18n("Ngspice RAW Binary"), AbstractFileFilter::NgspiceRawBinary); //hide widgets relevant for live data reading only ui.lSourceType->hide(); ui.cbSourceType->hide(); ui.gbUpdateOptions->hide(); } else { ui.cbFileType->addItem(i18n("ASCII data"), AbstractFileFilter::Ascii); ui.cbFileType->addItem(i18n("Binary data"), AbstractFileFilter::Binary); ui.cbFileType->addItem(i18n("Ngspice RAW ASCII"), AbstractFileFilter::NgspiceRawAscii); ui.cbFileType->addItem(i18n("Ngspice RAW Binary"), AbstractFileFilter::NgspiceRawBinary); ui.lePort->setValidator( new QIntValidator(ui.lePort) ); ui.cbBaudRate->addItems(LiveDataSource::supportedBaudRates()); ui.cbSerialPort->addItems(LiveDataSource::availablePorts()); ui.tabWidget->removeTab(2); #ifdef HAVE_MQTT m_searchTimer->setInterval(10000); m_connectTimeoutTimer->setInterval(6000); #endif } QStringList filterItems {i18n("Automatic"), i18n("Custom")}; ui.cbFilter->addItems(filterItems); //hide options that will be activated on demand ui.gbOptions->hide(); ui.gbUpdateOptions->hide(); setMQTTVisible(false); ui.cbReadingType->addItem(i18n("Whole file"), LiveDataSource::WholeFile); ui.bOpen->setIcon( QIcon::fromTheme(QLatin1String("document-open")) ); ui.bFileInfo->setIcon( QIcon::fromTheme(QLatin1String("help-about")) ); ui.bManageFilters->setIcon( QIcon::fromTheme(QLatin1String("configure")) ); ui.bSaveFilter->setIcon( QIcon::fromTheme(QLatin1String("document-save")) ); ui.bRefreshPreview->setIcon( QIcon::fromTheme(QLatin1String("view-refresh")) ); ui.tvJson->header()->setSectionResizeMode(QHeaderView::ResizeToContents); ui.tvJson->setAlternatingRowColors(true); showJsonModel(false); // the table widget for preview m_twPreview = new QTableWidget(ui.tePreview); m_twPreview->verticalHeader()->hide(); m_twPreview->setEditTriggers(QTableWidget::NoEditTriggers); auto* layout = new QHBoxLayout; layout->addWidget(m_twPreview); ui.tePreview->setLayout(layout); m_twPreview->hide(); #ifdef HAVE_MQTT ui.cbSourceType->addItem(QLatin1String("MQTT")); m_configPath = QStandardPaths::standardLocations(QStandardPaths::AppDataLocation).constFirst() + QLatin1String("MQTT_connections"); m_willSettings.MQTTUseWill = false; m_willSettings.willRetain = false; m_willSettings.willQoS = 0; m_willSettings.willMessageType = MQTTClient::WillMessageType::OwnMessage; m_willSettings.willUpdateType = MQTTClient::WillUpdateType::TimePeriod; m_willSettings.willTimeInterval = 10000; m_willSettings.willStatistics.fill(false, 15); ui.bSubscribe->setIcon(ui.bSubscribe->style()->standardIcon(QStyle::SP_ArrowRight)); ui.bSubscribe->setToolTip(i18n("Subscribe selected topics")); ui.bUnsubscribe->setIcon(ui.bUnsubscribe->style()->standardIcon(QStyle::SP_ArrowLeft)); ui.bUnsubscribe->setToolTip(i18n("Unsubscribe selected topics")); ui.bManageConnections->setIcon(QIcon::fromTheme(QLatin1String("network-server"))); ui.bManageConnections->setToolTip(i18n("Manage MQTT connections")); ui.bWillMessage->setEnabled(false); ui.bWillMessage->setToolTip(i18n("Manage MQTT connection's will settings")); ui.bWillMessage->setIcon(ui.bWillMessage->style()->standardIcon(QStyle::SP_FileDialogDetailedView)); #endif //TODO: implement save/load of user-defined settings later and activate these buttons again ui.bSaveFilter->hide(); ui.bManageFilters->hide(); } void ImportFileWidget::loadSettings() { m_suppressRefresh = true; //load last used settings QString confName; if (m_liveDataSource) confName = QLatin1String("LiveDataImport"); else confName = QLatin1String("FileImport"); KConfigGroup conf(KSharedConfig::openConfig(), confName); //read the source type first since settings in fileNameChanged() depend on this ui.cbSourceType->setCurrentIndex(conf.readEntry("SourceType").toInt()); //general settings AbstractFileFilter::FileType fileType = static_cast(conf.readEntry("Type", 0)); for (int i = 0; i < ui.cbFileType->count(); ++i) { AbstractFileFilter::FileType itemFileType = static_cast(ui.cbFileType->itemData(i).toInt()); if (itemFileType == fileType) { ui.cbFileType->setCurrentIndex(i); break; } } ui.cbFilter->setCurrentIndex(conf.readEntry("Filter", 0)); filterChanged(ui.cbFilter->currentIndex()); // needed if filter is not changed if (m_fileName.isEmpty()) ui.leFileName->setText(conf.readEntry("LastImportedFile", "")); else ui.leFileName->setText(m_fileName); //live data related settings ui.cbBaudRate->setCurrentIndex(conf.readEntry("BaudRate").toInt()); ui.cbReadingType->setCurrentIndex(conf.readEntry("ReadingType").toInt()); ui.cbSerialPort->setCurrentIndex(conf.readEntry("SerialPort").toInt()); ui.cbUpdateType->setCurrentIndex(conf.readEntry("UpdateType").toInt()); ui.leHost->setText(conf.readEntry("Host","")); ui.sbKeepNValues->setValue(conf.readEntry("KeepNValues").toInt()); ui.lePort->setText(conf.readEntry("Port","")); ui.sbSampleSize->setValue(conf.readEntry("SampleSize").toInt()); ui.sbUpdateInterval->setValue(conf.readEntry("UpdateInterval").toInt()); #ifdef HAVE_MQTT //MQTT related settings //read available connections m_initialisingMQTT = true; readMQTTConnections(); ui.cbConnection->setCurrentIndex(ui.cbConnection->findText(conf.readEntry("Connection", ""))); m_willSettings.willRetain = conf.readEntry("mqttWillRetain").toInt(); m_willSettings.willUpdateType = static_cast(conf.readEntry("mqttWillUpdateType").toInt()); m_willSettings.willQoS = conf.readEntry("mqttWillQoS").toInt(); m_willSettings.willOwnMessage = conf.readEntry("mqttWillOwnMessage",""); m_willSettings.willTimeInterval = conf.readEntry("mqttWillUpdateInterval","").toInt(); QString willStatistics = conf.readEntry("mqttWillStatistics",""); QStringList statisticsList = willStatistics.split('|', QString::SplitBehavior::SkipEmptyParts); for (auto value : statisticsList) { m_willSettings.willStatistics[value.toInt()] = true; } m_willSettings.willMessageType = static_cast(conf.readEntry("mqttWillMessageType").toInt()); m_willSettings.MQTTUseWill = conf.readEntry("mqttWillUse").toInt(); m_initialisingMQTT = false; #endif //initialize the slots after all settings were set in order to avoid unneeded refreshes initSlots(); //update the status of the widgets sourceTypeChanged(currentSourceType()); fileTypeChanged(fileType); readingTypeChanged(ui.cbReadingType->currentIndex()); //all set now, refresh the preview m_suppressRefresh = false; QTimer::singleShot(100, this, [=] () { refreshPreview(); }); } ImportFileWidget::~ImportFileWidget() { // save current settings QString confName; if (m_liveDataSource) confName = QLatin1String("LiveDataImport"); else confName = QLatin1String("FileImport"); KConfigGroup conf(KSharedConfig::openConfig(), confName); // general settings conf.writeEntry("Type", (int)currentFileType()); conf.writeEntry("Filter", ui.cbFilter->currentIndex()); conf.writeEntry("LastImportedFile", ui.leFileName->text()); //live data related settings conf.writeEntry("SourceType", (int)currentSourceType()); conf.writeEntry("UpdateType", ui.cbUpdateType->currentIndex()); conf.writeEntry("ReadingType", ui.cbReadingType->currentIndex()); conf.writeEntry("SampleSize", ui.sbSampleSize->value()); conf.writeEntry("KeepNValues", ui.sbKeepNValues->value()); conf.writeEntry("BaudRate", ui.cbBaudRate->currentIndex()); conf.writeEntry("SerialPort", ui.cbSerialPort->currentIndex()); conf.writeEntry("Host", ui.leHost->text()); conf.writeEntry("Port", ui.lePort->text()); conf.writeEntry("UpdateInterval", ui.sbUpdateInterval->value()); #ifdef HAVE_MQTT //MQTT related settings conf.writeEntry("Connection", ui.cbConnection->currentText()); conf.writeEntry("mqttWillMessageType", static_cast(m_willSettings.willMessageType)); conf.writeEntry("mqttWillUpdateType", static_cast(m_willSettings.willUpdateType)); conf.writeEntry("mqttWillQoS", QString::number(m_willSettings.willQoS)); conf.writeEntry("mqttWillOwnMessage", m_willSettings.willOwnMessage); conf.writeEntry("mqttWillUpdateInterval", QString::number(m_willSettings.willTimeInterval)); QString willStatistics; for (int i = 0; i < m_willSettings.willStatistics.size(); ++i) { if (m_willSettings.willStatistics[i]) willStatistics += QString::number(i)+"|"; } conf.writeEntry("mqttWillStatistics", willStatistics); conf.writeEntry("mqttWillRetain", static_cast(m_willSettings.willRetain)); conf.writeEntry("mqttWillUse", static_cast(m_willSettings.MQTTUseWill)); #endif // data type specific settings if (m_asciiOptionsWidget) m_asciiOptionsWidget->saveSettings(); if (m_binaryOptionsWidget) m_binaryOptionsWidget->saveSettings(); if (m_imageOptionsWidget) m_imageOptionsWidget->saveSettings(); if (m_jsonOptionsWidget) m_jsonOptionsWidget->saveSettings(); } void ImportFileWidget::initSlots() { //SLOTs for the general part of the data source configuration connect( ui.cbSourceType, SIGNAL(currentIndexChanged(int)), this, SLOT(sourceTypeChanged(int))); connect( ui.leFileName, SIGNAL(textChanged(QString)), SLOT(fileNameChanged(QString)) ); connect(ui.leHost, SIGNAL(textChanged(QString)), this, SIGNAL(hostChanged())); connect(ui.lePort, SIGNAL(textChanged(QString)), this, SIGNAL(portChanged())); connect( ui.tvJson, SIGNAL(clicked(QModelIndex)), this, SLOT(refreshPreview())); connect( ui.bOpen, SIGNAL(clicked()), this, SLOT (selectFile()) ); connect( ui.bFileInfo, SIGNAL(clicked()), this, SLOT (fileInfoDialog()) ); connect( ui.bSaveFilter, SIGNAL(clicked()), this, SLOT (saveFilter()) ); connect( ui.bManageFilters, SIGNAL(clicked()), this, SLOT (manageFilters()) ); connect( ui.cbFileType, SIGNAL(currentIndexChanged(int)), SLOT(fileTypeChanged(int)) ); connect( ui.cbUpdateType, SIGNAL(currentIndexChanged(int)), this, SLOT(updateTypeChanged(int))); connect( ui.cbReadingType, SIGNAL(currentIndexChanged(int)), this, SLOT(readingTypeChanged(int))); connect( ui.cbFilter, SIGNAL(activated(int)), SLOT(filterChanged(int)) ); connect( ui.bRefreshPreview, SIGNAL(clicked()), SLOT(refreshPreview()) ); #ifdef HAVE_MQTT connect(ui.cbConnection, static_cast(&QComboBox::currentIndexChanged), this, &ImportFileWidget::mqttConnectionChanged); connect(ui.bSubscribe, &QPushButton::clicked, this, &ImportFileWidget::mqttSubscribe); connect(ui.bUnsubscribe, &QPushButton::clicked, this,&ImportFileWidget::mqttUnsubscribe); connect(this, &ImportFileWidget::newTopic, this, &ImportFileWidget::setTopicCompleter); connect(m_searchTimer, &QTimer::timeout, this, &ImportFileWidget::topicTimeout); connect(m_connectTimeoutTimer, &QTimer::timeout, this, &ImportFileWidget::mqttConnectTimeout); connect(ui.cbFileType, static_cast(&QComboBox::currentIndexChanged), [this]() {emit checkFileType();}); connect(ui.leTopics, &QLineEdit::textChanged, this, &ImportFileWidget::scrollToTopicTreeItem); connect(ui.leSubscriptions, &QLineEdit::textChanged, this, &ImportFileWidget::scrollToSubsriptionTreeItem); connect(ui.bManageConnections, &QPushButton::clicked, this, &ImportFileWidget::showMQTTConnectionManager); connect(ui.bWillMessage, &QPushButton::clicked, this, &ImportFileWidget::showWillSettings); #endif } void ImportFileWidget::showAsciiHeaderOptions(bool b) { if (m_asciiOptionsWidget) m_asciiOptionsWidget->showAsciiHeaderOptions(b); } void ImportFileWidget::showJsonModel(bool b) { ui.tvJson->setVisible(b); ui.lField->setVisible(b); } void ImportFileWidget::showOptions(bool b) { ui.gbOptions->setVisible(b); if (m_liveDataSource) ui.gbUpdateOptions->setVisible(b); resize(layout()->minimumSize()); } QString ImportFileWidget::fileName() const { return ui.leFileName->text(); } QString ImportFileWidget::selectedObject() const { const QString& path = ui.leFileName->text(); //determine the file name only QString name = path.right(path.length() - path.lastIndexOf(QDir::separator()) - 1); //strip away the extension if available if (name.indexOf('.') != -1) name = name.left(name.lastIndexOf('.')); //for multi-dimensinal formats like HDF, netCDF and FITS add the currently selected object const auto format = currentFileType(); if (format == AbstractFileFilter::HDF5) { const QStringList& hdf5Names = m_hdf5OptionsWidget->selectedHDF5Names(); if (hdf5Names.size()) name += hdf5Names.first(); //the names of the selected HDF5 objects already have '/' } else if (format == AbstractFileFilter::NETCDF) { const QStringList& names = m_netcdfOptionsWidget->selectedNetCDFNames(); if (names.size()) name += QLatin1Char('/') + names.first(); } else if (format == AbstractFileFilter::FITS) { const QString& extensionName = m_fitsOptionsWidget->currentExtensionName(); if (!extensionName.isEmpty()) name += QLatin1Char('/') + extensionName; } else if (format == AbstractFileFilter::ROOT) { const QStringList& names = m_rootOptionsWidget->selectedROOTNames(); if (names.size()) name += QLatin1Char('/') + names.first(); } return name; } /*! * returns \c true if the number of lines to be imported from the currently selected file is zero ("file is empty"), * returns \c false otherwise. */ bool ImportFileWidget::isFileEmpty() const { return m_fileEmpty; } QString ImportFileWidget::host() const { return ui.leHost->text(); } QString ImportFileWidget::port() const { return ui.lePort->text(); } QString ImportFileWidget::serialPort() const { return ui.cbSerialPort->currentText(); } int ImportFileWidget::baudRate() const { return ui.cbBaudRate->currentText().toInt(); } /*! saves the settings to the data source \c source. */ void ImportFileWidget::saveSettings(LiveDataSource* source) const { AbstractFileFilter::FileType fileType = currentFileType(); auto updateType = static_cast(ui.cbUpdateType->currentIndex()); LiveDataSource::SourceType sourceType = currentSourceType(); auto readingType = static_cast(ui.cbReadingType->currentIndex()); source->setComment( ui.leFileName->text() ); source->setFileType(fileType); source->setFilter(this->currentFileFilter()); source->setSourceType(sourceType); source->setReadingType(readingType); if (updateType == LiveDataSource::UpdateType::TimeInterval) source->setUpdateInterval(ui.sbUpdateInterval->value()); else source->setFileWatched(true); source->setKeepNValues(ui.sbKeepNValues->value()); source->setUpdateType(updateType); if (readingType != LiveDataSource::ReadingType::TillEnd) source->setSampleSize(ui.sbSampleSize->value()); switch (sourceType) { case LiveDataSource::SourceType::FileOrPipe: source->setFileName(ui.leFileName->text()); source->setFileLinked(ui.chbLinkFile->isChecked()); break; case LiveDataSource::SourceType::LocalSocket: source->setFileName(ui.leFileName->text()); source->setLocalSocketName(ui.leFileName->text()); break; case LiveDataSource::SourceType::NetworkTcpSocket: case LiveDataSource::SourceType::NetworkUdpSocket: source->setHost(ui.leHost->text()); source->setPort((quint16)ui.lePort->text().toInt()); break; case LiveDataSource::SourceType::SerialPort: source->setBaudRate(ui.cbBaudRate->currentText().toInt()); source->setSerialPort(ui.cbSerialPort->currentText()); break; case LiveDataSource::SourceType::MQTT: break; default: break; } } #ifdef HAVE_MQTT /*! saves the settings to the MQTTClient \c client. */ void ImportFileWidget::saveMQTTSettings(MQTTClient* client) const { DEBUG("ImportFileWidget::saveMQTTSettings"); MQTTClient::UpdateType updateType = static_cast(ui.cbUpdateType->currentIndex()); MQTTClient::ReadingType readingType = static_cast(ui.cbReadingType->currentIndex()); client->setComment( ui.leFileName->text() ); client->setFilter(this->currentFileFilter()); client->setReadingType(readingType); if (updateType == MQTTClient::UpdateType::TimeInterval) client->setUpdateInterval(ui.sbUpdateInterval->value()); client->setKeepNValues(ui.sbKeepNValues->value()); client->setUpdateType(updateType); if (readingType != MQTTClient::ReadingType::TillEnd) client->setSampleSize(ui.sbSampleSize->value()); client->setMQTTClientHostPort(m_client->hostname(), m_client->port()); KConfig config(m_configPath, KConfig::SimpleConfig); KConfigGroup group = config.group(ui.cbConnection->currentText()); bool useID = group.readEntry("UseID").toUInt(); bool useAuthentication = group.readEntry("UseAuthentication").toUInt(); client->setMQTTUseAuthentication(useAuthentication); if (useAuthentication) client->setMQTTClientAuthentication(m_client->username(), m_client->password()); client->setMQTTUseID(useID); if (useID) client->setMQTTClientId(m_client->clientId()); for (int i = 0; i < m_mqttSubscriptions.count(); ++i) { client->addInitialMQTTSubscriptions(m_mqttSubscriptions[i]->topic(), m_mqttSubscriptions[i]->qos()); } bool retain = group.readEntry("Retain").toUInt(); client->setMQTTRetain(retain); client->setWillSettings(m_willSettings); } #endif /*! returns the currently used file type. */ AbstractFileFilter::FileType ImportFileWidget::currentFileType() const { return static_cast( ui.cbFileType->itemData(ui.cbFileType->currentIndex()).toInt() ); } LiveDataSource::SourceType ImportFileWidget::currentSourceType() const { return static_cast(ui.cbSourceType->currentIndex()); } /*! returns the currently used filter. */ AbstractFileFilter* ImportFileWidget::currentFileFilter() const { DEBUG("ImportFileWidget::currentFileFilter()"); AbstractFileFilter::FileType fileType = currentFileType(); switch (fileType) { case AbstractFileFilter::Ascii: { DEBUG(" ASCII"); //TODO std::unique_ptr filter(new AsciiFilter()); auto* filter = new AsciiFilter(); if (ui.cbFilter->currentIndex() == 0) //"automatic" filter->setAutoModeEnabled(true); else if (ui.cbFilter->currentIndex() == 1) { //"custom" filter->setAutoModeEnabled(false); if (m_asciiOptionsWidget) m_asciiOptionsWidget->applyFilterSettings(filter); } else filter->loadFilterSettings( ui.cbFilter->currentText() ); //save the data portion to import filter->setStartRow( ui.sbStartRow->value()); filter->setEndRow( ui.sbEndRow->value() ); filter->setStartColumn( ui.sbStartColumn->value()); filter->setEndColumn( ui.sbEndColumn->value()); return filter; } case AbstractFileFilter::Binary: { auto* filter = new BinaryFilter(); if ( ui.cbFilter->currentIndex() == 0 ) //"automatic" filter->setAutoModeEnabled(true); else if ( ui.cbFilter->currentIndex() == 1 ) { //"custom" filter->setAutoModeEnabled(false); if (m_binaryOptionsWidget) m_binaryOptionsWidget->applyFilterSettings(filter); } else { //TODO: load filter settings // filter->setFilterName( ui.cbFilter->currentText() ); } filter->setStartRow( ui.sbStartRow->value() ); filter->setEndRow( ui.sbEndRow->value() ); return filter; } case AbstractFileFilter::Image: { auto* filter = new ImageFilter(); filter->setImportFormat(m_imageOptionsWidget->currentFormat()); filter->setStartRow( ui.sbStartRow->value() ); filter->setEndRow( ui.sbEndRow->value() ); filter->setStartColumn( ui.sbStartColumn->value() ); filter->setEndColumn( ui.sbEndColumn->value() ); return filter; } case AbstractFileFilter::HDF5: { auto* filter = new HDF5Filter(); QStringList names = selectedHDF5Names(); if (!names.isEmpty()) filter->setCurrentDataSetName(names[0]); filter->setStartRow( ui.sbStartRow->value() ); filter->setEndRow( ui.sbEndRow->value() ); filter->setStartColumn( ui.sbStartColumn->value() ); filter->setEndColumn( ui.sbEndColumn->value() ); return filter; } case AbstractFileFilter::NETCDF: { auto* filter = new NetCDFFilter(); if (!selectedNetCDFNames().isEmpty()) filter->setCurrentVarName(selectedNetCDFNames()[0]); filter->setStartRow( ui.sbStartRow->value() ); filter->setEndRow( ui.sbEndRow->value() ); filter->setStartColumn( ui.sbStartColumn->value() ); filter->setEndColumn( ui.sbEndColumn->value() ); return filter; } case AbstractFileFilter::FITS: { auto* filter = new FITSFilter(); filter->setStartRow( ui.sbStartRow->value()); filter->setEndRow( ui.sbEndRow->value() ); filter->setStartColumn( ui.sbStartColumn->value()); filter->setEndColumn( ui.sbEndColumn->value()); return filter; } case AbstractFileFilter::JSON: { auto* filter = new JsonFilter(); m_jsonOptionsWidget->applyFilterSettings(filter, ui.tvJson->currentIndex()); filter->setStartRow( ui.sbStartRow->value() ); filter->setEndRow( ui.sbEndRow->value() ); filter->setStartColumn( ui.sbStartColumn->value()); filter->setEndColumn( ui.sbEndColumn->value()); return filter; } case AbstractFileFilter::ROOT: { auto* filter = new ROOTFilter(); QStringList names = selectedROOTNames(); if (!names.isEmpty()) filter->setCurrentHistogram(names.first()); filter->setStartBin( m_rootOptionsWidget->startBin() ); filter->setEndBin( m_rootOptionsWidget->endBin() ); filter->setColumns( m_rootOptionsWidget->columns() ); return filter; } case AbstractFileFilter::NgspiceRawAscii: { auto* filter = new NgspiceRawAsciiFilter(); filter->setStartRow( ui.sbStartRow->value() ); filter->setEndRow( ui.sbEndRow->value() ); return filter; } case AbstractFileFilter::NgspiceRawBinary: { auto* filter = new NgspiceRawBinaryFilter(); filter->setStartRow( ui.sbStartRow->value() ); filter->setEndRow( ui.sbEndRow->value() ); return filter; } } return nullptr; } /*! opens a file dialog and lets the user select the file data source. */ void ImportFileWidget::selectFile() { KConfigGroup conf(KSharedConfig::openConfig(), "ImportFileWidget"); QString dir = conf.readEntry("LastDir", ""); QString path = QFileDialog::getOpenFileName(this, i18n("Select the File Data Source"), dir); if (path.isEmpty()) //cancel was clicked in the file-dialog return; int pos = path.lastIndexOf(QDir::separator()); if (pos != -1) { QString newDir = path.left(pos); if (newDir != dir) conf.writeEntry("LastDir", newDir); } //process all events after the FileDialog was closed to repaint the widget - //bevor we start calculating the preview + //before we start calculating the preview QApplication::processEvents(QEventLoop::AllEvents, 0); ui.leFileName->setText(path); //TODO: decide whether the selection of several files should be possible // QStringList filelist = QFileDialog::getOpenFileNames(this,i18n("Select one or more files to open")); // if (! filelist.isEmpty() ) // ui.leFileName->setText(filelist.join(";")); } /*! hides the MQTT related items of the widget */ void ImportFileWidget::setMQTTVisible(bool visible) { ui.lConnections->setVisible(visible); ui.cbConnection->setVisible(visible); ui.bManageConnections->setVisible(visible); ui.cbQos->setVisible(visible); ui.lQos->setVisible(visible); //topics if (ui.cbConnection->currentIndex() != -1 && visible) { ui.lTopics->setVisible(true); ui.gbManageSubscriptions->setVisible(true); } else { ui.lTopics->setVisible(false); ui.gbManageSubscriptions->setVisible(false); } //will message ui.lWillMessage->setVisible(visible); ui.bWillMessage->setVisible(visible); } #ifdef HAVE_MQTT /*! * returns \c true if there is a valid connection to an MQTT broker and the user has subscribed to at least 1 topic, * returns \c false otherwise. */ bool ImportFileWidget::isMqttValid() { bool connected = (m_client->state() == QMqttClient::ClientState::Connected); bool subscribed = (ui.twSubscriptions->topLevelItemCount() > 0); bool fileTypeOk = false; if (this->currentFileType() == AbstractFileFilter::FileType::Ascii) fileTypeOk = true; return connected && subscribed && fileTypeOk; } /*! *\brief Checks if a topic contains another one * * \param superior the name of a topic * \param inferior the name of a topic * \return true if superior is equal to or contains(if superior contains wildcards) inferior, * false otherwise */ bool ImportFileWidget::checkTopicContains(const QString& superior, const QString& inferior) { if (superior == inferior) return true; if (!superior.contains('/')) return false; const QStringList& superiorList = superior.split('/', QString::SkipEmptyParts); const QStringList& inferiorList = inferior.split('/', QString::SkipEmptyParts); //a longer topic can't contain a shorter one if (superiorList.size() > inferiorList.size()) return false; bool ok = true; for (int i = 0; i < superiorList.size(); ++i) { if (superiorList.at(i) != inferiorList.at(i)) { if ((superiorList.at(i) != '+') && !(superiorList.at(i) == "#" && i == superiorList.size() - 1)) { //if the two topics differ, and the superior's current level isn't + or #(which can be only in the last position) //then superior can't contain inferior ok = false; break; } else if (i == superiorList.size() - 1 && (superiorList.at(i) == '+' && inferiorList.at(i) == "#") ) { //if the two topics differ at the last level //and the superior's current level is + while the inferior's is #(which can be only in the last position) //then superior can't contain inferior ok = false; break; } } } return ok; } /*! *\brief Returns the '+' wildcard containing topic name, which includes the given topic names * * \param first the name of a topic * \param second the name of a topic * \return The name of the common topic, if it exists, otherwise "" */ QString ImportFileWidget::checkCommonLevel(const QString& first, const QString& second) { const QStringList& firstList = first.split('/', QString::SkipEmptyParts); if (firstList.isEmpty()) return QString(); const QStringList& secondtList = second.split('/', QString::SkipEmptyParts); QString commonTopic = ""; //the two topics have to be the same size and can't be identic if (firstList.size() == secondtList.size() && (first != second)) { //the index where they differ int differIndex = -1; for (int i = 0; i < firstList.size(); ++i) { if (firstList.at(i) != secondtList.at(i)) { differIndex = i; break; } } //they can differ at only one level bool differ = false; if (differIndex > 0) { for (int j = differIndex + 1; j < firstList.size(); ++j) { if (firstList.at(j) != secondtList.at(j)) { differ = true; break; } } } else differ = true; if (!differ) { for (int i = 0; i < firstList.size(); ++i) { if (i != differIndex) { commonTopic.append(firstList.at(i)); } else { //we put '+' wildcard at the level where they differ commonTopic.append('+'); } if (i != firstList.size() - 1) commonTopic.append('/'); } } } qDebug() << "Common topic for " << first << " and " << second << " is: " << commonTopic; return commonTopic; } /*! *\brief Returns the index of level where the two topic names differ, if there is a common topic for them * * \param first the name of a topic * \param second the name of a topic * \return The index of the unequal level, if there is a common topic, otherwise -1 */ int ImportFileWidget::commonLevelIndex(const QString& first, const QString& second) { QStringList firstList = first.split('/', QString::SkipEmptyParts); QStringList secondtList = second.split('/', QString::SkipEmptyParts); QString commonTopic = ""; int differIndex = -1; if (!firstList.isEmpty()) { //the two topics have to be the same size and can't be identic if (firstList.size() == secondtList.size() && (first != second)) { //the index where they differ for (int i = 0; i < firstList.size(); ++i) { if (firstList.at(i) != secondtList.at(i)) { differIndex = i; break; } } //they can differ at only one level bool differ = false; if (differIndex > 0) { for (int j = differIndex + 1; j < firstList.size(); ++j) { if (firstList.at(j) != secondtList.at(j)) { differ = true; break; } } } else differ = true; if (!differ) { for (int i = 0; i < firstList.size(); ++i) { if (i != differIndex) commonTopic.append(firstList.at(i)); else commonTopic.append('+'); if (i != firstList.size() - 1) commonTopic.append('/'); } } } } //if there is a common topic we return the differIndex if (!commonTopic.isEmpty()) return differIndex; else return -1; } /*! *\brief Unsubscribes from the given topic, and removes any data connected to it * * \param topicName the name of a topic we want to unsubscribe from */ void ImportFileWidget::unsubscribeFromTopic(const QString& topicName) { if (topicName.isEmpty()) return; QMqttTopicFilter filter{topicName}; m_client->unsubscribe(filter); for (int i = 0; i< m_mqttSubscriptions.count(); ++i) if (m_mqttSubscriptions[i]->topic().filter() == topicName) { m_mqttSubscriptions.remove(i); break; } m_mqttReadyForPreview = false; QMapIterator i(m_messageArrived); while(i.hasNext()) { i.next(); if (checkTopicContains(topicName, i.key().name())) m_messageArrived.remove(i.key()); } QMapIterator j(m_lastMessage); while(j.hasNext()) { j.next(); if (checkTopicContains(topicName, j.key().name())) m_lastMessage.remove(j.key()); } for (int row = 0; row < ui.twSubscriptions->topLevelItemCount(); row++) { if (ui.twSubscriptions->topLevelItem(row)->text(0) == topicName) { ui.twSubscriptions->topLevelItem(row)->takeChildren(); ui.twSubscriptions->takeTopLevelItem(row); } } for (int i = 0; i < m_subscribedTopicNames.size(); ++i) { if (checkTopicContains(topicName, m_subscribedTopicNames[i])) { m_subscribedTopicNames.remove(i); i--; } } if (m_willSettings.willTopic == topicName) { QVector children; if (ui.twSubscriptions->topLevelItemCount() > 0) { findSubscriptionLeafChildren(children, ui.twSubscriptions->topLevelItem(0)); m_willSettings.willTopic = children[0]->text(0); } else m_willSettings.willTopic = ""; } //signals that there was a change among the subscribed topics emit subscriptionsChanged(); refreshPreview(); } /*! *\brief Adds to a # wildcard containing topic, every topic present in twTopics that the former topic contains * * \param topic pointer to the TreeWidgetItem which was selected before subscribing * \param subscription pointer to the TreeWidgetItem which represents the new subscirption, * we add all of the children to this item */ void ImportFileWidget::addSubscriptionChildren(QTreeWidgetItem * topic, QTreeWidgetItem * subscription) { //if the topic doesn't have any children we don't do anything if (topic->childCount() <= 0) return; for (int i = 0; i < topic->childCount(); ++i) { QTreeWidgetItem* temp = topic->child(i); QString name; //if it has children, then we add it as a # wildcrad containing topic if (topic->child(i)->childCount() > 0) { name.append(temp->text(0) + "/#"); while(temp->parent() != nullptr) { temp = temp->parent(); name.prepend(temp->text(0) + '/'); } } //if not then we simply add the topic itself else { name.append(temp->text(0)); while(temp->parent() != nullptr) { temp = temp->parent(); name.prepend(temp->text(0) + '/'); } } QStringList nameList; nameList.append(name); QTreeWidgetItem* childItem = new QTreeWidgetItem(nameList); subscription->addChild(childItem); //we use the function recursively on the given item addSubscriptionChildren(topic->child(i), childItem); } } /*! *\brief Fills the children vector, with the root item's (twSubscriptions) leaf children (meaning no wildcard containing topics) * * \param children vector of TreeWidgetItem pointers * \param root pointer to a TreeWidgetItem of twSubscriptions */ void ImportFileWidget::findSubscriptionLeafChildren(QVector& children, QTreeWidgetItem* root) { if (root->childCount() == 0) children.push_back(root); else for (int i = 0; i < root->childCount(); ++i) findSubscriptionLeafChildren(children, root->child(i)); } /*! *\brief Returns the amount of topics that the '+' wildcard will replace in the level position * * \param levelIdx the level currently being investigated * \param level the level where the new + wildcard will be placed * \param commonList the topic name split into levels * \param currentItem pointer to a TreeWidgetItem which represents the parent of the level * represented by levelIdx * \return returns the childCount, or -1 if some topics already represented by + wildcard have different * amount of children */ int ImportFileWidget::checkCommonChildCount(int levelIdx, int level, QStringList& commonList, QTreeWidgetItem* currentItem) { //we recursively check the number of children, until we get to level-1 if (levelIdx < level - 1) { if (commonList[levelIdx] != '+') { for (int j = 0; j < currentItem->childCount(); ++j) { if (currentItem->child(j)->text(0) == commonList[levelIdx]) { //if the level isn't represented by + wildcard we simply return the amount of children of the corresponding item, recursively return checkCommonChildCount(levelIdx + 1, level, commonList, currentItem->child(j)); } } } else { int childCount = -1; bool ok = true; //otherwise we check if every + wildcard represented topic has the same number of children, recursively for (int j = 0; j < currentItem->childCount(); ++j) { int temp = checkCommonChildCount(levelIdx + 1, level, commonList, currentItem->child(j)); if ((j > 0) && (temp != childCount)) { ok = false; break; } childCount = temp; } //if yes we return this number, otherwise -1 if (ok) return childCount; else return -1; } } else if (levelIdx == level - 1) { if (commonList[levelIdx] != '+') { for (int j = 0; j < currentItem->childCount(); ++j) { if (currentItem->child(j)->text(0) == commonList[levelIdx]) { //if the level isn't represented by + wildcard we simply return the amount of children of the corresponding item return currentItem->child(j)->childCount(); } } } else { int childCount = -1; bool ok = true; //otherwise we check if every + wildcard represented topic has the same number of children for (int j = 0; j < currentItem->childCount(); ++j) { if ((j > 0) && (currentItem->child(j)->childCount() != childCount)) { ok = false; break; } childCount = currentItem->child(j)->childCount(); } //if yes we return this number, otherwise -1 if (ok) return childCount; else return -1; } } else if (level == 1 && levelIdx == 1) return currentItem->childCount(); return -1; } /*! *\brief We search in twSubscriptions for topics that can be represented using + wildcards, then merge them. * We do this until there are no topics to merge */ void ImportFileWidget::manageCommonLevelSubscriptions() { bool foundEqual = false; do{ foundEqual = false; QMap> equalTopicsMap; QVector equalTopics; //compare the subscriptions present in the TreeWidget for (int i = 0; i < ui.twSubscriptions->topLevelItemCount() - 1; ++i) { for (int j = i + 1; j < ui.twSubscriptions->topLevelItemCount(); ++j) { QString commonTopic = checkCommonLevel(ui.twSubscriptions->topLevelItem(i)->text(0), ui.twSubscriptions->topLevelItem(j)->text(0)); //if there is a common topic for the 2 compared topics, we add them to the map (using the common topic as key) if (!commonTopic.isEmpty()) { if (!equalTopicsMap[commonTopic].contains(ui.twSubscriptions->topLevelItem(i)->text(0))) { equalTopicsMap[commonTopic].push_back(ui.twSubscriptions->topLevelItem(i)->text(0)); } if (!equalTopicsMap[commonTopic].contains(ui.twSubscriptions->topLevelItem(j)->text(0))) { equalTopicsMap[commonTopic].push_back(ui.twSubscriptions->topLevelItem(j)->text(0)); } } } } if (!equalTopicsMap.isEmpty()) { qDebug()<<"Manage common topics"; QVector commonTopics; QMapIterator> topics(equalTopicsMap); //check for every map entry, if the found topics can be merged or not while(topics.hasNext()) { topics.next(); int level = commonLevelIndex(topics.value().last(), topics.value().first()); QStringList commonList = topics.value().first().split('/', QString::SkipEmptyParts); QTreeWidgetItem* currentItem = nullptr; //search the corresponding item to the common topics first level(root) for (int i = 0; i < ui.twTopics->topLevelItemCount(); ++i) { if (ui.twTopics->topLevelItem(i)->text(0) == commonList.first()) { currentItem = ui.twTopics->topLevelItem(i); break; } } //calculate the number of topics the new + wildcard could replace int childCount = checkCommonChildCount(1, level, commonList, currentItem); if (childCount > 0) { //if the number of topics found and the calculated number of topics is equal, the topics can be merged if (topics.value().size() == childCount) { qDebug() << "Found common topic to manage: " << topics.key(); foundEqual = true; commonTopics.push_back(topics.key()); } } } if (foundEqual) { //if there are more common topics, the topics of which can be merged, we choose the one which has the lowest level new '+' wildcard int lowestLevel = INT_MAX; int topicIdx = -1; for (int i = 0; i < commonTopics.size(); ++i) { int level = commonLevelIndex(equalTopicsMap[commonTopics[i]].first(), commonTopics[i]); if (level < lowestLevel) { topicIdx = i; lowestLevel = level; } } qDebug() << "Manage: " << commonTopics[topicIdx]; equalTopics.append(equalTopicsMap[commonTopics[topicIdx]]); //Add the common topic ("merging") QString commonTopic; commonTopic = checkCommonLevel(equalTopics.first(), equalTopics.last()); QStringList nameList; nameList.append(commonTopic); QTreeWidgetItem* newTopic = new QTreeWidgetItem(nameList); ui.twSubscriptions->addTopLevelItem(newTopic); QMqttTopicFilter filter {commonTopic}; QMqttSubscription *temp_subscription = m_client->subscribe(filter, static_cast (ui.cbQos->currentText().toUInt()) ); if (temp_subscription) { m_mqttSubscriptions.push_back(temp_subscription); connect(temp_subscription, &QMqttSubscription::messageReceived, this, &ImportFileWidget::mqttSubscriptionMessageReceived); emit subscriptionsChanged(); } //remove the "merged" topics for (int i = 0; i < equalTopics.size(); ++i) { for (int j = 0; j < ui.twSubscriptions->topLevelItemCount(); ++j) { if (ui.twSubscriptions->topLevelItem(j)->text(0) == equalTopics[i]) { newTopic->addChild(ui.twSubscriptions->takeTopLevelItem(j)); unsubscribeFromTopic(equalTopics[i]); break; } } } //remove any subscription that the new subscription contains for (int i = 0; i < ui.twSubscriptions->topLevelItemCount(); ++i) { if (checkTopicContains(commonTopic, ui.twSubscriptions->topLevelItem(i)->text(0)) && commonTopic != ui.twSubscriptions->topLevelItem(i)->text(0) ) { unsubscribeFromTopic(ui.twSubscriptions->topLevelItem(i)->text(0)); i--; } } } } } while(foundEqual); } /*! *\brief Fills twSubscriptions with the subscriptions made by the client */ void ImportFileWidget::updateSubscriptionTree() { DEBUG("ImportFileWidget::updateSubscriptionTree()"); ui.twSubscriptions->clear(); for (int i = 0; i < m_mqttSubscriptions.size(); ++i) { QStringList name; name.append(m_mqttSubscriptions[i]->topic().filter()); bool found = false; for (int j = 0; j < ui.twSubscriptions->topLevelItemCount(); ++j) { if (ui.twSubscriptions->topLevelItem(j)->text(0) == m_mqttSubscriptions[i]->topic().filter()) { found = true; break; } } if (!found) { //Add the subscription to the tree widget QTreeWidgetItem* newItem = new QTreeWidgetItem(name); ui.twSubscriptions->addTopLevelItem(newItem); name.clear(); name = m_mqttSubscriptions[i]->topic().filter().split('/', QString::SkipEmptyParts); //find the corresponding "root" item in twTopics QTreeWidgetItem* topic = nullptr; for (int j = 0; j < ui.twTopics->topLevelItemCount(); ++j) { if (ui.twTopics->topLevelItem(j)->text(0) == name[0]) { topic = ui.twTopics->topLevelItem(j); break; } } //restore the children of the subscription if (topic != nullptr && topic->childCount() > 0) { restoreSubscriptionChildren(topic, newItem, name, 1); } } } m_searching = false; } /*! *\brief Restores the children of a top level item in twSubscriptions if it contains wildcards * * \param topic pointer to a top level item in twTopics which represents the root of the subscription topic * \param subscription pointer to a top level item in twSubscriptions, this is the item whose children will be restored * \param list QStringList containing the levels of the subscription topic * \param level the level's number which is being investigated */ void ImportFileWidget::restoreSubscriptionChildren(QTreeWidgetItem * topic, QTreeWidgetItem * subscription, const QStringList& list, int level) { DEBUG("ImportFileWidget::restoreSubscriptionChildren"); if (list[level] != '+' && list[level] != "#" && level < list.size() - 1) { for (int i = 0; i < topic->childCount(); ++i) { //if the current level isn't + or # wildcard we recursively continue with the next level if (topic->child(i)->text(0) == list[level]) { restoreSubscriptionChildren(topic->child(i), subscription, list, level + 1); break; } } } else if (list[level] == '+') { for (int i = 0; i < topic->childCount(); ++i) { //determine the name of the topic, contained by the subscription QString name; name.append(topic->child(i)->text(0)); for (int j = level + 1; j < list.size(); ++j) { name.append('/' + list[j]); } QTreeWidgetItem* temp = topic->child(i); while(temp->parent() != nullptr) { temp = temp->parent(); name.prepend(temp->text(0) + '/'); } //Add the topic as child of the subscription QStringList nameList; nameList.append(name); QTreeWidgetItem* newItem = new QTreeWidgetItem(nameList); subscription->addChild(newItem); //Continue adding children recursively to the new item restoreSubscriptionChildren(topic->child(i), newItem, list, level + 1); } } else if (list[level] == "#") { //add the children of the # wildcard containing subscription addSubscriptionChildren(topic, subscription); } } /*! *\brief Updates the completer for leSubscriptions */ void ImportFileWidget::updateSubscriptionCompleter() { QStringList subscriptionList; for (int i = 0; i < ui.twSubscriptions->topLevelItemCount(); ++i) subscriptionList.append(ui.twSubscriptions->topLevelItem(i)->text(0)); if (!subscriptionList.isEmpty()) { m_subscriptionCompleter = new QCompleter(subscriptionList, this); m_subscriptionCompleter->setCompletionMode(QCompleter::PopupCompletion); m_subscriptionCompleter->setCaseSensitivity(Qt::CaseSensitive); ui.leSubscriptions->setCompleter(m_subscriptionCompleter); } else ui.leSubscriptions->setCompleter(nullptr); } #endif /************** SLOTS **************************************************************/ QString absolutePath(const QString& fileName) { #ifndef HAVE_WINDOWS // make absolute path // FIXME if (!fileName.isEmpty() && fileName.at(0) != QDir::separator()) return QDir::homePath() + QDir::separator() + fileName; #endif return fileName; } /*! called on file name changes. Determines the file format (ASCII, binary etc.), if the file exists, and activates the corresponding options. */ void ImportFileWidget::fileNameChanged(const QString& name) { DEBUG("ImportFileWidget::fileNameChanged()"); const QString fileName = absolutePath(name); bool fileExists = QFile::exists(fileName); if (fileExists) ui.leFileName->setStyleSheet(""); else ui.leFileName->setStyleSheet("QLineEdit{background:red;}"); ui.gbOptions->setEnabled(fileExists); ui.bManageFilters->setEnabled(fileExists); ui.cbFilter->setEnabled(fileExists); ui.cbFileType->setEnabled(fileExists); ui.bFileInfo->setEnabled(fileExists); ui.gbUpdateOptions->setEnabled(fileExists); if (!fileExists) { //file doesn't exist -> delete the content preview that is still potentially //available from the previously selected file ui.tePreview->clear(); m_twPreview->clear(); //TODO: // m_hdf5OptionsWidget->clear(); // m_netcdfOptionsWidget->clear(); // m_fitsOptionsWidget->clear(); // m_jsonOptionsWidget->clearModel(); // m_rootOptionsWidget->clear(); emit fileNameChanged(); return; } if (currentSourceType() == LiveDataSource::FileOrPipe) { const AbstractFileFilter::FileType fileType = AbstractFileFilter::fileType(fileName); for (int i = 0; i < ui.cbFileType->count(); ++i) { AbstractFileFilter::FileType itemFileType = static_cast(ui.cbFileType->itemData(i).toInt()); if (itemFileType == fileType) { ui.cbFileType->setCurrentIndex(i); break; } } //TODO: updateContent(fileName, fileType); } refreshPreview(); emit fileNameChanged(); } /*! saves the current filter settings */ void ImportFileWidget::saveFilter() { bool ok; QString text = QInputDialog::getText(this, i18n("Save Filter Settings as"), i18n("Filter name:"), QLineEdit::Normal, i18n("new filter"), &ok); if (ok && !text.isEmpty()) { //TODO //AsciiFilter::saveFilter() } } /*! opens a dialog for managing all available predefined filters. */ void ImportFileWidget::manageFilters() { //TODO } /*! Depending on the selected file type, activates the corresponding options in the data portion tab and populates the combobox with the available pre-defined fllter settings for the selected type. */ void ImportFileWidget::fileTypeChanged(int index) { Q_UNUSED(index); AbstractFileFilter::FileType fileType = currentFileType();//(AbstractFileFilter::FileType)ui.cbFileType->itemData(index).toInt(); DEBUG("ImportFileWidget::fileTypeChanged " << ENUM_TO_STRING(AbstractFileFilter, FileType, fileType)); initOptionsWidget(fileType); //default ui.lFilter->show(); ui.cbFilter->show(); //different file types show different number of tabs in ui.tabWidget. //when switching from the previous file type we re-set the tab widget to its original state //and remove/add the required tabs further below for (int i = 0; icount(); ++i) ui.tabWidget->count(); ui.tabWidget->addTab(ui.tabDataFormat, i18n("Data format")); ui.tabWidget->addTab(ui.tabDataPreview, i18n("Preview")); ui.tabWidget->addTab(ui.tabDataPortion, i18n("Data portion to read")); ui.lPreviewLines->show(); ui.sbPreviewLines->show(); ui.lStartColumn->show(); ui.sbStartColumn->show(); ui.lEndColumn->show(); ui.sbEndColumn->show(); showJsonModel(false); switch (fileType) { case AbstractFileFilter::Ascii: break; case AbstractFileFilter::Binary: ui.lStartColumn->hide(); ui.sbStartColumn->hide(); ui.lEndColumn->hide(); ui.sbEndColumn->hide(); break; case AbstractFileFilter::ROOT: ui.tabWidget->removeTab(1); // falls through case AbstractFileFilter::HDF5: case AbstractFileFilter::NETCDF: case AbstractFileFilter::FITS: ui.lFilter->hide(); ui.cbFilter->hide(); // hide global preview tab. we have our own ui.tabWidget->setTabText(0, i18n("Data format && preview")); ui.tabWidget->removeTab(1); ui.tabWidget->setCurrentIndex(0); break; case AbstractFileFilter::Image: ui.lFilter->hide(); ui.cbFilter->hide(); ui.lPreviewLines->hide(); ui.sbPreviewLines->hide(); break; case AbstractFileFilter::NgspiceRawAscii: case AbstractFileFilter::NgspiceRawBinary: ui.lFilter->hide(); ui.cbFilter->hide(); ui.lStartColumn->hide(); ui.sbStartColumn->hide(); ui.lEndColumn->hide(); ui.sbEndColumn->hide(); ui.tabWidget->removeTab(0); ui.tabWidget->setCurrentIndex(0); break; case AbstractFileFilter::JSON: ui.lFilter->hide(); ui.cbFilter->hide(); showJsonModel(true); break; default: DEBUG("unknown file type"); } int lastUsedFilterIndex = ui.cbFilter->currentIndex(); ui.cbFilter->clear(); ui.cbFilter->addItem( i18n("Automatic") ); ui.cbFilter->addItem( i18n("Custom") ); //TODO: populate the combobox with the available pre-defined filter settings for the selected type ui.cbFilter->setCurrentIndex(lastUsedFilterIndex); filterChanged(lastUsedFilterIndex); if (currentSourceType() == LiveDataSource::FileOrPipe) { const QString fileName = absolutePath(ui.leFileName->text()); if (QFile::exists(fileName)) updateContent(fileName, static_cast(fileType)); } //for file types other than ASCII and binary we support re-reading the whole file only //select "read whole file" and deactivate the combobox if (m_liveDataSource && (fileType != AbstractFileFilter::Ascii && fileType != AbstractFileFilter::Binary)) { ui.cbReadingType->setCurrentIndex(3); ui.cbReadingType->setEnabled(false); } else ui.cbReadingType->setEnabled(true); refreshPreview(); } // file type specific option widgets void ImportFileWidget::initOptionsWidget(AbstractFileFilter::FileType fileType) { DEBUG("ImportFileWidget::initOptionsWidget for " << ENUM_TO_STRING(AbstractFileFilter, FileType, fileType)); switch(fileType) { case AbstractFileFilter::Ascii: if (!m_asciiOptionsWidget) { QWidget* asciiw = new QWidget(); m_asciiOptionsWidget = std::unique_ptr(new AsciiOptionsWidget(asciiw)); m_asciiOptionsWidget->loadSettings(); ui.swOptions->addWidget(asciiw); } ui.swOptions->setCurrentWidget(m_asciiOptionsWidget->parentWidget()); break; case AbstractFileFilter::Binary: if (!m_binaryOptionsWidget) { QWidget* binaryw = new QWidget(); m_binaryOptionsWidget = std::unique_ptr(new BinaryOptionsWidget(binaryw)); ui.swOptions->addWidget(binaryw); m_binaryOptionsWidget->loadSettings(); } ui.swOptions->setCurrentWidget(m_binaryOptionsWidget->parentWidget()); break; case AbstractFileFilter::Image: if (!m_imageOptionsWidget) { QWidget* imagew = new QWidget(); m_imageOptionsWidget = std::unique_ptr(new ImageOptionsWidget(imagew)); ui.swOptions->addWidget(imagew); m_imageOptionsWidget->loadSettings(); } ui.swOptions->setCurrentWidget(m_imageOptionsWidget->parentWidget()); break; case AbstractFileFilter::HDF5: if (!m_hdf5OptionsWidget) { QWidget* hdf5w = new QWidget(); m_hdf5OptionsWidget = std::unique_ptr(new HDF5OptionsWidget(hdf5w, this)); ui.swOptions->addWidget(hdf5w); } else m_hdf5OptionsWidget->clear(); ui.swOptions->setCurrentWidget(m_hdf5OptionsWidget->parentWidget()); break; case AbstractFileFilter::NETCDF: if (!m_netcdfOptionsWidget) { QWidget* netcdfw = new QWidget(); m_netcdfOptionsWidget = std::unique_ptr(new NetCDFOptionsWidget(netcdfw, this)); ui.swOptions->insertWidget(AbstractFileFilter::NETCDF, netcdfw); } else m_netcdfOptionsWidget->clear(); ui.swOptions->setCurrentWidget(m_netcdfOptionsWidget->parentWidget()); break; case AbstractFileFilter::FITS: if (!m_fitsOptionsWidget) { QWidget* fitsw = new QWidget(); m_fitsOptionsWidget = std::unique_ptr(new FITSOptionsWidget(fitsw, this)); ui.swOptions->addWidget(fitsw); } ui.swOptions->setCurrentWidget(m_fitsOptionsWidget->parentWidget()); break; case AbstractFileFilter::JSON: if (!m_jsonOptionsWidget) { QWidget* jsonw = new QWidget(); m_jsonOptionsWidget = std::unique_ptr(new JsonOptionsWidget(jsonw, this)); ui.tvJson->setModel(m_jsonOptionsWidget->model()); ui.swOptions->addWidget(jsonw); m_jsonOptionsWidget->loadSettings(); } ui.swOptions->setCurrentWidget(m_jsonOptionsWidget->parentWidget()); break; case AbstractFileFilter::ROOT: if (!m_rootOptionsWidget) { QWidget* rootw = new QWidget(); m_rootOptionsWidget = std::unique_ptr(new ROOTOptionsWidget(rootw, this)); ui.swOptions->addWidget(rootw); } else m_rootOptionsWidget->clear(); ui.swOptions->setCurrentWidget(m_rootOptionsWidget->parentWidget()); break; case AbstractFileFilter::NgspiceRawAscii: case AbstractFileFilter::NgspiceRawBinary: break; } } const QStringList ImportFileWidget::selectedHDF5Names() const { return m_hdf5OptionsWidget->selectedHDF5Names(); } const QStringList ImportFileWidget::selectedNetCDFNames() const { return m_netcdfOptionsWidget->selectedNetCDFNames(); } const QStringList ImportFileWidget::selectedFITSExtensions() const { return m_fitsOptionsWidget->selectedFITSExtensions(); } const QStringList ImportFileWidget::selectedROOTNames() const { return m_rootOptionsWidget->selectedROOTNames(); } /*! shows the dialog with the information about the file(s) to be imported. */ void ImportFileWidget::fileInfoDialog() { QStringList files = ui.leFileName->text().split(';'); auto* dlg = new FileInfoDialog(this); dlg->setFiles(files); dlg->exec(); } /*! enables the options if the filter "custom" was chosen. Disables the options otherwise. */ void ImportFileWidget::filterChanged(int index) { // ignore filter for these formats AbstractFileFilter::FileType fileType = currentFileType(); if (fileType != AbstractFileFilter::Ascii && fileType != AbstractFileFilter::Binary) { ui.swOptions->setEnabled(true); return; } if (index == 0) { // "automatic" ui.swOptions->setEnabled(false); ui.bSaveFilter->setEnabled(false); } else if (index == 1) { //custom ui.swOptions->setEnabled(true); ui.bSaveFilter->setEnabled(true); } else { // predefined filter settings were selected. //load and show them in the GUI. //TODO } } void ImportFileWidget::refreshPreview() { if (m_suppressRefresh) return; WAIT_CURSOR; QString fileName = absolutePath(ui.leFileName->text()); AbstractFileFilter::FileType fileType = currentFileType(); LiveDataSource::SourceType sourceType = currentSourceType(); int lines = ui.sbPreviewLines->value(); if (sourceType == LiveDataSource::SourceType::FileOrPipe) DEBUG("refreshPreview(): file name = " << fileName.toStdString()); // generic table widget if (fileType == AbstractFileFilter::Ascii || fileType == AbstractFileFilter::Binary || fileType == AbstractFileFilter::JSON || fileType == AbstractFileFilter::NgspiceRawAscii || fileType == AbstractFileFilter::NgspiceRawBinary) m_twPreview->show(); else m_twPreview->hide(); bool ok = true; QTableWidget* tmpTableWidget = m_twPreview; QVector importedStrings; QStringList vectorNameList; QVector columnModes; DEBUG("Data File Type: " << ENUM_TO_STRING(AbstractFileFilter, FileType, fileType)); switch (fileType) { case AbstractFileFilter::Ascii: { ui.tePreview->clear(); auto* filter = static_cast(this->currentFileFilter()); DEBUG("Data Source Type: " << ENUM_TO_STRING(LiveDataSource, SourceType, sourceType)); switch (sourceType) { case LiveDataSource::SourceType::FileOrPipe: { importedStrings = filter->preview(fileName, lines); break; } case LiveDataSource::SourceType::LocalSocket: { QLocalSocket lsocket{this}; DEBUG("Local socket: CONNECT PREVIEW"); lsocket.connectToServer(fileName, QLocalSocket::ReadOnly); if (lsocket.waitForConnected()) { DEBUG("connected to local socket " << fileName.toStdString()); if (lsocket.waitForReadyRead()) importedStrings = filter->preview(lsocket); DEBUG("Local socket: DISCONNECT PREVIEW"); lsocket.disconnectFromServer(); // read-only socket is disconnected immediately (no waitForDisconnected()) } else { DEBUG("failed connect to local socket " << fileName.toStdString() << " - " << lsocket.errorString().toStdString()); } break; } case LiveDataSource::SourceType::NetworkTcpSocket: { QTcpSocket tcpSocket{this}; tcpSocket.connectToHost(host(), port().toInt(), QTcpSocket::ReadOnly); if (tcpSocket.waitForConnected()) { DEBUG("connected to TCP socket"); if ( tcpSocket.waitForReadyRead() ) importedStrings = filter->preview(tcpSocket); tcpSocket.disconnectFromHost(); } else { DEBUG("failed to connect to TCP socket " << " - " << tcpSocket.errorString().toStdString()); } break; } case LiveDataSource::SourceType::NetworkUdpSocket: { QUdpSocket udpSocket{this}; DEBUG("UDP Socket: CONNECT PREVIEW, state = " << udpSocket.state()); udpSocket.bind(QHostAddress(host()), port().toInt()); udpSocket.connectToHost(host(), 0, QUdpSocket::ReadOnly); if (udpSocket.waitForConnected()) { DEBUG(" connected to UDP socket " << host().toStdString() << ':' << port().toInt()); if (!udpSocket.waitForReadyRead(2000) ) DEBUG(" ERROR: not ready for read after 2 sec"); if (udpSocket.hasPendingDatagrams()) { DEBUG(" has pending data"); } else { DEBUG(" has no pending data"); } importedStrings = filter->preview(udpSocket); DEBUG("UDP Socket: DISCONNECT PREVIEW, state = " << udpSocket.state()); udpSocket.disconnectFromHost(); } else { DEBUG("failed to connect to UDP socket " << " - " << udpSocket.errorString().toStdString()); } break; } case LiveDataSource::SourceType::SerialPort: { QSerialPort sPort{this}; DEBUG(" Port: " << serialPort().toStdString() << ", Settings: " << baudRate() << ',' << sPort.dataBits() << ',' << sPort.parity() << ',' << sPort.stopBits()); sPort.setPortName(serialPort()); sPort.setBaudRate(baudRate()); if (sPort.open(QIODevice::ReadOnly)) { if (sPort.waitForReadyRead(2000)) importedStrings = filter->preview(sPort); else DEBUG(" ERROR: not ready for read after 2 sec"); sPort.close(); } else { DEBUG(" ERROR: failed to open serial port. error: " << sPort.error()); } break; } case LiveDataSource::SourceType::MQTT: { #ifdef HAVE_MQTT qDebug()<<"Start MQTT preview, is it ready:"<vectorNames().clear(); QMapIterator i(m_lastMessage); while(i.hasNext()) { i.next(); filter->MQTTPreview(importedStrings, QString(i.value().payload().data()), i.key().name() ); if (importedStrings.isEmpty()) break; } QMapIterator j(m_messageArrived); while(j.hasNext()) { j.next(); m_messageArrived[j.key()] = false; } m_mqttReadyForPreview = false; } #endif break; } } vectorNameList = filter->vectorNames(); columnModes = filter->columnModes(); break; } case AbstractFileFilter::Binary: { ui.tePreview->clear(); auto* filter = (BinaryFilter*)this->currentFileFilter(); importedStrings = filter->preview(fileName, lines); break; } case AbstractFileFilter::Image: { ui.tePreview->clear(); QImage image(fileName); QTextCursor cursor = ui.tePreview->textCursor(); cursor.insertImage(image); RESET_CURSOR; return; } case AbstractFileFilter::HDF5: { auto* filter = (HDF5Filter*)this->currentFileFilter(); lines = m_hdf5OptionsWidget->lines(); importedStrings = filter->readCurrentDataSet(fileName, nullptr, ok, AbstractFileFilter::Replace, lines); tmpTableWidget = m_hdf5OptionsWidget->previewWidget(); break; } case AbstractFileFilter::NETCDF: { auto* filter = (NetCDFFilter*)this->currentFileFilter(); lines = m_netcdfOptionsWidget->lines(); importedStrings = filter->readCurrentVar(fileName, nullptr, AbstractFileFilter::Replace, lines); tmpTableWidget = m_netcdfOptionsWidget->previewWidget(); break; } case AbstractFileFilter::FITS: { auto* filter = (FITSFilter*)this->currentFileFilter(); lines = m_fitsOptionsWidget->lines(); // update file name (may be any file type) m_fitsOptionsWidget->updateContent(filter, fileName); QString extensionName = m_fitsOptionsWidget->extensionName(&ok); if (!extensionName.isEmpty()) { DEBUG(" extension name = " << extensionName.toStdString()); fileName = extensionName; } DEBUG(" file name = " << fileName.toStdString()); bool readFitsTableToMatrix; importedStrings = filter->readChdu(fileName, &readFitsTableToMatrix, lines); emit checkedFitsTableToMatrix(readFitsTableToMatrix); tmpTableWidget = m_fitsOptionsWidget->previewWidget(); break; } case AbstractFileFilter::JSON: { ui.tePreview->clear(); m_jsonOptionsWidget->loadDocument(fileName); auto* filter = (JsonFilter*)this->currentFileFilter(); m_jsonOptionsWidget->applyFilterSettings(filter, ui.tvJson->currentIndex()); importedStrings = filter->preview(fileName); vectorNameList = filter->vectorNames(); columnModes = filter->columnModes(); break; } case AbstractFileFilter::ROOT: { auto* filter = (ROOTFilter*)this->currentFileFilter(); lines = m_rootOptionsWidget->lines(); m_rootOptionsWidget->setNBins(filter->binsInCurrentHistogram(fileName)); importedStrings = filter->previewCurrentHistogram( fileName, m_rootOptionsWidget->startBin(), qMin(m_rootOptionsWidget->startBin() + m_rootOptionsWidget->lines() - 1, m_rootOptionsWidget->endBin()) ); tmpTableWidget = m_rootOptionsWidget->previewWidget(); // the last vector element contains the column names vectorNameList = importedStrings.last(); importedStrings.removeLast(); columnModes = QVector(vectorNameList.size(), AbstractColumn::Numeric); break; } case AbstractFileFilter::NgspiceRawAscii: { ui.tePreview->clear(); auto* filter = (NgspiceRawAsciiFilter*)this->currentFileFilter(); importedStrings = filter->preview(fileName, lines); vectorNameList = filter->vectorNames(); columnModes = filter->columnModes(); break; } case AbstractFileFilter::NgspiceRawBinary: { ui.tePreview->clear(); auto* filter = (NgspiceRawBinaryFilter*)this->currentFileFilter(); importedStrings = filter->preview(fileName, lines); vectorNameList = filter->vectorNames(); columnModes = filter->columnModes(); break; } } // fill the table widget tmpTableWidget->setRowCount(0); tmpTableWidget->setColumnCount(0); if ( !importedStrings.isEmpty() ) { //QDEBUG("importedStrings =" << importedStrings); if (!ok) { // show imported strings as error message tmpTableWidget->setRowCount(1); tmpTableWidget->setColumnCount(1); auto* item = new QTableWidgetItem(); item->setText(importedStrings[0][0]); tmpTableWidget->setItem(0, 0, item); } else { //TODO: maxrows not used const int rows = qMax(importedStrings.size(), 1); const int maxColumns = 300; tmpTableWidget->setRowCount(rows); for (int i = 0; i < rows; ++i) { // QDEBUG("imported string " << importedStrings[i]); int cols = importedStrings[i].size() > maxColumns ? maxColumns : importedStrings[i].size(); // new if (cols > tmpTableWidget->columnCount()) tmpTableWidget->setColumnCount(cols); for (int j = 0; j < cols; ++j) { auto* item = new QTableWidgetItem(importedStrings[i][j]); tmpTableWidget->setItem(i, j, item); } } // set header if columnMode available for (int i = 0; i < qMin(tmpTableWidget->columnCount(), columnModes.size()); ++i) { QString columnName = QString::number(i+1); if (i < vectorNameList.size()) columnName = vectorNameList[i]; auto* item = new QTableWidgetItem(columnName + QLatin1String(" {") + ENUM_TO_STRING(AbstractColumn, ColumnMode, columnModes[i]) + QLatin1String("}")); item->setTextAlignment(Qt::AlignLeft); item->setIcon(AbstractColumn::iconForMode(columnModes[i])); tmpTableWidget->setHorizontalHeaderItem(i, item); } } tmpTableWidget->horizontalHeader()->resizeSections(QHeaderView::ResizeToContents); m_fileEmpty = false; } else { m_fileEmpty = true; } emit previewRefreshed(); RESET_CURSOR; } void ImportFileWidget::updateContent(const QString& fileName, AbstractFileFilter::FileType fileType) { initOptionsWidget(fileType); if (auto* filter = currentFileFilter()) { switch (fileType) { case AbstractFileFilter::HDF5: m_hdf5OptionsWidget->updateContent((HDF5Filter*)filter, fileName); break; case AbstractFileFilter::NETCDF: m_netcdfOptionsWidget->updateContent((NetCDFFilter*)filter, fileName); break; case AbstractFileFilter::FITS: #ifdef HAVE_FITS m_fitsOptionsWidget->updateContent((FITSFilter*)filter, fileName); #endif break; case AbstractFileFilter::ROOT: m_rootOptionsWidget->updateContent((ROOTFilter*)filter, fileName); break; case AbstractFileFilter::Ascii: case AbstractFileFilter::Binary: case AbstractFileFilter::Image: case AbstractFileFilter::JSON: case AbstractFileFilter::NgspiceRawAscii: case AbstractFileFilter::NgspiceRawBinary: break; } } } void ImportFileWidget::updateTypeChanged(int idx) { const auto UpdateType = static_cast(idx); switch (UpdateType) { case LiveDataSource::UpdateType::TimeInterval: ui.lUpdateInterval->show(); ui.sbUpdateInterval->show(); break; case LiveDataSource::UpdateType::NewData: ui.lUpdateInterval->hide(); ui.sbUpdateInterval->hide(); } } void ImportFileWidget::readingTypeChanged(int idx) { const auto readingType = static_cast(idx); const LiveDataSource::SourceType sourceType = currentSourceType(); if (sourceType == LiveDataSource::SourceType::NetworkTcpSocket || sourceType == LiveDataSource::SourceType::LocalSocket || sourceType == LiveDataSource::SourceType::SerialPort || readingType == LiveDataSource::ReadingType::TillEnd || readingType == LiveDataSource::ReadingType::WholeFile) { ui.lSampleSize->hide(); ui.sbSampleSize->hide(); } else { ui.lSampleSize->show(); ui.sbSampleSize->show(); } if (readingType == LiveDataSource::ReadingType::WholeFile) { ui.lKeepLastValues->hide(); ui.sbKeepNValues->hide(); } else { ui.lKeepLastValues->show(); ui.sbKeepNValues->show(); } } void ImportFileWidget::sourceTypeChanged(int idx) { const auto sourceType = static_cast(idx); // enable/disable "on new data"-option const auto* model = qobject_cast(ui.cbUpdateType->model()); QStandardItem* item = model->item(LiveDataSource::UpdateType::NewData); switch (sourceType) { case LiveDataSource::SourceType::FileOrPipe: ui.lFileName->show(); ui.leFileName->show(); ui.bFileInfo->show(); ui.bOpen->show(); ui.chbLinkFile->show(); //option for sample size are available for "continuously fixed" and "from end" reading options if (ui.cbReadingType->currentIndex() < 2) { ui.lSampleSize->show(); ui.sbSampleSize->show(); } else { ui.lSampleSize->hide(); ui.sbSampleSize->hide(); } ui.cbBaudRate->hide(); ui.lBaudRate->hide(); ui.lHost->hide(); ui.leHost->hide(); ui.lPort->hide(); ui.lePort->hide(); ui.cbSerialPort->hide(); ui.lSerialPort->hide(); item->setFlags(Qt::ItemIsSelectable | Qt::ItemIsEnabled); fileNameChanged(ui.leFileName->text()); ui.cbFileType->show(); ui.lFileType->show(); setMQTTVisible(false); break; case LiveDataSource::SourceType::NetworkTcpSocket: case LiveDataSource::SourceType::NetworkUdpSocket: ui.lHost->show(); ui.leHost->show(); ui.lePort->show(); ui.lPort->show(); if (sourceType == LiveDataSource::SourceType::NetworkTcpSocket) { ui.lSampleSize->hide(); ui.sbSampleSize->hide(); } else { ui.lSampleSize->show(); ui.sbSampleSize->show(); } ui.lBaudRate->hide(); ui.cbBaudRate->hide(); ui.lSerialPort->hide(); ui.cbSerialPort->hide(); ui.lFileName->hide(); ui.leFileName->hide(); ui.bFileInfo->hide(); ui.bOpen->hide(); ui.chbLinkFile->hide(); item->setFlags(item->flags() & ~(Qt::ItemIsSelectable | Qt::ItemIsEnabled)); ui.gbOptions->setEnabled(true); ui.bManageFilters->setEnabled(true); ui.cbFilter->setEnabled(true); ui.cbFileType->setEnabled(true); ui.cbFileType->show(); ui.lFileType->show(); setMQTTVisible(false); break; case LiveDataSource::SourceType::LocalSocket: ui.lFileName->show(); ui.leFileName->show(); ui.bOpen->show(); ui.lSampleSize->hide(); ui.sbSampleSize->hide(); ui.bFileInfo->hide(); ui.cbBaudRate->hide(); ui.lBaudRate->hide(); ui.lHost->hide(); ui.leHost->hide(); ui.lPort->hide(); ui.lePort->hide(); ui.cbSerialPort->hide(); ui.lSerialPort->hide(); ui.chbLinkFile->hide(); item->setFlags(Qt::ItemIsSelectable | Qt::ItemIsEnabled); ui.gbOptions->setEnabled(true); ui.bManageFilters->setEnabled(true); ui.cbFilter->setEnabled(true); ui.cbFileType->setEnabled(true); ui.cbFileType->show(); ui.lFileType->show(); setMQTTVisible(false); break; case LiveDataSource::SourceType::SerialPort: ui.lBaudRate->show(); ui.cbBaudRate->show(); ui.lSerialPort->show(); ui.cbSerialPort->show(); ui.lSampleSize->show(); ui.sbSampleSize->show(); ui.lHost->hide(); ui.leHost->hide(); ui.lePort->hide(); ui.lPort->hide(); ui.lFileName->hide(); ui.leFileName->hide(); ui.bFileInfo->hide(); ui.bOpen->hide(); ui.chbLinkFile->hide(); item->setFlags(item->flags() & ~(Qt::ItemIsSelectable | Qt::ItemIsEnabled)); ui.cbFileType->setEnabled(true); ui.cbFileType->show(); ui.gbOptions->setEnabled(true); ui.bManageFilters->setEnabled(true); ui.cbFilter->setEnabled(true); ui.lFileType->show(); setMQTTVisible(false); break; case LiveDataSource::SourceType::MQTT: #ifdef HAVE_MQTT item->setFlags(Qt::ItemIsSelectable | Qt::ItemIsEnabled); //for MQTT we read ascii data only, hide the file type options int idx = ui.cbFileType->findText("ASCII data"); ui.cbFileType->setCurrentIndex(idx); ui.cbFileType->hide(); ui.lFileType->hide(); ui.lBaudRate->hide(); ui.cbBaudRate->hide(); ui.lSerialPort->hide(); ui.cbSerialPort->hide(); ui.lHost->hide(); ui.leHost->hide(); ui.lPort->hide(); ui.lePort->hide(); ui.lFileName->hide(); ui.leFileName->hide(); ui.bFileInfo->hide(); ui.bOpen->hide(); ui.chbLinkFile->hide(); setMQTTVisible(true); ui.cbFileType->setEnabled(true); ui.gbOptions->setEnabled(true); ui.bManageFilters->setEnabled(true); ui.cbFilter->setEnabled(true); //in case there are already connections defined, show the available topics for the currently selected connection mqttConnectionChanged(); #endif break; } //deactivate/activate options that are specific to file of pipe sources only auto* typeModel = qobject_cast(ui.cbFileType->model()); if (sourceType != LiveDataSource::FileOrPipe) { //deactivate file types other than ascii and binary for (int i = 2; i < ui.cbFileType->count(); ++i) typeModel->item(i)->setFlags(item->flags() & ~(Qt::ItemIsSelectable | Qt::ItemIsEnabled)); //"whole file" read option is available for file or pipe only, disable it typeModel = qobject_cast(ui.cbReadingType->model()); QStandardItem* item = typeModel->item(LiveDataSource::ReadingType::WholeFile); item->setFlags(item->flags() & ~(Qt::ItemIsSelectable | Qt::ItemIsEnabled)); //"update options" groupbox can be deactivated for "file and pipe" if the file is invalid. //Activate the groupbox when switching from "file and pipe" to a different source type. ui.gbUpdateOptions->setEnabled(true); } else { for (int i = 2; i < ui.cbFileType->count(); ++i) typeModel->item(i)->setFlags(Qt::ItemIsSelectable | Qt::ItemIsEnabled); //enable "whole file" item for file or pipe typeModel = qobject_cast(ui.cbReadingType->model()); QStandardItem* item = typeModel->item(LiveDataSource::ReadingType::WholeFile); item->setFlags(Qt::ItemIsSelectable | Qt::ItemIsEnabled); } emit sourceTypeChanged(); refreshPreview(); } #ifdef HAVE_MQTT /*! *\brief called when a different MQTT connection is selected in the connection ComboBox. * connects to the MQTT broker according to the connection settings. */ void ImportFileWidget::mqttConnectionChanged() { if (m_initialisingMQTT) return; if (m_client == nullptr || m_client->state() == QMqttClient::ClientState::Disconnected) { if (ui.cbConnection->currentIndex() == -1) return; WAIT_CURSOR; delete m_client; m_client = new QMqttClient; connect(m_client, &QMqttClient::connected, this, &ImportFileWidget::onMqttConnect); connect(m_client, &QMqttClient::disconnected, this, &ImportFileWidget::onMqttDisconnect); connect(m_client, &QMqttClient::messageReceived, this, &ImportFileWidget::mqttMessageReceived); connect(m_client, &QMqttClient::errorChanged, this, &ImportFileWidget::mqttErrorChanged); ui.cbConnection->setEnabled(false); ui.bManageConnections->setEnabled(false); //determine the current connection's settings KConfig config(m_configPath, KConfig::SimpleConfig); KConfigGroup group = config.group(ui.cbConnection->currentText()); m_client->setHostname(group.readEntry("Host")); m_client->setPort(group.readEntry("Port").toUInt()); const bool useID = group.readEntry("UseID").toUInt(); if (useID) m_client->setClientId(group.readEntry("ClientID")); else m_client->setClientId(""); const bool useAuthentication = group.readEntry("UseAuthentication").toUInt(); if (useAuthentication) { m_client->setUsername(group.readEntry("UserName")); m_client->setPassword(group.readEntry("Password")); } else { m_client->setUsername(""); m_client->setPassword(""); } qDebug()<< "Use ID" << useID << " " << m_client->clientId(); qDebug()<< "Use authentication" << useAuthentication << " " << m_client->username() << " " << m_client->password(); qDebug()<< m_client->hostname() << " " << m_client->port(); qDebug()<< "Trying to connect"; m_connectTimeoutTimer->start(); m_client->connectToHost(); } else if (m_client->state() == QMqttClient::ClientState::Connected) { WAIT_CURSOR; ui.cbConnection->setEnabled(false); ui.bManageConnections->setEnabled(false); qDebug()<<"Disconnecting from MQTT broker" ; m_client->disconnectFromHost(); } } /*! *\brief called when the client connects to the broker successfully, it subscribes to every topic (# wildcard) * in order to later list every available topic */ void ImportFileWidget::onMqttConnect() { if (m_client->error() == QMqttClient::NoError) { m_connectTimeoutTimer->stop(); ui.gbManageSubscriptions->setVisible(true); //subscribing to every topic (# wildcard) in order to later list every available topic QMqttTopicFilter globalFilter{"#"}; m_mainSubscription = m_client->subscribe(globalFilter, 1); if (!m_mainSubscription) QMessageBox::critical(this, i18n("Couldn't subscribe"), i18n("Couldn't subscribe. Something went wrong")); } ui.cbConnection->setEnabled(true); ui.bManageConnections->setEnabled(true); emit subscriptionsChanged(); RESET_CURSOR; } /*! *\brief called when the client disconnects from the broker successfully * removes every information about the former connection */ void ImportFileWidget::onMqttDisconnect() { m_lastMessage.clear(); m_messageArrived.clear(); m_mqttSubscriptions.clear(); m_topicList.clear(); ui.twSubscriptions->clear(); ui.twTopics->clear(); m_searchTimer->stop(); m_connectTimeoutTimer->stop(); ui.gbManageSubscriptions->setVisible(false); ui.cbSourceType->setEnabled(true); ui.cbConnection->setEnabled(true); ui.bManageConnections->setEnabled(true); m_mqttReadyForPreview = false; m_searching = false; m_topicCompleter = new QCompleter; m_subscriptionCompleter = new QCompleter; emit subscriptionsChanged(); RESET_CURSOR; if (!m_initialisingMQTT) { if (!m_connectionTimedOut) QTimer::singleShot(300, this, &ImportFileWidget::mqttConnectionChanged); else m_connectionTimedOut = false; } } /*! *\brief called when the subscribe button is pressed * subscribes to the topic represented by the current item of twTopics */ void ImportFileWidget::mqttSubscribe() { QTreeWidgetItem* item = ui.twTopics->currentItem(); if (!item) { QMessageBox::warning(this, i18n("Warning"), i18n("You didn't select any item from the Tree Widget")); return; } //determine the topic name that the current item represents QTreeWidgetItem* tempItem = item; QString name = item->text(0); if (item->childCount() != 0) name.append("/#"); while(tempItem->parent() != nullptr) { tempItem = tempItem->parent(); name.prepend(tempItem->text(0) + '/'); } //check if the subscription already exists const QList& topLevelList = ui.twSubscriptions->findItems(name, Qt::MatchExactly); if (topLevelList.isEmpty() || topLevelList.first()->parent() != nullptr) { qDebug() << "Subscribe to: " << name; bool foundSuperior = false; for (int i = 0; i < ui.twSubscriptions->topLevelItemCount(); ++i) { //if the new subscirptions contains an already existing one, we remove the inferior one if (checkTopicContains(name, ui.twSubscriptions->topLevelItem(i)->text(0)) && name != ui.twSubscriptions->topLevelItem(i)->text(0)) { unsubscribeFromTopic(ui.twSubscriptions->topLevelItem(i)->text(0)); --i; continue; } //if there is a subscription containing the new one we set foundSuperior true if (checkTopicContains(ui.twSubscriptions->topLevelItem(i)->text(0), name) && name != ui.twSubscriptions->topLevelItem(i)->text(0)) { foundSuperior = true; qDebug()<<"Can't continue subscribe. Found superior for " << name <<" : "<< ui.twSubscriptions->topLevelItem(i)->text(0); break; } } //if there wasn't a superior subscription we can subscribe to the new topic if (!foundSuperior) { QStringList toplevelName; toplevelName.push_back(name); QTreeWidgetItem* newTopLevelItem = new QTreeWidgetItem(toplevelName); ui.twSubscriptions->addTopLevelItem(newTopLevelItem); const QMqttTopicFilter filter {name}; QMqttSubscription *tempSubscription = m_client->subscribe(filter, static_cast(ui.cbQos->currentText().toUInt()) ); if (tempSubscription) { m_mqttSubscriptions.push_back(tempSubscription); connect(tempSubscription, &QMqttSubscription::messageReceived, this, &ImportFileWidget::mqttSubscriptionMessageReceived); emit subscriptionsChanged(); } if (name.endsWith('#')) { //adding every topic that the subscription contains to twSubscriptions addSubscriptionChildren(item, newTopLevelItem); //if an already existing subscription contains a topic that the new subscription also contains //we decompose the already existing subscription //by unsubscribing from its topics, that are present in the new subscription as well const QStringList nameList = name.split('/', QString::SkipEmptyParts); const QString& root = nameList.first(); QVector children; for (int i = 0; i < ui.twSubscriptions->topLevelItemCount(); ++i) { if (ui.twSubscriptions->topLevelItem(i)->text(0).startsWith(root) && name != ui.twSubscriptions->topLevelItem(i)->text(0)) { children.clear(); //get the "leaf" children of the inspected subscription findSubscriptionLeafChildren(children, ui.twSubscriptions->topLevelItem(i)); for (int j = 0; j < children.size(); ++j) { if (checkTopicContains(name, children[j]->text(0))) { //if the new subscription contains a topic, we unsubscribe from it ui.twSubscriptions->setCurrentItem(children[j]); mqttUnsubscribe(); --i; } } } } } manageCommonLevelSubscriptions(); updateSubscriptionCompleter(); if (!ui.bWillMessage->isEnabled()) ui.bWillMessage->setEnabled(true); } else QMessageBox::warning(this, i18n("Warning"), i18n("You already subscribed to a topic containing this one")); } else QMessageBox::warning(this, i18n("Warning"), i18n("You already subscribed to this topic")); } /*! *\brief called when the unsubscribe button is pressed * unsubscribes from the topic represented by the current item of twSubscription */ void ImportFileWidget::mqttUnsubscribe() { QTreeWidgetItem* unsubscribeItem = ui.twSubscriptions->currentItem(); if (!unsubscribeItem) { QMessageBox::warning(this, i18n("Warning"), i18n("You didn't select any item from the Tree Widget")); return; } qDebug() << "Unsubscribe from: " << unsubscribeItem->text(0); //if it is a top level item, meaning a topic that we really subscribed to(not one that belongs to a subscription) //we can simply unsubscribe from it if (unsubscribeItem->parent() == nullptr) unsubscribeFromTopic(unsubscribeItem->text(0)); //otherwise we remove the selected item, but subscribe to every other topic, that was contained by //the selected item's parent subscription(top level item of twSubscriptions) else { while(unsubscribeItem->parent() != nullptr) { for (int i = 0; i < unsubscribeItem->parent()->childCount(); ++i) { if (unsubscribeItem->text(0) != unsubscribeItem->parent()->child(i)->text(0)) { const QMqttTopicFilter filter {unsubscribeItem->parent()->child(i)->text(0)}; QMqttSubscription *tempSubscription = m_client->subscribe(filter, static_cast(ui.cbQos->currentText().toUInt()) ); ui.twSubscriptions->addTopLevelItem(unsubscribeItem->parent()->takeChild(i)); if (tempSubscription) { m_mqttSubscriptions.push_back(tempSubscription); connect(tempSubscription, &QMqttSubscription::messageReceived, this, &ImportFileWidget::mqttSubscriptionMessageReceived); emit subscriptionsChanged(); } --i; } } unsubscribeItem = unsubscribeItem->parent(); } unsubscribeFromTopic(unsubscribeItem->text(0)); //check if any common topics were subscribed, if possible merge them manageCommonLevelSubscriptions(); } updateSubscriptionCompleter(); if (ui.twSubscriptions->topLevelItemCount() <= 0) ui.bWillMessage->setEnabled(false); } /*! *\brief called when the client receives a message * if the message arrived from a new topic, the topic is put in twTopics */ void ImportFileWidget::mqttMessageReceived(const QByteArray& message, const QMqttTopicName& topic) { Q_UNUSED(message); if (m_addedTopics.contains(topic.name())) return; m_addedTopics.push_back(topic.name()); QStringList name; QString rootName; const QChar sep = '/'; if (topic.name().contains(sep)) { const QStringList& list = topic.name().split(sep, QString::SkipEmptyParts); if (!list.isEmpty()) { rootName = list.at(0); name.append(list.at(0)); int topItemIdx = -1; //check whether the first level of the topic can be found in twTopics for (int i = 0; i < ui.twTopics->topLevelItemCount(); ++i) { if (ui.twTopics->topLevelItem(i)->text(0) == list.at(0)) { topItemIdx = i; break; } } QTreeWidgetItem* currentItem = nullptr; //if not we simply add every level of the topic to the tree if (topItemIdx < 0) { currentItem = new QTreeWidgetItem(name); ui.twTopics->addTopLevelItem(currentItem); for (int i = 1; i < list.size(); ++i) { name.clear(); name.append(list.at(i)); currentItem->addChild(new QTreeWidgetItem(name)); currentItem = currentItem->child(0); } } //otherwise we search for the first level that isn't part of the tree, //then add every level of the topic to the tree from that certain level else { currentItem = ui.twTopics->topLevelItem(topItemIdx); int listIdx = 1; for (; listIdx < list.size(); ++listIdx) { QTreeWidgetItem* childItem = nullptr; bool found = false; for (int j = 0; j < currentItem->childCount(); ++j) { childItem = currentItem->child(j); if (childItem->text(0) == list.at(listIdx)) { found = true; currentItem = childItem; break; } } if (!found) { //this is the level that isn't present in the tree break; } } //add every level to the tree starting with the first level that isn't part of the tree for (; listIdx < list.size(); ++listIdx) { name.clear(); name.append(list.at(listIdx)); currentItem->addChild(new QTreeWidgetItem(name)); currentItem = currentItem->child(currentItem->childCount() - 1); } } } } else { rootName = topic.name(); name.append(topic.name()); ui.twTopics->addTopLevelItem(new QTreeWidgetItem(name)); } //if a subscribed topic contains the new topic, we have to update twSubscriptions for (int i = 0; i < ui.twSubscriptions->topLevelItemCount(); ++i) { const QStringList subscriptionName = ui.twSubscriptions->topLevelItem(i)->text(0).split('/', QString::SkipEmptyParts); if (!subscriptionName.isEmpty()) { if (rootName == subscriptionName.first()) { updateSubscriptionTree(); break; } } } //signals that a newTopic was added, in order to fill the completer of leTopics emit newTopic(rootName); } /*! *\brief called when a new topic is added to the tree(twTopics) * appends the topic's root to the topicList if it isn't in the list already * then sets the completer for leTopics */ void ImportFileWidget::setTopicCompleter(const QString& topic) { if (!m_topicList.contains(topic)) { m_topicList.append(topic); if (!m_searching) { m_topicCompleter = new QCompleter(m_topicList, this); m_topicCompleter->setCompletionMode(QCompleter::PopupCompletion); m_topicCompleter->setCaseSensitivity(Qt::CaseSensitive); ui.leTopics->setCompleter(m_topicCompleter); } } } /*! *\brief called when 10 seconds passed since the last time the user searched for a certain root in twTopics * enables updating the completer for le */ void ImportFileWidget::topicTimeout() { m_searching = false; m_searchTimer->stop(); } /*! *\brief called when the client receives a message from a subscribed topic (that isn't the "#" wildcard) */ void ImportFileWidget::mqttSubscriptionMessageReceived(const QMqttMessage &msg) { qDebug()<<"message received from: "< i(m_messageArrived); while(i.hasNext()) { i.next(); if (i.value() == false ) { check = false; break; } } //if there is a message from every subscribed topic, we refresh the preview if (check) { m_mqttReadyForPreview = true; refreshPreview(); } } /*! *\brief called when use will message is changed in the settings widget * Updates the will settings */ void ImportFileWidget::useWillMessage(int state) { if (state == Qt::Checked) m_willSettings.MQTTUseWill = true; else if (state == Qt::Unchecked) m_willSettings.MQTTUseWill = false; } /*! *\brief called when the selected will message type is changed in the settings widget * Updates the will settings */ void ImportFileWidget::willMessageTypeChanged(int type) { m_willSettings.willMessageType = static_cast(type); } /*! *\brief called when the selected will message' type's retain flag is changed in the settings widget * Updates the will settings */ void ImportFileWidget::willRetainChanged(bool retain) { m_willSettings.willRetain = retain; } /*! *\brief called when the selected will update interval is changed in the settings widget * Updates the will settings */ void ImportFileWidget::willTimeIntervalChanged(int interval) { m_willSettings.willTimeInterval = interval; } /*! *\brief called when the selected will own message is changed in the settings widget * Updates the will settings */ void ImportFileWidget::willOwnMessageChanged(const QString& msg) { m_willSettings.willOwnMessage = msg; } /*! *\brief called when the selected will topic is changed in the settings widget * Updates the will settings */ void ImportFileWidget::willTopicChanged(const QString& topic) { m_willSettings.willTopic = topic; } /*! *\brief called when the selected will statistics are changed in the settings widget * Updates the will settings */ void ImportFileWidget::willStatisticsChanged(int index) { m_willSettings.willStatistics[index] = !m_willSettings.willStatistics[index]; } /*! *\brief called when the selected will message's QoS is changed in the settings widget * Updates the will settings */ void ImportFileWidget::willQoSChanged(int qos) { m_willSettings.willQoS = qos; } /*! *\brief called when the selected will update type is changed in the settings widget * Updates the will settings */ void ImportFileWidget::willUpdateTypeChanged(int updateType) { qDebug() << "update type changed: " <(updateType); } /*! *\brief called when the clientError of the MQTT client changes * * \param clientError the current error of the client */ void ImportFileWidget::mqttErrorChanged(QMqttClient::ClientError clientError) { switch (clientError) { case QMqttClient::BadUsernameOrPassword: QMessageBox::critical(this, i18n("Couldn't connect"), i18n("Wrong username or password")); break; case QMqttClient::IdRejected: QMessageBox::critical(this, i18n("Couldn't connect"), i18n("The client ID wasn't accepted")); break; case QMqttClient::ServerUnavailable: QMessageBox::critical(this, i18n("Server unavailable"), i18n("The broker couldn't be reached.")); break; case QMqttClient::NotAuthorized: QMessageBox::critical(this, i18n("Not authorized"), i18n("The client is not authorized to connect.")); break; case QMqttClient::UnknownError: QMessageBox::critical(this, i18n("Unknown MQTT error"), i18n("An unknown error occurred.")); break; case QMqttClient::NoError: case QMqttClient::InvalidProtocolVersion: case QMqttClient::TransportInvalid: case QMqttClient::ProtocolViolation: break; default: break; } } /*! *\brief called when leTopics' text is changed * if the rootName can be found in twTopics, then we scroll it to the top of the tree widget * * \param rootName the current text of leTopics */ void ImportFileWidget::scrollToTopicTreeItem(const QString& rootName) { m_searching = true; m_searchTimer->start(); int topItemIdx = -1; for (int i = 0; i < ui.twTopics->topLevelItemCount(); ++i) if (ui.twTopics->topLevelItem(i)->text(0) == rootName) { topItemIdx = i; break; } if (topItemIdx >= 0) ui.twTopics->scrollToItem(ui.twTopics->topLevelItem(topItemIdx), QAbstractItemView::ScrollHint::PositionAtTop); } /*! *\brief called when leSubscriptions' text is changed * if the rootName can be found in twSubscriptions, then we scroll it to the top of the tree widget * * \param rootName the current text of leSubscriptions */ void ImportFileWidget::scrollToSubsriptionTreeItem(const QString& rootName) { m_searching = true; m_searchTimer->start(); int topItemIdx = -1; for (int i = 0; i < ui.twSubscriptions->topLevelItemCount(); ++i) if (ui.twSubscriptions->topLevelItem(i)->text(0) == rootName) { topItemIdx = i; break; } if (topItemIdx >= 0) ui.twSubscriptions->scrollToItem(ui.twSubscriptions->topLevelItem(topItemIdx), QAbstractItemView::ScrollHint::PositionAtTop); } /*! *\brief called when m_connectTimeoutTimer ticks, * meaning that the client couldn't connect to the broker in 5 seconds * disconnects the client, stops the timer, and warns the user */ void ImportFileWidget::mqttConnectTimeout() { m_connectionTimedOut = true; m_client->disconnectFromHost(); m_connectTimeoutTimer->stop(); QMessageBox::warning(this, i18n("Warning"), i18n("Connecting to the given broker timed out! Try changing the settings")); RESET_CURSOR; } /*! Shows the MQTT connection manager where the connections are created and edited. The selected connection is selected in the connection combo box in this widget. */ void ImportFileWidget::showMQTTConnectionManager() { bool previousConnectionChanged = false; MQTTConnectionManagerDialog* dlg = new MQTTConnectionManagerDialog(this, ui.cbConnection->currentText(), previousConnectionChanged); if (dlg->exec() == QDialog::Accepted) { //re-read the available connections to be in sync with the changes in MQTTConnectionManager m_initialisingMQTT = true; const QString& prevConn = ui.cbConnection->currentText(); ui.cbConnection->clear(); readMQTTConnections(); m_initialisingMQTT = false; //select the connection the user has selected in MQTTConnectionManager const QString& conn = dlg->connection(); int index = ui.cbConnection->findText(conn); if (conn != prevConn) {//Current connection isn't the previous one if (ui.cbConnection->currentIndex() != index) ui.cbConnection->setCurrentIndex(index); else mqttConnectionChanged(); } else if (dlg->initialConnectionChanged()) {//Current connection is the same with previous one but it changed if (ui.cbConnection->currentIndex() == index) mqttConnectionChanged(); else ui.cbConnection->setCurrentIndex(index); } else { //Previous connection wasn't changed m_initialisingMQTT = true; ui.cbConnection->setCurrentIndex(index); m_initialisingMQTT = false; } } delete dlg; } /*! loads all available saved MQTT nconnections */ void ImportFileWidget::readMQTTConnections() { qDebug()<< "ImportFileWidget: reading available MQTT connections"; KConfig config(m_configPath, KConfig::SimpleConfig); for (const auto& name : config.groupList()) ui.cbConnection->addItem(name); } /*! * \brief Shows the mqtt will settings widget, which allows the user to modify the will settings */ void ImportFileWidget::showWillSettings() { QMenu menu; QVector children; for (int i = 0; i < ui.twSubscriptions->topLevelItemCount(); ++i) findSubscriptionLeafChildren(children, ui.twSubscriptions->topLevelItem(i)); QVector topics; for (int i = 0; i < children.size(); ++i) topics.append(children[i]->text(0)); MQTTWillSettingsWidget willSettings(&menu, m_willSettings, topics); connect(&willSettings, &MQTTWillSettingsWidget::useChanged, this, &ImportFileWidget::useWillMessage); connect(&willSettings, &MQTTWillSettingsWidget::messageTypeChanged, this, &ImportFileWidget::willMessageTypeChanged); connect(&willSettings, &MQTTWillSettingsWidget::updateTypeChanged, this, &ImportFileWidget::willUpdateTypeChanged); connect(&willSettings, &MQTTWillSettingsWidget::retainChanged, this, &ImportFileWidget::willRetainChanged); connect(&willSettings, &MQTTWillSettingsWidget::intervalChanged, this, &ImportFileWidget::willTimeIntervalChanged); connect(&willSettings, &MQTTWillSettingsWidget::ownMessageChanged, this, &ImportFileWidget::willOwnMessageChanged); connect(&willSettings, &MQTTWillSettingsWidget::topicChanged, this, &ImportFileWidget::willTopicChanged); connect(&willSettings, &MQTTWillSettingsWidget::statisticsChanged, this, &ImportFileWidget::willStatisticsChanged); connect(&willSettings, &MQTTWillSettingsWidget::QoSChanged, this, &ImportFileWidget::willQoSChanged); connect(&willSettings, SIGNAL(canceled()), &menu, SLOT(close())); QWidgetAction* widgetAction = new QWidgetAction(this); widgetAction->setDefaultWidget(&willSettings); menu.addAction(widgetAction); const QPoint pos(ui.bWillMessage->sizeHint().width(),ui.bWillMessage->sizeHint().height()); menu.exec(ui.bWillMessage->mapToGlobal(pos)); } #endif