diff --git a/ChangeLog b/ChangeLog index 66ccc8f4d..fcd05bb56 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,274 +1,275 @@ -----2.7----- New features: * Allow to move objects (drag&drop) to different folders in the project explorer * [spreadsheet] allow to insert multiple rows and columns in one step * [spreadsheet] improved the handling of calculated columns (values calculated via a formula): * Allow to automatically update the calculated column on data changes in the variable columns * Invalidate the calculated column if one of the variable columns was deleted * In "Functions Values" dialog highlight the variable column combobox red if the variable column was deleted in the project * In "Functions Values" dialog don't allow to select columns to be calculated as variable columns (avoid circular dependencies) + * In "Used in" context menu of a column show all other calculated columns where this column is used as a variable * [spreadsheet] when pasting data into empty columns, automatically convert their type to the type of the data to be pasted * BUG 406805 - support für directory structures in ROOT files * Automatically update curves if a data column deleted before was re-added again (during the import or manually) * When auto-scaling in the plot take also the error bars into account, if available * For live data sources allow to save the relative path to the data source UX improvements: * in the "Import Data" dialog show the history of the recently imported files * Improved the selection of curves in the plot that are near to or behind each other * BUG 408529 - Allow to use DEL to delete cells in spreadsheet and matrix * BUG 408537 - Allow to insert multiple rows and columns in the spreadsheet in one step * Added translations to Windows and macOS packages * Allow to stop the current selection in the plot with ESC * BUG 410855 - Seed the random number generator with the current time in order not to generate repetitive numbers in the spreadsheet. Performance: * Increased the plotting speed for curves with many data points by removing points which are shown on the same pixel in the scene Bug fixes: * Disable autoscale when zooming, otherwise wrong (= auto scaled) data region is shown after project save&open * [macOS] fix determining number of lines of data files * Bug 408530 - correctly show error bars for data source columns having NANs * Bug 407847 - fixed the broken project import * [data fitting] handle weighted fitting with zero errors correctly * [data fitting] allow weighted fitting with data source from curve * [data fitting] fix function for calculating bounded values * [nsl] fix memory leak in line simplification (Douglas Peuker variant) * [nsl] fix bug in calculating Bessel polynomials on Windows (used in Bessel filter) -----2.6----- New features: * Histogram * Import from MQTT sources * Import of ROOT (CERN) TH1 histograms * Import of Ngspice raw files (ASCII and binary) * Import of data in JSON format (JSON arrays and objects) * Improved import of NetCDF files * Added file type specific summary and content for special file types in info box of import dialog * Convolution/Deconvolution of data sets (sampling interval, linear/circular, normalization, wrap, standard kernel) * Cross-/Autocorrelation of data sets (sampling interval, linear/circular, normalization) * Allow to specify the number format when exporting spreadsheet and matrix * Improved user interface for data fitting (add fit function preview, show parameters directly, make options foldable) * [spreadsheet] when filling a float column with row numbers, automatically convert its type to integer * [spreadsheet] when filling an integer column with function values, automatically convert its type to float * [spreadsheet] data manipulation: add/subtract/multiply/divide for column values * [spreadsheet] export to SQLite * [matrix] data manipulation: add/subtract/multiply/divide for matrix values * [worksheet] Allow to specify different border shapes for labels (rectangle, eclipse, etc.) * [worksheet] Allow to rotate plot legends * [worksheet] Better positioning of rotated axis tick labels * [worksheet] Allow to make plots not-interactive (ignore mouse drag and wheel events) to avoid unwanted occasional panning and zooming * Allow to connect to SQL databases via ODBC * Show the amount of consumed memory in the status bar (optional) * Allow to change the settings for different computer algebra systems (Maxima, etc.) directly in LabPlot (embedd Cantor's settings widgets) Bug fixes: * Fixed several problems in live data support * [spreadsheet] properly calculate function values out of integer x-values * [matrix] fix editing integer values * [import] fix preview update and add missing close of netcdf files * Don't crash when a scaling factor equal to zero was set for axis -----2.5----- New features: * Support for reading and plotting of live-data * Improved data fitting * Automatically guess parameter of custom models * Better result presentation * Support different weight types * Consider given x- and y-error when fitting (can be switched off) * Show t statistics, P > |t| and confidence interval * Calculate p-value for chi-square and F test in nonlinear fitting * added fit models for most statistical distributions * Improved theming * Apply themes to worksheet and to all its children * Respect theme settings also in plot legends and labels * Allow to disable theming in worksheets and plots after a theme was selected * Show currently active theme in the "Apply theme" button * New application option in the settings for the default theme used for new worksheets. * Support different data types * auto detect integer and datetime data in import * support number locale and datetime formats * improved data type support in spreadsheets * Import from SQL databases (tables or custom queries) * Import Origin OPJ projects * Much better support for Windows and macOS * Syntax highlighting for LaTeX in the text label * Allow to set the background color for LaTeX labels * Support Hermite polynomials from GSL 2.4 * Support error functions and related functions from libcerf * "Used in" sub-menu in column contex menu for faster navigation to the curves consuming the column * Direct application of analysis functions (smoothing, interpolating, etc.) on the ploted data via curve's context menu * Direct application of analysis functions on the data in the spreadsheet and plotting of the results via spreadsheet's context menu * Drag columns in the project explorer and drop them on plots (either in a worksheet view or in the project explorer) to create curves * "Show last N points" and "Show first N points" data ranges in cartesian plot * Added CLI option --presenter to start LabPlot directly in the presenter mode * Added CLI parameter to directly open project files (LabPlot or Origin) * Allow drag&drop of projects files (LabPlot and Origin) on the main window to load the project * Allow drag&drop of data files on the main window to import the data * Show tooltips for the supported mathematical functions and constants in the expression text field * Automatically switch to the scientific representation for numbers bigger than 10^4 on the axis tick labels * Automatically allow the latex typesetting in the application after the latex environment was installed later without making the user to go to the settings dialog * Allow to change the color scheme for the application * Smooth and animated zooming in the worksheet view * Allow to add text labels to plots * Improved building with MSVC, Intel and PGI compiler Performance improvements: * Faster copy&paste in the spreadsheet Bug fixes: * Bug 379877 - masked rows in spreadsheet not restored in project * Calculation of fit results corrected * Axes now support values larger than FLT_MAX (~10^38) and smaller than FLT_MIN (~10^-38) * When a LabPlot project is being droped in the main window, load the project directly instead of showing the import file dialog * Correctly save and restore masked cells * Don't crash if the rc-file was not found during the startup -----2.4----- New features: * Support themes for plots * Import and editing of FITS data files * Data reduction by removing data points using multiple algorithms * Numerical differentiation and integration with several options * Many new pre-defined fit models (Gompertz, Weibull, Log-Normal, Gumbel, etc.) sorted in categories * Fit parameter now support fixed values, lower/upper limits and use Unicode * Fit model and random number distribution formulas are now rendered with LaTeX * Support user specified x range in all analysis functions * Allow to enter complete LaTeX documents in text labels * Configuration parameter to use different LaTex engines (LuaLaTex, XeLatex, pdfLaTex, LaTex) * Disable LaTeX typesetting if no LaTex installation (and other required tools) were found at runtime * Presenter mode for worksheets * Support for Mac OS * Support for Julia's vectors and tuples in CAS worksheets (requires Cantor v. 16.12 or higher) * Allow to jump directly to the data source spreadsheet via XYCurve's context menu * Select and delete multiple objects in project explorer * Improved and extended internal parser for mathematical expressions * Copy of worksheet elements as image to the clipboard via CTRL+C Bug fixes: * BUG: 361326 - Allow to select curves with overlapping bounding boxes * Correctly load worksheet sizes from saved templates * Fixed crash when removing columns in spreadsheet * Fixed crash when fitting using GSL >= 2 * List of available functions corrected * Constants are now available with full accuracy * Windows: Import of files and open recent files fixed -----2.3----- New features: * Integration of Cantor - Support for different open-source computer algebra systems * Statistics on spreadsheets and matrices * Export of spreadsheets and matrices to LaTeX tables * Interpolation of data including different splines, cosine, exponential, cubic Hermite (Catmull-Rom, cardinal, Kochanek-Bartels) and rational functions * Data smoothing using moving average (centered or lagged), percentile filter or Savitzky-Golay algorithm * Fourier filter (low pass, high pass, band pass, band reject) with ideal, Butterworth, Chebychev I+II, Legendre or Bessel-Thomson filter * Fourier transform with many window functions (Welch, Hann, Hamming, etc.) calculating magnitude, amplitude, power, phase, dB, etc. and supporting one/two sided spectrum with or without shift and x scaling to frequency, index or period * Filter and search capabilities in the drop down box for the selection of data sources * Sigmoid function as a new pre-defined fit model * Support for compiling on Microsoft Windows Performance improvements: * Faster generation of random values * Global option to enable/disable the double-buffering for faster painting of curves (enabled on default) Bug fixes: * Save and restore last used setting in RandomValuesDialog * Update axis title shape on title rotations correctly * Save and restore custom column widths in the spreadsheet * Fixed sporadic crashes during project close -----2.2----- New features: * Datapicker - tool for extracting curves and data points from imported images * Custom point on the plot with user-defined position and appearance * Accept drag&drop events * Support GSL 2.x * Import and export dialogs save and restore their settings and sizes Performance improvements: * Faster rendering of the image view of the matrix Bug fixes: * BUG: 354744 - make setting of range auto scaling in CartesianPlot undo/redo-able * Removed couple of hard coded sizes given in pixels for better user-experience on hidpi-displays * Fixes the bug with disabled undo/redo-actions in after the undo-history was cleared * Keep the information about the columns to be shown in the project explorer after project close * Fixed some bugs in painting of the selection band on the worksheet * Allow to open gz- and bz2-compressed LabPlot project files on the command line interface -----2.1----- New features: * New Matrix view for handling matrix data. * Workbook - a new container for grouping several objects of type Spreadsheet and/or Matrix. * Import of binary, image, NetCDF and HDF data into spreadsheet or matrix. * Visual HDF and NetCDF parser to view content of files and import data sets. * Preview of all supported file types in import dialog. * Transparently import compressed data files. * In xy-curve the points may not be connected by the line if there are NANs in-between. This behaviour is controlled by the parameter "skip gaps". * Multiplier of Pi format of the plot axis for periodical functions. * New operations on columns in Spreadsheet - reverse, drop values and mask values. * Formula used to generate the values in a column is stored and can be changed/adjusted in the formula dialog afterwards. * Curve filling: the area below, under, left to or right to the curve can be filled. * Support for multiple variables in "Function Values"-dialog - new data in the spreadsheet can be calculated out of multiple columns. Performance improvements: * Speeded up the creation of new columns during the import Bug fixes: * Fixed wrong behaviour when doing "zoom&select" in a plot and then deselecting the plot - it was not possible anymore to select the plot again on the worksheet. -----2.0.2----- New features: * Plot 2D-curves defined by mathematical equations in cartesian and polar coordinates or via a parametric equation. * Linear and non-linear regression analysis. Several predefined fit-models are provided. User-defined models are also possible. * Besides fixed worksheet sizes (predefined sizes like A4 etc. or user-defined), complete view size can be used. All sizes are automatically adjusted on resize. * Different axis arrow types. * "select region and zoom in", "select x-region and zoom in", "select y-region and zoom in" functions for cartesian plot. * Rounded border for several worksheet objects (plot area, legend etc.) * Hover effect for axis, curve and text label. * Added a MessageBox - ask befor deleting worksheet objects. * Added three new types for drop lines - "zero baseline", "min baseline" and "max baseline" * Fill the selection in Spreadsheet with a constant value provided by the user * Fill columns with uniform and non-uniform random numbers, several distributions are available. * Fill columns with function values * Allow custom resolutions for PNG-export * Export of the spreadsheet to a text file. * Simultaneous zooming and navigation accross multiple plots. * Implemented "Powers of 10/2/e" for the axis tick labels Bug fixes: * Don't crash when trying to create a plot in case no rc-file was installed. * Don't allow to select unwanted objects in TreeViewComboBox in ImportDialog and XYCurveDock. * Corrected painting of background images in plot area and legend. * BUG: 330723 - fixed weird selection in spreadsheet. * BUG: 330774 - fixed wrong positioning of axis on orientation changes. * Update main window title when project name is changed -----2.0.1----- Bug fix release. Solved issues: * Fixed wrong scaling of legend's text labels in pdf-export * Fixed memory corruption in CartesianPlotDock that can lead to crashes -----2.0.0----- First stable release of LabPlot2. LabPlot2 is a complete rewrite of LabPlot1 and lacks in this release a lot of features available in the predecessor. On the other hand, the GUI and the usability is more superior as compared to LabPlot1 and there are several new features that were not available in LabPlot1. Brief summary of features and capabilities of LabPlot2 implemented in the first release: * project-based management of data * created objects are organized in a tree and are visualized and accessible in the project explorer * for a better management of objects, additional folders and sub-folders can be created within a project * spreadsheet with very basic functionality is available for manual data entry * "file data source" can be used to import a file and to let LabPlot2 watch for changes in that file * external data from an ascii file can be also directly imported to a spreadsheet for further editing * worksheet is the main object where plots, labels etc. are placed on * several zooming functions for the worksheet * only cartesian plot is implemented in the first release * arbitrary number of freely positionable axes is possible * xy-curve is implemented. As the data source for the x- and y-values columns either from a spreadsheet or from a file data source can be used * several zooming and "movement" functions are available for the plots which help to navigate through data * legend for xy-plots * a lot of properties of the worksheet elements can be edited in a very easy way in the corresponding dock widgets * plots on the worksheet can be put into a horizontal, vertical or grid layouts * export of worksheet (entire worksheet or current seleciton) to pdf, eps, png and svg diff --git a/src/backend/core/column/Column.cpp b/src/backend/core/column/Column.cpp index f967573b1..30f67eaa4 100644 --- a/src/backend/core/column/Column.cpp +++ b/src/backend/core/column/Column.cpp @@ -1,1921 +1,1936 @@ /*************************************************************************** File : Column.cpp Project : LabPlot Description : Aspect that manages a column -------------------------------------------------------------------- Copyright : (C) 2007-2009 Tilman Benkert (thzs@gmx.net) Copyright : (C) 2013-2017 Alexander Semke (alexander.semke@web.de) Copyright : (C) 2017 Stefan Gerlach (stefan.gerlach@uni.kn) ***************************************************************************/ /*************************************************************************** * * * This program is free software; you can redistribute it and/or modify * * it under the terms of the GNU General Public License as published by * * the Free Software Foundation; either version 2 of the License, or * * (at your option) any later version. * * * * This program is distributed in the hope that it will be useful, * * but WITHOUT ANY WARRANTY; without even the implied warranty of * * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * * GNU General Public License for more details. * * * * You should have received a copy of the GNU General Public License * * along with this program; if not, write to the Free Software * * Foundation, Inc., 51 Franklin Street, Fifth Floor, * * Boston, MA 02110-1301 USA * * * ***************************************************************************/ #include "backend/core/column/Column.h" #include "backend/core/column/ColumnPrivate.h" #include "backend/core/column/ColumnStringIO.h" #include "backend/core/column/columncommands.h" #include "backend/core/Project.h" #include "backend/lib/XmlStreamReader.h" #include "backend/core/datatypes/String2DateTimeFilter.h" #include "backend/core/datatypes/DateTime2StringFilter.h" #include "backend/worksheet/plots/cartesian/XYCurve.h" #include "backend/worksheet/plots/cartesian/XYAnalysisCurve.h" extern "C" { #include } #include #include #include #include #include #include /** * \class Column * \brief Aspect that manages a column * * This class represents a column, i.e., (mathematically) a 1D vector of * values with a header. It provides a public reading and (undo aware) writing * interface as defined in AbstractColumn. A column * can have one of currently three data types: double, QString, or * QDateTime. The string representation of the values can differ depending * on the mode of the column. * * Column inherits from AbstractAspect and is intended to be a child * of the corresponding Spreadsheet in the aspect hierarchy. Columns don't * have a view as they are intended to be displayed inside a spreadsheet. */ Column::Column(const QString& name, ColumnMode mode) : AbstractColumn(name, AspectType::Column), d(new ColumnPrivate(this, mode)) { init(); } /** * \brief Common part of ctors */ void Column::init() { m_string_io = new ColumnStringIO(this); d->inputFilter()->input(0, m_string_io); d->outputFilter()->input(0, this); d->inputFilter()->setHidden(true); d->outputFilter()->setHidden(true); addChild(d->inputFilter()); addChild(d->outputFilter()); m_suppressDataChangedSignal = false; m_usedInActionGroup = new QActionGroup(this); connect(m_usedInActionGroup, &QActionGroup::triggered, this, &Column::navigateTo); } Column::~Column() { delete m_string_io; delete d; } QMenu* Column::createContextMenu() { QMenu* menu = AbstractAspect::createContextMenu(); QAction* firstAction{nullptr}; //insert after "rename" and "delete" actions, if available. //MQTTTopic columns don't have these actions if (menu->actions().size() > 1) firstAction = menu->actions().at(1); //add actions available in SpreadsheetView //TODO: we don't need to add anything from the view for MQTTTopic columns. //at the moment it's ok to check to the null pointer for firstAction here. //later, once we have some actions in the menu also for MQTT topics we'll //need to explicitly to dynamic_cast for MQTTTopic if (firstAction) emit requestProjectContextMenu(menu); //"Used in" menu containing all curves where the column is used QMenu* usedInMenu = new QMenu(i18n("Used in")); usedInMenu->setIcon(QIcon::fromTheme("go-next-view")); + usedInMenu->addSection(i18n("Curves")); //remove previously added actions for (auto* action : m_usedInActionGroup->actions()) m_usedInActionGroup->removeAction(action); //add curves where the column is currently in use QVector curves = project()->children(AbstractAspect::Recursive); for (const auto* curve : curves) { bool used = false; const auto* analysisCurve = dynamic_cast(curve); if (analysisCurve) { if (analysisCurve->dataSourceType() == XYAnalysisCurve::DataSourceSpreadsheet && (analysisCurve->xDataColumn() == this || analysisCurve->yDataColumn() == this || analysisCurve->y2DataColumn() == this) ) used = true; } else { if (curve->xColumn() == this || curve->yColumn() == this) used = true; } if (used) { QAction* action = new QAction(curve->icon(), curve->name(), m_usedInActionGroup); action->setData(curve->path()); usedInMenu->addAction(action); } } + //add calculated columns where the column is used in formula variables + usedInMenu->addSection(i18n("Calculated Columns")); + QVector columns = project()->children(AbstractAspect::Recursive); + const QString& path = this->path(); + for (const auto* column : columns) { + auto paths = column->formulaVariableColumnPaths(); + if (paths.indexOf(path) != -1) { + QAction* action = new QAction(column->icon(), column->name(), m_usedInActionGroup); + action->setData(column->path()); + usedInMenu->addAction(action); + } + } + + if (firstAction) menu->insertSeparator(firstAction); menu->insertMenu(firstAction, usedInMenu); menu->insertSeparator(firstAction); return menu; } void Column::navigateTo(QAction* action) { project()->navigateTo(action->data().toString()); } /*! * */ void Column::setSuppressDataChangedSignal(bool b) { m_suppressDataChangedSignal = b; } /** * \brief Set the column mode * * This sets the column mode and, if * necessary, converts it to another datatype. */ void Column::setColumnMode(AbstractColumn::ColumnMode mode) { if (mode == columnMode()) return; DEBUG("Column::setColumnMode()"); beginMacro(i18n("%1: change column type", name())); auto* old_input_filter = d->inputFilter(); auto* old_output_filter = d->outputFilter(); exec(new ColumnSetModeCmd(d, mode)); if (d->inputFilter() != old_input_filter) { removeChild(old_input_filter); addChild(d->inputFilter()); d->inputFilter()->input(0, m_string_io); } if (d->outputFilter() != old_output_filter) { removeChild(old_output_filter); addChild(d->outputFilter()); d->outputFilter()->input(0, this); } endMacro(); DEBUG("Column::setColumnMode() DONE"); } void Column::setColumnModeFast(AbstractColumn::ColumnMode mode) { if (mode == columnMode()) return; auto* old_input_filter = d->inputFilter(); auto* old_output_filter = d->outputFilter(); exec(new ColumnSetModeCmd(d, mode)); if (d->inputFilter() != old_input_filter) { removeChild(old_input_filter); addChildFast(d->inputFilter()); d->inputFilter()->input(0, m_string_io); } if (d->outputFilter() != old_output_filter) { removeChild(old_output_filter); addChildFast(d->outputFilter()); d->outputFilter()->input(0, this); } } bool Column::isDraggable() const { return true; } QVector Column::dropableOn() const { return QVector{AspectType::CartesianPlot}; } /** * \brief Copy another column of the same type * * This function will return false if the data type * of 'other' is not the same as the type of 'this'. * Use a filter to convert a column to another type. */ bool Column::copy(const AbstractColumn* other) { Q_CHECK_PTR(other); if (other->columnMode() != columnMode()) return false; exec(new ColumnFullCopyCmd(d, other)); return true; } /** * \brief Copies a part of another column of the same type * * This function will return false if the data type * of 'other' is not the same as the type of 'this'. * \param other pointer to the column to copy * \param src_start first row to copy in the column to copy * \param dest_start first row to copy in * \param num_rows the number of rows to copy */ bool Column::copy(const AbstractColumn* source, int source_start, int dest_start, int num_rows) { Q_CHECK_PTR(source); if (source->columnMode() != columnMode()) return false; exec(new ColumnPartialCopyCmd(d, source, source_start, dest_start, num_rows)); return true; } /** * \brief Insert some empty (or initialized with zero) rows */ void Column::handleRowInsertion(int before, int count) { AbstractColumn::handleRowInsertion(before, count); exec(new ColumnInsertRowsCmd(d, before, count)); if (!m_suppressDataChangedSignal) emit dataChanged(this); d->statisticsAvailable = false; d->hasValuesAvailable = false; d->propertiesAvailable = false; } /** * \brief Remove 'count' rows starting from row 'first' */ void Column::handleRowRemoval(int first, int count) { AbstractColumn::handleRowRemoval(first, count); exec(new ColumnRemoveRowsCmd(d, first, count)); if (!m_suppressDataChangedSignal) emit dataChanged(this); d->statisticsAvailable = false; d->hasValuesAvailable = false; d->propertiesAvailable = false; } /** * \brief Set the column plot designation */ void Column::setPlotDesignation(AbstractColumn::PlotDesignation pd) { if (pd != plotDesignation()) exec(new ColumnSetPlotDesignationCmd(d, pd)); } /** * \brief Get width */ int Column::width() const { return d->width(); } /** * \brief Set width */ void Column::setWidth(int value) { d->setWidth(value); } /** * \brief Clear the whole column */ void Column::clear() { exec(new ColumnClearCmd(d)); } //////////////////////////////////////////////////////////////////////////////// //@} //////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////// //! \name Formula related functions //@{ //////////////////////////////////////////////////////////////////////////////// /** * \brief Returns the formula used to generate column values */ QString Column:: formula() const { return d->formula(); } const QStringList& Column::formulaVariableNames() const { return d->formulaVariableNames(); } const QVector& Column::formulaVariableColumns() const { return d->formulaVariableColumns(); } const QStringList& Column::formulaVariableColumnPaths() const { return d->formulaVariableColumnPaths(); } void Column::setformulVariableColumnsPath(int index, const QString path) { d->setformulVariableColumnsPath(index, path); } void Column::setformulVariableColumn(int index, Column* column) { d->setformulVariableColumn(index, column); } bool Column::formulaAutoUpdate() const { return d->formulaAutoUpdate(); } /** * \brief Sets the formula used to generate column values */ void Column::setFormula(const QString& formula, const QStringList& variableNames, const QVector& columns, bool autoUpdate) { exec(new ColumnSetGlobalFormulaCmd(d, formula, variableNames, columns, autoUpdate)); } /*! * in case the cell values are calculated via a global column formula, * updates the values on data changes in all the dependent changes in the * "variable columns". */ void Column::updateFormula() { d->updateFormula(); } /** * \brief Set a formula string for an interval of rows */ void Column::setFormula(const Interval& i, const QString& formula) { exec(new ColumnSetFormulaCmd(d, i, formula)); } /** * \brief Overloaded function for convenience */ void Column::setFormula(int row, const QString& formula) { setFormula(Interval(row, row), formula); } /** * \brief Clear all formulas */ void Column::clearFormulas() { exec(new ColumnClearFormulasCmd(d)); } //////////////////////////////////////////////////////////////////////////////// //@} //////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////// //! \name type specific functions //@{ //////////////////////////////////////////////////////////////////////////////// /** * \brief Set the content of row 'row' * * Use this only when columnMode() is Text */ void Column::setTextAt(int row, const QString& new_value) { DEBUG("Column::setTextAt()"); d->statisticsAvailable = false; d->propertiesAvailable = false; exec(new ColumnSetTextCmd(d, row, new_value)); } /** * \brief Replace a range of values * * Use this only when columnMode() is Text */ void Column::replaceTexts(int first, const QVector& new_values) { DEBUG("Column::replaceTexts()"); if (!new_values.isEmpty()) { //TODO: do we really need this check? d->statisticsAvailable = false; d->propertiesAvailable = false; exec(new ColumnReplaceTextsCmd(d, first, new_values)); } } /** * \brief Set the content of row 'row' * * Use this only when columnMode() is DateTime, Month or Day */ void Column::setDateAt(int row, QDate new_value) { d->statisticsAvailable = false; d->propertiesAvailable = false; setDateTimeAt(row, QDateTime(new_value, timeAt(row))); } /** * \brief Set the content of row 'row' * * Use this only when columnMode() is DateTime, Month or Day */ void Column::setTimeAt(int row, QTime new_value) { d->statisticsAvailable = false; d->propertiesAvailable = false; setDateTimeAt(row, QDateTime(dateAt(row), new_value)); } /** * \brief Set the content of row 'row' * * Use this only when columnMode() is DateTime, Month or Day */ void Column::setDateTimeAt(int row, const QDateTime& new_value) { d->statisticsAvailable = false; d->propertiesAvailable = false; exec(new ColumnSetDateTimeCmd(d, row, new_value)); } /** * \brief Replace a range of values * * Use this only when columnMode() is DateTime, Month or Day */ void Column::replaceDateTimes(int first, const QVector& new_values) { if (!new_values.isEmpty()) { d->statisticsAvailable = false; d->propertiesAvailable = false; exec(new ColumnReplaceDateTimesCmd(d, first, new_values)); } } /** * \brief Set the content of row 'row' * * Use this only when columnMode() is Numeric */ void Column::setValueAt(int row, const double new_value) { // DEBUG("Column::setValueAt()"); d->statisticsAvailable = false; d->hasValuesAvailable = false; d->propertiesAvailable = false; exec(new ColumnSetValueCmd(d, row, new_value)); } /** * \brief Replace a range of values * * Use this only when columnMode() is Numeric */ void Column::replaceValues(int first, const QVector& new_values) { DEBUG("Column::replaceValues()"); if (!new_values.isEmpty()) { d->statisticsAvailable = false; d->hasValuesAvailable = false; d->propertiesAvailable = false; exec(new ColumnReplaceValuesCmd(d, first, new_values)); } } /** * \brief Set the content of row 'row' * * Use this only when columnMode() is Integer */ void Column::setIntegerAt(int row, const int new_value) { DEBUG("Column::setIntegerAt()"); d->statisticsAvailable = false; d->hasValuesAvailable = false; d->propertiesAvailable = false; exec(new ColumnSetIntegerCmd(d, row, new_value)); } /** * \brief Replace a range of values * * Use this only when columnMode() is Integer */ void Column::replaceInteger(int first, const QVector& new_values) { DEBUG("Column::replaceInteger()"); if (!new_values.isEmpty()) { d->statisticsAvailable = false; d->hasValuesAvailable = false; d->propertiesAvailable = false; exec(new ColumnReplaceIntegersCmd(d, first, new_values)); } } /*! * \brief Column::properties * Returns the column properties of this curve (monoton increasing, monoton decreasing, ... ) * \see AbstractColumn::properties */ AbstractColumn::Properties Column::properties() const{ if (!d->propertiesAvailable) d->updateProperties(); return d->properties; } const Column::ColumnStatistics& Column::statistics() const { if (!d->statisticsAvailable) calculateStatistics(); return d->statistics; } void Column::calculateStatistics() const { d->statistics = ColumnStatistics(); ColumnStatistics& statistics = d->statistics; // TODO: support other data types? auto* rowValues = reinterpret_cast*>(data()); size_t notNanCount = 0; double val; double columnSum = 0.0; double columnProduct = 1.0; double columnSumNeg = 0.0; double columnSumSquare = 0.0; statistics.minimum = INFINITY; statistics.maximum = -INFINITY; QMap frequencyOfValues; QVector rowData; rowData.reserve(rowValues->size()); for (int row = 0; row < rowValues->size(); ++row) { val = rowValues->value(row); if (std::isnan(val) || isMasked(row)) continue; if (val < statistics.minimum) statistics.minimum = val; if (val > statistics.maximum) statistics.maximum = val; columnSum+= val; columnSumNeg += (1.0 / val); columnSumSquare += pow(val, 2.0); columnProduct *= val; if (frequencyOfValues.contains(val)) frequencyOfValues.operator [](val)++; else frequencyOfValues.insert(val, 1); ++notNanCount; rowData.push_back(val); } if (notNanCount == 0) { d->statisticsAvailable = true; return; } if (rowData.size() < rowValues->size()) rowData.squeeze(); statistics.arithmeticMean = columnSum / notNanCount; statistics.geometricMean = pow(columnProduct, 1.0 / notNanCount); statistics.harmonicMean = notNanCount / columnSumNeg; statistics.contraharmonicMean = columnSumSquare / columnSum; double columnSumVariance = 0; double columnSumMeanDeviation = 0.0; double columnSumMedianDeviation = 0.0; double sumForCentralMoment_r3 = 0.0; double sumForCentralMoment_r4 = 0.0; gsl_sort(rowData.data(), 1, notNanCount); statistics.median = (notNanCount%2) ? rowData.at((int)((notNanCount-1)/2)) : (rowData.at((int)((notNanCount-1)/2)) + rowData.at((int)(notNanCount/2)))/2.0; QVector absoluteMedianList; absoluteMedianList.reserve((int)notNanCount); absoluteMedianList.resize((int)notNanCount); int idx = 0; for (int row = 0; row < rowValues->size(); ++row) { val = rowValues->value(row); if (std::isnan(val) || isMasked(row) ) continue; columnSumVariance += pow(val - statistics.arithmeticMean, 2.0); sumForCentralMoment_r3 += pow(val - statistics.arithmeticMean, 3.0); sumForCentralMoment_r4 += pow(val - statistics.arithmeticMean, 4.0); columnSumMeanDeviation += fabs( val - statistics.arithmeticMean ); absoluteMedianList[idx] = fabs(val - statistics.median); columnSumMedianDeviation += absoluteMedianList[idx]; idx++; } statistics.meanDeviationAroundMedian = columnSumMedianDeviation / notNanCount; statistics.medianDeviation = (notNanCount%2) ? absoluteMedianList.at((int)((notNanCount-1)/2)) : (absoluteMedianList.at((int)((notNanCount-1)/2)) + absoluteMedianList.at((int)(notNanCount/2)))/2.0; const double centralMoment_r3 = sumForCentralMoment_r3 / notNanCount; const double centralMoment_r4 = sumForCentralMoment_r4 / notNanCount; statistics.variance = columnSumVariance / notNanCount; statistics.standardDeviation = sqrt(statistics.variance); statistics.skewness = centralMoment_r3 / pow(statistics.standardDeviation, 3.0); statistics.kurtosis = (centralMoment_r4 / pow(statistics.standardDeviation, 4.0)) - 3.0; statistics.meanDeviation = columnSumMeanDeviation / notNanCount; double entropy = 0.0; for (const auto& v : frequencyOfValues) { const double frequencyNorm = static_cast(v) / notNanCount; entropy += (frequencyNorm * log2(frequencyNorm)); } statistics.entropy = -entropy; d->statisticsAvailable = true; } ////////////////////////////////////////////////////////////////////////////////////////////// void* Column::data() const { return d->data(); } /*! * return \c true if the column has numeric values, \false otherwise. */ bool Column::hasValues() const { if (d->hasValuesAvailable) return d->hasValues; bool foundValues = false; if (columnMode() == AbstractColumn::Numeric) { for (int row = 0; row < rowCount(); ++row) { if (!std::isnan(valueAt(row))) { foundValues = true; break; } } } else if (columnMode() == AbstractColumn::Integer) { //integer column has always valid values foundValues = true; } else if (columnMode() == AbstractColumn::DateTime) { for (int row = 0; row < rowCount(); ++row) { if (dateTimeAt(row).isValid()) { foundValues = true; break; } } } d->hasValues = foundValues; d->hasValuesAvailable = true; return d->hasValues; } //TODO: support all data types /** * \brief Return the content of row 'row'. * * Use this only when columnMode() is Text */ QString Column::textAt(int row) const { return d->textAt(row); } /** * \brief Return the date part of row 'row' * * Use this only when columnMode() is DateTime, Month or Day */ QDate Column::dateAt(int row) const { return d->dateAt(row); } /** * \brief Return the time part of row 'row' * * Use this only when columnMode() is DateTime, Month or Day */ QTime Column::timeAt(int row) const { return d->timeAt(row); } /** * \brief Return the QDateTime in row 'row' * * Use this only when columnMode() is DateTime, Month or Day */ QDateTime Column::dateTimeAt(int row) const { return d->dateTimeAt(row); } /** * \brief Return the double value in row 'row' */ double Column::valueAt(int row) const { return d->valueAt(row); } /** * \brief Return the int value in row 'row' */ int Column::integerAt(int row) const { return d->integerAt(row); } /* * call this function if the data of the column was changed directly via the data()-pointer * and not via the setValueAt() in order to emit the dataChanged-signal. * This is used e.g. in \c XYFitCurvePrivate::recalculate() */ void Column::setChanged() { d->propertiesAvailable = false; if (!m_suppressDataChangedSignal) emit dataChanged(this); d->statisticsAvailable = false; d->hasValuesAvailable = false; } //////////////////////////////////////////////////////////////////////////////// //@} //////////////////////////////////////////////////////////////////////////////// /** * \brief Return an icon to be used for decorating the views and spreadsheet column headers */ QIcon Column::icon() const { return iconForMode(columnMode()); } //////////////////////////////////////////////////////////////////////////////////////////////////// //! \name serialize/deserialize //@{ //////////////////////////////////////////////////////////////////////////////////////////////////// /** * \brief Save the column as XML */ void Column::save(QXmlStreamWriter* writer) const { writer->writeStartElement("column"); writeBasicAttributes(writer); writer->writeAttribute("rows", QString::number(rowCount())); writer->writeAttribute("designation", QString::number(plotDesignation())); writer->writeAttribute("mode", QString::number(columnMode())); writer->writeAttribute("width", QString::number(width())); //save the formula used to generate column values, if available if (!formula().isEmpty() ) { writer->writeStartElement("formula"); writer->writeAttribute("autoUpdate", QString::number(d->formulaAutoUpdate())); writer->writeTextElement("text", formula()); writer->writeStartElement("variableNames"); for (const auto& name : formulaVariableNames()) writer->writeTextElement("name", name); writer->writeEndElement(); writer->writeStartElement("columnPathes"); for (const auto path : formulaVariableColumnPaths()) writer->writeTextElement("path", path); writer->writeEndElement(); writer->writeEndElement(); } writeCommentElement(writer); writer->writeStartElement("input_filter"); d->inputFilter()->save(writer); writer->writeEndElement(); writer->writeStartElement("output_filter"); d->outputFilter()->save(writer); writer->writeEndElement(); XmlWriteMask(writer); //TODO: formula in cells is not implemented yet // QVector< Interval > formulas = formulaIntervals(); // foreach(const Interval& interval, formulas) { // writer->writeStartElement("formula"); // writer->writeAttribute("start_row", QString::number(interval.start())); // writer->writeAttribute("end_row", QString::number(interval.end())); // writer->writeCharacters(formula(interval.start())); // writer->writeEndElement(); // } int i; switch (columnMode()) { case AbstractColumn::Numeric: { const char* data = reinterpret_cast(static_cast< QVector* >(d->data())->constData()); size_t size = d->rowCount() * sizeof(double); writer->writeCharacters(QByteArray::fromRawData(data, (int)size).toBase64()); break; } case AbstractColumn::Integer: { const char* data = reinterpret_cast(static_cast< QVector* >(d->data())->constData()); size_t size = d->rowCount() * sizeof(int); writer->writeCharacters(QByteArray::fromRawData(data, (int)size).toBase64()); break; } case AbstractColumn::Text: for (i = 0; i < rowCount(); ++i) { writer->writeStartElement("row"); writer->writeAttribute("index", QString::number(i)); writer->writeCharacters(textAt(i)); writer->writeEndElement(); } break; case AbstractColumn::DateTime: case AbstractColumn::Month: case AbstractColumn::Day: for (i = 0; i < rowCount(); ++i) { writer->writeStartElement("row"); writer->writeAttribute("index", QString::number(i)); writer->writeCharacters(dateTimeAt(i).toString("yyyy-dd-MM hh:mm:ss:zzz")); writer->writeEndElement(); } break; } writer->writeEndElement(); // "column" } //TODO: extra header class DecodeColumnTask : public QRunnable { public: DecodeColumnTask(ColumnPrivate* priv, const QString& content) { m_private = priv; m_content = content; }; void run() override { QByteArray bytes = QByteArray::fromBase64(m_content.toLatin1()); if (m_private->columnMode() == AbstractColumn::Numeric) { auto* data = new QVector(bytes.size()/(int)sizeof(double)); memcpy(data->data(), bytes.data(), bytes.size()); m_private->replaceData(data); } else { auto* data = new QVector(bytes.size()/(int)sizeof(int)); memcpy(data->data(), bytes.data(), bytes.size()); m_private->replaceData(data); } } private: ColumnPrivate* m_private; QString m_content; }; /** * \brief Load the column from XML */ bool Column::load(XmlStreamReader* reader, bool preview) { if (!readBasicAttributes(reader)) return false; KLocalizedString attributeWarning = ki18n("Attribute '%1' missing or empty, default value is used"); QXmlStreamAttributes attribs = reader->attributes(); QString str = attribs.value("rows").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("rows").toString()); else d->resizeTo(str.toInt()); str = attribs.value("designation").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("designation").toString()); else d->setPlotDesignation( AbstractColumn::PlotDesignation(str.toInt()) ); str = attribs.value("mode").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("mode").toString()); else setColumnModeFast( AbstractColumn::ColumnMode(str.toInt()) ); str = attribs.value("width").toString(); if (str.isEmpty()) reader->raiseWarning(attributeWarning.subs("width").toString()); else d->setWidth(str.toInt()); // read child elements while (!reader->atEnd()) { reader->readNext(); if (reader->isEndElement()) break; if (reader->isStartElement()) { bool ret_val = true; if (reader->name() == "comment") ret_val = readCommentElement(reader); else if (reader->name() == "input_filter") ret_val = XmlReadInputFilter(reader); else if (reader->name() == "output_filter") ret_val = XmlReadOutputFilter(reader); else if (reader->name() == "mask") ret_val = XmlReadMask(reader); else if (reader->name() == "formula") ret_val = XmlReadFormula(reader); else if (reader->name() == "row") ret_val = XmlReadRow(reader); else { // unknown element reader->raiseWarning(i18n("unknown element '%1'", reader->name().toString())); if (!reader->skipToEndElement()) return false; } if (!ret_val) return false; } if (!preview) { QString content = reader->text().toString().trimmed(); if (!content.isEmpty() && ( columnMode() == AbstractColumn::Numeric || columnMode() == AbstractColumn::Integer)) { auto* task = new DecodeColumnTask(d, content); QThreadPool::globalInstance()->start(task); } } } return !reader->error(); } void Column::finalizeLoad() { d->finalizeLoad(); } /** * \brief Read XML input filter element */ bool Column::XmlReadInputFilter(XmlStreamReader* reader) { Q_ASSERT(reader->isStartElement() == true && reader->name() == "input_filter"); if (!reader->skipToNextTag()) return false; if (!d->inputFilter()->load(reader, false)) return false; if (!reader->skipToNextTag()) return false; Q_ASSERT(reader->isEndElement() == true && reader->name() == "input_filter"); return true; } /** * \brief Read XML output filter element */ bool Column::XmlReadOutputFilter(XmlStreamReader* reader) { Q_ASSERT(reader->isStartElement() == true && reader->name() == "output_filter"); if (!reader->skipToNextTag()) return false; if (!d->outputFilter()->load(reader, false)) return false; if (!reader->skipToNextTag()) return false; Q_ASSERT(reader->isEndElement() == true && reader->name() == "output_filter"); return true; } /** * \brief Read XML formula element */ bool Column::XmlReadFormula(XmlStreamReader* reader) { QString formula; QStringList variableNames; QStringList columnPathes; bool autoUpdate = reader->attributes().value("autoUpdate").toInt(); while (reader->readNext()) { if (reader->isEndElement()) break; if (reader->name() == "text") formula = reader->readElementText(); else if (reader->name() == "variableNames") { while (reader->readNext()) { if (reader->name() == "variableNames" && reader->isEndElement()) break; if (reader->isStartElement()) variableNames << reader->readElementText(); } } else if (reader->name() == "columnPathes") { while (reader->readNext()) { if (reader->name() == "columnPathes" && reader->isEndElement()) break; if (reader->isStartElement()) columnPathes << reader->readElementText(); } } } d->setFormula(formula, variableNames, columnPathes, autoUpdate); return true; } //TODO: read cell formula, not implemented yet // bool Column::XmlReadFormula(XmlStreamReader* reader) // { // Q_ASSERT(reader->isStartElement() && reader->name() == "formula"); // // bool ok1, ok2; // int start, end; // start = reader->readAttributeInt("start_row", &ok1); // end = reader->readAttributeInt("end_row", &ok2); // if (!ok1 || !ok2) // { // reader->raiseError(i18n("invalid or missing start or end row")); // return false; // } // setFormula(Interval(start,end), reader->readElementText()); // // return true; // } /** * \brief Read XML row element */ bool Column::XmlReadRow(XmlStreamReader* reader) { Q_ASSERT(reader->isStartElement() == true && reader->name() == "row"); // QXmlStreamAttributes attribs = reader->attributes(); bool ok; int index = reader->readAttributeInt("index", &ok); if (!ok) { reader->raiseError(i18n("invalid or missing row index")); return false; } QString str = reader->readElementText(); switch (columnMode()) { case AbstractColumn::Numeric: { double value = str.toDouble(&ok); if (!ok) { reader->raiseError(i18n("invalid row value")); return false; } setValueAt(index, value); break; } case AbstractColumn::Integer: { int value = str.toInt(&ok); if (!ok) { reader->raiseError(i18n("invalid row value")); return false; } setIntegerAt(index, value); break; } case AbstractColumn::Text: setTextAt(index, str); break; case AbstractColumn::DateTime: case AbstractColumn::Month: case AbstractColumn::Day: QDateTime date_time = QDateTime::fromString(str,"yyyy-dd-MM hh:mm:ss:zzz"); setDateTimeAt(index, date_time); break; } return true; } //////////////////////////////////////////////////////////////////////////////// //@} //////////////////////////////////////////////////////////////////////////////// /** * \brief Return whether the object is read-only */ bool Column::isReadOnly() const { return false; } /** * \brief Return the column mode * * This function is mostly used by spreadsheets but can also be used * by plots. The column mode specifies how to interpret * the values in the column additional to the data type. */ AbstractColumn::ColumnMode Column::columnMode() const { return d->columnMode(); } /** * \brief Return the data vector size * * This returns the number of rows that actually contain data. * Rows beyond this can be masked etc. but should be ignored by filters, * plots etc. */ int Column::rowCount() const { return d->rowCount(); } /** * \brief Return the column plot designation */ AbstractColumn::PlotDesignation Column::plotDesignation() const { return d->plotDesignation(); } QString Column::plotDesignationString() const { switch (plotDesignation()) { case AbstractColumn::NoDesignation: return QString(""); case AbstractColumn::X: return QLatin1String("[X]"); case AbstractColumn::Y: return QLatin1String("[Y]"); case AbstractColumn::Z: return QLatin1String("[Z]"); case AbstractColumn::XError: return QLatin1String("[") + i18n("X-error") + QLatin1Char(']'); case AbstractColumn::XErrorPlus: return QLatin1String("[") + i18n("X-error +") + QLatin1Char(']'); case AbstractColumn::XErrorMinus: return QLatin1String("[") + i18n("X-error -") + QLatin1Char(']'); case AbstractColumn::YError: return QLatin1String("[") + i18n("Y-error") + QLatin1Char(']'); case AbstractColumn::YErrorPlus: return QLatin1String("[") + i18n("Y-error +") + QLatin1Char(']'); case AbstractColumn::YErrorMinus: return QLatin1String("[") + i18n("Y-error -") + QLatin1Char(']'); } return QString(""); } AbstractSimpleFilter* Column::outputFilter() const { return d->outputFilter(); } /** * \brief Return a wrapper column object used for String I/O. */ ColumnStringIO* Column::asStringColumn() const { return m_string_io; } //////////////////////////////////////////////////////////////////////////////// //! \name IntervalAttribute related functions //@{ //////////////////////////////////////////////////////////////////////////////// /** * \brief Return the formula associated with row 'row' */ QString Column::formula(int row) const { return d->formula(row); } /** * \brief Return the intervals that have associated formulas * * This can be used to make a list of formulas with their intervals. * Here is some example code: * * \code * QStringList list; * QVector< Interval > intervals = my_column.formulaIntervals(); * foreach(Interval interval, intervals) * list << QString(interval.toString() + ": " + my_column.formula(interval.start())); * \endcode */ QVector< Interval > Column::formulaIntervals() const { return d->formulaIntervals(); } void Column::handleFormatChange() { DEBUG("Column::handleFormatChange() mode = " << ENUM_TO_STRING(AbstractColumn, ColumnMode, columnMode())); if (columnMode() == AbstractColumn::DateTime) { auto* input_filter = static_cast(d->inputFilter()); auto* output_filter = static_cast(d->outputFilter()); DEBUG("change format " << input_filter->format().toStdString() << " to " << output_filter->format().toStdString()); input_filter->setFormat(output_filter->format()); } emit aspectDescriptionChanged(this); // the icon for the type changed if (!m_suppressDataChangedSignal) emit dataChanged(this); // all cells must be repainted d->statisticsAvailable = false; d->hasValuesAvailable = false; d->propertiesAvailable = false; DEBUG("Column::handleFormatChange() DONE"); } /*! * calculates the minimal value in the column. * for \c count = 0, the minimum of all elements is returned. * for \c count > 0, the minimum of the first \count elements is returned. * for \c count < 0, the minimum of the last \count elements is returned. */ double Column::minimum(int count) const { double min = INFINITY; if (count == 0 && d->statisticsAvailable) min = const_cast(this)->statistics().minimum; else { int start, end; if (count == 0) { start = 0; end = rowCount(); } else if (count > 0) { start = 0; end = qMin(rowCount(), count); } else { start = qMax(rowCount() + count, 0); end = rowCount(); } return minimum(start, end); } return min; } /*! * \brief Column::minimum * Calculates the minimum value in the column between the \p startIndex and \p endIndex, endIndex is excluded. * If startIndex is greater than endIndex the indices are swapped * \p startIndex * \p endIndex */ double Column::minimum(int startIndex, int endIndex) const { double min = INFINITY; if (rowCount() == 0) return min; if (startIndex > endIndex && startIndex >= 0 && endIndex >= 0) std::swap(startIndex, endIndex); startIndex = qMax(startIndex, 0); endIndex = qMax(endIndex, 0); startIndex = qMin(startIndex, rowCount() - 1); endIndex = qMin(endIndex, rowCount() - 1); int foundIndex = 0; ColumnMode mode = columnMode(); Properties property = properties(); if (property == Properties::No) { switch (mode) { case Numeric: { auto* vec = static_cast*>(data()); for (int row = startIndex; row < endIndex; ++row) { const double val = vec->at(row); if (std::isnan(val)) continue; if (val < min) min = val; } break; } case Integer: { auto* vec = static_cast*>(data()); for (int row = startIndex; row < endIndex; ++row) { const int val = vec->at(row); if (val < min) min = val; } break; } case Text: break; case DateTime: { auto* vec = static_cast*>(data()); for (int row = startIndex; row < endIndex; ++row) { const qint64 val = vec->at(row).toMSecsSinceEpoch(); if (val < min) min = val; } break; } case Day: case Month: default: break; } return min; } // use the properties knowledge to determine maximum faster if (property == Properties::Constant || property == Properties::MonotonicIncreasing) foundIndex = 0; else if (property == Properties::MonotonicDecreasing) foundIndex = endIndex; switch (mode) { case Numeric: case Integer: return valueAt(foundIndex); case DateTime: case Month: case Day: return dateTimeAt(foundIndex).toMSecsSinceEpoch(); case Text: default: break; } return min; } /*! * calculates the maximal value in the column. * for \c count = 0, the maximum of all elements is returned. * for \c count > 0, the maximum of the first \count elements is returned. * for \c count < 0, the maximum of the last \count elements is returned. */ double Column::maximum(int count) const { double max = -INFINITY; if (count == 0 && d->statisticsAvailable) max = const_cast(this)->statistics().maximum; else { int start, end; if (count == 0) { start = 0; end = rowCount(); } else if (count > 0) { start = 0; end = qMin(rowCount(), count); } else { start = qMax(rowCount() + count, 0); end = rowCount(); } return maximum(start, end); } return max; } /*! * \brief Column::maximum * Calculates the maximum value in the column between the \p startIndex and \p endIndex. * If startIndex is greater than endIndex the indices are swapped * \p startIndex * \p endIndex */ double Column::maximum(int startIndex, int endIndex) const { double max = -INFINITY; if (rowCount() == 0) return max; if (startIndex > endIndex && startIndex >= 0 && endIndex >= 0) std::swap(startIndex, endIndex); startIndex = qMax(startIndex, 0); endIndex = qMax(endIndex, 0); startIndex = qMin(startIndex, rowCount() - 1); endIndex = qMin(endIndex, rowCount() - 1); int foundIndex = 0; ColumnMode mode = columnMode(); Properties property = properties(); if (property == Properties::No) { switch (mode) { case Numeric: { auto* vec = static_cast*>(data()); for (int row = startIndex; row < endIndex; ++row) { const double val = vec->at(row); if (std::isnan(val)) continue; if (val > max) max = val; } break; } case Integer: { auto* vec = static_cast*>(data()); for (int row = startIndex; row < endIndex; ++row) { const int val = vec->at(row); if (val > max) max = val; } break; } case Text: break; case DateTime: { auto* vec = static_cast*>(data()); for (int row = startIndex; row < endIndex; ++row) { const qint64 val = vec->at(row).toMSecsSinceEpoch(); if (val > max) max = val; } break; } case Day: case Month: default: break; } return max; } // use the properties knowledge to determine maximum faster if (property == Properties::Constant || property == Properties::MonotonicDecreasing) foundIndex = 0; else if (property == Properties::MonotonicIncreasing) foundIndex = endIndex; switch (mode) { case Numeric: case Integer: return valueAt(foundIndex); case DateTime: case Month: case Day: return dateTimeAt(foundIndex).toMSecsSinceEpoch(); case Text: default: break; } return max; } /*! * calculates log2(x)+1 for an integer value. * Used in y(double x) to calculate the maximum steps * source: https://stackoverflow.com/questions/11376288/fast-computing-of-log2-for-64-bit-integers * source: http://graphics.stanford.edu/~seander/bithacks.html#IntegerLogLookup * @param value * @return returns calculated value */ // TODO: testing if it is faster than calculating log2. int Column::calculateMaxSteps (unsigned int value) { const signed char LogTable256[256] = { -1,0,1,1,2,2,2,2,3,3,3,3,3,3,3,3, 4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4, 5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5, 5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7 }; unsigned int r; // r will be lg(v) unsigned int t, tt; // temporaries if ((tt = value >> 16)) r = (t = tt >> 8) ? 24 + LogTable256[t] : 16 + LogTable256[tt]; else r = (t = value >> 8) ? 8 + LogTable256[t] : LogTable256[value]; return r+1; } /*! * Find index which corresponds to a @p x . In a vector of values * When monotonic increasing or decreasing a different algorithm will be used, which needs less steps (mean) (log_2(rowCount)) to find the value. * @param x * @return -1 if index not found, otherwise the index */ int Column::indexForValue(double x, QVector& column, Properties properties) { int rowCount = column.count(); if (rowCount == 0) return -1; double prevValue = 0; //qint64 prevValueDateTime = 0; if (properties == AbstractColumn::Properties::MonotonicIncreasing || properties == AbstractColumn::Properties::MonotonicDecreasing) { // bisects the index every time, so it is possible to find the value in log_2(rowCount) steps bool increase = true; if(properties == AbstractColumn::Properties::MonotonicDecreasing) increase = false; int lowerIndex = 0; int higherIndex = rowCount-1; unsigned int maxSteps = calculateMaxSteps(static_cast(rowCount))+1; for (unsigned int i = 0; i < maxSteps; i++) { // so no log_2(rowCount) needed int index = lowerIndex + round(static_cast(higherIndex - lowerIndex)/2); double value = column[index]; if (higherIndex - lowerIndex < 2) { if (qAbs(column[lowerIndex] - x) < qAbs(column[higherIndex] - x)) index = lowerIndex; else index = higherIndex; return index; } if (value > x && increase) higherIndex = index; else if (value >= x && !increase) lowerIndex = index; else if (value <= x && increase) lowerIndex = index; else if (value < x && !increase) higherIndex = index; } } else if (properties == AbstractColumn::Properties::Constant) { return 0; } else { // AbstractColumn::Properties::No // naiv way int index = 0; prevValue = column[0]; for (int row = 0; row < rowCount; row++) { double value = column[row]; if (qAbs(value - x) <= qAbs(prevValue - x)) { // "<=" prevents also that row - 1 become < 0 prevValue = value; index = row; } } return index; } return -1; } /*! * Find index which corresponds to a @p x . In a vector of values * When monotonic increasing or decreasing a different algorithm will be used, which needs less steps (mean) (log_2(rowCount)) to find the value. * @param x * @return -1 if index not found, otherwise the index */ int Column::indexForValue(const double x, const QVector& points, Properties properties) { int rowCount = points.count(); if (rowCount == 0) return -1; double prevValue = 0; //qint64 prevValueDateTime = 0; if (properties == AbstractColumn::Properties::MonotonicIncreasing || properties == AbstractColumn::Properties::MonotonicDecreasing) { // bisects the index every time, so it is possible to find the value in log_2(rowCount) steps bool increase = true; if(properties == AbstractColumn::Properties::MonotonicDecreasing) increase = false; int lowerIndex = 0; int higherIndex = rowCount - 1; unsigned int maxSteps = calculateMaxSteps(static_cast(rowCount))+1; for (unsigned int i = 0; i < maxSteps; i++) { // so no log_2(rowCount) needed int index = lowerIndex + round(static_cast(higherIndex - lowerIndex)/2); double value = points[index].x(); if (higherIndex - lowerIndex < 2) { if (qAbs(points[lowerIndex].x() - x) < qAbs(points[higherIndex].x() - x)) index = lowerIndex; else index = higherIndex; return index; } if (value > x && increase) higherIndex = index; else if (value >= x && !increase) lowerIndex = index; else if (value <= x && increase) lowerIndex = index; else if (value < x && !increase) higherIndex = index; } } else if (properties == AbstractColumn::Properties::Constant) { return 0; } else { // AbstractColumn::Properties::No // naiv way prevValue = points[0].x(); int index = 0; for (int row = 0; row < rowCount; row++) { double value = points[row].x(); if (qAbs(value - x) <= qAbs(prevValue - x)) { // "<=" prevents also that row - 1 become < 0 prevValue = value; index = row; } } return index; } return -1; } /*! * Find index which corresponds to a @p x . In a vector of values * When monotonic increasing or decreasing a different algorithm will be used, which needs less steps (mean) (log_2(rowCount)) to find the value. * @param x * @return -1 if index not found, otherwise the index */ int Column::indexForValue(double x, QVector& lines, Properties properties) { int rowCount = lines.count(); if (rowCount == 0) return -1; // use only p1 to find index double prevValue = 0; //qint64 prevValueDateTime = 0; if (properties == AbstractColumn::Properties::MonotonicIncreasing || properties == AbstractColumn::Properties::MonotonicDecreasing) { // bisects the index every time, so it is possible to find the value in log_2(rowCount) steps bool increase = true; if(properties == AbstractColumn::Properties::MonotonicDecreasing) increase = false; int lowerIndex = 0; int higherIndex = rowCount-1; unsigned int maxSteps = calculateMaxSteps(static_cast(rowCount))+1; for (unsigned int i = 0; i < maxSteps; i++) { // so no log_2(rowCount) needed int index = lowerIndex + round(static_cast(higherIndex - lowerIndex)/2); double value = lines[index].p1().x(); if (higherIndex - lowerIndex < 2) { if (qAbs(lines[lowerIndex].p1().x() - x) < qAbs(lines[higherIndex].p1().x() - x)) index = lowerIndex; else index = higherIndex; return index; } if (value > x && increase) higherIndex = index; else if (value >= x && !increase) lowerIndex = index; else if (value <= x && increase) lowerIndex = index; else if (value < x && !increase) higherIndex = index; } } else if (properties == AbstractColumn::Properties::Constant) { return 0; } else { // AbstractColumn::Properties::No // naiv way int index = 0; prevValue = lines[0].p1().x(); for (int row = 0; row < rowCount; row++) { double value = lines[row].p1().x(); if (qAbs(value - x) <= qAbs(prevValue - x)) { // "<=" prevents also that row - 1 become < 0 prevValue = value; index = row; } } return index; } return -1; } int Column::indexForValue(double x) const { double prevValue = 0; qint64 prevValueDateTime = 0; AbstractColumn::ColumnMode mode = columnMode(); int property = properties(); if (property == AbstractColumn::Properties::MonotonicIncreasing || property == AbstractColumn::Properties::MonotonicDecreasing) { // bisects the index every time, so it is possible to find the value in log_2(rowCount) steps bool increase = (property != AbstractColumn::Properties::MonotonicDecreasing); int lowerIndex = 0; int higherIndex = rowCount() - 1; unsigned int maxSteps = calculateMaxSteps(static_cast(rowCount()))+1; if ((mode == AbstractColumn::ColumnMode::Numeric || mode == AbstractColumn::ColumnMode::Integer)) { for (unsigned int i = 0; i < maxSteps; i++) { // so no log_2(rowCount) needed int index = lowerIndex + round(static_cast(higherIndex - lowerIndex)/2); double value = valueAt(index); if (higherIndex - lowerIndex < 2) { if (qAbs(valueAt(lowerIndex) - x) < qAbs(valueAt(higherIndex) - x)) index = lowerIndex; else index = higherIndex; return index; } if (value > x && increase) higherIndex = index; else if (value >= x && !increase) lowerIndex = index; else if (value <= x && increase) lowerIndex = index; else if (value < x && !increase) higherIndex = index; } } else if ((mode == AbstractColumn::ColumnMode::DateTime || mode == AbstractColumn::ColumnMode::Month || mode == AbstractColumn::ColumnMode::Day)) { qint64 xInt64 = static_cast(x); for (unsigned int i = 0; i < maxSteps; i++) { // so no log_2(rowCount) needed int index = lowerIndex + round(static_cast(higherIndex - lowerIndex)/2); qint64 value = dateTimeAt(index).toMSecsSinceEpoch(); if (higherIndex - lowerIndex < 2) { if (abs(dateTimeAt(lowerIndex).toMSecsSinceEpoch() - xInt64) < abs(dateTimeAt(higherIndex).toMSecsSinceEpoch() - xInt64)) index = lowerIndex; else index = higherIndex; return index; } if (value > xInt64 && increase) higherIndex = index; else if (value >= xInt64 && !increase) lowerIndex = index; else if (value <= xInt64 && increase) lowerIndex = index; else if (value < xInt64 && !increase) higherIndex = index; } } } else if (property == AbstractColumn::Properties::Constant) { if (rowCount() > 0) return 0; else return -1; } else { // naiv way int index = 0; if ((mode == AbstractColumn::ColumnMode::Numeric || mode == AbstractColumn::ColumnMode::Integer)) { for (int row = 0; row < rowCount(); row++) { if (isValid(row)) { if (row == 0) prevValue = valueAt(row); double value = valueAt(row); if (abs(value - x) <= abs(prevValue - x)) { // <= prevents also that row - 1 become < 0 if (row < rowCount() - 1) { prevValue = value; index = row; } } } } return index; } else if ((mode == AbstractColumn::ColumnMode::DateTime || mode == AbstractColumn::ColumnMode::Month || mode == AbstractColumn::ColumnMode::Day)) { qint64 xInt64 = static_cast(x); int index = 0; for (int row = 0; row < rowCount(); row++) { if (isValid(row)) { if (row == 0) prevValueDateTime = dateTimeAt(row).toMSecsSinceEpoch(); qint64 value = dateTimeAt(row).toMSecsSinceEpoch(); if (abs(value - xInt64) <= abs(prevValueDateTime - xInt64)) { // "<=" prevents also that row - 1 become < 0 prevValueDateTime = value; index = row; } } } return index; } } return -1; } /*! * Finds the minimal and maximal index which are between v1 and v2 * \brief Column::indicesForX * \param x1 * \param x2 * \param start * \param end * \return */ bool Column::indicesMinMax(double v1, double v2, int& start, int& end) const { start = -1; end = -1; if (rowCount() == 0) return false; // Assumption: v1 is always the smaller value if (v1 > v2) qSwap(v1, v2); Properties property = properties(); if (property == Properties::MonotonicIncreasing || property == Properties::MonotonicDecreasing) { start = indexForValue(v1); end = indexForValue(v2); switch (columnMode()) { case Integer: case Numeric: { if (start > 0 && valueAt(start -1) <= v2 && valueAt(start -1) >= v1) start--; if (end < rowCount() - 1 && valueAt(end + 1) <= v2 && valueAt(end + 1) >= v1) end++; break; } case DateTime: case Month: case Day: { qint64 v1int64 = v1; qint64 v2int64 = v2; qint64 value; if (start > 0) { value = dateTimeAt(start -1).toMSecsSinceEpoch(); if (value <= v2int64 && value >= v1int64) start--; } if (end > rowCount() - 1) { value = dateTimeAt(end + 1).toMSecsSinceEpoch(); if (value <= v2int64 && value >= v1int64) end++; } break; } case Text: return false; } return true; } else if (property == Properties::Constant) { start = 0; end = rowCount() - 1; return true; } switch (columnMode()) { case Integer: case Numeric: { double value; for (int i = 0; i < rowCount(); i++) { value = valueAt(i); if (value <= v2 && value >= v1) { end = i; if (start < 0) start = i; } } break; } case DateTime: case Month: case Day: { qint64 value; qint64 v2int64 = v2; qint64 v1int64 = v2; for (int i = 0; i < rowCount(); i++) { value = dateTimeAt(i).toMSecsSinceEpoch(); if (value <= v2int64 && value >= v1int64) { end = i; if (start < 0) start = i; } } break; } case Text: return false; } return true; }