java.lang.Object
org.rrd4j.data.DataProcessor
All Implemented Interfaces:
DataHolder
public class DataProcessorextends Object implements DataHolder
Class which should be used for all calculations based on the data fetched from RRD files. This class supports ordinary DEF datasources (defined in RRD files), CDEF datasources (RPN expressions evaluation), SDEF (static datasources - extension of Rrd4j) and PDEF (plottables, see Plottable for more information.

Typical class usage:

final long t1 = ...
final long t2 = ...
DataProcessor dp = new DataProcessor(t1, t2);
// DEF datasource
dp.addDatasource("x", "demo.rrd", "some_source", "AVERAGE");
// DEF datasource
dp.addDatasource("y", "demo.rrd", "some_other_source", "AVERAGE");
// CDEF datasource, z = (x + y) / 2
dp.addDatasource("z", "x,y,+,2,/");
// ACTION!
dp.processData();
// Dump calculated values
System.out.println(dp.dump());

  • Field Details

    • DEFAULT_PERCENTILE

      public static final double DEFAULT_PERCENTILE
      Constant DEFAULT_PERCENTILE=95.0
      See Also:

    • pixelCount

      private int pixelCount
      Pixel count

    • poolUsed

      private boolean poolUsed
      Whether the shared database pool is used.

    • pool

      private RrdDbPool pool
      The pool databases are opened through.

    • tStart

      private long tStart
      First time of the data being processed.

    • tEnd

      private long tEnd
      Last time of the data being processed.

    • timestamps

      private long[] timestamps
      Array of timestamps for the data series

    • lastRrdArchiveUpdateTime

      private long lastRrdArchiveUpdateTime
      Last rrd archive update time

    • step

      private long step
      Time step in seconds between data points; adjusted during processing to match the archives actually read.

    • fetchRequestResolution

      private long fetchRequestResolution
      Resolution of the fetch from each RRD file, in seconds.

    • tz

      private TimeZone tz
      Timezone dates are interpreted in.

    • sources

      private final Map<String,Source> sources
      Sources by name, in insertion order - a plain HashMap would lose that order.

    • defSources

      private Def[] defSources
      Sources defined by a DEF graph definition rather than read from a file.

  • Constructor Details

    • DataProcessor

      public DataProcessor(long t1, long t2)
      Creates new DataProcessor object for the given time span. Ending timestamp may be set to zero. In that case, the class will try to find the optimal ending timestamp based on the last update time of RRD files processed with the processData() method.
      Parameters:
      t1 - Starting timestamp in seconds without milliseconds
      t2 - Ending timestamp in seconds without milliseconds

    • DataProcessor

      public DataProcessor(Date d1, Date d2)
      Creates new DataProcessor object for the given time span. Ending date may be set to null. In that case, the class will try to find optimal ending date based on the last update time of RRD files processed with the processData() method.
      Parameters:
      d1 - start of the range, in seconds since the epoch
      d2 - end of the range, in seconds since the epoch; null leaves the end to be chosen from the last update time of the processed files

    • DataProcessor

      public DataProcessor(Calendar gc1, Calendar gc2)
      Creates new DataProcessor object for the given time span. Ending date may be set to null. In that case, the class will try to find optimal ending date based on the last update time of RRD files processed with the processData() method.

      It use the time zone for starting calendar date.

      Parameters:
      gc1 - Starting Calendar date
      gc2 - Ending Calendar date

    • DataProcessor

      public DataProcessor(java.time.temporal.TemporalAmount d)
      Creates new DataProcessor object for the given time duration. The given duration will be subtracted from current time.
      Parameters:
      d - duration to subtract.

  • Method Details

    • isPoolUsed

      public boolean isPoolUsed()
      Returns boolean value representing RrdDbPool usage policy.
      Specified by:
      isPoolUsed in interface DataHolder
      Returns:
      true, if the pool will be used internally to fetch data from RRD files, false otherwise.

    • setPoolUsed

      public void setPoolUsed(boolean poolUsed)
      Sets the RrdDbPool usage policy.
      Specified by:
      setPoolUsed in interface DataHolder
      Parameters:
      poolUsed - true, if the pool should be used to fetch data from RRD files, false otherwise.

    • getPool

      public RrdDbPool getPool()
      Return the RRD database pool, null when none is configured.
      Specified by:
      getPool in interface DataHolder
      Returns:
      the current pool, or null

    • setPool

      public void setPool(RrdDbPool pool)
      Defines the RrdDbPool to use. If not defined, but {setPoolUsed(boolean) set to true, the default RrdDbPool.getInstance() will be used.
      Specified by:
      setPool in interface DataHolder
      Parameters:
      pool - an optional pool to use.

    • setPixelCount

      public void setPixelCount(int pixelCount)
      Sets the number of pixels (target graph width). This number is used only to calculate pixel coordinates for Rrd4j graphs (methods getValuesPerPixel(String) and getTimestampsPerPixel()), but has influence neither on datasource values calculated with the processData() method nor on aggregated values returned from getAggregates(String) and similar methods. In other words, aggregated values will not change once you decide to change the dimension of your graph.
      Parameters:
      pixelCount - The number of pixels. If you process RRD data in order to display it on the graph, this should be the width of your graph.

    • getPixelCount

      public int getPixelCount()
      Returns the number of pixels (target graph width). See setPixelCount(int) for more information.
      Returns:
      Target graph width

    • setStep

      public void setStep(long step)
      Once data are fetched, the step value will be used to generate values. If not defined, or set to 0, a optimal step will be calculated.
      Specified by:
      setStep in interface DataHolder
      Parameters:
      step - Default to 0.

    • getStep

      public long getStep()
      Returns the time step used for data processing. Initially, this method returns zero. Once processData() is finished, the method will return the time stamp interval.
      Specified by:
      getStep in interface DataHolder
      Returns:
      Step used for data processing.

    • getFetchRequestResolution

      public long getFetchRequestResolution()
      Returns desired RRD archive step (resolution) in seconds to be used while fetching data from RRD files. In other words, this value will used as the last parameter of RrdDb.createFetchRequest() method when this method is called internally by this DataProcessor.
      Returns:
      Desired archive step (fetch resolution) in seconds.

    • setFetchRequestResolution

      public void setFetchRequestResolution(long fetchRequestResolution)
      Sets desired RRD archive step in seconds to be used internally while fetching data from RRD files. In other words, this value will used as the last parameter of RrdDb.createFetchRequest() method when this method is called internally by this DataProcessor. If this method is never called, fetch request resolution defaults to 1 (smallest possible archive step will be chosen automatically).
      Parameters:
      fetchRequestResolution - Desired archive step (fetch resolution) in seconds.

    • getTimeZone

      public TimeZone getTimeZone()
      Return the time zone timestamps are interpreted in.
      Specified by:
      getTimeZone in interface DataHolder
      Returns:
      the time zone

    • setTimeZone

      public void setTimeZone(TimeZone tz)
      Set the time zone timestamps are interpreted in.
      Specified by:
      setTimeZone in interface DataHolder
      Parameters:
      tz - the time zone to set

    • getTimestamps

      public long[] getTimestamps()
      /** Returns consolidated timestamps created with the processData() method.
      Returns:
      array of timestamps in seconds

    • getValues

      public double[] getValues(String sourceName)
      Returns calculated values for a single datasource. Corresponding timestamps can be obtained from the getTimestamps() method.
      Parameters:
      sourceName - Datasource name
      Returns:
      an array of datasource values
      Throws:
      IllegalArgumentException - Thrown if invalid datasource name is specified, or if datasource values are not yet calculated (method processData() was not called)

    • getVariable

      public Variable.Value getVariable(String sourceName)
      Extract the variable value from an already define Variable datasource (a VDEF)
      Parameters:
      sourceName - Datasource name
      Returns:
      A combined time and value extracted calculated from the datasource

    • getVariable

      public Variable.Value getVariable(String sourceName, Variable var)
      Returns single aggregated value for a single datasource.
      Parameters:
      sourceName - Datasource name
      var - variable that will generate value
      Returns:
      A combined time and value extracted calculated from the datasource

    • getSourceNames

      public String[] getSourceNames()
      Returns array of datasource names defined in this DataProcessor.
      Returns:
      array of datasource names

    • getValues

      public double[][] getValues()
      Returns an array of all datasource values for all datasources. Each row in this two-dimensional array represents an array of calculated values for a single datasource. The order of rows is the same as the order in which datasources were added to this DataProcessor object.
      Returns:
      All datasource values for all datasources. The first index is the index of the datasource, the second index is the index of the datasource value. The number of datasource values is equal to the number of timestamps returned with getTimestamps() method.
      Throws:
      IllegalArgumentException - Thrown if invalid datasource name is specified, or if datasource values are not yet calculated (method processData() was not called)

    • getSource

      Source getSource(String sourceName)
      Datasource name @return the matching Source.
      Parameters:
      sourceName - datasource name @return the matching Source
      Returns:
      the datasource registered under that name

    • datasource

      public void datasource(String name, IPlottable plottable)
      Adds a custom, plottable datasource (PDEF). The datapoints should be made available by a class extending Plottable class.
      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - source name.
      plottable - class that extends Plottable class and is suited for graphing.

    • datasource

      public void datasource(String name, String rpnExpression)
      Adds complex source (CDEF). Complex sources are evaluated using the supplied RPN expression. Complex source name can be used:
      • To specify sources for line, area and stack plots.
      • To define other complex sources.
      The supported RPN functions, operators and constants are detailed at RRD4J's wiki.

      Rrd4j does not force you to specify at least one simple source name as RRDTool.

      For more details on RPN see RRDTool's rrdgraph man page.

      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - source name.
      rpnExpression - RPN expression containing comma delimited simple and complex source names, RPN constants, functions and operators.

    • datasource

      public void datasource(String name, String defName, Variable var)
      Creates a datasource that performs a variable calculation on an another named datasource to yield a single combined timestamp/value.

      Requires that the other datasource has already been defined; otherwise, it'll end up with no data

      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - - the new virtual datasource name
      defName - - the datasource from which to extract the percentile. Must be a previously defined virtual datasource
      var - - a new instance of a Variable used to do the calculation

    • datasource

      public void datasource(String name, String file, String dsName, ConsolFun consolFunc)
      Adds simple datasource (DEF). Simple source name can be used:
      • To specify sources for line, area and stack plots.
      • To define complex sources
      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - source name.
      file - Path to RRD file.
      dsName - Datasource name defined in the RRD file.
      consolFunc - Consolidation function that will be used to extract data from the RRD

    • datasource

      public void datasource(String name, URI rrdUri, String dsName, ConsolFun consolFunc)
      Adds simple datasource (DEF). Simple source name can be used:
      • To specify sources for line, area and stack plots.
      • To define complex sources
      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - source name.
      rrdUri - URI to RRD file.
      dsName - Datasource name defined in the RRD file.
      consolFunc - Consolidation function that will be used to extract data from the RRD

    • datasource

      public void datasource(String name, String file, String dsName, ConsolFun consolFunc, RrdBackendFactory backend)
      Adds simple source (DEF). Source name can be used:
      • To specify sources for line, area and stack plots.
      • To define complex sources
      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - Source name.
      file - Path to RRD file.
      dsName - Data source name defined in the RRD file.
      consolFunc - Consolidation function that will be used to extract data from the RRD file ("AVERAGE", "MIN", "MAX" or "LAST" - these string constants are conveniently defined in the ConsolFun class).
      backend - Name of the RrdBackendFactory that should be used for this RrdDb.

    • datasource

      public void datasource(String name, URI uri, String dsName, ConsolFun consolFunc, RrdBackendFactory backend)
      Adds simple source (DEF). Source name can be used:
      • To specify sources for line, area and stack plots.
      • To define complex sources
      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - Source name.
      uri - URI to RRD file.
      dsName - Data source name defined in the RRD file.
      consolFunc - Consolidation function that will be used to extract data from the RRD file ("AVERAGE", "MIN", "MAX" or "LAST" - these string constants are conveniently defined in the ConsolFun class).
      backend - Name of the RrdBackendFactory that should be used for this RrdDb.

    • datasource

      public void datasource(String name, FetchData fetchData)
      Adds DEF datasource with datasource values already available in the FetchData object. This method is used internally by Rrd4j and probably has no purpose outside of it.
      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - Source name.
      fetchData - Fetched data containing values for the given source name.

    • datasource

      public void datasource(String name, String dsName, FetchData fetchData)
      Adds DEF datasource with datasource values already available in the FetchData object. This method is used internally by Rrd4j and probably has no purpose outside of it. The values will be extracted from dsName in fetchData.
      Specified by:
      datasource in interface DataHolder
      Parameters:
      name - Source name.
      dsName - Source name in the fetch data.
      fetchData - Fetched data containing values for the given source name.

    • processData

      public void processData() throws IOException
      Method that should be called once all datasources are defined. Data will be fetched from RRD files, RPN expressions will be calculated, etc.
      Throws:
      IOException - Thrown in case of I/O error (while fetching data from RRD files)

    • getValuesPerPixel

      public double[] getValuesPerPixel(String sourceName, int pixelCount)
      Method used to calculate datasource values which should be presented on the graph based on the desired graph width. Each value returned represents a single pixel on the graph. Corresponding timestamp can be found in the array returned from getTimestampsPerPixel() method.
      Parameters:
      sourceName - Datasource name
      pixelCount - graph width in pixels, retained for later calls
      Returns:
      Per-pixel datasource values
      Throws:
      IllegalArgumentException - Thrown if datasource values are not yet calculated (method processData() was not called)

    • getValuesPerPixel

      public double[] getValuesPerPixel(String sourceName)
      Method used to calculate datasource values which should be presented on the graph based on the graph width set with a setPixelCount(int) method call. Each value returned represents a single pixel on the graph. Corresponding timestamp can be found in the array returned from getTimestampsPerPixel(int) method.
      Parameters:
      sourceName - Datasource name
      Returns:
      Per-pixel datasource values
      Throws:
      IllegalArgumentException - Thrown if datasource values are not yet calculated (method processData() was not called)

    • getTimestampsPerPixel

      public long[] getTimestampsPerPixel(int pixelCount)
      Calculates timestamps which correspond to individual pixels on the graph. It also set the timestampsPerPixel value.
      Parameters:
      pixelCount - graph width in pixels, retained for later calls
      Returns:
      Array of timestamps

    • getTimestampsPerPixel

      public long[] getTimestampsPerPixel()
      Calculates timestamps which correspond to individual pixels on the graph based on the graph width set with a setPixelCount(int) method call.
      Returns:
      Array of timestamps

    • dump

      public String dump()
      Dumps timestamps and values of all datasources in a tabular form. Very useful for debugging.
      Returns:
      Dumped object content.

    • getLastRrdArchiveUpdateTime

      public long getLastRrdArchiveUpdateTime()
      Returns time when last RRD archive was updated (all RRD files are considered).
      Returns:
      Last archive update time for all RRD files in this DataProcessor

    • extractDefs

      private void extractDefs()
      Reads each DEF source's values from the RRD file it names.

    • fetchRrdData

      private void fetchRrdData() throws IOException
      Fetch rrd data
      Throws:
      IOException - if an RRD file cannot be read while fetching its datapoints

    • fixZeroEndingTimestamp

      private void fixZeroEndingTimestamp()
      Fix zero ending timestamp

    • chooseOptimalStep

      private void chooseOptimalStep()
      Choose optimal step. Tricky and ugly. Should be redesigned some time in the future.

    • createTimestamps

      private void createTimestamps()
      Create timestamps

    • assignTimestampsToSources

      private void assignTimestampsToSources()
      Assign timestamps to sources

    • normalizeRrdValues

      private void normalizeRrdValues()
      Normalize rrd values

    • calculateNonRrdSources

      private void calculateNonRrdSources()
      Calculate non rrd sources

    • format

      private static String format(String s, int length)
      Right-pad a string with spaces to the requested length.
      Parameters:
      s - the string to pad
      length - the minimum width of the result
      Returns:
      the padded string

    • setEndTime

      public void setEndTime(long time)
      Ending timestamp.
      Specified by:
      setEndTime in interface DataHolder
      Parameters:
      time - end of the range, in seconds since the epoch

    • getEndTime

      public long getEndTime()
      Ending timestamp.
      Specified by:
      getEndTime in interface DataHolder
      Returns:
      end of the range, in seconds since the epoch

    • setStartTime

      public void setStartTime(long time)
      Starting timestamp.
      Specified by:
      setStartTime in interface DataHolder
      Parameters:
      time - start of the range, in seconds since the epoch

    • getStartTime

      public long getStartTime()
      Start of the data time range.
      Specified by:
      getStartTime in interface DataHolder
      Returns:
      Starting timestamp in seconds

    • setTimeSpan

      public void setTimeSpan(long startTime, long endTime)
      Set the time range to process.
      Specified by:
      setTimeSpan in interface DataHolder
      Parameters:
      startTime - Starting time in seconds since epoch
      endTime - Ending time in seconds since epoch