java.lang.Object
net.i2p.util.SimpleTimer2.SaturationWatchdog
Enclosing class:
SimpleTimer2
private class SimpleTimer2.SaturationWatchdogextends Object
Samples this timer on its own daemon thread and reports a pool that has stopped keeping up.

It cannot run on the pool it watches: a wedged pool has no spare worker to run the watchdog on, which is the whole failure. It also does nothing but read counters, so no event running on this timer can delay it. Every sample is four counter reads and a few field writes - no allocation, no synchronization beyond the monitor it sleeps on - so it is free to leave running for the life of the router.

Everything below is touched only by the watchdog thread, except _stopped, which is guarded by the watchdog monitor.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    private long
    _lastCompleted
    Completed count at the previous sample.
    private long
    _lastProgressAt
    Wall clock at which we last saw the completion count rise.
    private long
    _maxObservedGapMs
    Longest quiet period this timer has previously come back from.
    private int
    _progressSamples
    Consecutive samples that completed a task; recovery needs this many.
    private long
    _saturatedCompleted
    Completed count seen at the last saturated sample.
    private int
    _saturatedSamples
    Samples seen with the queue saturated.
    private boolean
    _saturationReported
    True once the saturation was logged.
    private int
    _stalledSamples
    Samples seen with no completions.
    private boolean
    _stallReported
    True once the stall was logged.
    private long
    _stallStartCompleted
    Completion count when the current quiet episode began.
    private boolean
    _stopped
    Guarded by this.
    private boolean
    _workDueThisEpisode
    Whether any task was due during the current quiet episode.

  • Constructor Summary

    Constructors
    Constructor
    Description
    SaturationWatchdog()
    Builds the saturation watchdog.

  • Method Summary

    Private Methods
    Modifier and Type
    Method
    Description
    private void
    checkSaturation(int active, int pool, int queued, long completed)
    Warn once per episode of a fully occupied pool with a backlog.
    private void
    checkStall(long completed, boolean workDue)
    Warn once per episode of a pool that completes nothing at all.
    private boolean
    isWorkDue()
    Is a task actually due on this timer right now?
    private Log
    log()
    The timer log, looked up per message rather than cached: the router replaces its LogManager after start-up, and a cached Log would go quiet from then on.
    private void
    run()
    Sample every SimpleTimer2.WATCHDOG_INTERVAL_MS until shutdown().
    private void
    sample()
    Take one sample and act on it.
    (package private) void
    shutdown()
    Stop sampling.
    private void
    start()
    Start the sampling thread, warning if the JVM will not run it - a watchdog that silently failed to start is the same invisible failure this class exists to prevent.

    Methods inherited from class Object

    clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • _saturatedSamples

      private int _saturatedSamples
      Samples seen with the queue saturated.

    • _saturationReported

      private boolean _saturationReported
      True once the saturation was logged.

    • _saturatedCompleted

      private long _saturatedCompleted
      Completed count seen at the last saturated sample.

    • _stalledSamples

      private int _stalledSamples
      Samples seen with no completions.

    • _stallReported

      private boolean _stallReported
      True once the stall was logged.

    • _progressSamples

      private int _progressSamples
      Consecutive samples that completed a task; recovery needs this many.

    • _workDueThisEpisode

      private boolean _workDueThisEpisode
      Whether any task was due during the current quiet episode.

    • _stallStartCompleted

      private long _stallStartCompleted
      Completion count when the current quiet episode began.

    • _lastCompleted

      private long _lastCompleted
      Completed count at the previous sample.

    • _lastProgressAt

      private long _lastProgressAt
      Wall clock at which we last saw the completion count rise.

    • _maxObservedGapMs

      private long _maxObservedGapMs
      Longest quiet period this timer has previously come back from. Grows only while the watchdog observes recovery, so it is a floor learned from the timer's own behaviour rather than a guess.

    • _stopped

      private boolean _stopped
      Guarded by this.

  • Constructor Details

    • SaturationWatchdog

      SaturationWatchdog()
      Builds the saturation watchdog.

  • Method Details

    • start

      private void start()
      Start the sampling thread, warning if the JVM will not run it - a watchdog that silently failed to start is the same invisible failure this class exists to prevent. No reference to the thread is kept: it runs until shutdown(), and being a daemon it cannot hold up JVM exit either way.

    • shutdown

      void shutdown()
      Stop sampling. Idempotent, cannot throw, and safe when start() failed. The thread is a daemon, so it cannot hold up JVM exit even if this is never called.

    • run

      private void run()
      Sample every SimpleTimer2.WATCHDOG_INTERVAL_MS until shutdown().

      The stopped test and the wait share the monitor, so shutdown() cannot land between them and leave this thread parked for the rest of the interval - hence no interrupt is needed.

    • sample

      private void sample()
      Take one sample and act on it. All the executor counters are safe to read after shutdown, so a sample racing SimpleTimer2.stop() degrades to reporting nothing rather than throwing.

    • isWorkDue

      private boolean isWorkDue()
      Is a task actually due on this timer right now?

      A pool that has completed nothing may be idle or wedged, and completion counts cannot tell them apart - only this can. It is a property of the work offered, not of the timer's history, so unlike a calibrated gap threshold it cannot be tuned into uselessness by a timer that has always been quiet: a streaming retransmission shard that simply has nothing to retransmit has no due task and is correctly silent, while a shard with a due task and no progress is the thing worth reporting.

      Returns:
      true if the queue holds a task whose delay has already elapsed

    • checkSaturation

      private void checkSaturation(int active, int pool, int queued, long completed)
      Warn once per episode of a fully occupied pool with a backlog.
      Parameters:
      active - the number of currently active threads
      pool - the configured pool size
      queued - the number of tasks queued behind the active threads
      completed - the total number of tasks completed so far

    • checkStall

      private void checkStall(long completed, boolean workDue)
      Warn once per episode of a pool that completes nothing at all.

      The bar is the timer's own longest observed gap, learned as the timer comes and goes, rather than its mean interval. See SimpleTimer2.evaluateStall(long, long, int, int, boolean, long).

      Parameters:
      completed - the total number of tasks completed so far
      workDue - true if a task is due to run right now

    • log

      private Log log()
      The timer log, looked up per message rather than cached: the router replaces its LogManager after start-up, and a cached Log would go quiet from then on. Only log paths pay for this, never a sample.
      Returns:
      the current log for this class