RouterConsoleRunner that checks the configuration for stats to be tracked via jrobin, and adds or deletes RRDs as necessary. This also contains methods to generate xml or graph image output. The rendering for graphs is in GraphRenderer.
To control memory, the number of simultaneous renderings is limited.
Nested Class Summary
Nested ClassesModifier and TypeClassDescription(package private) static final classCount and oldest age for oneGraphGenerator.StaleCause.(package private) static enumWhy coalesces are turning into samples that never reach a listener.(package private) static enumWhat the watchdog may do about one rate's listener.(package private) static final classDecide whether a repeated report is due, and remember the decision.private classMake sure any persistent RRDs are closed.(package private) static enumWhy a graph listener is not producing RRD writes.Field Summary
Modifier and TypeFieldDescriptionprivate intConsecutive watchdog ticks on which no tracked rate coalesced at all.private final RouterContextThe router context.private booleanTrue once the rrd drift was logged.private final GraphGenerator.ReportThrottleThrottles the ledger-drift report independently of the stall report.private ScheduledExecutorServiceHealth watchdog scheduler, separate from_schedulerso a stuck sync task cannot stop the thing that is meant to notice it is stuck.private booleanGuards the watchdog against being scheduled twice; seestartHealthWatchdog().private booleanTrue while the generator runs.private longCoalesce delta across all tracked listeners at the previous watchdog tick.private final Map<Rate, GraphListener> Index of listener by rate for O(1) lookup during rendering.private final List<GraphListener> List of GraphListener instances, preserved for iteration by UI helpers.private final LogLogger for the graph renderer.private booleanWhether this instance configured the shared RRD backend factory.private ScheduledExecutorServiceScheduler running the periodic renders.private final SemaphorePermits limiting the concurrent renders.private booleanSet when a stall or drift report is logged, so the matching recovery line is emitted exactly once when the condition clears.private final GraphGenerator.ReportThrottleThrottles the stall report so the watchdog and the sync task cannot double-log it.private final longWall-clock time when graph generation started; lets the never-yet-written staleness test distinguish an empty startup window from a genuinely stalled writer.private intRenders completed by the current job.(package private) static final longGrace period before a coalesce sweep that has never run counts as a stall.private static final longLongest gap between coalesces on any tracked rate before "the rates stopped".private static final intCOALESCE_STALL_MSexpressed in watchdog ticks.static final StringThe stats graphed whenstat.summariesis unset, as statName.period pairs.private static final intWatchdog period.private static final intEmit a liveness heartbeat every N sync ticks (~27min at the 90s period).private static final booleanTrue when running on Windows.(package private) static final intUpper bound on how many retained samples one backfill is asked for.private static final intGraph renders allowed at once.(package private) static final intMost listener names a single tally will list before summarising the rest as a count.(package private) static final longFloor applied to a rate's period when deciding whether its listener is stale.private static final StringThread name of the generator.private static final longRepeat period for a report whose numbers have not changed.private static final StringOrigin wording for an age measured from the last successful write.private static final StringOrigin wording for an age with no earlier write to measure back from.Constructor Summary
ConstructorsConstructorDescriptionInstalls the rrd4j log bridge and the shutdown hook.Method Summary
Modifier and TypeMethodDescriptionprivate StringTranslate a string for display on graphs.private booleanStart tracking a rate by creating a new GraphListener for it.private StringadjustDatabases(String oldSpecs) Rewrites the rrd file paths in a data source spec.private voidWalk every listener, heal what can be healed, and report what could not be.(package private) static intbackfillWindow(long drift) How many retained samples one backfill attempt should ask for.(package private) static GraphGenerator.HealActionchooseHeal(boolean mapped, boolean detached, boolean registered, boolean writable, boolean backfillable) Cheapest action that can restore recording for one rate.(package private) static GraphGenerator.DriftCauseclassifyDrift(int pending, long oldestPendingMs, long overruns, boolean coalesceAdvancing) Name the mechanism behind positive ledger drift.(package private) static GraphGenerator.StaleCauseclassifyStaleness(boolean detached, boolean registered, long lastUpdateSuccess, long now, long startedMs, long ratePeriod) Decide whether a listener is stalled, and why, assuming the coalesce sweep has run recently enough to have fed these rates.(package private) static GraphGenerator.StaleCauseclassifyStaleness(boolean detached, boolean registered, long lastUpdateSuccess, long now, long startedMs, long ratePeriod, boolean coalesceStalled) Decide whether a listener is stalled, and why.private voidClose the shared RRD backend factory, but only if this instance configured it.private StringcoalesceSweepAbsence(long now) How long the coalesce sweep has been absent, for the report.intHow many stats are being recorded right now.private voidDelete the old rrd dir if we are no longer persistent.(package private) static StringformatCoalesceSweepAbsence(long sweepAgeMs, long startedMs, long now) Render how long the coalesce sweep has been absent.(package private) static StringformatLedgerDrift(int totalListeners, int drifters, long maxDrift, long permanentSteps, long healedSteps, int rebuilt, int reopened, long coalesced, GraphGenerator.DriftCause mechanism, long overruns, long superseded, int pending, long oldestPendingMs, long backlogCollapses, long coalesceSkips) Compose the ledger-drift log line.(package private) static StringformatStaleness(int totalListeners, GraphGenerator.CauseTally neverWritten, GraphGenerator.CauseTally unregistered, GraphGenerator.CauseTally writesStopped) Compose the write-stall log line.(package private) static StringformatStaleness(int totalListeners, GraphGenerator.CauseTally neverWritten, GraphGenerator.CauseTally unregistered, GraphGenerator.CauseTally writesStopped, GraphGenerator.CauseTally coalesceStalled) Compose the write-stall log line, counting the coalesce fault alongside the per-listener causes.The app's display name.getGroupListeners(String groupId, Set<String> enabledStats) Collect the listeners for the enabled members of a graph group.List of GraphListener instances.getName()The app's internal name.getState()Always RUNNING.booleangetXML(Rate rate, OutputStream out) Export rate data as XML.private voidOne watchdog tick: report a stalled data path, then apply the heal ladder.private voidRoute rrd4j diagnostics into the router log.static GraphGeneratorinstance(I2PAppContext ctx) The generator registered with the app manager, which only exists while it is running.private booleanisCoalesceStalled(long now) Whether the coalesce sweep is currently stalled, perisCoalesceStalledNow(boolean, long, long, long).(package private) static booleanisCoalesceStalledNow(boolean everSwept, long sweepAgeMs, long startedMs, long now) Whether the shared coalesce sweep should be reported as stalled right now.(package private) static booleanisCoalesceSweepStalled(long sweepAgeMs) Whether the coalesce sweep has missed enough cycles that no rate can be recording.static booleanisDisabled(I2PAppContext ctx) Whether graph generation is currently switched off.private booleanlocked_getXML(Rate rate, OutputStream out) Export rate data as XML under the semaphore lock.private booleanlocked_renderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) Render the two-data bandwidth graph under the semaphore lock.private booleanlocked_renderGraph(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) Render a single-data graph under the semaphore lock.(package private) static booleanneedsRevive(boolean mapped, boolean detached, boolean registered) Does a configured rate need a listener rebuilt?private booleannoteCoalesceProgress(long coalesced) Track whether the tracked rates are still coalescing at all.parseSpecs(String specs) Resolve astat.summariesstring to the Rates it names.private static StringName a rate for a log line, tolerating a Rate with no stat behind it.private booleanrebuildListener(Rate rate, GraphListener lsnr, String why) Replace a listener for a rate, releasing the old one first.(package private) static GraphGenerator.StaleCauserefineForIdleRate(GraphGenerator.StaleCause cause, long lastEventCount) Split "the rate has nothing to say" out of the causes that lose data.private voidRemove a rate from tracking and stop its listener.booleanrenderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit) Render the two-data bandwidth graph with the specified options.booleanrenderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) Render the two-data bandwidth graph with the specified options.booleanrenderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) AsrenderCombinedGraph(OutputStream, int, int, boolean, boolean, boolean, boolean, int, int, boolean), but emitting the plot geometry and series as JSON.booleanrenderCombinedGraphMeta(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) The two-data bandwidth graph's metadata, as JSON.booleanrenderGraph(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) Render a single-data graph with the specified options.booleanrenderGraph(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) AsrenderGraph(Rate, OutputStream, int, int, boolean, boolean, boolean, boolean, int, int, boolean, boolean), but emitting the plot geometry and series as JSON.booleanrenderGraphMeta(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) A single stat's metadata, as JSON.booleanrenderGroupedGraph(OutputStream out, String groupId, Set<String> enabledStats, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) Render a combined graph for one group.booleanrenderGroupedGraph(OutputStream out, String groupId, Set<String> enabledStats, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) AsrenderGroupedGraph(OutputStream, String, Set, int, int, boolean, boolean, boolean, boolean, int, int, boolean, boolean), but emitting the plot geometry and series as JSON.booleanrenderGroupedGraphMeta(OutputStream out, String groupId, Set<String> enabledStats, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) A grouped graph's metadata, as JSON.private voidLoud failure for a silently stalled data path: logs an ERROR when any attached listener is not recording, naming the cause per listener class rather than reporting one ambiguous count.private voidRe-create any listener that has stopped recording, so neither a transient write failure nor a lost Rate registration can silently retire a graph for the remaining life of the router.voidrun()Starts graph generation, disabling the app if it fails before registration.private voidBody ofrun(), separated so any failure before the app registers itself is logged.(package private) voidDisable graph generation until restart.(package private) static voidsetDisabled(I2PAppContext ctx) Disable graph generation until restart.voidDoes nothing, we aren't tracked.(package private) static longstaleAge(long now, long startedMs, long lastUpdateSuccess) Age of a listener's data path, measured from whichever origin exists.(package private) static StringstaleAgeOrigin(boolean sinceLastWrite) Wording for astaleAge(long, long, long)value: which event the age is measured from.private voidStart the 10s health watchdog, at most once.voidstartup()Does nothing, we aren't tracked.voidstop()Shuts down the health watchdog and the scheduler, unregistering the app.private voidStop the health watchdog if one was started.private voidsyncSpecsHolder(String[] specsHolder) Sync RRD4J stats to disk on a fixed schedule.
Field Details
_context
The router context._log
Logger for the graph renderer._listeners
List of GraphListener instances, preserved for iteration by UI helpers._listenerByRate
Index of listener by rate for O(1) lookup during rendering.MAX_CONCURRENT_RENDER
private static final int MAX_CONCURRENT_RENDERGraph renders allowed at once._sem
Permits limiting the concurrent renders._isRunning
private volatile boolean _isRunningTrue while the generator runs._scheduler
Scheduler running the periodic renders._healthScheduler
Health watchdog scheduler, separate from_schedulerso a stuck sync task cannot stop the thing that is meant to notice it is stuck._healthStarted
private boolean _healthStartedGuards the watchdog against being scheduled twice; seestartHealthWatchdog()._ownsBackendFactory
private volatile boolean _ownsBackendFactoryWhether this instance configured the shared RRD backend factory.RrdBackendFactory.getDefaultFactory()is a process-wide singleton, so closing it from aGraphGenerator.Shutdownthat never configured it would pull the backend out from under every other graph listener still recording. Set at the point this instance callssetSyncPoolSize, which is the only place it takes responsibility for the shared configuration.- See Also:
_stallThrottle
Throttles the stall report so the watchdog and the sync task cannot double-log it._driftThrottle
Throttles the ledger-drift report independently of the stall report._stallReported
private boolean _stallReportedSet when a stall or drift report is logged, so the matching recovery line is emitted exactly once when the condition clears. Without it a healthy router would log a recovery on every watchdog tick, and a router that never stalled would log one at all._driftReported
private boolean _driftReportedTrue once the rrd drift was logged._lastCoalesceSum
private long _lastCoalesceSumCoalesce delta across all tracked listeners at the previous watchdog tick._coalesceStalledTicks
private int _coalesceStalledTicksConsecutive watchdog ticks on which no tracked rate coalesced at all.NAME
HEARTBEAT_TICKS
private static final int HEARTBEAT_TICKSEmit a liveness heartbeat every N sync ticks (~27min at the 90s period).- See Also:
HEALTH_INTERVAL_MS
private static final int HEALTH_INTERVAL_MSWatchdog period.Short enough that the next occurrence of a silently dead data path names its own cause while somebody is still looking at the log, and long enough that a healthy router does no measurable work. Deliberately separate from the 90s sync task: a fault severe enough to wedge that task is exactly the fault the watchdog exists to report, so sharing a thread would hide the symptom it is looking for.
- See Also:
COALESCE_STALL_MS
private static final long COALESCE_STALL_MSLongest gap between coalesces on any tracked rate before "the rates stopped".- See Also:
COALESCE_STALL_TICKS
private static final int COALESCE_STALL_TICKSCOALESCE_STALL_MSexpressed in watchdog ticks.- See Also:
MAX_BACKFILL_STEPS
static final int MAX_BACKFILL_STEPSUpper bound on how many retained samples one backfill is asked for.- See Also:
REPORT_REPEAT_MS
private static final long REPORT_REPEAT_MSRepeat period for a report whose numbers have not changed.- See Also:
_startedMs
private final long _startedMsWall-clock time when graph generation started; lets the never-yet-written staleness test distinguish an empty startup window from a genuinely stalled writer._ticks
private int _ticksRenders completed by the current job.MIN_STALENESS_PERIOD_MS
static final long MIN_STALENESS_PERIOD_MSFloor applied to a rate's period when deciding whether its listener is stale.Sub-minute rates can legitimately go minutes between coalesces, and 2x a few hundred milliseconds would flag them from their first tick onwards.
- See Also:
MAX_NAMED
static final int MAX_NAMEDMost listener names a single tally will list before summarising the rest as a count.- See Also:
staleAgeSinceLastWrite
Origin wording for an age measured from the last successful write.- See Also:
staleAgeSinceLastWriteAttach
Origin wording for an age with no earlier write to measure back from.The listener's own attach time, not the router's graphing start: only the listener knows how long it has had the chance to write.
- See Also:
COALESCE_STALL_GRACE_MS
static final long COALESCE_STALL_GRACE_MSGrace period before a coalesce sweep that has never run counts as a stall.One coalesce interval (
Router.COALESCE_TIME, 50s) plus one shortest rate period, rounded up to two staleness periods.- See Also:
DEFAULT_DATABASES
The stats graphed whenstat.summariesis unset, as statName.period pairs.- See Also:
IS_WIN
private static final boolean IS_WINTrue when running on Windows.
Constructor Details
GraphGenerator
Installs the rrd4j log bridge and the shutdown hook. Recording starts in run().- Parameters:
ctx- the router context, for properties, stats, the log and the app manager
Method Details
installRrdLogging
private void installRrdLogging()Route rrd4j diagnostics into the router log.The vendored rrd4j in apps/jrobin is built without a classpath, so it cannot log through I2P itself and ships with all of its own logging stripped. That is why a permanently cancelled flush schedule produced no output at all: the .jrb files simply stopped changing. This installs the bridge so that failure is visible from now on.
instance
The generator registered with the app manager, which only exists while it is running.- Parameters:
ctx- the context whose app manager holds the registration- Returns:
- null if disabled
run
runGenerator
private void runGenerator()Body ofrun(), separated so any failure before the app registers itself is logged.startHealthWatchdog
private void startHealthWatchdog()Start the 10s health watchdog, at most once.Idempotent on purpose:
runGenerator()is the only caller and runs once, but a second scheduler would mean two threads applying the heal ladder to the same listeners concurrently, which is how a REOPEN ends up racing a REBUILD into a half-closed handle. Guarded here rather than trusted to the caller.stopHealthWatchdog
private void stopHealthWatchdog()Stop the health watchdog if one was started.Never called from a watchdog task: a scheduler that awaits its own termination from inside a task cannot terminate, and the bounded wait would become an unbounded stall.
syncSpecsHolder
Sync RRD4J stats to disk on a fixed schedule. Stops the generator if the router is no longer alive.- Parameters:
specsHolder- single-element holder for the active specs string
classifyStaleness
static GraphGenerator.StaleCause classifyStaleness(boolean detached, boolean registered, long lastUpdateSuccess, long now, long startedMs, long ratePeriod) Decide whether a listener is stalled, and why, assuming the coalesce sweep has run recently enough to have fed these rates.- Parameters:
detached-GraphListener.isDetached()- the RRD is closedregistered-GraphListener.isRegistered()- the Rate still points at the listenerlastUpdateSuccess- wall-clock ms of the last successful write, or 0 if nonenow- current wall-clock msstartedMs- wall-clock ms when graphing beganratePeriod- the rate's period in ms- Returns:
GraphGenerator.StaleCause.OKwhen there is nothing to report, otherwise the cause- See Also:
classifyStaleness
static GraphGenerator.StaleCause classifyStaleness(boolean detached, boolean registered, long lastUpdateSuccess, long now, long startedMs, long ratePeriod, boolean coalesceStalled) Decide whether a listener is stalled, and why.Pure decision logic, split out of the sync task so the thresholds can be pinned by unit tests without a router or an RRD file.
- Parameters:
detached-GraphListener.isDetached()- the RRD is closedregistered-GraphListener.isRegistered()- the Rate still points at the listenerlastUpdateSuccess- wall-clock ms of the last successful write, or 0 if nonenow- current wall-clock msstartedMs- wall-clock ms when graphing beganratePeriod- the rate's period in mscoalesceStalled- true when the coalesce sweep has not run recently enough to have fed these rates; seeStatManager.getCoalesceSweepAgeMs(long)- Returns:
GraphGenerator.StaleCause.OKwhen there is nothing to report, otherwise the cause
staleAge
static long staleAge(long now, long startedMs, long lastUpdateSuccess) Age of a listener's data path, measured from whichever origin exists.- Parameters:
now- current wall-clock msstartedMs- wall-clock ms when graphing beganlastUpdateSuccess- wall-clock ms of the last successful write, or 0 if none- Returns:
- ms since the last write, or ms since graphing began when nothing was ever written
staleAgeOrigin
Wording for astaleAge(long, long, long)value: which event the age is measured from.- Parameters:
sinceLastWrite- true if a successful write has ever happened- Returns:
- the origin to quote alongside the age
refineForIdleRate
static GraphGenerator.StaleCause refineForIdleRate(GraphGenerator.StaleCause cause, long lastEventCount) Split "the rate has nothing to say" out of the causes that lose data.A listener that has never written, or has stopped writing, looks identical whether its RRD writes are failing or whether nothing in the router is feeding its rate at all. Only the first loses data. Event count settles it: a rate with no events in its last completed period had nothing to hand the listener, so there is nothing being lost.
Event count rather than coalesce count, deliberately. Coalescing is driven by the StatManager sweep and happens on schedule whether or not any data arrives, so it keeps advancing for a rate nobody is updating. That made the coalesce form unable to distinguish "idle" from "fed but unwritten" for any listener that had been running, which is precisely the case a stopped service produces.
- Parameters:
cause- classification fromclassifyStaleness(boolean, boolean, long, long, long, long)lastEventCount- events the rate accrued in its last completed period, or 0 when it accrued none- Returns:
GraphGenerator.StaleCause.RATE_IDLEwhen the rate produced no events in its last period, otherwise cause unchanged
formatStaleness
static String formatStaleness(int totalListeners, GraphGenerator.CauseTally neverWritten, GraphGenerator.CauseTally unregistered, GraphGenerator.CauseTally writesStopped) Compose the write-stall log line.Pure: a function of the three tallies, so both the wording and the count attribution can be unit tested without a router.
- Parameters:
totalListeners- number of listeners being trackedneverWritten- tally forGraphGenerator.StaleCause.NEVER_WRITTENunregistered- tally forGraphGenerator.StaleCause.UNREGISTEREDwritesStopped- tally forGraphGenerator.StaleCause.WRITES_STOPPED- Returns:
- a single-line message beginning "RRD data stalled:"
formatStaleness
static String formatStaleness(int totalListeners, GraphGenerator.CauseTally neverWritten, GraphGenerator.CauseTally unregistered, GraphGenerator.CauseTally writesStopped, GraphGenerator.CauseTally coalesceStalled) Compose the write-stall log line, counting the coalesce fault alongside the per-listener causes.- Parameters:
totalListeners- number of listeners being trackedneverWritten- tally forGraphGenerator.StaleCause.NEVER_WRITTENunregistered- tally forGraphGenerator.StaleCause.UNREGISTEREDwritesStopped- tally forGraphGenerator.StaleCause.WRITES_STOPPEDcoalesceStalled- tally forGraphGenerator.StaleCause.COALESCE_STALLED- Returns:
- a single-line message beginning "RRD data stalled:"
reportWriteStaleness
private void reportWriteStaleness()Loud failure for a silently stalled data path: logs an ERROR when any attached listener is not recording, naming the cause per listener class rather than reporting one ambiguous count.Called from both the 90s sync task and the 10s watchdog, so the report is throttled on its own numbers rather than on which task happened to run: an unchanged stall is repeated every
REPORT_REPEAT_MS, a changed one immediately, and the first occurrence is never throttled at all.A single INFO follows once the stall clears, so the ERROR has a visible ending rather than being indistinguishable from a router still losing data.
isCoalesceSweepStalled
static boolean isCoalesceSweepStalled(long sweepAgeMs) Whether the coalesce sweep has missed enough cycles that no rate can be recording.Pure threshold test, separated from the manager lookup so it can be pinned by unit tests. A sweep is considered stalled once it is older than twice the shortest rate period: the coalesce timer runs every 50s and the shortest period is one minute, so two missed sweeps is already unambiguous.
- Parameters:
sweepAgeMs- ms since the last completed sweep, or -1 if none ever has- Returns:
- true when the sweep is stalled
isCoalesceStalledNow
static boolean isCoalesceStalledNow(boolean everSwept, long sweepAgeMs, long startedMs, long now) Whether the shared coalesce sweep should be reported as stalled right now.Separate from
isCoalesceSweepStalled(long)so the start-up window can be pinned by tests. Until a sweep has ever completed its age is reported as -1, meaning "none ever", which that function quite correctly calls stalled. At start-up, though, that is not a fault - it is a sweep that has not come round yet, because the coalesce timer only fires everyRouter.COALESCE_TIME. Judging it as stalled made every boot log an ERROR naming the coalesce timer as the fault within seconds of start-up, and pointed the operator at the one subsystem the evidence could not implicate.- Parameters:
everSwept- whether any sweep has completed since graphing begansweepAgeMs- ms since the last completed sweep, or -1 if none ever hasstartedMs- wall-clock ms when graphing begannow- current wall-clock ms- Returns:
- true only once a sweep should have come round and has not
isCoalesceStalled
private boolean isCoalesceStalled(long now) Whether the coalesce sweep is currently stalled, perisCoalesceStalledNow(boolean, long, long, long).- Parameters:
now- the current time in ms since the epoch- Returns:
- true only once a sweep should have come round and has not
coalesceSweepAbsence
How long the coalesce sweep has been absent, for the report.Always a real, non-negative figure: the caller only reaches this once the absence has already exceeded the normal cadence, so any sentinel would be meaningless noise in an operator-facing line. A sweep that has never run has no age to quote, so the age of the graphing window is reported in its place and labelled as such - a lower bound on the absence, and never the
-1the accessor uses to mean "none ever".- Parameters:
now- current wall-clock ms- Returns:
- a human-readable duration that always starts with a non-negative number
formatCoalesceSweepAbsence
Render how long the coalesce sweep has been absent.Pure, so the sentinel handling can be pinned by test. The bug this exists to prevent: the accessor reports an unknown age as
-1, and an earlier version interpolated that straight into the operator-facing ERROR, producing the nonsensical"has not completed for -1s". A negative duration is not a duration. The unknown case is now reported as the age of the graphing window and labelled, which is a real lower bound on the absence.- Parameters:
sweepAgeMs- ms since the last completed sweep, or negative if none ever hasstartedMs- wall-clock ms when graphing begannow- current wall-clock ms- Returns:
- a duration that always begins with a non-negative number of seconds
needsRevive
static boolean needsRevive(boolean mapped, boolean detached, boolean registered) Does a configured rate need a listener rebuilt?Three distinct faults, each of which leaves the rate unrecorded for the rest of the router's life while every observable state still reads healthy: the RRD was closed after repeated write failures (
detached), the map lost its listener (mappedfalse), or a newer listener took the Rate's single registration so this one can never be called again (registeredfalse).- Parameters:
mapped- false if the rate has no listener entry at alldetached- true if the listener's RRD is closedregistered- true if the Rate still points at the listener- Returns:
- true if the rate needs a new listener
reviveDetachedListeners
private void reviveDetachedListeners()Re-create any listener that has stopped recording, so neither a transient write failure nor a lost Rate registration can silently retire a graph for the remaining life of the router.A detached listener stays in the rate-to-listener map, so it is never rebuilt by
adjustDatabases(String): that only adds rates missing from the old spec. The same holds for a listener that kept its open RRD but lost the registration, and for a rate whose mapping lost its listener. This runs on the same tick as the spec sync, reusing the existing scheduled task rather than adding a thread.rebuildListener
Replace a listener for a rate, releasing the old one first.Shared with the health watchdog's REBUILD step so there is exactly one definition of what retiring a listener means: the old handle is released before the replacement re-opens the same file, and the stale entry leaves the listener list so it keeps counting only what actually records.
- Parameters:
rate- the series being rebuilt; re-opens its RRD file and names the rate in the log linelsnr- the listener being replaced, or null if the map lost itwhy- short reason for the log line- Returns:
- true if a replacement listener is now recording
rateName
chooseHeal
static GraphGenerator.HealAction chooseHeal(boolean mapped, boolean detached, boolean registered, boolean writable, boolean backfillable) Cheapest action that can restore recording for one rate.Pure decision logic, ordered cheapest first, so the cost ordering is pinned by unit tests rather than by reading the ladder at the call site.
The first row is also the spec-churn guard: a listener that is mapped, attached and registered can only ever reach BACKFILL or NONE, never REOPEN or REBUILD, so re-reading a changed
stat.summariesstring cannot close and reopen a healthy RRD.- Parameters:
mapped- false when the rate has no listener entrydetached- true when the listener's RRD is closedregistered- true when the Rate still points at the listenerwritable- true when the handle can still take a write (not closed)backfillable- true when the rate still retains unstored samples- Returns:
- the action to take, cheapest first
backfillWindow
static int backfillWindow(long drift) How many retained samples one backfill attempt should ask for.Bounded by the drift, since asking for more steps than are missing wastes a list copy every tick, and capped at
MAX_BACKFILL_STEPSso a listener that fell thousands of steps behind does not ask for a list that long - the accessor's own clamp would discard the excess anyway, and the excess is exactly the part that is permanently gone.- Parameters:
drift- coalesces minus stored steps- Returns:
- number of retained samples to request, never below one
classifyDrift
static GraphGenerator.DriftCause classifyDrift(int pending, long oldestPendingMs, long overruns, boolean coalesceAdvancing) Name the mechanism behind positive ledger drift.These are the three candidates that were indistinguishable from the console during the incident this exists for: a coalesce that never happened, a sample that queued and never drained, and a sample that arrived and was not written. Each has a different fix, so the report has to pick one.
Pure, so the mapping from counters to mechanism is unit tested without a router, a StatManager or a clock.
- Parameters:
pending- samples waiting for deliveryoldestPendingMs- age of the oldest waiting sampleoverruns- samples evicted at the queue's hard capcoalesceAdvancing- whether any tracked rate coalesced recently- Returns:
- the most specific cause the counters support
formatLedgerDrift
static String formatLedgerDrift(int totalListeners, int drifters, long maxDrift, long permanentSteps, long healedSteps, int rebuilt, int reopened, long coalesced, GraphGenerator.DriftCause mechanism, long overruns, long superseded, int pending, long oldestPendingMs, long backlogCollapses, long coalesceSkips) Compose the ledger-drift log line.Pure and single-line, so the wording and the six StatManager counters can be asserted in a unit test. The counters are the point of the line: an operator reading "pending climbing with oldest-pending-age climbing" knows the consumer is wedged, "pending zero with a flat coalesce count" knows the rate stopped coalescing, and a climbing skip count points at the coalesce early return.
- Parameters:
totalListeners- listeners being trackeddrifters- number of listeners with positive driftmaxDrift- largest positive driftpermanentSteps- steps proven beyond the retained-sample ringhealedSteps- steps backfilled since the previous reportrebuilt- listeners rebuilt this tickreopened- listeners reopened this tickcoalesced- coalesce delta across the tracked listenersmechanism- the named cause fromclassifyDrift(int, long, long, boolean)overruns- StatManager sample overrunssuperseded- StatManager same-step supersessionspending- StatManager delivery queue deptholdestPendingMs- StatManager age of the oldest queued samplebacklogCollapses- StatManager coalesce backlog collapsescoalesceSkips- StatManager coalesce skips- Returns:
- a single-line message beginning "RRD ledger drift:"
healthCheck
private void healthCheck()One watchdog tick: report a stalled data path, then apply the heal ladder.Holds no lock across I/O and does no RRD work itself beyond the backfill the ladder asks for. Wrapped in a Throwable catch because an escaping exception silences a fixed-rate scheduled task permanently, which would remove the very watchdog that is supposed to notice the data path dying.
applyHealLadder
private void applyHealLadder()Walk every listener, heal what can be healed, and report what could not be.Iterates
_listenersrather than the rate map so that "mapped" is a real question: a listener that is alive but has lost its map entry is exactly the REBUILD case the ladder exists for, and iterating the map would never see it. The map's own dead entries are handled byreviveDetachedListeners().The ledger reports at WARN, not ERROR, because it describes drift the ladder has usually already repaired, and a single INFO marks its clearing.
noteCoalesceProgress
private boolean noteCoalesceProgress(long coalesced) Track whether the tracked rates are still coalescing at all.Tested over a window rather than a single tick: a 60s rate coalesces on one tick in six, so comparing against the previous tick alone would report "the rates stopped" five times out of six.
Not synchronized, and deliberately so: only the single watchdog thread ever calls it, whereas
stop()holds this instance's monitor across a bounded scheduler shutdown wait. Synchronizing here would let a shutdown stall the watchdog for the length of that wait.- Parameters:
coalesced- coalesce delta summed across the tracked listeners- Returns:
- true if some rate coalesced within
COALESCE_STALL_MS
stop
public void stop()Shuts down the health watchdog and the scheduler, unregistering the app.The watchdog goes first: it is the task that would notice the sync task hanging, so it has to be shut down before the thing it watches, not after.
isDisabled
Whether graph generation is currently switched off.- Parameters:
ctx- the context whose app manager holds the registration- Returns:
- true if no generator is registered, so nothing is being recorded
setDisabled
Disable graph generation until restart.- Parameters:
ctx- the context whose registered generator is stopped
setDisabled
void setDisabled()Disable graph generation until restart.startup
shutdown
getState
Always RUNNING.getName
getDisplayName
The app's display name.- Specified by:
getDisplayNamein interfaceClientApp- Returns:
- non-null
getListeners
List of GraphListener instances.- Returns:
- the registered listeners, empty if none
countGraphs
public int countGraphs()How many stats are being recorded right now.- Returns:
- the number of live listeners
adjustDatabases
removeDb
Remove a rate from tracking and stop its listener.- Parameters:
r- the rate to remove
addDb
Start tracking a rate by creating a new GraphListener for it.Single choke point for every listener creation, so it is also where "one live listener per rate" is enforced. Both the 90s sync task and the 10s watchdog can decide to rebuild the same rate within one interval; without this the second rebuild would register a second listener on the rate, orphan the first into the listener list, and leave an attached-but-never-called listener accumulating one per tick - which is precisely the fault the ladder exists to remove.
- Parameters:
r- the rate to track- Returns:
- true if a new listener was created and is recording
renderGraph
public boolean renderGraph(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) throws IOException Render a single-data graph with the specified options. For the two-data bandwidth graph seerenderCombinedGraph(OutputStream, int, int, boolean, boolean, boolean, boolean, int, int, boolean). Synchronized to conserve memory.- Parameters:
rate- the rate to graphout- the output stream to write the graph image towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, draw event markersperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, show the I2P+ credit lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" label- Returns:
- true on success, false if the stat is not currently renderable
- Throws:
IOException- if rendering fails
renderGraphMeta
public boolean renderGraphMeta(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) throws IOException A single stat's metadata, as JSON.- Parameters:
rate- the rate to graphout- the output stream to write the metadata towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, plot the event count rather than the statperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, keep the signature lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" label- Returns:
- true on success, false if the stat is not currently renderable
- Throws:
IOException- if rendering fails
renderGraph
public boolean renderGraph(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) throws IOException AsrenderGraph(Rate, OutputStream, int, int, boolean, boolean, boolean, boolean, int, int, boolean, boolean), but emitting the plot geometry and series as JSON.- Parameters:
rate- the rate to graphout- the output stream to write the image or the metadata towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, plot the event count rather than the statperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, keep the signature lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" labelmeta- true to write the metadata instead of the image- Returns:
- true on success, false if the stat is not currently renderable
- Throws:
IOException- if rendering fails
locked_renderGraph
private boolean locked_renderGraph(Rate rate, OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) throws IOException Render a single-data graph under the semaphore lock.- Parameters:
rate- the rate to graphout- the output stream to write towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, draw event markersperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, show the I2P+ credit lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" labelmeta- true to write the metadata instead of the image- Returns:
- true on success, false if the stat is not currently renderable
- Throws:
IOException- if rendering fails
getXML
Export rate data as XML.- Parameters:
rate- the rate to exportout- the output stream to write the XML to- Returns:
- true if the data was exported successfully
- Throws:
IOException- if export fails
locked_getXML
Export rate data as XML under the semaphore lock.- Parameters:
rate- the rate to exportout- the output stream to write to- Returns:
- true if the data was exported successfully
- Throws:
IOException- if export fails
renderCombinedGraph
public boolean renderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit) throws IOException Render the two-data bandwidth graph with the specified options. For all other graphs seerenderGraph(Rate, OutputStream, int, int, boolean, boolean, boolean, boolean, int, int, boolean, boolean)above. Synchronized to conserve memory.- Parameters:
out- the output stream to write the graph image towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, draw event markersperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, show the I2P+ credit line- Returns:
- true on success, false if the stat is not currently renderable
- Throws:
IOException- if rendering fails
renderCombinedGraph
public boolean renderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) throws IOException Render the two-data bandwidth graph with the specified options.- Parameters:
out- the output stream to write towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, draw event markersperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, show the I2P+ credit lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" label- Returns:
- true on success, false if the stat is not currently renderable
- Throws:
IOException- if rendering fails
renderCombinedGraphMeta
public boolean renderCombinedGraphMeta(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) throws IOException The two-data bandwidth graph's metadata, as JSON.- Parameters:
out- the output stream to write the metadata towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, plot the event count rather than the statperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, keep the signature lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" label- Returns:
- true on success
- Throws:
IOException- if rendering fails
renderCombinedGraph
public boolean renderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) throws IOException AsrenderCombinedGraph(OutputStream, int, int, boolean, boolean, boolean, boolean, int, int, boolean), but emitting the plot geometry and series as JSON.- Parameters:
out- the output stream to write the image or the metadata towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, plot the event count rather than the statperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, keep the signature lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" labelmeta- true to write the metadata instead of the image- Returns:
- true on success
- Throws:
IOException- if rendering fails
locked_renderCombinedGraph
private boolean locked_renderCombinedGraph(OutputStream out, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) throws IOException Render the two-data bandwidth graph under the semaphore lock.- Parameters:
out- the output stream to write towidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, draw event markersperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, show the I2P+ credit lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" labelmeta- true to write the metadata instead of the image- Returns:
- true on success, false if the stat is not currently renderable
- Throws:
IOException- if rendering fails
parseSpecs
Resolve astat.summariesstring to the Rates it names.A token that is not statName.period, names an unknown stat or carries a non-numeric period is skipped rather than failing the sync tick.
- Parameters:
specs- a comma-separated list of statName.period tokens; malformed tokens are skipped rather than failing the sync tick- Returns:
- list of Rate objects
getGroupListeners
Collect the listeners for the enabled members of a graph group.Members are returned in the group's legend order, not in listener order, so the colours and legend entries stay put as stats are enabled and disabled. Members whose RRD has not been created yet are skipped: a stat that has never been sampled has no series to draw, and including it would silently stretch the axis to zero.
- Parameters:
groupId- a group id fromGraphGroupsenabledStats- stat names enabled by the user, without period suffixes- Returns:
- the members that have data, in legend order; empty when fewer than two
renderGroupedGraph
public boolean renderGroupedGraph(OutputStream out, String groupId, Set<String> enabledStats, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) throws IOException Render a combined graph for one group.- Parameters:
out- the output stream to write the graph image togroupId- a group id fromGraphGroupsenabledStats- stat names enabled by the user, without period suffixeswidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, plot the event count rather than the statperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, keep the signature lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" label- Returns:
- true if a graph was written; false when the group had too few usable members
- Throws:
IOException- if rendering fails
renderGroupedGraphMeta
public boolean renderGroupedGraphMeta(OutputStream out, String groupId, Set<String> enabledStats, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts) throws IOException A grouped graph's metadata, as JSON.- Parameters:
out- the output stream to write the metadata togroupId- a group id fromGraphGroupsenabledStats- stat names enabled by the user, without period suffixeswidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, plot the event count rather than the statperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, keep the signature lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" label- Returns:
- true on success
- Throws:
IOException- if rendering fails
renderGroupedGraph
public boolean renderGroupedGraph(OutputStream out, String groupId, Set<String> enabledStats, int width, int height, boolean hideLegend, boolean hideGrid, boolean hideTitle, boolean showEvents, int periodCount, int end, boolean showCredit, boolean showRestarts, boolean meta) throws IOException AsrenderGroupedGraph(OutputStream, String, Set, int, int, boolean, boolean, boolean, boolean, int, int, boolean, boolean), but emitting the plot geometry and series as JSON.- Parameters:
out- the output stream to write the image or the metadata togroupId- a group id fromGraphGroupsenabledStats- stat names enabled by the user, without period suffixeswidth- image width in pixelsheight- image height in pixelshideLegend- if true, omit the legendhideGrid- if true, omit the grid lineshideTitle- if true, omit the titleshowEvents- if true, plot the event count rather than the statperiodCount- number of time periods to display, or -1 for defaultend- number of periods before now to end atshowCredit- if true, keep the signature lineshowRestarts- if true, draw the vertical restart lines and "Router restarted" labelmeta- true to write the metadata instead of the image- Returns:
- true on success
- Throws:
IOException- if rendering fails
deleteOldRRDs
private void deleteOldRRDs()Delete the old rrd dir if we are no longer persistent._t
closeBackendFactory
private void closeBackendFactory()Close the shared RRD backend factory, but only if this instance configured it.RrdBackendFactory.getDefaultFactory()is a singleton, so a secondGraphGeneratorshutting down after the one that configured it would otherwise stop the flush pool out from under every listener still recording.