- All Implemented Interfaces:
Job
First tests are not queued one job at a time: offers go to a bounded first-test buffer (offerFirstTest(RouterContext, PooledTunnelCreatorConfig, TunnelPool)) that a single paced pump job drains in small batches under the in-flight gates, so build bursts cannot flood the job queue. Retests reuse this instance via scheduleRetest(boolean).
Nested Class Summary
Nested ClassesModifier and TypeClassDescription(package private) static final classRuntime state owned by a single router context: the first-test buffer and its pump flag, the instance and in-flight gauges, and the running-test map.(package private) static enumWhy a round was refused its in-flight permits.(package private) static final classThe instance slots one TestJob owns: the total counter (memory bound), the per-pool claim, and the running-test registration.private classHandles the reply to one test message.private classTimes out one test message.(package private) static final classA first-test candidate waiting in the batch buffer.private static classSingle queue entry that drains the first-test buffer in small batches.private classMatches the reply for one test message.(package private) static final classIdentity of one dispatched test round: the generation it belongs to, the pool it counts against, the deadline it expires at, and the release switches for the permits it reserved.Field Summary
Modifier and TypeFieldDescriptionprivate static final AtomicIntegerCounter handing out test ids.private static intCached base limit on queued tests.private static intCached hard limit on queued tests.private static intCached test limit for client pools.private static intCached limit on concurrent tests.private static intCached test limit for exploratory pools.private static intCached longest delay before a test times out.private static intCached longest interval between tests.private static intCached shortest delay before a test times out.private static intCached shortest interval between tests.private static doubleCached pool coverage needed to skip testing.private final PooledTunnelCreatorConfigThe pool's tunnel configuration.private static RouterContextContext the cached config values were read from.private static longWhen the cached config values were last read.private TestJob.InstanceClaimsThe instance slots this job owns: the total counter, the per-pool claim (with the pool id captured when it was taken), and the running-test registration.private booleanTrue once this failure has already been re-tested against a different partner without charging the tunnel.private intTimes this test was deferred by the concurrency cap.private SessionTagSession tag for the ElGamal test message.private TunnelInfoThe partner whose failure triggered the current confirmation.Reply partner router hash to the time it was last blamed for a failed round, per direction.private final LogLogger for tunnel test results.Outbound test results tallied per reply partner peer.private TunnelInfoThe outbound tunnel carrying the test.private intTimes the paired pool kicked this test.private intConsecutive test rounds deferred because the partner tunnel was suspect.private booleanTrue when the partner for the current round came from the exploratory pool because the paired pool had nothing to offer.private OutNetMessageThe last pending message registered for this instance's round, kept so a reset or drop can unregister it even when the round was registered but never armed (permit refusal).private final TunnelPoolThe pool whose tunnels are tested.private intDeferrals since a pool kick.private RatchetSessionTagSession tag for the ratchet test message.private intDeferrals waiting for a reply partner.private TunnelInfoThe inbound tunnel expected to carry the reply.private final AtomicReference<TestJob.RoundToken> The round currently in flight on this instance, or the most recently completed one.private final AtomicLongMonotonic generation for round tokens; diagnostics only.private intSuccesses held in the success history.private boolean[]Rolling window of recent test results.private intWrite index into the success history.private booleanSet when a restart, shutdown, or queue death cancelled this instance while it was still queued.private intThis test's id, matched against the reply.private intTest period for the current round, computed once at send time so the reply is judged against the same window used to set the expiration.private static intTuner-set longest test timeout, -1 when untuned.private static intTuner-set longest test interval, -1 when untuned.private static intTuner-set shortest test timeout, -1 when untuned.private static intTuned test-job params, -1 = use router config.private booleanFlag to indicate if this job is valid and should be queued.(package private) static final intIn-flight cap divisor defining an "ebb" for active-GOOD retests: when fewer thanmaxConcurrent / ACTIVE_GOOD_EBB_DIVISORtests are on the wire there is spare capacity to verify a busy tunnel.(package private) static final longLongest a retest may be pushed out while its tunnel is busy.private static final ConcurrentHashMap<RouterContext, TestJob.BatchState> OneTestJob.BatchStateper router context.private static final longHow often the cached config values are refreshed.private static final intMaximum number of TestJob instances that should be queued before deferring new ones.private static final floatSuccess rate assumed for a tunnel with no history.(package private) static final longMinimum spacing between batch-denial INFO lines (ms).(package private) static final String[]The per-direction event names registered byregisterDirectionStats(StatManager, long[]), excluding the pass/fail pair whichdirectionStat(boolean, boolean)spells itself.(package private) static final intCandidates popped from the buffer per drain.(package private) static final longShortest wait before re-checking an active-GOOD deferral.(package private) static final doubleFailure rate at or above which a reply partner counts as implicated in the round that just failed.(package private) static final intBatched first-test dispatch.private static final intNumber of times a test can be deferred for lack of a partner tunnel before the paired pool is kicked to rebuild.private static final intJob queue lag in ms above which testing is deferred.private static final intLifetime budget of paired-pool kicks issued by the deferral escape inrunJob().(package private) static final intMaximum consecutive test rounds that can be deferred because the partner tunnel was suspect before the failure is counted regardless.private static final intMaximum number of deferrals allowed after the paired-pool kick budget is exhausted before the tunnel is failed.(package private) static final intConsecutive rounds that may be deferred because the same reply partner was already blamed for a failed round, before the tunnel is charged regardless.private static final intUpper bound on remembered reply partners, so a router that churns through many destinations cannot grow this without limit.private static final intHard ceiling on consecutive test failures for server pool tunnels.(package private) static final intStranded-UNTESTED candidates a pool sweep offers per pass — bounds one ensure cycle's contribution to the buffer.static intThe queue ceiling actually in force.private static final longMinimum gap between two tests of the same already-tested tunnel.(package private) static final intCandidates claimed from the buffer per pump run.(package private) static final longJob-queue lag above which the pump uses the busy delay (ms).(package private) static final longPump requeue delay while the job queue is busy.(package private) static final longPump requeue delay while the job queue is quiet.(package private) static final longPump requeue delay when the job queue is badly lagging.(package private) static final longJob-queue lag above which the pump uses the lagged delay (ms).private static final longHow recently data must have flowed through a tunnel for us to trust it over the test result.private static final intRecent test results kept for the success rate.(package private) static final intBytes credited to a tunnel for one completed test round.(package private) static final longHow long after the last real traffic a tunnel test is held back.(package private) static final longHow recently a tunnel must have carried real (non-test) traffic for the pool to clear its FAILING flag and mark it GOOD.(package private) static final longGrace period before a tunnel's first test failure can count against it.Constructor Summary
ConstructorsConstructorDescriptionTestJob(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) TestJob.TestJob(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool, TestJob.InstanceClaims claims) TestJob for a caller that already holds the instance slots, such as the batch pump, which must claim before it constructs so a refused candidate can go back to the buffer instead of being dropped.Method Summary
Modifier and TypeMethodDescription(package private) voidabandonRound(TestJob.RoundToken token) Give up a round that was armed but never made it onto the wire: mark it claimed so any straggler callback is a no-op, then return its permits.(package private) static longactiveGoodDeferDelay(long remainingDeferMs, long normalDelayMs) Delay for a retest that was held back because its tunnel is busy: the time left in the traffic window, clamped to [EBB_RETRY_FLOOR_MS,ACTIVE_GOOD_EBB_RETRY_MS], or the normal retest delay when there is nothing left to wait for.private static voidRefreshmaxQueuedTestsfrom the configured property when the Tuner has not overridden the static dynamically.(package private) static intapplyMinRetestGap(int delayMs, long minGapMs, boolean expedited) Apply the minimum retest gap to a computed delay.(package private) static voidarmRound(TestJob.BatchState state, AtomicReference<TestJob.RoundToken> active, TestJob.RoundToken next) Armnextas the active round and return the permits held by the round it supersedes.private voidattributeRoundToReplyPartner(boolean inbound, TunnelInfo replyPartner, boolean failed) Tally this round's result against the peer that relayed the reply.(package private) static intbaseRemovalThreshold(boolean degraded) Base removal bar for data-carrying and client/exploratory tunnels.(package private) static StringbatchDenialReason(int inFlight, int maxConcurrent, int poolInFlight, int poolBudget, int total, int hardLimit) Why a batch candidate cannot dispatch yet, or null when it can.(package private) static StringbatchDenialReason(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) Live gate evaluation for one buffered candidate: reads the in-flight gauges and configured budgets from the context's runtime state, then defers to the pure overload.(package private) static TestJob.BatchStatebatchState(RouterContext ctx) The runtime state for a router context, created on first use.(package private) static voidStart a per-context reset of the batch-dispatch subsystem: a router restart, as called fromTunnelPoolManager.restart()before the pools are torn down.private booleanbeginConfirmationRound(TunnelInfo partner) Arm a confirming round: mark the failure as unconfirmed, remember the partner that must not be reused, and schedule the round ASAP.(package private) voidCancel this instance for a restart, a queue shutdown, or an overload drop: unregister its pending message so no callback can arrive for a round nobody is waiting on, claim the active round (so a reply or timeout that races in cannot complete it), return that round's permits, and release this instance's slots.(package private) static booleancanClaimInstance(int current, int hardLimit) Sole authority for allocating a test instance slot: a slot is available while the current count is strictly below the hard limit.(package private) static <T> List<T> claimBatch(Deque<T> buf, int max) Pop up tomaxelements from the front ofbufwith no eligibility gate.(package private) static <T> List<T> claimBatch(Deque<T> buf, int max, java.util.function.Predicate<T> eligible) Drain up tomaxremovable elements from the front ofbuf.(package private) static TestJob.InstanceClaimsclaimBatchSlot(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) Reserve the instance slots a batched TestJob needs: the total counter (memory bound) and the per-pool claim.(package private) static intclaimPoolTestSlot(TestJob.BatchState state, String poolId) Claim one per-pool test-instance slot forpoolIdin a router context's runtime state.(package private) booleanclaimRound(TestJob.RoundToken token) Claim the round for this completion attempt.(package private) static booleanclaimRoundCompletion(TestJob.RoundToken active, TestJob.RoundToken token) First path to claim a round wins: the round must still be the active one and must not have been claimed before.private voidClean up this test job from tunnel tracking.private voidClears session tags used in the current test to avoid leaks.(package private) static intcomputeMaxTestJobs(int maxQueuedTests, int activeRunners, int numPools) Test-job capacity derived from the queue state: at least 12 jobs (or 3 per pool), scaled up by active runners, but never above the hard queued limit.(package private) static intcomputeTestPeriod(double successTimeAvgMs, double sendProcessingMs, int outLen, int replyLen, int minPeriod, int maxPeriod) Compute the tunnel test failure window: how long a test may take before the tunnel is declared failed.private TunnelInfoThe partner the current round was actually routed through.private voidRelease this instance's total-counter slot, once.private booleanDefer a round because the same reply partner was already blamed for a recent failure, and kick the reply pool so the side that is actually degraded is the side that rebuilds.private voiddeferForMissingPartner(TunnelPool paired, String direction) Handle a test round whose paired pool had no tunnel to pair with.private booleanExcuse a failed round because the tunnel is still inside its grace period: it has not yet had a realistic chance to settle, join the LeaseSet, or carry the traffic that would prove it, so the round is observed but not charged.(package private) static StringdirectionStat(boolean inbound, boolean success) Name of a per-direction test outcome stat.(package private) static StringdirectionStat(boolean inbound, String event) Name of a per-direction stat for an outcome that is not simply pass/fail — a confirmation result, or condemnation by the pool.(package private) static intdrainBuffer(RouterContext ctx) Drain one batch from the first-test buffer and dispatch admitted candidates directly (no per-tunnel queue entry).voiddropped()Called when the job is dropped due to router overload, when the batch pump's catch sees a dispatch throw, or when a reset cancels it.(package private) static voidFinish a reset started bybeginBatchReset(RouterContext): resume offers and pump dispatch.(package private) static StringfarEndVerdict(boolean unreachable, boolean established) Classify a failed round's far endpoint into exactly one bucket.(package private) static booleanWhether a tunnel has never produced a latency reading, i.e.(package private) longgetAdaptiveTimeout(long baseTimeout) Get the current test timeout, adapted to the rolling average duration.private static intBase max queued tests value, read from PROP or static default.private static intgetClientPoolTestBudget(RouterContext ctx, TunnelPool pool, int activeCount) Per-pool test budget for client pools, bounded by three knobs: the Tuner budget (dynamic), the pool-coverage cap (at most coverageThreshold of a pool's active tunnels under test at once), and the absolute per-pool cap.static intCurrent number of queued + active test jobs across all router contexts, for capacity planning byBuildExecutor.private intgetDelay()Returns the delay before the next test pass.static intHard limit for total TestJob instances (queued + active).private static longAdaptive latency threshold for demoting a peer as "high latency" after a tunnel test.private static intReturns the test limit for client pools.private static intMaximum number of tunnel tests that can run concurrently.private static intReturns the test limit for exploratory pools.static intThe max test delay in effect, tuned value if set else config.static intMaximum number of queued test jobs allowed before deferring.private intThe maximum test period from config or default (15s).static intThe max test period in effect, tuned value if set else config.static intThe min test delay in effect, tuned value if set else config.private intThe minimum test period from config or default (3s).static intThe min test period in effect, tuned value if set else config.getName()The name of this job.private static doubleReturns the pool coverage needed to skip testing.(package private) static StringgetPoolId(TunnelPool pool) Generate a stable, collision-free identifier for a tunnel pool, used as the key for the per-pool instance and in-flight gauges.private RatchetSKMReturns the ratchet session key manager for test tags.private SessionKeyManagerReturns the session key manager for test tags.private floatReturns the recent test success rate for this pool.private intReturns the interval before the next test of this pool.private static LongGenerate a unique key for a tunnel to track running tests.(package private) static booleanhasPoolTestHeadroom(TestJob.BatchState state, RouterContext ctx, TunnelPool pool, int activeCount) Whether the per-pool instance budget still has room for one more test inpool.(package private) static booleanWhether the tunnel config has non-zero tunnel IDs at hop 0, i.e.private booleanWhether tunnel build success has fallen below the attack threshold.(package private) static booleanisEarlyExpiry(PooledTunnelCreatorConfig cfg, long now) Whether the tunnel expires so soon it is not worth testing — it has already been pruned for early expiry.(package private) static booleanWhether the tunnel belongs to a ping pool, which is short-lived and does not need testing.(package private) static booleanisPoolCritical(int activeCount, int target) Whether a pool is critically low on tunnels: no active tunnels at all, or at most 2 active against a larger target.(package private) static booleanisPoolDeficit(int activeCount, int target) Whether a pool is running below its target active-tunnel count.(package private) static booleanisPoolMember(TunnelPool pool, PooledTunnelCreatorConfig cfg) Whether a pool still owns a tunnel config, i.e.(package private) static booleanisPoolScopedDenial(String reason) Whether a denial reason is scoped to one pool, and therefore can be worked around by trying the candidates behind it, rather than being a global gate that stops the whole scan.(package private) static booleanisRecentDataTransfer(long lastTransferMs, long nowMs) Whether a tunnel's last real traffic is recent enough to trust over a test failure.(package private) static booleanisTestQueueOverloaded(boolean critical, int readyCount, int activeRunners, long maxLag, long expeditedLagLimit, int currentTestJobs, int expeditedJobLimit) Whether the job queue is too backed up to accept another test job.booleanisValid()Check if this TestJob instance is valid and should be queued.(package private) static booleanisZeroActivePool(TunnelPool pool) Whether a client pool has zero active tunnels, i.e.(package private) static StringkeepDataCarryingLine(String tunnel, long verified, int failures, int maxFailures, long staleMs) The line for a failed test on a tunnel that is still carrying data, kept because failing it would drop live traffic.private static voidlogBatchDenial(TestJob.BatchState state, RouterContext ctx, String reason) Rate-limited INFO line for a batch denial, so the reason a backlog is not draining stays visible without flooding the log.private voidRecord blame evidence when a tunnel test fails and our own next hop has no session.(package private) static voidnoteReplyPartnerRound(Map<Hash, long[]> tally, TunnelInfo replyPartner, boolean failed) Record one round against its reply partner: the use always, the failure only when the round failed.private voidnoteSuccess(long ms, TunnelInfo tunnel) Records a successful round trip and its time.(package private) static <T> List<T> offerBounded(Deque<T> buf, T elem, int bound) Appendelemto the buffer, evicting from the front while at or aboveboundso the buffer can never grow without limit.(package private) static booleanofferFirstTest(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) Offer a tunnel's first test to the batched dispatch buffer instead of creating a queue entry per tunnel.static voidThe job queue is going away.(package private) static intpoolInFlightBudget(RouterContext ctx, TunnelPool pool) Per-pool in-flight budget for a pool in its current state, or -1 when the pool gate is skipped.private TunnelPoolManagerThe pool manager, reached through this job's own pool rather thanRouterContext.tunnelManager().(package private) static List<TestJob.PendingTest> prioritizeBatch(List<TestJob.PendingTest> scan, long now) Reorder one scan so the candidates that are about to expire go first: last-chance admitted tests sorted by soonest expiration, then everything else in its original buffer order.(package private) static List<TestJob.PendingTest> prioritizeBatch(List<TestJob.PendingTest> scan, long now, RouterContext ctx) Reorder one scan: last-chance candidates first (soonest expiry), then the rest by peer tier.private voidProbe the far endpoint of a tunnel whose test round just failed, and record whether it was still reachable.(package private) static voidpruneReplyPartnerTally(Map<Hash, long[]> tally) Keep the tally bounded.(package private) static longpumpDelayMs(long maxLag) Pump requeue delay for the current job-queue lag: brisk while quiet, slower when busy, slowest when lagging, so draining a test backlog never competes with router-critical jobs.(package private) static voidrebufferAll(TestJob.BatchState state, RouterContext ctx, List<TestJob.PendingTest> pending) Return candidates to the buffer head, preserving their order, after a gate denial or a lost claim race so the next pump run retries them instead of waiting for a sweep re-offer.(package private) static <T> List<T> rebufferBounded(Deque<T> buf, List<T> pending, int bound) Insertpendingat the buffer head, preserving their order, and evict from the tail while the buffer is over its bound, returning what was evicted so the caller can drop their keys.(package private) static voidrebufferOne(TestJob.BatchState state, TestJob.PendingTest p) Put one candidate back at the buffer tail, used to rotate a candidate that is denied only for its own pool behind the rest of the scan so the drain is not blocked at the head by it.(package private) static voidrecordDirectionStat(StatManager stats, boolean inbound, boolean success) Record one per-direction test outcome event.(package private) static voidrecordTestRoundBytes(TunnelInfo outTunnel) Credit a completed test round's bytes to the tunnel that carried it, for the per-peer throughput profile only.private static voidRefresh the cached test-job configuration from properties at most once per CONFIG_REFRESH_MS, or immediately when the context changes.(package private) static StringrefusalReasonText(TestJob.InFlightRefusal refusal) Short log fragment naming which gate refused a round.(package private) static voidregisterBatchStats(StatManager stats, long[] periods) Register the batch-dispatch rate stats this class writes, soaddRateData()records real rates instead of reporting invalid names.(package private) static voidregisterDirectionStats(StatManager stats, long[] periods) Register the per-direction test outcome stats written bytestSuccessful(int)and by the test-failure handler.(package private) static voidreleasePoolTestSlot(TestJob.BatchState state, String poolId) Release one per-pool test-instance slot claimed byclaimPoolTestSlot(TestJob.BatchState, String), removing the counter at zero so the map stays bounded by the number of pools.(package private) voidreleaseRound(TestJob.RoundToken token) Release the in-flight permits a round reserved, once.(package private) static voidreleaseTokenPermits(TestJob.BatchState state, TestJob.RoundToken token) Release the in-flight permits a round reserved, once.(package private) static intremovalThreshold(int base, int remainingTunnels) Raise the removal bar when the pool is nearly empty so a burst of concurrent test failures cannot cascade the last tunnels out before replacements finish building.(package private) static doublereplyPartnerFailureRate(Map<Hash, long[]> tally, TunnelInfo replyPartner) Failure rate of a reply partner: failures divided by rounds it carried.replyPartnerMemory(boolean inbound) The per-context tally of reply partners, holding for each one the number of rounds it carried and the time it was last blamed.(package private) booleanReserve the in-flight permits a round needs before it goes on the wire, and arm the round as this instance's active one.(package private) TestJob.InFlightRefusalReserve in-flight permits and arm the round, reporting which gate refused it so the caller can log a specific reason.(package private) static TestJob.InFlightRefusalreserveInFlightWithReason(TestJob.BatchState state, int maxConcurrent, String poolId, int poolBudget, TestJob.RoundToken token) Take the global and per-pool in-flight permits for a round, atomically, reporting which gate refused it.voidrunJob()Run the tunnel test and report success or failure.(package private) static voidsaturatingDecrement(AtomicInteger gauge) Decrement a gauge without letting it fall below zero.(package private) static intscaleHardLimit(int base, long maxLag) Scale the TestJob hard limit down when the job queue is lagging, with the scale-down floored at half the base limit.private booleanscheduleRetest(boolean asap) Schedules another test of the same tunnels.(package private) booleanscheduleRetest(boolean asap, boolean deferWhileActive) Schedule a retest of this tunnel.private booleansendTest(I2NPMessage m, int testPeriod, TestJob.RoundToken token) Send the tunnel test message through the selected tunnels.static voidsetMaxTestDelay(int ms) Set the max test delay (called by Tuner).static voidsetMaxTestPeriod(int ms) Set the max test period (called by Tuner).static voidsetMinTestDelay(int ms) Set the min test delay (called by Tuner).static voidsetMinTestPeriod(int ms) Set the min test period (called by Tuner).(package private) static booleanshouldDeferActiveGoodTunnel(long lastTraffic, long now, TunnelTestStatus status, int inFlight, int maxConcurrent) Whether a retest of a GOOD tunnel carrying recent real traffic must wait for the traffic to stop.(package private) static booleanshouldDeferFailingRetest(int remainingTunnels, int failures) Whether an ASAP retest should be deferred because the pool is thin and the tunnel already has failures.(package private) static booleanshouldDeferForPartner(TunnelInfo partner, boolean partnerFallback) Whether a failed round should be deferred instead of charged to the tunnel under test, because the partner tunnel — the one the round was actually routed through — is itself suspect.(package private) static booleanshouldDropPending(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) Whether a buffered candidate should be discarded instead of tested: its pool died, it was never fully built, it is a ping tunnel or was pruned for early expiry, its test already started (registered while buffered), or it is no longer UNTESTED (a prior dispatch tested or traffic-proven it while it waited).(package private) static booleanshouldRejectAtOffer(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) Whether an offer must be refused outright: no pool owns the tunnel, it was never fully built, it is a ping tunnel, it is already expired (only the drain-time early-expiry window is left toshouldDropPending(RouterContext, PooledTunnelCreatorConfig, TunnelPool), since last-chance offers arrive inside it on purpose), its test already started, or it is no longer UNTESTED.static booleanStatic method to check if a TestJob should be created and scheduled.private TestJob.BatchStatestate()This job's per-context runtime state.private voidtestFailed(long timeToFail) Handles a test that timed out or failed.voidtestSuccessful(int ms) Called when the tunnel test completes successfully.(package private) static StringthinPoolRetestLine(int poolSize, int failures) The line for an ASAP retest deferred because the pool is too thin to afford losing another tunnel to a test right now.(package private) static inttierRank(TestJob.PendingTest p, RouterContext ctx) Tier rank of a candidate's first hop, best first:0fast,1high-capacity,2untiered.(package private) static booleantryReserveInFlight(TestJob.BatchState state, int maxConcurrent, String poolId, int poolBudget, TestJob.RoundToken token) Take the global and per-pool in-flight permits for a round, atomically.private voidupdateSuccessHistory(boolean success) Records one test result in the rolling window.private voidupdateTestDuration(long durationMs) Update the rolling average test duration and adapt the timeout multiplier.private booleanWhether a candidate partner is usable for this round, i.e.(package private) static booleanusablePartner(TunnelInfo candidate, TunnelInfo excluded) Whether a candidate partner may carry the confirming round.(package private) static booleanwithinYoungTunnelGrace(long creationTimeMs, long nowMs) Whether a failed round should be excused because the tunnel is too new to have been fairly judged.Methods inherited from class JobImpl
getContext, getJobId, getMadeReadyOn, getTiming, madeReady, requeue, toStringModifier and TypeMethodDescriptionfinal RouterContextThe router context that this job belongs to.longgetJobId()The unique job ID for this job.longThe timestamp when this job was last made ready for execution.The timing object for this job.voidmadeReady(long now) For JobQueue only, not for external use.protected voidrequeue(long delayMs) Requeue this job to run after the specified delay.toString()
Field Details
_log
Logger for tunnel test results._pool
The pool whose tunnels are tested._cfg
The pool's tunnel configuration._outTunnel
The outbound tunnel carrying the test._replyTunnel
The inbound tunnel expected to carry the reply._partnerFallback
private boolean _partnerFallbackTrue when the partner for the current round came from the exploratory pool because the paired pool had nothing to offer. A round routed through a borrowed partner can fail for reasons that have nothing to do with the tunnel under test, sotestFailed(long)defers instead of counting it. Reset at the start of every selection._partnerDeferrals
private int _partnerDeferralsConsecutive test rounds deferred because the partner tunnel was suspect. Bounded so a tunnel that genuinely fails tests is eventually marked even when its partner is also degraded. Otherwise cascading deferrals delay failure detection indefinitely._confirmPending
private boolean _confirmPendingTrue once this failure has already been re-tested against a different partner without charging the tunnel. Set on the first ambiguous failure so the following round is a confirmation. Cleared on any success and once the failure is finally charged, so each charge needs a fresh confirmation._excludedPartner
The partner whose failure triggered the current confirmation. The confirming round must not reuse it, or the retest proves nothing._encryptTag
Session tag for the ElGamal test message._ratchetEncryptTag
Session tag for the ratchet test message.__id
Counter handing out test ids._testId
private int _testIdThis test's id, matched against the reply._testPeriod
private int _testPeriodTest period for the current round, computed once at send time so the reply is judged against the same window used to set the expiration.MAX_DEFERRED
private static final int MAX_DEFERREDNumber of times a test can be deferred for lack of a partner tunnel before the paired pool is kicked to rebuild. Prevents test deadlock where both inbound and outbound pools are degraded and neither can test. Reaching the cap does NOT fail the tunnel: a pool-wide shortage is not evidence against this tunnel, and on a server pool that would drop a healthy tunnel out of the published LeaseSet.- See Also:
_deferredCount
private int _deferredCountTimes this test was deferred by the concurrency cap.MAX_PAIRED_POOL_KICKS
private static final int MAX_PAIRED_POOL_KICKSLifetime budget of paired-pool kicks issued by the deferral escape inrunJob(). A partner shortage is not evidence against this tunnel, so it is never failed for one — but the kick must still be bounded, or a paired pool that cannot recover turns into an unbounded rebuild loop against a tunnel that has no say in the matter.- See Also:
MAX_PARTNER_DEFERRALS
static final int MAX_PARTNER_DEFERRALSMaximum consecutive test rounds that can be deferred because the partner tunnel was suspect before the failure is counted regardless. A tunnel that genuinely fails tests must eventually be marked, even if its partner is also degraded.- See Also:
_pairedPoolKicks
private int _pairedPoolKicksTimes the paired pool kicked this test.MAX_POST_KICK_DEFERRALS
private static final int MAX_POST_KICK_DEFERRALSMaximum number of deferrals allowed after the paired-pool kick budget is exhausted before the tunnel is failed. A tunnel that cannot be tested because its paired pool is permanently gone must not sit in the pool forever: it inflatesuntestedCount, which blocks new builds via theaddTunnel()cap. This bound gives the paired pool time to recover while still guaranteeing termination.- See Also:
_postKickDeferrals
private int _postKickDeferralsDeferrals since a pool kick.MAX_REPLY_PARTNER_MEMORY
private static final int MAX_REPLY_PARTNER_MEMORYUpper bound on remembered reply partners, so a router that churns through many destinations cannot grow this without limit. Reaching the cap evicts the oldest entries.- See Also:
MAX_REPLY_PARTNER_DEFERRALS
static final int MAX_REPLY_PARTNER_DEFERRALSConsecutive rounds that may be deferred because the same reply partner was already blamed for a failed round, before the tunnel is charged regardless. Separate fromMAX_PARTNER_DEFERRALS, which covers a partner that is known bad: this one covers a partner that merely looks implicated, and it is deliberately smaller, because the evidence is weaker. Reaching the cap means the reply path is not going to recover by trying another round, and continuing to defer would let a genuinely dead tunnel sit in the pool indefinitely.- See Also:
TEST_ROUND_BYTES
static final int TEST_ROUND_BYTESBytes credited to a tunnel for one completed test round. Nominal: it feeds the throughput profile, not the traffic proof.- See Also:
YOUNG_TUNNEL_GRACE_MS
static final long YOUNG_TUNNEL_GRACE_MSGrace period before a tunnel's first test failure can count against it.A tunnel that is condemned before it has had a realistic chance to settle and carry traffic can never prove itself, and a pool in that state cannot reach steady state. Measured on a live router: outbound pools were condemning a tunnel every ~1.8 min against a 10-12 min configured lifetime, because the fastest retest interval and a four-strike threshold condemn a failing tunnel in ~90s. Nothing distinguished "this tunnel is dead" from "this tunnel is ninety seconds old".
Must exceed the fastest path to condemnation. Four strikes at the 30s minimum retest interval is 120s — or a young tunnel can still reach the threshold inside its own grace and the loop is not broken. Three minutes leaves margin over that 120s while staying well inside a 10-12 minute lifetime, so a genuinely dead tunnel is delayed by at most this much: the grace defers the first charge, it does not excuse the tunnel.
- See Also:
_replyPartnerDeferrals
private int _replyPartnerDeferralsDeferrals waiting for a reply partner.IMPLICATED_REPLY_PARTNER_RATE
static final double IMPLICATED_REPLY_PARTNER_RATEFailure rate at or above which a reply partner counts as implicated in the round that just failed. A rate rather than a repeat flag: a flag tracks how often a partner is re-picked, which measures pool churn instead of partner health.- See Also:
_inboundReplyPartnerTally
Reply partner router hash to the time it was last blamed for a failed round, per direction. Split by direction because the diagnosis is asymmetric: a reply leg that repeatedly fails for inbound tests is a different finding from one that does so for outbound._outboundReplyPartnerTally
_cfgCtx
Context the cached config values were read from._cfgRefreshed
private static volatile long _cfgRefreshedWhen the cached config values were last read._cachedMaxConcurrent
private static volatile int _cachedMaxConcurrentCached limit on concurrent tests._cachedMinTestPeriod
private static volatile int _cachedMinTestPeriodCached shortest interval between tests._cachedMaxTestPeriod
private static volatile int _cachedMaxTestPeriodCached longest interval between tests._cachedMinTestDelay
private static volatile int _cachedMinTestDelayCached shortest delay before a test times out._cachedMaxTestDelay
private static volatile int _cachedMaxTestDelayCached longest delay before a test times out._cachedPoolCoverageThreshold
private static volatile double _cachedPoolCoverageThresholdCached pool coverage needed to skip testing._cachedMaxExploratoryPerPool
private static volatile int _cachedMaxExploratoryPerPoolCached test limit for exploratory pools._cachedMaxClientPerPool
private static volatile int _cachedMaxClientPerPoolCached test limit for client pools._cachedBaseMaxQueued
private static volatile int _cachedBaseMaxQueuedCached base limit on queued tests._cachedHardLimit
private static volatile int _cachedHardLimitCached hard limit on queued tests.CONFIG_REFRESH_MS
private static final long CONFIG_REFRESH_MSHow often the cached config values are refreshed.- See Also:
_tunedMinTestPeriod
private static volatile int _tunedMinTestPeriodTuned test-job params, -1 = use router config. Set by the Tuner when autotuning the corresponding testJob params._tunedMaxTestPeriod
private static volatile int _tunedMaxTestPeriodTuner-set longest test interval, -1 when untuned._tunedMinTestDelay
private static volatile int _tunedMinTestDelayTuner-set shortest test timeout, -1 when untuned._tunedMaxTestDelay
private static volatile int _tunedMaxTestDelayTuner-set longest test timeout, -1 when untuned.MIN_RETEST_GAP_MS
private static final long MIN_RETEST_GAP_MSMinimum gap between two tests of the same already-tested tunnel.The retest delay is derived from the observed success rate and can bottom out near
getMinTestDelay(RouterContext), so a pool full of healthy, fast-testing tunnels can re-enter the queue faster than tests complete. That fills the job queue with test work and starves the builds and dispatches that actually move a pool. A floor bounds the retest rate.Applies only to retests. A tunnel's first test is never delayed, so pool population is unaffected.
An expedited test bypasses the floor. Expedited is set when a pool is critical, in deficit, or has zero active tunnels, and those are exactly the states where a long gap would leave a pool empty — the pre-emptive recovery path depends on those tests being prompt.
- See Also:
SUCCESS_HISTORY_SIZE
private static final int SUCCESS_HISTORY_SIZERecent test results kept for the success rate.- See Also:
MAX_LAG_FOR_SCHEDULE
private static final int MAX_LAG_FOR_SCHEDULEJob queue lag in ms above which testing is deferred.- See Also:
MAX_SERVER_POOL_TEST_FAILURES
private static final int MAX_SERVER_POOL_TEST_FAILURESHard ceiling on consecutive test failures for server pool tunnels. Without this, dead server pool tunnels keep their slot indefinitely because incrementTestFailures() holds them for the LS republish cycle. At 10+ failures the tunnel is clearly dead — mark it conclusively.- See Also:
DEFAULT_SUCCESS_RATE
private static final float DEFAULT_SUCCESS_RATESuccess rate assumed for a tunnel with no history.- See Also:
RECENT_TRAFFIC_MS
private static final long RECENT_TRAFFIC_MSHow recently data must have flowed through a tunnel for us to trust it over the test result. If data was seen within this window, the test failure is treated as a false negative (reply-path issue).- See Also:
TRAFFIC_PROOF_MS
static final long TRAFFIC_PROOF_MSHow recently a tunnel must have carried real (non-test) traffic for the pool to clear its FAILING flag and mark it GOOD. Real traffic is proof the tunnel works — a tunnel receiving or sending data has not failed. Only FAILING (not FAILED) tunnels are cleared, so a genuinely dead tunnel still accumulates failures during quiet periods and is removed.- See Also:
TRAFFIC_DEFER_MS
static final long TRAFFIC_DEFER_MSHow long after the last real traffic a tunnel test is held back. Tests are deferred while the tunnel is carrying recent traffic: the traffic proves it works, and testing now would waste a job-queue slot and risk a false negative under load. The defer is load-aware, not absolute — during an ebb in test traffic the retest runs anyway (seeshouldDeferActiveGoodTunnel(long, long, TunnelTestStatus, int, int)). This is the recency window that arms the defer; the wait itself is bounded byACTIVE_GOOD_EBB_RETRY_MS.- See Also:
EBB_RETRY_FLOOR_MS
static final long EBB_RETRY_FLOOR_MSShortest wait before re-checking an active-GOOD deferral. Below this the recheck is not worth a queue slot: it would fire almost immediately and re-evaluate the same contention it just saw.- See Also:
ACTIVE_GOOD_EBB_RETRY_MS
static final long ACTIVE_GOOD_EBB_RETRY_MSLongest a retest may be pushed out while its tunnel is busy. Waiting the fullTRAFFIC_DEFER_MSwindow let the tunnel expire out from under the retest chain (the expiry pre-check compares against delay + 3 test periods, so a long delay makesscheduleRetest(boolean)give up and drop the test entirely); a bounded retry instead brings the job back to re-check contention while the tunnel is still alive.- See Also:
ACTIVE_GOOD_EBB_DIVISOR
static final int ACTIVE_GOOD_EBB_DIVISORIn-flight cap divisor defining an "ebb" for active-GOOD retests: when fewer thanmaxConcurrent / ACTIVE_GOOD_EBB_DIVISORtests are on the wire there is spare capacity to verify a busy tunnel. With the default cap of 64 the ebb is below 16 in flight, leaving the rest of the cap for first-test dispatch so untested tunnels keep priority.- See Also:
DEFAULT_QUEUED_LIMIT
private static final int DEFAULT_QUEUED_LIMITMaximum number of TestJob instances that should be queued before deferring new ones. Prevents job queue saturation from too many waiting tunnel tests. Tunable viai2p.tunnel.testJob.maxQueued(default: 96 fast / 64 slow). Dynamically overridden by the Tuner at runtime; when it still holds the built-in default, the configured property value is applied on first use.maxQueuedTests
public static volatile int maxQueuedTestsThe queue ceiling actually in force. Starts atDEFAULT_QUEUED_LIMITand is replaced either by the Tuner or by thei2p.tunnel.testJob.maxQueuedproperty, so read it rather than assuming either.BATCH_STATES
OneTestJob.BatchStateper router context. Identity-keyed, since RouterContext does not override equals; production holds exactly one entry and tests hold one per mock context.DIRECTION_EVENTS
The per-direction event names registered byregisterDirectionStats(StatManager, long[]), excluding the pass/fail pair whichdirectionStat(boolean, boolean)spells itself. Held in registration order.Declared in one place so a test can assert the registered set. A stat that is registered but never emitted sits at zero forever and reads as "this never happens" rather than "this is broken", which is how two dead per-direction counters survived a review here.
_valid
private boolean _validFlag to indicate if this job is valid and should be queued._claims
The instance slots this job owns: the total counter, the per-pool claim (with the pool id captured when it was taken), and the running-test registration. Every release goes through here so a claim is dropped at most once and only by the instance that took it, even when the pool id would differ today because pool settings changed mid-flight._round
The round currently in flight on this instance, or the most recently completed one. Reply and timeout callbacks capture the token they were created with and only complete it while it is still the active round, so a delayed callback from an earlier retest can never complete, release, or clear state belonging to a newer round._pendingMessage
The last pending message registered for this instance's round, kept so a reset or drop can unregister it even when the round was registered but never armed (permit refusal). Re-registering overwrites it. The registry's unregister is idempotent, so a stale reference is harmless._roundGen
Monotonic generation for round tokens; diagnostics only._terminated
private volatile boolean _terminatedSet when a restart, shutdown, or queue death cancelled this instance while it was still queued. A cancelled instance must not start a round or requeue itself when the job queue eventually runs it.MAX_BUFFERED_FIRST_TESTS
static final int MAX_BUFFERED_FIRST_TESTSBatched first-test dispatch. First tests, i.e. fresh builds, stranded sweeps and last-chance offers, flow through a bounded buffer drained by a single PumpJob instead of one queue entry per tunnel. The pump claims candidates under the in-flight gates and runs their TestJob directly, so draining a large UNTESTED backlog costs one ready-queue slot per batch - paced by queue health - instead of flooding the queue and having the queued cap drop (and lose) the excess.This is the maximum first-test candidates held in the batch buffer. When full the oldest is evicted, which simply returns that tunnel to the unbuffered state: recovery depends on the pool sweep re-offering it, so the buffer is best-effort, not lossless. An eviction delays a test rather than discarding it, but only for as long as the sweep runs again.
- See Also:
PUMP_BATCH_SIZE
static final int PUMP_BATCH_SIZECandidates claimed from the buffer per pump run. Small batches keep a burst from hitting the network at once while still retiring up to this many tunnels per ready-queue slot.- See Also:
DRAIN_SCAN_LIMIT
static final int DRAIN_SCAN_LIMITCandidates popped from the buffer per drain. Larger than the dispatch batch so a candidate denied only for its own pool can rotate to the tail while the scan continues to the candidates behind it, instead of blocking the whole run at the head.- See Also:
MAX_STRANDED_OFFERS_PER_SWEEP
static final int MAX_STRANDED_OFFERS_PER_SWEEPStranded-UNTESTED candidates a pool sweep offers per pass — bounds one ensure cycle's contribution to the buffer.- See Also:
PUMP_DELAY_HEALTHY_MS
static final long PUMP_DELAY_HEALTHY_MSPump requeue delay while the job queue is quiet. @since 0.9.71+.- See Also:
PUMP_DELAY_BUSY_MS
static final long PUMP_DELAY_BUSY_MSPump requeue delay while the job queue is busy. @since 0.9.71+.- See Also:
PUMP_DELAY_LAGGED_MS
static final long PUMP_DELAY_LAGGED_MSPump requeue delay when the job queue is badly lagging. @since 0.9.71+.- See Also:
PUMP_BUSY_LAG_MS
static final long PUMP_BUSY_LAG_MSJob-queue lag above which the pump uses the busy delay (ms). @since 0.9.71+.- See Also:
PUMP_LAGGED_LAG_MS
static final long PUMP_LAGGED_LAG_MSJob-queue lag above which the pump uses the lagged delay (ms). @since 0.9.71+.- See Also:
DENIAL_LOG_INTERVAL_MS
static final long DENIAL_LOG_INTERVAL_MSMinimum spacing between batch-denial INFO lines (ms). @since 0.9.71+.- See Also:
_successHistory
private volatile boolean[] _successHistoryRolling window of recent test results._successHistoryIndex
private volatile int _successHistoryIndexWrite index into the success history._successCount
private volatile int _successCountSuccesses held in the success history.
Constructor Details
TestJob
TestJob. Claims the instance slots itself, so a direct construction cannot run without them; returns an invalid instance (nothing claimed) when the memory bound or a concurrent claim refuses.- Parameters:
ctx- the router contextcfg- the tunnel configpool- the owning pool (may be null; read from cfg)
TestJob
TestJob(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool, TestJob.InstanceClaims claims) TestJob for a caller that already holds the instance slots, such as the batch pump, which must claim before it constructs so a refused candidate can go back to the buffer instead of being dropped.- Parameters:
ctx- the router contextcfg- the tunnel configpool- the owning pool (may be null; read from cfg)claims- the slots already claimed for this instance; null refuses it
Method Details
probeFarEndpoint
Probe the far endpoint of a tunnel whose test round just failed, and record whether it was still reachable.A round times out, the tunnel is charged, and the pool loses it. The charge assumes the far endpoint was gone, but nothing records whether that was true. Two very different faults produce the same verdict: the endpoint having died (the tunnel is genuinely broken, so the charge is correct), and the endpoint being alive with the reply simply never arriving (the tunnel is being condemned for something it does not own). The first is a data-path fault. The second is a fault in the test's own pass criterion, and the remedy is opposite. One counter cannot tell them apart, so this splits the population.
Observational only: it never gates selection, never alters the charge, and never changes timing, so it cannot itself change behaviour. It runs only on rounds that have already failed and are already being charged.
Two signals are read rather than one, because
CommSystemFacade.isEstablished(Hash)andCommSystemFacade.wasUnreachable(Hash)both default tofalsein the interface and only their implementations answer. A lookup that misses therefore reports "reachable" silently; recording both makes that failure mode visible in the data instead of hiding it.Caveats, so the result is not over-read: "established" means reachable now, not that the peer was reachable when the message was sent and chose not to answer; and it does not separate a send failure from a lost reply, since both look the same from here. Only failed rounds are probed, so the counters describe the failure population only.
- Parameters:
cfg- the tunnel whose round failed
farEndVerdict
Classify a failed round's far endpoint into exactly one bucket.Unreachable outranks established, because a peer that is both currently reachable and recently unreachable is a peer that was gone and has since returned — which is the case where the tunnel's condemnation was correct at the time. "Unknown" is kept separate rather than folded into either side, so a missed lookup cannot inflate the established share that the whole diagnosis turns on.
- Parameters:
unreachable- the far endpoint was recently unreachableestablished- the far endpoint has a live transport session- Returns:
- the event name
deferIfTooYoungToJudge
private boolean deferIfTooYoungToJudge()Excuse a failed round because the tunnel is still inside its grace period: it has not yet had a realistic chance to settle, join the LeaseSet, or carry the traffic that would prove it, so the round is observed but not charged. Without this, the fastest retest interval plus a four-strike threshold condemns a new tunnel in about 90s — a sixth of its lifetime — and the pool never settles.Split out of
testFailed(long), which is already well past the complexity budget, so this does not add to it.- Returns:
- true when the round was deferred
withinYoungTunnelGrace
static boolean withinYoungTunnelGrace(long creationTimeMs, long nowMs) Whether a failed round should be excused because the tunnel is too new to have been fairly judged. Static and clock-injected so the boundary is testable, and so "never built" is representable.- Parameters:
creationTimeMs- when the tunnel was built, 0 if unknownnowMs- current time in ms- Returns:
- true when the tunnel is still inside its grace period
poolManager
The pool manager, reached through this job's own pool rather thanRouterContext.tunnelManager(). The latter is the publicTunnelManagerFacadeinterface, and adding the ranked-partner methods there would break every out-of-tree implementor for the sake of two test-cycle call sites.- Returns:
- the pool manager, or null if this job has no pool
replyPartnerFailureRate
Failure rate of a reply partner: failures divided by rounds it carried. A rate is the only figure comparable between partners. A raw "blamed before" flag reads a stable pool as bad and a churning pool as good, because it tracks how often a partner is re-picked rather than how often it fails. That confound was visible in the field: outbound tests drew partners from the stable inbound pool and showed a 74% repeat share, while inbound tests drew from the churning outbound pool and showed 15%. Neither number said anything about partner health.- Parameters:
tally- partner to failure/use counts, may be nullreplyPartner- the tunnel the reply came back through, may be null- Returns:
- failures per round carried, 0 when nothing is recorded
noteReplyPartnerRound
Record one round against its reply partner: the use always, the failure only when the round failed. Bounded, because this rolls over every destination the router serves.- Parameters:
tally- partner to failure/use countsreplyPartner- the tunnel the reply came back through, may be nullfailed- true when the round failed
pruneReplyPartnerTally
Keep the tally bounded. Drops an arbitrary entry once over the cap: the per-partner rate is a running diagnostic, so evicting the least interesting entry is preferable to the scan that an oldest-first eviction needs on every round.- Parameters:
tally- partner to failure/use counts
replyPartnerMemory
The per-context tally of reply partners, holding for each one the number of rounds it carried and the time it was last blamed.- Parameters:
inbound- true for the inbound tally, false for the outbound one- Returns:
- the live tally map, which the caller must synchronize on
attributeRoundToReplyPartner
Tally this round's result against the peer that relayed the reply.- Parameters:
inbound- true for an inbound testreplyPartner- the peer that relayed the replyfailed- true if the round failed
setMinTestPeriod
public static void setMinTestPeriod(int ms) Set the min test period (called by Tuner).- Parameters:
ms- the min test period in ms
setMaxTestPeriod
public static void setMaxTestPeriod(int ms) Set the max test period (called by Tuner).- Parameters:
ms- the max test period in ms
setMinTestDelay
public static void setMinTestDelay(int ms) Set the min test delay (called by Tuner).- Parameters:
ms- the min test delay in ms
setMaxTestDelay
public static void setMaxTestDelay(int ms) Set the max test delay (called by Tuner).- Parameters:
ms- the max test delay in ms
getMinTestPeriod
The min test period in effect, tuned value if set else config.- Parameters:
ctx- the router context- Returns:
- the min test period in ms
getMaxTestPeriod
The max test period in effect, tuned value if set else config.- Parameters:
ctx- the router context- Returns:
- the max test period in ms
getMinTestDelay
The min test delay in effect, tuned value if set else config.- Parameters:
ctx- the router context- Returns:
- the min test delay in ms
getMaxTestDelay
The max test delay in effect, tuned value if set else config.- Parameters:
ctx- the router context- Returns:
- the max test delay in ms
refreshTestJobConfig
Refresh the cached test-job configuration from properties at most once per CONFIG_REFRESH_MS, or immediately when the context changes. Benign race: duplicate refreshes are idempotent writes.- Parameters:
ctx- the router context supplying the clock and properties
getMaxConcurrentTests
Maximum number of tunnel tests that can run concurrently. Prevents overwhelming the router with too many simultaneous tunnel tests. This value can be adjusted based on system capacity. Tunable viai2p.tunnel.testJob.maxConcurrent(default: 64 fast / 32 slow).- Parameters:
ctx- the router context whose properties are cached- Returns:
- the max concurrent tests
getMinTestPeriod
private int getMinTestPeriod()The minimum test period from config or default (3s). Tunable viai2p.tunnel.testJob.minTestPeriod(default: 3000).- Returns:
- the min test period
getMaxTestPeriod
private int getMaxTestPeriod()The maximum test period from config or default (15s). Tunable viai2p.tunnel.testJob.maxTestPeriod(default: 15000).- Returns:
- the max test period
getPoolCoverageThreshold
Returns the pool coverage needed to skip testing.- Parameters:
ctx- the router context- Returns:
- the coverage fraction
getMaxExploratoryPerPool
Returns the test limit for exploratory pools.- Parameters:
ctx- the router context- Returns:
- the maximum concurrent tests
getMaxClientPerPool
Returns the test limit for client pools.- Parameters:
ctx- the router context- Returns:
- the maximum concurrent tests
getClientPoolTestBudget
Per-pool test budget for client pools, bounded by three knobs: the Tuner budget (dynamic), the pool-coverage cap (at most coverageThreshold of a pool's active tunnels under test at once), and the absolute per-pool cap. With defaults the Tuner budget (effectively unlimited) is superseded by the coverage cap.Takes a pre-fetched active count: callers that already needed it (criticality checks, zero-active bypasses) avoid a second O(tunnels) scan inside this method.
- Parameters:
ctx- the router contextpool- the client pool, never nullactiveCount- the pool's active tunnel count, fetched by the caller- Returns:
- maximum concurrent test claims for this pool
applyConfiguredQueuedLimit
RefreshmaxQueuedTestsfrom the configured property when the Tuner has not overridden the static dynamically.- Parameters:
ctx- the router context holding the configured limit
getBaseMaxQueuedTests
Base max queued tests value, read from PROP or static default.- Parameters:
ctx- the router context whose configuration supplies the limit- Returns:
- configured base max queued tests
getHardLimit
Hard limit for total TestJob instances (queued + active). Above this threshold, no new tests are scheduled until count decreases. Prevents ever-increasing backlogs that could cause job lag.Scales down under job queue pressure to prevent TestJobs from starving critical router jobs, but never below half the base limit — see
scaleHardLimit(int, long).Tunable via
i2p.tunnel.testJob.hardLimit(default: 512 fast / 384 slow).- Parameters:
ctx- the router context- Returns:
- the pressure-scaled hard limit
scaleHardLimit
static int scaleHardLimit(int base, long maxLag) Scale the TestJob hard limit down when the job queue is lagging, with the scale-down floored at half the base limit. Quartering under sustained lag starved the tunnel test queue — UNTESTED tunnels piled up while the shrunken limit blocked scheduling, and the pool never recovered enough GOOD tunnels to publish — so extreme lag now gets the same half-limit response as moderate lag.- Parameters:
base- configured hard limit; values <= 0 pass through untouchedmaxLag- current job queue max lag (ms)- Returns:
- base unchanged when lag is at or below 5s (or base is non-positive), otherwise at least
base / 2
batchState
The runtime state for a router context, created on first use. Every buffer, gauge, permit, and pump flag lives in the returned state, so two contexts can never observe each other's dispatch state.- Parameters:
ctx- the router context, never null- Returns:
- that context's batch state, never null
registerBatchStats
Register the batch-dispatch rate stats this class writes, soaddRateData()records real rates instead of reporting invalid names. Called once per router context fromTunnelPoolManager's constructor.Units: every batch rate is an event count (value 1 per event), so a rate of N means N events in the period.
testBufferOfferedandtestBufferRejectedcount candidates,testBufferDroppedcounts candidates evicted or discarded as stale,testBatchDeniedcounts drain passes refused by a gate, andtestBatchDispatchedcounts rounds actually sent.- Parameters:
stats- the context's stat manager, never nullperiods- the rate periods to record on, never null
directionStat
Name of a per-direction test outcome stat.A successful test emits no log line at all, so counters are the only way to see test health by direction. Without them "no outbound tests passed" and "passing tests are not logged" are indistinguishable in a log review. Registration and emission both route through this method, so a stat can never be written without being declared:
addRateData()silently drops names it has not seen.- Parameters:
inbound- true for an inbound tunnel, false for outboundsuccess- true for a test that passed, false for one that failed- Returns:
- stat name, e.g.
tunnel.testOutboundFailed
directionStat
Name of a per-direction stat for an outcome that is not simply pass/fail — a confirmation result, or condemnation by the pool. Shares thetunnel.test{Inbound,Outbound}<event>scheme so every per-direction test stat is spelled from one place.- Parameters:
inbound- true for an inbound tunnel, false for outboundevent- the outcome, e.g. ConfirmPassed, ConfirmFailed, Condemned- Returns:
- stat name, e.g.
tunnel.testOutboundCondemned
registerDirectionStats
Register the per-direction test outcome stats written bytestSuccessful(int)and by the test-failure handler.The failure counters are written before the inbound traffic exemption is evaluated, so they include failures the exemption later suppresses; the size of that gap is how much the exemption is masking.
- Parameters:
stats- manager to declare the stats againstperiods- rate periods to record them over
recordDirectionStat
Record one per-direction test outcome event.All four stats are declared
(count), so every emission must pass a unit value. The emitters already keep the round-trip time intunnel.testSuccessTime/tunnel.testFailedTime; handing that duration to a count stat makesAveragea sum of milliseconds and hides the pass/fail rate entirely. Centralising the write here is what makes that mistake unrepresentable at the call sites.- Parameters:
stats- manager to record againstinbound- true for an inbound tunnel, false for outboundsuccess- true for a test that passed, false for one that failed
beginConfirmationRound
Arm a confirming round: mark the failure as unconfirmed, remember the partner that must not be reused, and schedule the round ASAP. The tunnel is not charged for this failure. Only a confirming round that also fails charges it.- Parameters:
partner- the partner whose failure triggered this, may be null- Returns:
- true if the confirming round was scheduled, false if it could not be and the caller should charge the tunnel as before
currentPartner
The partner the current round was actually routed through.- Returns:
- the reply tunnel for an inbound test, the outbound for an outbound test; null when none was selected
usablePartner
Whether a candidate partner is usable for this round, i.e. present and not the one a confirming round is meant to avoid. A confirming round that reused the failed partner would prove nothing, so it must prefer a different one. If none exists the round still proceeds with it, which simply leaves the tunnel to be charged as it was before.- Parameters:
t- candidate partner, may be null- Returns:
- true when the candidate may be used
usablePartner
Whether a candidate partner may carry the confirming round. Identity based, not status based: the point of the confirming round is that the reply came back through a *different* tunnel, and reusing the one that just failed would prove nothing. A null candidate is never usable, and with nothing excluded every live candidate is.- Parameters:
candidate- the partner the round would use, may be nullexcluded- the partner that must not be reused, may be null- Returns:
- true when the candidate may be used
recordTestRoundBytes
Credit a completed test round's bytes to the tunnel that carried it, for the per-peer throughput profile only.realTrafficis false and must stay false. A test round is the tunnel under test carrying its own probe, so crediting it as real traffic would stampTunnelCreatorConfig.getLastTransferred()— and for an outbound test the outgoing tunnel is the tunnel under test. A tunnel that had never carried a single production byte would then look recently proven and could grant itself the data-verified test trust, which is decided on exactly that clock. The exemption budget is unaffected, since onlyTunnelCreatorConfig.recordRealTraffic()refills it.Static and taking the tunnel as an argument, so the call site itself is testable: asserting on
TunnelCreatorConfigdirectly would pass even if this passed the wrong flag.- Parameters:
outTunnel- the tunnel the round was sent through, may be null
isRecentDataTransfer
static boolean isRecentDataTransfer(long lastTransferMs, long nowMs) Whether a tunnel's last real traffic is recent enough to trust over a test failure. Static and clock-injected so the staleness bound is testable without a running router, and so the direction of the comparison is pinned: a tunnel that never carried production traffic reportslastTransfer == 0, which is maximally stale and must therefore never qualify.- Parameters:
lastTransferMs- when the tunnel last carried production traffic, 0 if nevernowMs- current time in ms- Returns:
- true when the traffic is recent enough to shield a test failure
beginBatchReset
Start a per-context reset of the batch-dispatch subsystem: a router restart, as called fromTunnelPoolManager.restart()before the pools are torn down. Offers and dispatch are refused while the state is quiesced, every live instance is cancelled (its pending message, its round, and its slots), and the first-test buffer, its key set, the pump flag, and the gauges are cleared — a buffered candidate or an inflated gauge would otherwise outlive the tunnels it describes.Must be paired with
endBatchReset(RouterContext); until then the context accepts no new test work.- Parameters:
ctx- the router context
endBatchReset
Finish a reset started bybeginBatchReset(RouterContext): resume offers and pump dispatch. The buffer was emptied by the reset, so there is nothing queued to revive — the first new offer restarts the pump.- Parameters:
ctx- the router context
onJobQueueShutdown
The job queue is going away. Its shutdown clears queued jobs without runningdropped()or their timeout callbacks, so anything the queue held would otherwise leak its slots, keep its pending selector, and leave the pump flag set forever. Reset the context's batch state so those leaks cannot outlive the queue; there is no matchingendBatchResetbecause a dead queue never accepts work again (offers are additionally refused by theisAlive()check).- Parameters:
ctx- the router context
getTunnelKey
Generate a unique key for a tunnel to track running tests. Uses combination of receive and send tunnel IDs.- Parameters:
cfg- the tunnel configuration- Returns:
- unique key for the tunnel, or null if unavailable
getPoolId
Generate a stable, collision-free identifier for a tunnel pool, used as the key for the per-pool instance and in-flight gauges. Distinct pools must never share a key or their budgets merge: client pools are keyed by the full destination hash plus direction (nicknames may be shared and an eight-character base32 prefix is not unique), while exploratory pools have exactly one inbound and one outbound instance per context.- Parameters:
pool- the tunnel pool- Returns:
- a stable identifier for the pool, never null
getCurrentTestJobCount
public static int getCurrentTestJobCount()Current number of queued + active test jobs across all router contexts, for capacity planning byBuildExecutor.- Returns:
- the current test job count
getMaxTestJobs
public static int getMaxTestJobs()Maximum number of queued test jobs allowed before deferring.- Returns:
- the max test jobs
state
This job's per-context runtime state.- Returns:
- the BatchState shared by all TestJobs of this context
shouldSchedule
Static method to check if a TestJob should be created and scheduled. This prevents creating invalid job objects that would have timing issues. Note: the standard first-test entry points (build complete, stranded sweep, last chance) now route throughofferFirstTest(RouterContext, PooledTunnelCreatorConfig, TunnelPool), which applies the in-flight gates at drain time; this gate remains for direct scheduling.Advisory only: it decides whether an instance is worth creating but claims nothing, so a rejection needs no rollback. The atomic slot claim happens once, in
claimBatchSlot(RouterContext, PooledTunnelCreatorConfig, TunnelPool), when the instance is actually constructed — a read-only check can only be stale, never leak.- Parameters:
ctx- the router contextcfg- the tunnel config- Returns:
- true if the job should be created and scheduled, false otherwise
hasPoolTestHeadroom
static boolean hasPoolTestHeadroom(TestJob.BatchState state, RouterContext ctx, TunnelPool pool, int activeCount) Whether the per-pool instance budget still has room for one more test inpool. Read-only twin ofclaimPoolTestSlot(TestJob.BatchState, String): the caller may be stale by the time it claims, but it can never leak a claim by rejecting.- Parameters:
state- the context's batch state, never nullctx- the router context (budget configuration)pool- the pool to check, never nullactiveCount- pool.getActiveTunnelCount() already read by the caller- Returns:
- true when one more test fits under the pool's budget
canClaimInstance
static boolean canClaimInstance(int current, int hardLimit) Sole authority for allocating a test instance slot: a slot is available while the current count is strictly below the hard limit. The bound is the read itself, not a separate pre-check, so a caller can never observe "under the limit", act, and leave the count at limit + 1.- Parameters:
current- the current instance counthardLimit- the configured maximum instance count- Returns:
- true when one more instance fits
claimPoolTestSlot
Claim one per-pool test-instance slot forpoolIdin a router context's runtime state. The claim is done inside the map's compute() so it can never land on a counter that a concurrentreleasePoolTestSlot(TestJob.BatchState, String)just removed (which would silently lose the claim and weaken the pool budget).- Parameters:
state- the context's batch state, never nullpoolId- pool key, never null- Returns:
- the pool's claim count after this claim; exact when no other claim or release interleaves, otherwise a close reading used only for the soft budget check
releasePoolTestSlot
Release one per-pool test-instance slot claimed byclaimPoolTestSlot(TestJob.BatchState, String), removing the counter at zero so the map stays bounded by the number of pools. Decrement and removal happen in one computeIfPresent() step: a plain decrement-then-remove can unlink a counter a concurrent claim just incremented, losing the claim.- Parameters:
state- the context's batch state, never nullpoolId- pool key, never null
hasValidTunnelIds
Whether the tunnel config has non-zero tunnel IDs at hop 0, i.e. the tunnel is fully built and can carry a test message. Outbound tunnels only have a send tunnel ID at hop 0 (the gateway), inbound tunnels only a receive tunnel ID. Any failure reading the IDs (not yet built) is treated as not ready; the caller logs the debug detail.- Parameters:
cfg- the tunnel config, never null- Returns:
- true if the applicable hop-0 ID is non-zero
isPingTunnel
Whether the tunnel belongs to a ping pool, which is short-lived and does not need testing. Ping pools are named "I2Ping" or "Ping*[n]".- Parameters:
cfg- the tunnel config, never null- Returns:
- true if the pool nickname identifies a ping pool
isEarlyExpiry
Whether the tunnel expires so soon it is not worth testing — it has already been pruned for early expiry.A tunnel the pool explicitly admitted for a last-chance test is exempt until that admission's deadline: the sweep keeps stale UNTESTED tunnels with
TunnelPool.LAST_CHANCE_MIN_LIFE_MS(60s) of life precisely so they get one first test, while this gate would otherwise drop anything inside the 120s early-expiry window the moment the test reaches the front of the queue.- Parameters:
cfg- the tunnel config, never nullnow- current time in ms- Returns:
- true if the tunnel expires within the early-expiry window and no last-chance admission covers it
isZeroActivePool
Whether a client pool has zero active tunnels, i.e. it is collapsed and its test jobs must bypass the queue cap and per-pool budget. Exploratory pools are never critical — they have no LeaseSet to feed.- Parameters:
pool- the pool, may be null (then not critical)- Returns:
- true if a non-exploratory pool with no active tunnels
isPoolCritical
static boolean isPoolCritical(int activeCount, int target) Whether a pool is critically low on tunnels: no active tunnels at all, or at most 2 active against a larger target. Critical pools get test priority so replacement tunnels are validated before the pool drains. NOTE:BuildExecutor.calculatePairedBuilds(List, List)carries an identical inline predicate. Dedupe candidate if either side changes again.- Parameters:
activeCount- current active tunnel counttarget- configured total quantity- Returns:
- true if the pool is critical
isPoolDeficit
static boolean isPoolDeficit(int activeCount, int target) Whether a pool is running below its target active-tunnel count. Deficit pools get expedited test treatment (see the expedited request inshouldSchedule(RouterContext, PooledTunnelCreatorConfig)) so their UNTESTED backlog drains before the pool drops further behind — but unlikeisPoolCritical(int, int)they do not bypass queue caps or per-pool budgets, which stay reserved for collapsed pools.- Parameters:
activeCount- current active tunnel counttarget- configured total quantity- Returns:
- true when the pool has fewer active tunnels than its target
computeMaxTestJobs
static int computeMaxTestJobs(int maxQueuedTests, int activeRunners, int numPools) Test-job capacity derived from the queue state: at least 12 jobs (or 3 per pool), scaled up by active runners, but never above the hard queued limit.- Parameters:
maxQueuedTests- hard cap frommaxQueuedTestsactiveRunners- current job-queue runner countnumPools- number of tunnel pools- Returns:
- the capacity in test jobs
isTestQueueOverloaded
static boolean isTestQueueOverloaded(boolean critical, int readyCount, int activeRunners, long maxLag, long expeditedLagLimit, int currentTestJobs, int expeditedJobLimit) Whether the job queue is too backed up to accept another test job. Non-critical pools are deferred when the ready queue outruns the runners, the max lag exceeds the (possibly expedited) limit, or the current test job count meets the (possibly expedited) job limit. Critical pools always get through.- Parameters:
critical- whether the pool is critical (bypasses the gate)readyCount- jobs waiting in the queueactiveRunners- jobs being processedmaxLag- measured queue lag in msexpeditedLagLimit- lag threshold (doubled for expedited)currentTestJobs- running test job countexpeditedJobLimit- job threshold (1.5x for expedited)- Returns:
- true if scheduling should be deferred
offerFirstTest
Offer a tunnel's first test to the batched dispatch buffer instead of creating a queue entry per tunnel. All first-test producers (fresh builds, stranded-UNTESTED sweeps, last-chance offers) funnel through here: the buffer absorbs bursts, a singlePumpJobdrains it in small batches paced by queue health, and the in-flight gates decide what actually dispatches. Candidates are validated at admission byshouldRejectAtOffer(RouterContext, PooledTunnelCreatorConfig, TunnelPool), so a stale, built-but-rejected, or no longer owned config never evicts a valid one from the bounded buffer; duplicate offers of the same tunnel key are dropped as well. Gates are re-evaluated at drain time, so a denial here is only a later retry. Every call also (re)starts the pump when it is idle, so a pump lost to queue overload is revived by the next offer (the sweeps re-offer at least every minute).- Parameters:
ctx- the router contextcfg- the tunnel config (first test only; retests keep the existing queue path so delayed retests never sit in the buffer)pool- the pool owning the tunnel; may be null (read from cfg)- Returns:
- true if the candidate was newly buffered
claimBatch
Drain up tomaxremovable elements from the front ofbuf. An element is removed wheneligiblereturns true (dispatch it, or discard it as stale). The first element that is not eligible stays at the head and ends the claim, so a temporarily denied candidate pauses the batch behind it instead of being skipped and retried out of order, while permanently-stale entries never wedge the head (they test as eligible and are dropped by the caller).- Type Parameters:
T- the element type held in the buffer and returned in the claimed list- Parameters:
buf- the buffer to claim from, modified in placemax- maximum number of elements to claimeligible- removal predicate; false leaves the element in place- Returns:
- the claimed elements in buffer order; never null
claimBatch
Pop up tomaxelements from the front ofbufwith no eligibility gate. Callers that must not evaluate anything (expensive checks, lock-ordering rules) use this and triage the popped elements themselves, re-buffering the ones that cannot proceed.- Type Parameters:
T- the element type held in the buffer and returned in the claimed list- Parameters:
buf- the buffer to claim from, modified in placemax- maximum number of elements to claim- Returns:
- the claimed elements in buffer order; never null
offerBounded
Appendelemto the buffer, evicting from the front while at or aboveboundso the buffer can never grow without limit.- Type Parameters:
T- the element type of the buffer, the appended element and the evicted list- Parameters:
buf- the buffer to append toelem- the element to appendbound- maximum buffer size after the append; non-positive means unbounded- Returns:
- the evicted elements oldest first; empty when nothing was evicted
pumpDelayMs
static long pumpDelayMs(long maxLag) Pump requeue delay for the current job-queue lag: brisk while quiet, slower when busy, slowest when lagging, so draining a test backlog never competes with router-critical jobs.- Parameters:
maxLag- current job-queue max lag (ms)- Returns:
- requeue delay in ms
batchDenialReason
static String batchDenialReason(int inFlight, int maxConcurrent, int poolInFlight, int poolBudget, int total, int hardLimit) Why a batch candidate cannot dispatch yet, or null when it can. Side-effect free (claims happen inclaimBatchSlot(RouterContext, PooledTunnelCreatorConfig, TunnelPool)after the gate passes) and measured on in-flight pressure — tests actually on the wire — so delayed retests holding queue slots no longer starve first tests.- Parameters:
inFlight- dispatched tests awaiting completion (all pools)maxConcurrent- configured in-flight cappoolInFlight- dispatched tests for this poolpoolBudget- per-pool in-flight budget, or -1 to skip the pool gatetotal- current TestJob instances (memory bound)hardLimit- instance memory bound- Returns:
- denial reason, or null when the candidate is admitted
poolInFlightBudget
Per-pool in-flight budget for a pool in its current state, or -1 when the pool gate is skipped. -1 covers a null pool and a collapsed client pool (zero active tunnels): their UNTESTED tunnels must drain or the LeaseSet never republishes, mirroring the critical-pool bypass inshouldSchedule(RouterContext, PooledTunnelCreatorConfig). Exploratory pools use their fixed cap.- Parameters:
ctx- the router contextpool- the owning pool, or null- Returns:
- the budget, or -1 to skip the pool gate
batchDenialReason
Live gate evaluation for one buffered candidate: reads the in-flight gauges and configured budgets from the context's runtime state, then defers to the pure overload.- Parameters:
ctx- the router contextcfg- the tunnel configpool- the owning pool (may be null; read from cfg)- Returns:
- denial reason, or null when the candidate is admitted
shouldDropPending
Whether a buffered candidate should be discarded instead of tested: its pool died, it was never fully built, it is a ping tunnel or was pruned for early expiry, its test already started (registered while buffered), or it is no longer UNTESTED (a prior dispatch tested or traffic-proven it while it waited).- Parameters:
ctx- the router context (clock for the early-expiry check)cfg- the buffered tunnel configpool- the owning pool (may be null; read from cfg)- Returns:
- true when the candidate must be dropped rather than tested
isPoolMember
Whether a pool still owns a tunnel config, i.e. the config is one of the pool's live tunnels. Approximates the pool-ownership half ofTunnelPool.addTunnel(TunnelInfo), which drops a config it does not own; rejected builds therefore never enter the buffer, and a config trimmed while buffered is dropped before it can dispatch.- Parameters:
pool- the pool to check, never nullcfg- the tunnel config to look for, never null- Returns:
- true when the pool currently lists the config
shouldRejectAtOffer
static boolean shouldRejectAtOffer(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) Whether an offer must be refused outright: no pool owns the tunnel, it was never fully built, it is a ping tunnel, it is already expired (only the drain-time early-expiry window is left toshouldDropPending(RouterContext, PooledTunnelCreatorConfig, TunnelPool), since last-chance offers arrive inside it on purpose), its test already started, or it is no longer UNTESTED. Keeps invalid candidates out of the buffer so the pump never pays a rejected dispatch for them.- Parameters:
ctx- the router context (clock and running-test registry)cfg- the offered tunnel configpool- the owning pool (may be null; read from cfg)- Returns:
- true when the offer must be refused
firstReadingPending
Whether a tunnel has never produced a latency reading, i.e. no test has ever completed against it. True for every UNTESTED tunnel, and for tunnels promoted to GOOD byTunnelPool-side traffic proof — those are untested in every way that matters: they have no measurement, the console shows no latency, and no retest chain exists for them. Such tunnels are first-test candidates and must outrank retests of already-measured tunnels.- Parameters:
cfg- the tunnel to inspect, may be null- Returns:
- true when no latency reading exists yet
saturatingDecrement
Decrement a gauge without letting it fall below zero.beginBatchReset(RouterContext)normalizes the gauges while instances are still alive, so an instance releasing after the reset must not drag its counter negative. A negative instance count would admit more thangetHardLimit(RouterContext)jobs.- Parameters:
gauge- the counter to decrement, never null
claimBatchSlot
static TestJob.InstanceClaims claimBatchSlot(RouterContext ctx, PooledTunnelCreatorConfig cfg, TunnelPool pool) Reserve the instance slots a batched TestJob needs: the total counter (memory bound) and the per-pool claim. The pool slot is claimed whenever the instance has a pool — exactly the condition under which the constructor's invalidation paths andcleanupTunnelTracking()release it — so claims and releases can never drift apart as pool state (active count, exploratory) changes between claim and release. Called only afterbatchDenialReason(int, int, int, int, int, int)passed, so a lost CAS race is a rare skip the caller re-buffers.- Parameters:
ctx- the router contextcfg- the tunnel configpool- the owning pool (may be null; read from cfg)- Returns:
- the acquired claims, or null when the memory bound is reached or a concurrent claim won the CAS race
rebufferBounded
Insertpendingat the buffer head, preserving their order, and evict from the tail while the buffer is over its bound, returning what was evicted so the caller can drop their keys. One call does both so a re-buffered batch can never push the buffer past its limit, and the batch just returned to the head can never be the batch immediately evicted from the tail (insert-then-trim is ordered; trim-then-insert is not).- Type Parameters:
T- the element type of the buffer, the pending batch and the evicted list- Parameters:
buf- the buffer to insert into, modified in placepending- candidates in original buffer order, head-most firstbound- maximum buffer size after the insert; non-positive means unbounded- Returns:
- the candidates evicted from the tail; never null
rebufferAll
static void rebufferAll(TestJob.BatchState state, RouterContext ctx, List<TestJob.PendingTest> pending) Return candidates to the buffer head, preserving their order, after a gate denial or a lost claim race so the next pump run retries them instead of waiting for a sweep re-offer. An entry whose tunnel key is already buffered again (a concurrent re-offer while it was popped) is dropped rather than duplicated, and any tail eviction it causes has its keys removed so bufferedKeys stays an exact mirror of the buffer.- Parameters:
state- the context's batch statectx- the router context (for the eviction stat)pending- candidates in original buffer order; empty is a no-op
rebufferOne
Put one candidate back at the buffer tail, used to rotate a candidate that is denied only for its own pool behind the rest of the scan so the drain is not blocked at the head by it. Refused when the buffer is full. The candidate simply returns to the unbuffered state and the sweeps re-offer it, rather than evicting an older candidate to make room for one that was just taken out.- Parameters:
state- the context's batch statep- the candidate to re-queue
isPoolScopedDenial
Whether a denial reason is scoped to one pool, and therefore can be worked around by trying the candidates behind it, rather than being a global gate that stops the whole scan.- Parameters:
reason- denial reason frombatchDenialReason(int, int, int, int, int, int)- Returns:
- true when only the candidate's own pool is over budget
prioritizeBatch
Reorder one scan so the candidates that are about to expire go first: last-chance admitted tests sorted by soonest expiration, then everything else in its original buffer order. Last-chance tunnels exist only because they are withinTunnelPool.LAST_CHANCE_MIN_LIFE_MSof expiry, so letting an ordinary candidate ahead of one in the same scan can burn the only test the dying tunnel will ever get.- Parameters:
scan- the popped candidates, in buffer ordernow- the current time- Returns:
- the prioritized order; never null, same elements
prioritizeBatch
static List<TestJob.PendingTest> prioritizeBatch(List<TestJob.PendingTest> scan, long now, RouterContext ctx) Reorder one scan: last-chance candidates first (soonest expiry), then the rest by peer tier.- Parameters:
scan- the popped candidates, in buffer ordernow- the current timectx- router context used to read peer tiers; may be null, in which case tier ordering is skipped and buffer order is preserved- Returns:
- the prioritized order; never null, same elements
tierRank
Tier rank of a candidate's first hop, best first:0fast,1high-capacity,2untiered.Reads the tier live from the profile rather than from a snapshot, so a promotion that lands while a candidate waits is reflected without the buffer having to be rewritten.
Only the first hop is consulted: it is the peer the test's probe reaches first, so its quality decides whether the tunnel is worth testing promptly. Deeper hops sit behind peers this router has not yet spoken to on this build.
Without a context, or for a candidate whose config has no readable peer array, the candidate ranks untiered rather than being dropped from the batch.
- Parameters:
p- the pending test whose first hop determines the rankctx- router context, may be null- Returns:
- the tier rank 0, 1 or 2; 2 when the tier cannot be read
logBatchDenial
Rate-limited INFO line for a batch denial, so the reason a backlog is not draining stays visible without flooding the log.- Parameters:
state- the context's batch state (denial-log timestamp)ctx- the router contextreason- the denial reason from the gate
drainBuffer
Drain one batch from the first-test buffer and dispatch admitted candidates directly (no per-tunnel queue entry).The pop runs under the buffer lock with zero evaluation. Gate checks call pool.getActiveTunnelCount(), which takes the pool's tunnel lock, and taking pool locks under the buffer lock both stalls every offer behind an O(tunnels) scan and invites lock-order inversion. More than one dispatch batch is popped so a candidate that must go to the back can rotate past the ones behind it; stale candidates are discarded, a pool-scoped denial rotates just that candidate to the tail, and a global denial or lost claim race re-buffers the untested remainder in order (FIFO preserved, nothing skipped) and ends the run.
- Parameters:
ctx- the router context- Returns:
- tests dispatched this run
claimRoundCompletion
First path to claim a round wins: the round must still be the active one and must not have been claimed before. Pure so the reply-vs-timeout race is testable without a live job. The reply job and the timeout job can both reach it, and exactly one may proceed to completion.- Parameters:
active- the round the instance currently considers live, or nulltoken- the round the caller is completing- Returns:
- true when the caller owns the completion
claimRound
Claim the round for this completion attempt.- Parameters:
token- the round the callback was built for- Returns:
- true when the caller owns the completion
reserveInFlightWithReason
static TestJob.InFlightRefusal reserveInFlightWithReason(TestJob.BatchState state, int maxConcurrent, String poolId, int poolBudget, TestJob.RoundToken token) Take the global and per-pool in-flight permits for a round, atomically, reporting which gate refused it. SeetryReserveInFlight(TestJob.BatchState, int, String, int, TestJob.RoundToken)for the reservation rules; this is the same logic with the reason retained.- Parameters:
state- the context's batch statemaxConcurrent- global in-flight cappoolId- pool identity captured on the token, or null to skip the per-pool gatepoolBudget- per-pool budget; negative means unboundedtoken- the round being armed- Returns:
- the gate that refused the round, or
TestJob.InFlightRefusal.NONE
tryReserveInFlight
static boolean tryReserveInFlight(TestJob.BatchState state, int maxConcurrent, String poolId, int poolBudget, TestJob.RoundToken token) Take the global and per-pool in-flight permits for a round, atomically. Static over itsTestJob.BatchStateso the reservation rules are testable without a live instance: the global count is taken with a CAS loop so two dispatchers cannot both read "one slot left" and both take it; the pool count is reserved inside the map's compute(), which either increments a counter it checked or leaves the map untouched when the budget is exhausted. A failed pool reserve releases the global count it just took, so a refusal never leaks a permit; a token that already holds its permits is refused rather than double-counted.- Parameters:
state- the context's batch statemaxConcurrent- global in-flight cappoolId- pool identity captured on the token, or null to skip the per-pool gatepoolBudget- per-pool budget; negative means unboundedtoken- the round being armed- Returns:
- true when both permits were taken and the token holds them
reserveInFlight
Reserve the in-flight permits a round needs before it goes on the wire, and arm the round as this instance's active one. Both gates are taken atomically bytryReserveInFlight(TestJob.BatchState, int, String, int, TestJob.RoundToken), and arming happens only on success, so a refused round can never be completed later by a straggler callback. Retests reserve here too: without this they would bypass the per-pool budget that first tests pay at drain time.- Parameters:
token- the round being armed- Returns:
- true when both permits were taken and the round may dispatch
armRound
static void armRound(TestJob.BatchState state, AtomicReference<TestJob.RoundToken> active, TestJob.RoundToken next) Armnextas the active round and return the permits held by the round it supersedes.This closes a permit leak. Every release path — the reply handler, the timeout handler,
abandonRound(TestJob.RoundToken), andcancelForReset()— reaches the permits only afterclaimRound(TestJob.RoundToken), which requires the token to still be the active round. So a round that is still holding permits when a new round arms becomes unreachable: nothing can ever claim it, and its permits are never returned. The gauge then drifts upward by one per occurrence until it sits atmaxConcurrent, at which point every subsequent test is refused withTestJob.InFlightRefusal.GLOBAL_CAPand no test dispatches at all while tunnel builds continue normally.Swapping and releasing together makes the accounting correct by construction rather than by auditing each caller. The release is a no-op when the prior round already returned its own permit (a CAS on the token's held flag guards it), so this is safe on every path, including the common one where the prior round completed normally.
- Parameters:
state- the context's batch stateactive- the active-round slot for this jobnext- the round to arm
reserveInFlightForToken
Reserve in-flight permits and arm the round, reporting which gate refused it so the caller can log a specific reason. Split fromreserveInFlight(TestJob.RoundToken)because the two gates bind for different reasons: a global cap means router-wide saturation, a pool budget means one pool is oversubscribed. A single "budget exhausted" line for both made the log useless for deciding which knob to tune.- Parameters:
token- the round being armed- Returns:
- the gate that refused the round, or
TestJob.InFlightRefusal.NONE
releaseTokenPermits
Release the in-flight permits a round reserved, once. The CAS makes a second terminal for the same round (reply racing timeout, a drop after completion) a no-op, so the gauges cannot drift negative; the per-pool decrement unlinks its counter at zero atomically, so a concurrent reserve can never increment an unlinked counter. Static so the exact-release and double-release rules are testable without a live instance.- Parameters:
state- the context's batch statetoken- the round whose permits are being returned, or null
releaseRound
Release the in-flight permits a round reserved, once.- Parameters:
token- the round whose permits are being returned, or null- See Also:
isValid
public boolean isValid()Check if this TestJob instance is valid and should be queued.- Returns:
- true if valid, false if it should not be queued
decrementIfCounted
private void decrementIfCounted()Release this instance's total-counter slot, once. Idempotent so a path that runs twice (a completion racing a queue drop) cannot drive the count negative, which would break the scheduling caps permanently.cleanupTunnelTracking
private void cleanupTunnelTracking()Clean up this test job from tunnel tracking. Must be called when a test job completes or is cancelled. Idempotent: the pool slot is released only when this call actually removes the instance's running-test registration, so a duplicate cleanup (defer path racing a terminal path) cannot double-release. Note: This does NOT affect the total counter, whichdecrementIfCounted()owns.getName
The name of this job.- Returns:
- the name this job is listed under in the job queue
runJob
public void runJob()Run the tunnel test and report success or failure.sendTest
Send the tunnel test message through the selected tunnels.- Parameters:
m- the message to sendtestPeriod- time in ms before message expirationtoken- the round this dispatch belongs to- Returns:
- true if message was sent successfully, false otherwise
refusalReasonText
Short log fragment naming which gate refused a round. Pure so the wording is testable and the two cases cannot drift back together.- Parameters:
refusal- the gate that refused- Returns:
- a log fragment, never null
abandonRound
Give up a round that was armed but never made it onto the wire: mark it claimed so any straggler callback is a no-op, then return its permits.- Parameters:
token- the round being abandoned
updateTestDuration
private void updateTestDuration(long durationMs) Update the rolling average test duration and adapt the timeout multiplier. Called after each test completion (success or failure).- Parameters:
durationMs- the test duration in ms
getAdaptiveTimeout
long getAdaptiveTimeout(long baseTimeout) Get the current test timeout, adapted to the rolling average duration.The adaptive state lives on the pool, not on this job: a
TestJobis constructed per build-request batch, so per-instance state resets to 1.0 before it can ever accumulate and the adaptation never engages.- Parameters:
baseTimeout- the base timeout in ms- Returns:
- the adapted timeout in ms
testSuccessful
public void testSuccessful(int ms) Called when the tunnel test completes successfully. Updates statistics and schedules the next test.- Parameters:
ms- time in milliseconds the test took to succeed
isDegraded
private boolean isDegraded()Whether tunnel build success has fallen below the attack threshold.- Returns:
- true when build success is below the attack threshold, indicating the router is struggling to find suitable peers. In this state, the test cycle should run more slowly and tolerate more failures to avoid wasting pool build capacity.
applyMinRetestGap
static int applyMinRetestGap(int delayMs, long minGapMs, boolean expedited) Apply the minimum retest gap to a computed delay.Pure, so the floor and its exemption are testable without a router.
- Parameters:
delayMs- the delay the success-rate scaling producedminGapMs- the floor; non-positive disables itexpedited- true when the tunnel is marked expedited and must not be held back- Returns:
- the delay to actually use, in ms
getDelay
private int getDelay()Returns the delay before the next test pass.- Returns:
- the delay in ms
getSuccessRate
private float getSuccessRate()Returns the recent test success rate for this pool.- Returns:
- the success fraction
updateSuccessHistory
private void updateSuccessHistory(boolean success) Records one test result in the rolling window.- Parameters:
success- true if the test passed
clearTestTags
private void clearTestTags()Clears session tags used in the current test to avoid leaks. Consolidated cleanup logic to reduce code duplication.noteSuccess
Records a successful round trip and its time.- Parameters:
ms- the round trip timetunnel- the tunnel that carried the reply
baseRemovalThreshold
static int baseRemovalThreshold(boolean degraded) Base removal bar for data-carrying and client/exploratory tunnels. Under degraded mode (low build success), allow more consecutive failures before removal so pool churn does not waste scarce build capacity.- Parameters:
degraded- true while the pool's build success is low, raising the base bar from 3 to 5- Returns:
- the base max consecutive failures before removal
removalThreshold
static int removalThreshold(int base, int remainingTunnels) Raise the removal bar when the pool is nearly empty so a burst of concurrent test failures cannot cascade the last tunnels out before replacements finish building. Thin pools (remaining ≤ 2) get +2 extra failures of grace.- Parameters:
base- the base threshold frombaseRemovalThreshold(boolean)remainingTunnels- tunnels still in the pool after this removal- Returns:
- the effective max consecutive failures before removal
shouldDeferFailingRetest
static boolean shouldDeferFailingRetest(int remainingTunnels, int failures) Whether an ASAP retest should be deferred because the pool is thin and the tunnel already has failures. Scheduling ASAP retries for a failing tunnel while the pool has ≤ 2 remaining tunnels competes with replacement builds and risks cascading the last survivors out.- Parameters:
remainingTunnels- tunnels still in the poolfailures- consecutive test failures for this tunnel- Returns:
- true when the retest should use the normal (non-ASAP) delay
shouldDeferActiveGoodTunnel
static boolean shouldDeferActiveGoodTunnel(long lastTraffic, long now, TunnelTestStatus status, int inFlight, int maxConcurrent) Whether a retest of a GOOD tunnel carrying recent real traffic must wait for the traffic to stop. The traffic defer exists so busy tunnels don't burn test slots or risk load-induced false negatives, but deferring unconditionally starves a continuously-busy GOOD tunnel of any test at all — it keeps no fresh latency sample, so the console never shows a latency for it. So the defer is load-aware: it holds only while test slots are actually contended (in-flight at or abovemaxConcurrent / ACTIVE_GOOD_EBB_DIVISOR); during an ebb the retest runs now and refreshes the latency sample. Untested tunnels keep strict priority — they are never subject to this defer, and the ebb line leaves most of the in-flight cap free for first tests.- Parameters:
lastTraffic- ms timestamp of the tunnel's last real traffic, or <= 0 when it has carried nonenow- current time in msstatus- the tunnel's current test statusinFlight- tests currently dispatched to the network (all pools)maxConcurrent- configured in-flight cap- Returns:
- true when the retest should wait for the traffic to ebb
activeGoodDeferDelay
static long activeGoodDeferDelay(long remainingDeferMs, long normalDelayMs) Delay for a retest that was held back because its tunnel is busy: the time left in the traffic window, clamped to [EBB_RETRY_FLOOR_MS,ACTIVE_GOOD_EBB_RETRY_MS], or the normal retest delay when there is nothing left to wait for.Unclamped, the remaining window is up to
TRAFFIC_DEFER_MS, which is long enough for the tunnel to expire out from under the retest chain — scheduleRetest() refuses a delay it cannot fit before expiration and drops the test entirely. Clamping turns the wait into a short poll: come back while the tunnel is still alive, re-check contention, and run as soon as it has ebbed.- Parameters:
remainingDeferMs- ms left in the traffic window; <= 0 when nonenormalDelayMs- the delay to use when nothing is being waited out- Returns:
- the delay in ms, never negative
shouldDeferForPartner
Whether a failed round should be deferred instead of charged to the tunnel under test, because the partner tunnel — the one the round was actually routed through — is itself suspect.A test of an inbound tunnel is sent out through an outbound partner and answered back down the tunnel under test. A test of an outbound tunnel is the mirror image. The round therefore fails for two independent reasons and the failure is charged to only one of them. When the partner is a stub, already marked failing, or borrowed from the exploratory pool because the paired pool had nothing, the round is evidence about the partner pool — counting it here cascades one degraded pool's failures into the other. The tunnel under test is not left immune: the defer runs through
deferForMissingPartner(TunnelPool, String), which kicks the paired pool to rebuild and eventually fails the tunnel if it never recovers, and data-phase send failures still report independently of any test.A null partner defers: with no reply leg at all there was no reply for any reason, so the round is no evidence about the tunnel.
- Parameters:
partner- the tunnel the round was routed through, may be nullpartnerFallback- true when the partner came from the exploratory pool because the paired pool could not supply one- Returns:
- true to defer the round rather than count the failure
deferForImplicatedReplyPartner
private boolean deferForImplicatedReplyPartner()Defer a round because the same reply partner was already blamed for a recent failure, and kick the reply pool so the side that is actually degraded is the side that rebuilds. Bounded byMAX_REPLY_PARTNER_DEFERRALS: once spent the caller falls through and charges the tunnel, so a reply path that never recovers cannot shield a dead tunnel indefinitely.Split out of
testFailed(long), which is already well past the complexity budget, so this does not add to it.- Returns:
- true when the round was deferred
noteLocalHopAttribution
private void noteLocalHopAttribution()Record blame evidence when a tunnel test fails and our own next hop has no session.A tunnel test is an onion: the message traverses every hop and any hop that cannot forward drops it silently, so the round trip reports only "somewhere between us and the far end". The single hop we can name is the peer we send through, and only when we hold no session to it is the fault local rather than beyond us. Anything else stays unattributed -- blaming a peer for a failure we cannot place on it is how a good peer ends up ghosted.
Uses
BuildRequestor.getBuildRequestPeer(PooledTunnelCreatorConfig)rather than a fixed index: the peer we send through isgetPeer(0)inbound butgetPeer(1)outbound, wheregetPeer(0)is the gateway we are trying to reach. Checking the wrong index would blame a peer we never contacted.Records the observation only. It does not fail the tunnel early -- the failure path owns that decision and its threshold -- and it does not penalise the peer, because one missed session is indistinguishable from a peer restarting, and a single such event must not remove a peer that is otherwise serving us.
testFailed
private void testFailed(long timeToFail) Handles a test that timed out or failed.- Parameters:
timeToFail- ms spent before the failure was noticed
computeTestPeriod
static int computeTestPeriod(double successTimeAvgMs, double sendProcessingMs, int outLen, int replyLen, int minPeriod, int maxPeriod) Compute the tunnel test failure window: how long a test may take before the tunnel is declared failed.When a recent successful-test measurement exists, the window is 2x that average. A test answered in ~2.3s is almost certainly dead if it needs more than ~4.6s. Before any measurements exist, fall back to mainline's formula (3x transport send processing + 2.5s per hop), which assumes a slow network so early tests don't false-fail.
Always clamped to [minPeriod, maxPeriod], with min <= max enforced by construction so a crossed config can't widen the window unintentionally. The pending-message expiration (and thus the reply selector) is tied to the period itself, so a short window never strands a selector.
- Parameters:
successTimeAvgMs- averagetunnel.testSuccessTimeover the last minute, NaN if never measuredsendProcessingMs- averagetransport.sendProcessingTime, NaN if unknownoutLen- outbound tunnel length, <= 0 if no outbound tunnelreplyLen- reply tunnel length, <= 0 if no reply tunnelminPeriod- the configured minTestPeriod, the lower clamp in msmaxPeriod- the configured maxTestPeriod, the upper clamp in ms- Returns:
- the failure window in ms
getTestPeriod
private int getTestPeriod()Returns the interval before the next test of this pool.- Returns:
- the interval in ms
getLatencyDemotionThreshold
Adaptive latency threshold for demoting a peer as "high latency" after a tunnel test. Uses the network's current baseline RTT (udp.sendConfirmTime, the time to send a message and receive its ACK) so that a network-wide latency spike does not demote every peer and starve the fast-peer pool. Floored at a fixed minimum so a genuinely slow peer is still demoted on a fast network.- Parameters:
ctx- the router context whose configuration supplies the limit- Returns:
- demotion threshold in ms
scheduleRetest
private boolean scheduleRetest(boolean asap) Schedules another test of the same tunnels.- Parameters:
asap- true to test as soon as possible- Returns:
- true if the retest was scheduled
deferForMissingPartner
Handle a test round whose paired pool had no tunnel to pair with.A partner shortage is a property of the pools, not of this tunnel, so the round defers instead of failing: on a server pool a failed test would drop a healthy inbound tunnel out of the published LeaseSet. Recovery is requested by kicking the paired pool to rebuild, but the kick is bounded by
MAX_PAIRED_POOL_KICKSfor the life of this job — a paired pool that cannot recover must not be rebuilt once per deferral window against a tunnel that has no say in the matter. Once the kick budget is spent the round keeps deferring at the normal retest cadence, but only forMAX_POST_KICK_DEFERRALSmore rounds: a tunnel whose paired pool is permanently gone must not sit in the pool forever inflatinguntestedCountand blocking new builds via theaddTunnel()cap, so the post-kick deferral budget is the termination condition and the tunnel is failed when it is exhausted.- Parameters:
paired- pool that could not supply a partner, may be nulldirection- "inbound" or "outbound", for the log line
scheduleRetest
boolean scheduleRetest(boolean asap, boolean deferWhileActive) Schedule a retest of this tunnel.- Parameters:
asap- expedite the retest (failing tunnels)deferWhileActive- when true, replace the delay with the bounded ebb retry fromactiveGoodDeferDelay(long, long), so the job comes back to re-check contention shortly instead of vanishing for the full traffic window — a long push here is what used to let the tunnel expire out from under the retest chain- Returns:
- true if the retest was scheduled
getSessionKeyManager
Returns the session key manager for test tags.- Returns:
- the key manager
getRatchetKeyManager
Returns the ratchet session key manager for test tags.- Returns:
- the ratchet key manager
dropped
public void dropped()Called when the job is dropped due to router overload, when the batch pump's catch sees a dispatch throw, or when a reset cancels it. Equivalent tocancelForReset().cancelForReset
void cancelForReset()Cancel this instance for a restart, a queue shutdown, or an overload drop: unregister its pending message so no callback can arrive for a round nobody is waiting on, claim the active round (so a reply or timeout that races in cannot complete it), return that round's permits, and release this instance's slots. Every step is idempotent, so a cancel racing a normal completion is safe.keepDataCarryingLine
static String keepDataCarryingLine(String tunnel, long verified, int failures, int maxFailures, long staleMs) The line for a failed test on a tunnel that is still carrying data, kept because failing it would drop live traffic. Split onto continuation lines so an emitted line stays readable.- Parameters:
tunnel- the tunnel as described by its own toString()verified- bytes verified as transferred on this tunnelfailures- the consecutive test failures so farmaxFailures- the failures tolerated before the tunnel is failedstaleMs- ms since the last verified transfer- Returns:
- the log entry, never null
thinPoolRetestLine
The line for an ASAP retest deferred because the pool is too thin to afford losing another tunnel to a test right now.- Parameters:
poolSize- tunnels left in the poolfailures- the tunnel's consecutive failures- Returns:
- the log entry, never null