java.lang.Object
net.i2p.router.peermanager.ProfilePersistenceHelper
class ProfilePersistenceHelperextends Object
Write profiles to disk at shutdown, read at startup. The files are gzip compressed, we previously stored them with a ".dat" extension instead of ".txt.gz", so it wasn't apparent. Now migrated to a ".txt.gz" extension.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    private static final class 
    ProfilePersistenceHelper.FileMetadata
    A profile file considered for deletion, with the age that decides its order.
    private static class 
    ProfilePersistenceHelper.ProfileFilter
    Filter selecting the profile files.

  • Field Summary

    Modifier and Type
    Field
    Description
    private final RouterContext
    _context
    The router context.
    private final Log
    _log
    Logger for the profile storage.
    private Set<Hash>
    _missingRouterInfo
    Peers whose profile was loaded while the netdb held no RouterInfo, collected during readProfiles().
    private final File
    _profileDir
    Directory holding the profile files.
    private int
    _storedProfileCount
    Last known number of profile files on disk, from the most recent load or purge.
    private Hash
    _us
    Our router hash.
    private static final String
    B64
    Alphabet of the base64 hash in the file names.
    static final String
    DEFAULT_PEER_PROFILE_DIR
    Default peer profile directory name, relative to the router directory.
    private static final String
    DIR_PREFIX
    Prefix of the profile subdirectory names.
    private static final String
    HR
    Comment rule line of the profile file.
    private static final int
    LIMIT_PROFILES
    Profiles kept on disk.
    private static final int
    MIN_NAME_LENGTH
    Shortest valid profile file name.
    private static final String
    NL
    The platform line separator.
    private static final String
    OLD_SUFFIX
    Filename suffix of the old serialized profile files.
    private static final String
    PREFIX
    Filename prefix of the profile files.
    static final String
    PROP_PEER_PROFILE_DIR
    Config property naming the base directory that peer profiles are stored under.
    private static final long[]
    RATES
    Rate stat periods for the profile statistics.
    (package private) static final long
    STALE_PROFILE_AGE_MS
    Age beyond which a peer that is also absent from the network database is treated as gone, making its profile file deletable at any store size.
    private static final String
    STAT_LOADED_LOW_LATENCY
    Profiles that loaded carrying the persisted low-latency flag.
    private static final String
    STAT_LOADED_RTT_EXPIRED
    Profiles whose stored first-hop RTT had aged past the validity window.
    private static final String
    STAT_LOADED_WITH_RTT
    Profiles that loaded with a first-hop RTT still inside its validity window.
    private static final String
    STAT_STORED_PROFILES
    Stat name for the number of stored profiles.
    private static final String
    SUFFIX
    Filename suffix of the compressed profile files.
    private static final String
    UNCOMPRESSED_SUFFIX
    Filename suffix of the uncompressed profile files.

  • Constructor Summary

    Constructors
    Constructor
    Description
    ProfilePersistenceHelper(RouterContext ctx)
    Construct a helper that reads and writes peer profiles under the router directory.

  • Method Summary

    Modifier and Type
    Method
    Description
    private static void
    add(StringBuilder buf, boolean addComments, String name, long val, String description)
    Write a profile property, with a comment if requested.
    private static void
    addDate(StringBuilder buf, boolean addComments, String name, long val, String description)
    Write a date property, with a comment if requested.
    int
    deleteOldProfiles(long age)
    Delete profile files with timestamps older than 'age' ago.
    private void
    fixupFirstHeardAbout(PeerProfile p)
    First heard about wasn't always set correctly before, set it to the minimum of all recorded timestamps.
    private static final float
    getFloat(Properties props, String key)
    Reads a float property with a default of zero.
    private Hash
    getHash(String name)
    Extracts the router hash from a profile file name.
    (package private) static long
    getLong(Properties props, String key)
    Parse a long value from the properties, or 0 if missing or invalid.
    int
    getStoredProfileCount()
    Last known number of profile files on disk, from the most recent load, stale-profile cleanup, or purge.
    (package private) static boolean
    isGenuinelyStale(boolean absentFromNetDb, long lastActivity, long now, long staleAge)
    Is a profile file genuinely stale, and therefore deletable whatever the store size?
    private void
    loadProps(Properties props, File file)
    Loads one profile file's properties.
    private void
    migrate(File[] files)
    Migrate from one-level to two-level directory structure.
    private static long
    newestOf(long a, long b, long c)
    Newest of three timestamps, treating absent (0) as oldest.
    private File
    pickFile(PeerProfile profile)
    Chooses the profile file for a peer.
    (package private) static void
    purgeCapacityBonus(Properties props)
    Drop the capacity-bonus keys from a loaded profile's properties.
    void
    purgeExcessProfiles(Set<Hash> keepPeers, int retentionFloor)
    Delete profile files that are no longer worth keeping.
    PeerProfile
    readProfile(File file, long cutoff)
    Read and parse a single stored profile.
    List<PeerProfile>
    readProfiles()
    Load all existing profiles from the profile directory.
    private List<File>
    selectFiles()
    Picks the profile files to load, newest first.
    (package private) void
    setMissingRouterInfoSink(Set<Hash> missingRouterInfo)
    Supply the set to collect peers-without-RouterInfo into.
    private void
    setStoredProfileCount(int count)
    Record the number of profile files on disk and publish it for graphing.
    void
    setUs(Hash routerIdentHash)
    Store the local router identity hash for filtering out the router's own profile.
    (package private) static int
    surplusToDelete(int storedFiles, int retentionFloor, int surplusAvailable)
    How many surplus (non-stale) profiles to delete to bring the store back to the retention floor.
    boolean
    writeProfile(PeerProfile profile)
    Write the data from the profile to the file.
    void
    writeProfile(PeerProfile profile, OutputStream out)
    Write the data from the profile to the stream includes comments.
    void
    writeProfile(PeerProfile profile, OutputStream out, boolean addComments)
    Write the data from the profile to the stream.

    Methods inherited from class Object

    clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • _log

      private final Log _log
      Logger for the profile storage.

    • _context

      private final RouterContext _context
      The router context.

    • STALE_PROFILE_AGE_MS

      static final long STALE_PROFILE_AGE_MS
      Age beyond which a peer that is also absent from the network database is treated as gone, making its profile file deletable at any store size.

      Seven days. The previous logic used a 24-hour activity threshold and separately deleted K/L/M/Unknown bandwidth peers regardless of age, so a peer merely quiet overnight could lose its learned history. Those peers are no longer written at all; this threshold now governs only what is left on disk from before that. A peer that has been out of the network for a week will not be selected by a RouterInfo that has since expired anyway.

      See Also:

    • PROP_PEER_PROFILE_DIR

      public static final String PROP_PEER_PROFILE_DIR
      Config property naming the base directory that peer profiles are stored under.
      See Also:

    • DEFAULT_PEER_PROFILE_DIR

      public static final String DEFAULT_PEER_PROFILE_DIR
      Default peer profile directory name, relative to the router directory.
      See Also:

    • NL

      private static final String NL
      The platform line separator.

    • HR

      private static final String HR
      Comment rule line of the profile file.
      See Also:

    • PREFIX

      private static final String PREFIX
      Filename prefix of the profile files.
      See Also:

    • SUFFIX

      private static final String SUFFIX
      Filename suffix of the compressed profile files.
      See Also:

    • UNCOMPRESSED_SUFFIX

      private static final String UNCOMPRESSED_SUFFIX
      Filename suffix of the uncompressed profile files.
      See Also:

    • OLD_SUFFIX

      private static final String OLD_SUFFIX
      Filename suffix of the old serialized profile files.
      See Also:

    • MIN_NAME_LENGTH

      private static final int MIN_NAME_LENGTH
      Shortest valid profile file name.

    • DIR_PREFIX

      private static final String DIR_PREFIX
      Prefix of the profile subdirectory names.
      See Also:

    • B64

      private static final String B64
      Alphabet of the base64 hash in the file names.
      See Also:

    • LIMIT_PROFILES

      private static final int LIMIT_PROFILES
      Profiles kept on disk.

    • _profileDir

      private final File _profileDir
      Directory holding the profile files.

    • _us

      private Hash _us
      Our router hash.

    • _storedProfileCount

      private volatile int _storedProfileCount
      Last known number of profile files on disk, from the most recent load or purge.

    • STAT_STORED_PROFILES

      private static final String STAT_STORED_PROFILES
      Stat name for the number of stored profiles.
      See Also:

    • STAT_LOADED_WITH_RTT

      private static final String STAT_LOADED_WITH_RTT
      Profiles that loaded with a first-hop RTT still inside its validity window. A non-zero value here is the confirmation that a transport is actually supplying latency: without one, every profile would fall into STAT_LOADED_RTT_EXPIRED or neither, and the fast tier would be seeded from persisted flags alone.
      See Also:

    • STAT_LOADED_RTT_EXPIRED

      private static final String STAT_LOADED_RTT_EXPIRED
      Profiles whose stored first-hop RTT had aged past the validity window.
      See Also:

    • STAT_LOADED_LOW_LATENCY

      private static final String STAT_LOADED_LOW_LATENCY
      Profiles that loaded carrying the persisted low-latency flag.
      See Also:

    • RATES

      private static final long[] RATES
      Rate stat periods for the profile statistics.

    • _missingRouterInfo

      private Set<Hash> _missingRouterInfo
      Peers whose profile was loaded while the netdb held no RouterInfo, collected during readProfiles().

      These profiles are kept on disk but inert until the RouterInfo is fetched again, so the caller needs the list to go and request it. Empty when no collection has been supplied.

  • Constructor Details

    • ProfilePersistenceHelper

      public ProfilePersistenceHelper(RouterContext ctx)
      Construct a helper that reads and writes peer profiles under the router directory.
      Parameters:
      ctx - the router context, whose router directory holds the profile files

  • Method Details

    • setStoredProfileCount

      private void setStoredProfileCount(int count)
      Record the number of profile files on disk and publish it for graphing.

      Every path that recomputes the count routes through here rather than assigning the field directly, so the graphed gauge cannot drift from the value the store actually holds.

      Parameters:
      count - number of profile files currently on disk

    • getStoredProfileCount

      public int getStoredProfileCount()
      Last known number of profile files on disk, from the most recent load, stale-profile cleanup, or purge.
      Returns:
      the stored profile count, 0 if the store has never been scanned

    • setUs

      public void setUs(Hash routerIdentHash)
      Store the local router identity hash for filtering out the router's own profile.
      Parameters:
      routerIdentHash - this router's identity hash, or null to stop filtering it out

    • writeProfile

      public boolean writeProfile(PeerProfile profile)
      Write the data from the profile to the file.

      A rare, benign failure mode: a concurrent purge (purgeExcessProfiles(Set, int)) or expiry sweep can remove the profile, or its shard directory (peerProfiles/pX), between pickFile(PeerProfile) and the open. When the shard directory vanished it is re-created and the write retried once so the store stays self-healing; anything else is a race or a genuine store problem and is worth a WARN without a stacktrace, not an ERROR.

      Parameters:
      profile - the profile to serialize
      Returns:
      true if the profile was written, false if it could not be stored

    • writeProfile

      public void writeProfile(PeerProfile profile, OutputStream out) throws IOException
      Write the data from the profile to the stream includes comments.
      Parameters:
      profile - the profile to serialize
      out - the stream to write the serialized profile to, not closed by this method
      Throws:
      IOException - if the stream cannot be written

    • writeProfile

      public void writeProfile(PeerProfile profile, OutputStream out, boolean addComments) throws IOException
      Write the data from the profile to the stream.
      Parameters:
      profile - the profile to serialize
      out - the stream to write the serialized profile to, not closed by this method
      addComments - add comment lines to the output
      Throws:
      IOException - if the stream cannot be written

    • addDate

      private static void addDate(StringBuilder buf, boolean addComments, String name, long val, String description)
      Write a date property, with a comment if requested.
      Parameters:
      buf - the builder to append the line to
      addComments - true to append the comment line, false the property line
      name - the property name to write
      val - the timestamp in ms, 0 or less meaning never
      description - comment text, the rendered date is appended to it

    • add

      private static void add(StringBuilder buf, boolean addComments, String name, long val, String description)
      Write a profile property, with a comment if requested.
      Parameters:
      buf - the builder to append the line to
      addComments - true to append the comment line, false the property line
      name - the property name to write
      val - the numeric value to write
      description - comment text for the property

    • readProfiles

      public List<PeerProfile> readProfiles()
      Load all existing profiles from the profile directory.
      Returns:
      the profiles read, most recently modified first, capped at the store limit

    • selectFiles

      private List<File> selectFiles()
      Picks the profile files to load, newest first.
      Returns:
      the files selected

    • migrate

      private void migrate(File[] files)
      Migrate from one-level to two-level directory structure.
      Parameters:
      files - the flat profile files to move into the two-level structure

    • deleteOldProfiles

      public int deleteOldProfiles(long age)
      Delete profile files with timestamps older than 'age' ago.
      Parameters:
      age - how far back in milliseconds a profile must predate to be deleted
      Returns:
      the number of profile files removed

    • readProfile

      public PeerProfile readProfile(File file, long cutoff)
      Read and parse a single stored profile.
      Parameters:
      file - the profile file to read
      cutoff - delete and return null if older than this (absolute time)
      Returns:
      the parsed profile, or null if the file was deleted as stale or malformed

    • fixupFirstHeardAbout

      private void fixupFirstHeardAbout(PeerProfile p)
      First heard about wasn't always set correctly before, set it to the minimum of all recorded timestamps.
      Parameters:
      p - the profile whose firstHeardAbout is corrected in place

    • purgeCapacityBonus

      static void purgeCapacityBonus(Properties props)
      Drop the capacity-bonus keys from a loaded profile's properties. A capacity bonus is a live, manual nudge set on the configpeers page; it must not survive a restart, and old profile files could carry one with no expiry timestamp, which the 4-hour decay would then treat as valid forever. Removing the keys here also clears them from disk on the next save.
      Parameters:
      props - the parsed profile properties, modified in place

    • getLong

      static long getLong(Properties props, String key)
      Parse a long value from the properties, or 0 if missing or invalid.
      Parameters:
      props - the parsed profile properties
      key - the property name to read
      Returns:
      the long value or 0

    • getFloat

      private static final float getFloat(Properties props, String key)
      Reads a float property with a default of zero.
      Parameters:
      props - the properties read
      key - the property name
      Returns:
      the value, zero if unparseable

    • loadProps

      private void loadProps(Properties props, File file) throws IOException
      Loads one profile file's properties.
      Parameters:
      props - the properties to fill
      file - the file to read
      Throws:
      IOException - if the file cannot be read

    • setMissingRouterInfoSink

      void setMissingRouterInfoSink(Set<Hash> missingRouterInfo)
      Supply the set to collect peers-without-RouterInfo into.
      Parameters:
      missingRouterInfo - the set to fill during readProfiles(), or null to collect nothing

    • getHash

      private Hash getHash(String name)
      Extracts the router hash from a profile file name.
      Parameters:
      name - the file name
      Returns:
      the hash, null if the name is malformed

    • pickFile

      private File pickFile(PeerProfile profile)
      Chooses the profile file for a peer.
      Parameters:
      profile - the profile to name
      Returns:
      the file to use

    • isGenuinelyStale

      static boolean isGenuinelyStale(boolean absentFromNetDb, long lastActivity, long now, long staleAge)
      Is a profile file genuinely stale, and therefore deletable whatever the store size?

      Both conditions are required. Absence from the network database alone means the RouterInfo expired, which happens routinely while a peer is offline for an hour; inactivity alone means the peer has not been heard from recently, which is equally routine. Only the conjunction identifies a profile whose peer has been gone long enough that its learned history is worth nothing.

      Pure decision, safe for unit tests.

      Parameters:
      absentFromNetDb - true if no usable RouterInfo is held for the peer
      lastActivity - newest of last-sent-successfully, last-heard-from and last-heard-about, or 0 if the profile records none
      now - current time in ms
      staleAge - age beyond which an absent peer is treated as gone, in ms
      Returns:
      true if the file may be deleted regardless of store size

    • surplusToDelete

      static int surplusToDelete(int storedFiles, int retentionFloor, int surplusAvailable)
      How many surplus (non-stale) profiles to delete to bring the store back to the retention floor.

      Stale profiles are deleted independently of this, so this counts only the additional trimming that the store being over its floor justifies. It is zero below the floor, which is what makes "never delete below the floor unless genuinely stale" hold regardless of how many profiles are active.

      Pure decision, safe for unit tests.

      Parameters:
      storedFiles - profile files currently on disk
      retentionFloor - file count to keep, exclusive of stale profiles
      surplusAvailable - surplus profiles not already counted as stale
      Returns:
      number of surplus files to delete, never negative and never more than surplusAvailable

    • purgeExcessProfiles

      public void purgeExcessProfiles(Set<Hash> keepPeers, int retentionFloor)
      Delete profile files that are no longer worth keeping.

      Two rules, in this order:

      1. a profile that is genuinely stale is deleted at any store size, oldest first;
      2. a profile that is merely surplus is deleted only while the store is above retentionFloor, least-recently-active first.
      Staleness therefore decides whether a file may go and the store size decides only how many.

      Low-bandwidth peers are excluded where it belongs, on the write path in PeerManager.storeProfile(), so no K/L/M/Unknown profile is created in the first place. This method used to remove them instead, ahead of any age check, which deleted the file of a peer heard from minutes earlier and only by accident got the right answer: files already on disk from before that check existed still age out here, through the stale bucket or the surplus trim.

      Ordering uses the profile's own recorded activity rather than the file's modification time, because profiles are rewritten whenever they are stored and so carry a fresh mtime even for a peer long gone.

      Parameters:
      keepPeers - peers whose files must be preserved, may be null
      retentionFloor - file count to keep before trimming surplus

    • newestOf

      private static long newestOf(long a, long b, long c)
      Newest of three timestamps, treating absent (0) as oldest.
      Parameters:
      a - first timestamp, 0 if unknown
      b - second timestamp, 0 if unknown
      c - third timestamp, 0 if unknown
      Returns:
      the largest non-zero value, or 0 if all are 0