java.lang.Object
net.i2p.router.networkdb.kademlia.TransientDataStore
net.i2p.router.networkdb.kademlia.PersistentDataStore
All Implemented Interfaces:
DataStore
public class PersistentDataStoreextends TransientDataStore
Persistent network database storage with disk I/O and memory management.

Writes RouterInfo entries to disk as received and periodically scans for unknown entries to load into memory. Newly discovered routers are automatically added to the routing table. Implements both flat and hierarchical directory structures for efficient file organization and access.

Public only for access to static methods by startup classes.

  • Field Details

    • _dbDir

      private final File _dbDir
      Database directory.

    • _facade

      private final KademliaNetworkDatabaseFacade _facade
      Reference to the Kademlia facade for netdb operations.

    • _writer

      private final PersistentDataStore.Writer _writer
      Background thread for persisting RouterInfo files to disk.

    • _readJob

      private final PersistentDataStore.ReadJob _readJob
      Read job that loads the database from disk.

    • _banLogger

      private final BanLogger _banLogger
      Ban logger.

    • _initialized

      private volatile boolean _initialized
      Initialized flag.

    • _flat

      private final boolean _flat
      Flat mode flag.

    • _networkID

      private final int _networkID
      Network ID.

    • READ_DELAY

      private static final int READ_DELAY
      Initial delay before first netdb scan.
      See Also:

    • READ_INTERVAL

      private static final int READ_INTERVAL
      Interval between periodic netdb rescans.
      See Also:

    • PROP_FLAT

      private static final String PROP_FLAT
      Config property to enable flat directory layout.
      See Also:

    • DIR_PREFIX

      static final String DIR_PREFIX
      Directory prefix for netDb files.
      See Also:

    • B64

      private static final String B64
      Base64 alphabet used for directory naming.
      See Also:

    • MAX_ROUTERS_INIT

      private static final int MAX_ROUTERS_INIT
      Max router files to load during startup initialization.
      See Also:

    • MIN_VERSION

      private static final String MIN_VERSION
      Minimum router version to persist to disk.
      See Also:

    • RATES

      private static final long[] RATES
      Stat rate periods for NetDb write metrics.

    • WRITE_LIMIT

      private static final int WRITE_LIMIT
      How many files to write every 10 minutes. Doesn't make sense to limit it, they just back up in the queue hogging memory.
      See Also:

    • WRITE_DELAY

      private static final long WRITE_DELAY
      Interval between batch disk write cycles.
      See Also:

    • cachedRiCount

      private int cachedRiCount
      Cached count of the router info entries.

    • lastRiCountTime

      private long lastRiCountTime
      When the cached count was taken.

    • RI_CACHE_TTL

      private static final long RI_CACHE_TTL
      How long the cached count stays valid.
      See Also:

    • ROUTERINFO_PREFIX

      private static final String ROUTERINFO_PREFIX
      Prefix for RouterInfo filenames on disk.
      See Also:

    • ROUTERINFO_SUFFIX

      private static final String ROUTERINFO_SUFFIX
      Suffix for RouterInfo filenames on disk.
      See Also:

    • RI_FILTER

      public static final FileFilter RI_FILTER
      File filter for RouterInfo files.

  • Constructor Details

    • PersistentDataStore

      public PersistentDataStore(RouterContext ctx, String dbDir, KademliaNetworkDatabaseFacade facade) throws IOException
      Creates a new persistent data store.
      Parameters:
      ctx - the router context
      dbDir - directory name resolved under the router directory
      facade - the network database facade
      Throws:
      IOException - if the database directory cannot be created or accessed

  • Method Details

    • isInitialized

      public boolean isInitialized()
      Whether the data store has completed initial loading, either because the read job finished or because enough RouterInfos have been loaded for the netdb to be usable.
      Specified by:
      isInitialized in interface DataStore
      Overrides:
      isInitialized in class TransientDataStore
      Returns:
      true if the data store has completed initial loading

    • stop

      public void stop()
      Stop the writer thread and clear the in-memory store.

      The read job is not stopped: it is a recurring SimpleTimer2 event with no cancellation flag, so it keeps rescanning until the router's graceful shutdown begins and runJob() returns early. A rescan after this point re-reads an empty store and is harmless, but it is an open question whether stop() should cancel the reader outright.

      Specified by:
      stop in interface DataStore
      Overrides:
      stop in class TransientDataStore

    • rescan

      public void rescan()
      Trigger an immediate rescan if the store is initialized.
      Specified by:
      rescan in interface DataStore
      Overrides:
      rescan in class TransientDataStore

    • get

      public DatabaseEntry get(Hash key)
      The entry stored for the key, or null if not found.
      Specified by:
      get in interface DataStore
      Overrides:
      get in class TransientDataStore
      Parameters:
      key - the key to look up
      Returns:
      the entry or null if not found

    • get

      public DatabaseEntry get(Hash key, boolean persist)
      Prepare for having only a partial set in memory and the rest on disk.
      Specified by:
      get in interface DataStore
      Overrides:
      get in class TransientDataStore
      Parameters:
      key - the hash key to look up
      persist - if false, call super only, don't access disk
      Returns:
      the database entry, or null if not found

    • remove

      public DatabaseEntry remove(Hash key)
      Remove from memory and queue the file for disk deletion.
      Specified by:
      remove in interface DataStore
      Overrides:
      remove in class TransientDataStore
      Parameters:
      key - the key to remove
      Returns:
      the removed entry, or null if none was stored

    • remove

      public DatabaseEntry remove(Hash key, boolean persist)
      Remove from memory and optionally queue disk deletion.
      Specified by:
      remove in interface DataStore
      Overrides:
      remove in class TransientDataStore
      Parameters:
      key - the key to remove
      persist - if false, call super only, don't access disk
      Returns:
      the removed entry, or null if none was stored

    • put

      public boolean put(Hash key, DatabaseEntry data)
      Store the entry and persist it to disk. Only RouterInfos are written; LeaseSets stay in memory.
      Specified by:
      put in interface DataStore
      Overrides:
      put in class TransientDataStore
      Parameters:
      key - the key to store under
      data - the entry to store
      Returns:
      true if the entry was stored

    • put

      public boolean put(Hash key, DatabaseEntry data, boolean persist)
      Store the entry and optionally persist it to disk. Only RouterInfos are written; LeaseSets stay in memory.
      Specified by:
      put in interface DataStore
      Overrides:
      put in class TransientDataStore
      Parameters:
      key - the key to store under
      data - the entry to store, null to fail
      persist - if false, call super only, don't access disk
      Returns:
      true if the entry was stored

    • forcePut

      public boolean forcePut(Hash key, DatabaseEntry data)
      Unconditionally store, bypassing all newer/older checks. Persists for RouterInfos only.
      Specified by:
      forcePut in interface DataStore
      Overrides:
      forcePut in class TransientDataStore
      Parameters:
      key - non-null
      data - non-null
      Returns:
      true if the entry was stored in the Kademlia structure

    • extractRiFlags

      private PersistentDataStore.RiFlags extractRiFlags(Hash key, RouterInfo ri)
      Extract classification flags from raw RouterInfo data.
      Parameters:
      key - the hash of the router, used for the banlist checks
      ri - the router info to classify
      Returns:
      a RiFlags snapshot of the version, capability, bandwidth, IP and banlist state used by the store decision

    • shouldStoreRi

      private static boolean shouldStoreRi(PersistentDataStore.RiFlags f)
      Should this RouterInfo be persisted to disk? Returns true only for reachable, non-degraded peers with sufficient bandwidth.
      Parameters:
      f - the pre-computed classification flags for the peer
      Returns:
      false if the RI should be skipped or deleted

    • checkSpoofAndBan

      private boolean checkSpoofAndBan(Hash key, RouterInfo ri, PersistentDataStore.RiFlags f)
      Detect IP spoofing and apply ban decisions for non-storeable peers. Called only for peers that failed shouldStoreRi(RiFlags) and are not banned.
      Parameters:
      key - the hash of the peer being checked
      ri - the peer's router info, used for the spoof check and ban logging
      f - the pre-computed classification flags for the peer
      Returns:
      true if the on-disk file should be deleted

    • writeRiFile

      private boolean writeRiFile(Hash key, DatabaseEntry data, File dbFile) throws IOException
      Write a single RouterInfo to disk. Skips if the file is already current.
      Parameters:
      key - the hash of the router, used for the log messages
      data - the RouterInfo entry to serialize
      dbFile - the destination file; skipped when its last-modified time is already at or after the entry's publish date
      Returns:
      true if the write failed (file should be deleted)
      Throws:
      IOException - if the output stream cannot be opened or the data cannot be written

    • write

      private void write(Hash key, DatabaseEntry data)
      Persist a single RouterInfo to disk, applying spoof/ban checks.
      Parameters:
      key - the hash of the router to write
      data - the entry to write; a null entry is ignored

    • countStoredRIs

      public int countStoredRIs()
      Count the number of RouterInfo files currently stored on disk. This includes files in subdirectories if not using flat mode.
      Returns:
      number of RI files on disk

    • getPublishDate

      private static long getPublishDate(DatabaseEntry data)
      Returns the publish date of an entry.
      Parameters:
      data - the entry to read
      Returns:
      the publish date

    • purgeSlowRouters

      private void purgeSlowRouters()
      At startup, this method scans existing RouterInfos and removes any categorized as slow, determined by the total number stored on disk.

    • getDbDir

      private File getDbDir(String dbDir) throws IOException
      Resolve and prepare the database directory, creating subdirs if needed.
      Parameters:
      dbDir - the database directory path, relative to the router directory
      Returns:
      the directory, created and verified readable and writable
      Throws:
      IOException - if the directory cannot be created, is not a directory, or is not readable or writable

    • unmigrate

      private static void unmigrate(File dbdir)
      Migrate from two-level to one-level directory structure.
      Parameters:
      dbdir - the NetDb root directory whose subdirs are emptied

    • migrate

      private static void migrate(File dbdir, File[] files)
      Migrate from one-level to two-level directory structure.
      Parameters:
      dbdir - the NetDb root directory the files currently sit in
      files - the RouterInfo files to move into the subdirectories

    • getRouterInfoName

      private String getRouterInfoName(Hash hash)
      Builds the router info file name for a hash.
      Parameters:
      hash - the hash to name
      Returns:
      the file name

    • getRouterInfoFile

      public static File getRouterInfoFile(RouterContext ctx, Hash hash)
      The persistent RI file for a hash. This is available before the netdb subsystem is running, so we can delete our old RI.
      Parameters:
      ctx - the router context
      hash - the router hash
      Returns:
      non-null, should be absolute, does not necessarily exist

    • getRouterInfoHash

      static Hash getRouterInfoHash(String filename)
      Package private for installer BundleRouterInfos.
      Parameters:
      filename - the routerinfo filename, including the prefix and suffix added by getRouterInfoFile(RouterContext, Hash)
      Returns:
      the hash encoded in the name, or null if the name is malformed or the key does not decode

    • getHash

      private static Hash getHash(String filename, String prefix, String suffix)
      Extract hash from a prefixed+suffixed filename.
      Parameters:
      filename - the file name as stored on disk
      prefix - the leading part to strip, e.g. "routerInfo-"
      suffix - the trailing part to strip, e.g. ".dat"
      Returns:
      the decoded hash, or null if the name is malformed or the embedded base64 key does not decode

    • removeFile

      private void removeFile(Hash key, File dir)
      Delete the RouterInfo file for a hash, if it is present.
      Parameters:
      key - the hash of the router whose file name is derived
      dir - the NetDb root directory the file path is resolved against

    • isShuttingDown

      public static boolean isShuttingDown(RouterContext ctx)
      Check if the router is in a graceful or hard shutdown state.
      Parameters:
      ctx - the router context
      Returns:
      true if shutdown is in progress