java.lang.Object
net.i2p.client.naming.NamingService
net.i2p.client.naming.DummyNamingService
net.i2p.router.naming.BlockfileNamingService
public class BlockfileNamingServiceextends DummyNamingService
A naming service using the net.metanotion BlockFile database.

This database contains the following skiplists:


"%%__INFO__%%" is the master database skiplist, containing one entry:
    "info": a Properties, serialized with DataHelper functions:
            "version": "4"
            "created": Java long time (ms)
            "upgraded": Java long time (ms) (as of database version 2)
            "lists":   Comma-separated list of host databases, to be
                       searched in-order for lookups

"%%__REVERSE__%%" is the reverse lookup skiplist
    (as of database version 2):
    The skiplist keys are Integers, the first 4 bytes of the hash of the dest.
    The skiplist values are Properties.
        There may be multiple entries in the properties, each one is a reverse mapping,
           as there may be more than one hostname for a given destination,
           or there could be collisions with the same first 4 bytes of the hash.
        Each property key is a hostname.
        Each property value is the empty string.

For each host database, there is a skiplist containing
the hosts for that database.
The keys/values in these skiplists are as follows:
     key: a UTF-8 String
     value: a DestEntry, which is:
            a one-byte count of the Properties/Destination pairs to follow
              (as of database version 4, otherwise one)
            that many pairs of:
              Properties (serialized with DataHelper)
              Destination (serialized as usual).


The DestEntry Properties typically contains:
            "a":   The time added (Java long time in ms)
            "s":   The original source of the entry (typically a file name or subscription URL)
            others TBD

All host names are converted to lower case.

  • Field Details

    • _bf

      private final BlockFile _bf
      The block file holding the naming data.

    • _raf

      private final RAIFile _raf
      Random access view of the block file.

    • _lists

      private final List<String> _lists
      Names of the lists in the database.

    • _invalid

      private final List<BlockfileNamingService.InvalidEntry> _invalid
      Entries that failed to parse.

    • _negativeCache

      private final Map<String,String> _negativeCache
      Cached lookups that found nothing.

    • _isClosed

      private volatile boolean _isClosed
      True once the service is closed.

    • _readOnly

      private final boolean _readOnly
      True if the database may not be written.

    • _version

      private String _version
      Version string of the database format.

    • _isVersion4

      private volatile boolean _isVersion4
      True if the database uses the version 4 format.

    • _needsUpgrade

      private boolean _needsUpgrade
      True if the database needs a format upgrade.

    • _infoSerializer

      private static final Serializer<Properties> _infoSerializer
      Serializer for the info block.

    • _stringSerializer

      private static final Serializer<String> _stringSerializer
      Serializer for host names.

    • _destSerializerV1

      private static final Serializer<BlockfileNamingService.DestEntry> _destSerializerV1
      Serializer for version 1 destination entries.

    • _destSerializerV4

      private static final Serializer<BlockfileNamingService.DestEntry> _destSerializerV4
      Serializer for version 4 destination entries.

    • _destSerializer

      private volatile Serializer<BlockfileNamingService.DestEntry> _destSerializer
      Serializer in effect for destination entries.

    • _hashIndexSerializer

      private static final Serializer<Integer> _hashIndexSerializer
      Serializer for hash index keys.

    • HOSTS_DB

      private static final String HOSTS_DB
      File name of the block file database.
      See Also:

    • FALLBACK_LIST

      private static final String FALLBACK_LIST
      File name of the fallback hosts list.
      See Also:

    • PROP_FORCE

      private static final String PROP_FORCE
      Config key allowing writes in the app context.
      See Also:

    • INFO_SKIPLIST

      private static final String INFO_SKIPLIST
      Skiplist holding the info block.
      See Also:

    • REVERSE_SKIPLIST

      private static final String REVERSE_SKIPLIST
      Skiplist holding the reverse lookups.
      See Also:

    • PROP_INFO

      private static final String PROP_INFO
      Info block key in the skiplist.
      See Also:

    • PROP_VERSION

      private static final String PROP_VERSION
      Version key in the info properties.
      See Also:

    • PROP_LISTVERSION

      private static final String PROP_LISTVERSION
      List version key in the info properties.
      See Also:

    • PROP_LISTS

      private static final String PROP_LISTS
      Lists key in the info properties.
      See Also:

    • PROP_CREATED

      private static final String PROP_CREATED
      Creation date key in the info properties.
      See Also:

    • PROP_UPGRADED

      private static final String PROP_UPGRADED
      Upgrade date key in the info properties.
      See Also:

    • VERSION

      private static final String VERSION
      Database format version this code writes.
      See Also:

    • PROP_ADDED

      private static final String PROP_ADDED
      Entry added-date key.
      See Also:

    • PROP_SOURCE

      private static final String PROP_SOURCE
      Entry source key.
      See Also:

    • DUMMY

      private static final String DUMMY
      Placeholder value where none is needed.
      See Also:

    • NEGATIVE_CACHE_SIZE

      private static final int NEGATIVE_CACHE_SIZE
      Entries kept in the negative cache.
      See Also:

    • MAX_VALUE_LENGTH

      private static final int MAX_VALUE_LENGTH
      Longest destination value accepted.
      See Also:

    • MAX_DESTS_PER_HOST

      private static final int MAX_DESTS_PER_HOST
      Destinations allowed for one host.
      See Also:

  • Constructor Details

    • BlockfileNamingService

      public BlockfileNamingService(I2PAppContext context)
      Opens the database at hostsdb.blockfile or creates a new one and imports entries from hosts.txt, userhosts.txt, and privatehosts.txt.

      If not in router context, the database will be opened read-only unless the property i2p.naming.blockfile.writeInAppContext is true. Not designed for multiple instantiations or simultaneous use by multiple JVMs.

      Parameters:
      context - the application context whose router directory holds hostsdb.blockfile
      Throws:
      RuntimeException - on fatal error

  • Method Details

    • initNew

      private BlockFile initNew(RAIFile f) throws IOException
      Create a new database and initialize it from the local files privatehosts.txt, userhosts.txt, and hosts.txt, creating a skiplist in the database for each.
      Parameters:
      f - open file the new database is written into
      Returns:
      the new database, with one skiplist per hosts file found
      Throws:
      IOException - if the file cannot be written, or initialization fails and the cause is wrapped into an IOException

    • initExisting

      private BlockFile initExisting(RAIFile raf) throws IOException
      Read the info block of an existing database.
      Parameters:
      raf - open file of the existing database
      Returns:
      the database, with version, creation date and list names read
      Throws:
      IOException - if the header, its info, the list names or the version are missing, or the version is newer than this implementation supports

    • needsUpgrade

      private boolean needsUpgrade(BlockFile bf)
      Whether the database is in an old format.
      Parameters:
      bf - database being considered, only writable files can upgrade
      Returns:
      true if the version is old and the file is writable

    • upgrade

      private boolean upgrade()
      Blockfile must be writable of course.

      Version 1->2: Add reverse skiplist and populate Version 2->3: Re-populate reverse skiplist as version 2 didn't keep it updated after the upgrade. No change to format. Version 3->4: Change format to support multiple destinations per hostname.

      Returns:
      true if upgraded successfully

    • setVersion

      private void setVersion(String version) throws IOException
      Save new version number in blockfile after upgrade. Blockfile must be writable, of course. Side effect: sets _version field

      Caller must synchronize.

      Parameters:
      version - version string to record, e.g. "2", "3" or "4"
      Throws:
      IOException - if the header or its info block is missing

    • getEntry

      private BlockfileNamingService.DestEntry getEntry(String listname, String key) throws IOException
      For either v1 or v4. Caller must synchronize.
      Parameters:
      listname - skiplist to read, one of the configured hosts file names
      key - lower-case hostname to look up
      Returns:
      the stored entry, or null if the list or the name is absent
      Throws:
      IOException - if the database read fails, or a runtime error is wrapped into one

    • addEntry

      private void addEntry(BlockFile bf, String listname, String key, Destination dest, String source) throws IOException
      Caller must synchronize.
      Parameters:
      bf - database to write into, must be writable
      listname - skiplist the entry goes into, created if absent
      key - lower-case hostname to add
      dest - destination stored under the key
      source - how the entry was added, may be null
      Throws:
      IOException - if the index cannot be read or written, or a runtime error is wrapped into one

    • addEntry

      private static void addEntry(SkipList<String, BlockfileNamingService.DestEntry> sl, String key, Destination dest, Properties props)
      Single dest version. Caller must synchronize.
      Parameters:
      sl - skiplist to write into
      key - lower-case hostname, any previous entry is replaced
      dest - destination stored under the key
      props - per-entry properties such as add time and source, may be null
      Throws:
      RuntimeException - if the skiplist write fails

    • addEntry

      private static void addEntry(SkipList<String, BlockfileNamingService.DestEntry> sl, String key, List<Destination> dests, List<Properties> propsList)
      Multiple dests version. DB MUST be version 4. Caller must synchronize.
      Parameters:
      sl - skiplist to write into
      key - lower-case hostname, any previous entry is replaced
      dests - destinations for the name, the first is also stored as dest
      propsList - properties parallel to dests, may be null, or entries may be null
      Throws:
      RuntimeException - if the skiplist write fails

    • getFilenames

      private static List<String> getFilenames(String list)
      Returns the files backing one naming list.
      Parameters:
      list - the list name
      Returns:
      the backing file names

    • removeEntry

      private static <V> V removeEntry(SkipList<String,V> sl, String key)
      Caller must synchronize.
      Type Parameters:
      V - value type held by the skiplist
      Parameters:
      sl - skiplist to remove the key from
      key - lower-case hostname to remove
      Returns:
      removed object or null
      Throws:
      RuntimeException - if the skiplist write fails

    • getReverseEntries

      private List<String> getReverseEntries(Hash hash)
      Caller must synchronize. Returns null without exception on error (logs only). Returns without logging if no reverse skiplist (version 1).
      Parameters:
      hash - destination hash to look the names up by
      Returns:
      all hostnames found for the hash, or null if none or on error

    • addReverseEntry

      private void addReverseEntry(String key, Destination dest)
      Caller must synchronize. Fails without exception on error (logs only).
      Parameters:
      key - lower-case hostname to record against the hash
      dest - destination whose hash is the reverse index key

    • addReverseEntry

      private static void addReverseEntry(BlockFile bf, String key, Destination dest, Log log)
      Caller must synchronize. Fails without exception on error (logs only). Returns without logging if no reverse skiplist (version 1).

      We store one or more hostnames for a given hash. The skiplist key is a signed Integer, the first 4 bytes of the dest hash. For convenience (since we have a serializer already) we use a Properties as the value, with a null string as the value for each hostname property. We could in the future use the property value for something.

      Parameters:
      bf - database to write into, must be writable
      key - lower-case hostname to record against the hash
      dest - destination whose hash is the reverse index key
      log - where a failed write is logged, nothing is rethrown

    • removeReverseEntry

      private void removeReverseEntry(String key, Destination dest)
      Caller must synchronize. Fails without exception on error (logs only).
      Parameters:
      key - lower-case hostname to forget
      dest - destination whose hash is the reverse index key

    • getReverseKey

      private static Integer getReverseKey(Destination dest)
      Reverse lookup key for the destination.
      Parameters:
      dest - destination whose hash is keyed
      Returns:
      the first 4 hash bytes as a signed Integer

    • getReverseKey

      private static Integer getReverseKey(Hash hash)
      Reverse lookup key for the hash.
      Parameters:
      hash - destination hash to key on
      Returns:
      the first 4 hash bytes as a signed Integer

    • lookup

      public Destination lookup(String hostname, Properties lookupOptions, Properties storedOptions)
      Lookup a hostname. Will strip a "www." prefix and retry if lookup fails.
      Overrides:
      lookup in class DummyNamingService
      Parameters:
      hostname - upper/lower case ok
      lookupOptions - If non-null and contains the key "list", lookup in that list only, otherwise all lists
      storedOptions - output parameter, may be null. If non-null, the properties stored with the entry ("a" date added, "s" source, and any options given when the name was added) are added to it when the name is found. A non-null value also disables the Base32 cache, since the cache does not retain the stored properties.
      Returns:
      the destination, or null if not found

    • lookup2

      private Destination lookup2(String hostname, Properties lookupOptions, Properties storedOptions)
      Single dest version lookup.
      Parameters:
      hostname - name to look up, mixed case ok, lower-cased for the key
      lookupOptions - If non-null and contains the key "list", lookup in that list only, otherwise all lists
      storedOptions - If non-null, stored properties will be added
      Returns:
      the destination, or null if not found

    • lookupAll2

      private List<Destination> lookupAll2(String hostname, Properties lookupOptions, List<Properties> storedOptions)
      Multiple dests version lookup. DB MUST be version 4.
      Parameters:
      hostname - the host to look up, in one of the configured lists
      lookupOptions - If non-null and contains the key "list", lookup in that list only, otherwise all lists
      storedOptions - output list for stored properties, may be null
      Returns:
      non-null list of destinations, or null if not found

    • put

      public boolean put(String hostname, Destination d, Properties options)
      Store a destination, replacing any previous entry for the name.
      Overrides:
      put in class NamingService
      Parameters:
      hostname - the host name to map to the destination, overwriting any existing entry
      d - the destination for the hostname
      options - If non-null and contains the key "list", add to that list (default "hosts.txt") Use the key "s" for the source
      Returns:
      true if the entry was stored; this base implementation always returns false

    • putIfAbsent

      public boolean putIfAbsent(String hostname, Destination d, Properties options)
      Store a destination only if the name is unused.
      Overrides:
      putIfAbsent in class NamingService
      Parameters:
      hostname - the host name to map to the destination, which must not already be present
      d - the destination for the hostname
      options - If non-null and contains the key "list", add to that list (default "hosts.txt") Use the key "s" for the source. Key "a" will be added with the current time, unless "a" is present in options.
      Returns:
      true if no entry existed and the new one was stored; this base implementation always returns false

    • put

      private boolean put(String hostname, Destination d, Properties options, boolean checkExisting)
      Single dest version This does not prevent adding b32. Caller must check.
      Parameters:
      hostname - name to store, mixed case ok, lower-cased for the key
      d - destination to store
      options - If non-null and contains the key "list", add to that list (default "hosts.txt"). Use the key "s" for the source
      checkExisting - if true, fail if entry already exists
      Returns:
      true if the entry was written, false if the database is read-only or closed, the name was taken, or the write failed

    • put

      private boolean put(String hostname, List<Destination> dests, List<Properties> propsList, boolean checkExisting)
      Multiple dests version. DB MUST be version 4. This does not prevent adding b32. Caller must check.
      Parameters:
      hostname - name to store, mixed case ok, lower-cased for the key
      dests - destinations for the name, at least one is required
      propsList - may be null, or entries may be null
      checkExisting - if true, fail if entry already exists
      Returns:
      true if the entry was written, false if the database is read-only or closed, the name was taken, or the write failed
      Throws:
      IllegalArgumentException - if dests is empty

    • remove

      public boolean remove(String hostname, Properties options)
      Remove the named entry.
      Overrides:
      remove in class NamingService
      Parameters:
      hostname - the host name whose entry is to be deleted
      options - If non-null and contains the key "list", remove from that list (default "hosts.txt", NOT all lists)
      Returns:
      true if removed successfully, false on error or if it did not exist

    • getEntries

      public Map<String, Destination> getEntries(Properties options)
      All entries as destination objects.
      Overrides:
      getEntries in class NamingService
      Parameters:
      options - If non-null and contains the key "list", get from that list (default "hosts.txt", NOT all lists) Key "skip": skip that many entries Key "limit": max number to return Key "search": return only those matching substring Key "startsWith": return only those starting with ("[0-9]" allowed) Key "beginWith": start here in the iteration Don't use both startsWith and beginWith. Search, startsWith, and beginWith values must be lower case.
      Returns:
      the matching hostnames mapped to their destinations

    • getBase64Entries

      public Map<String,String> getBase64Entries(Properties options)
      All entries as Base64 strings.
      Overrides:
      getBase64Entries in class NamingService
      Parameters:
      options - If non-null and contains the key "list", get from that list (default "hosts.txt", NOT all lists) Key "skip": skip that many entries Key "limit": max number to return Key "search": return only those matching substring Key "startsWith": return only those starting with ("[0-9]" allowed) Key "beginWith": start here in the iteration Don't use both startsWith and beginWith. Search, startsWith, and beginWith values must be lower case.
      Returns:
      the base64 entries

    • export

      public void export(Writer out, Properties options) throws IOException
      Export in a hosts.txt format. Output is sorted. Caller must close writer.
      Overrides:
      export in class NamingService
      Parameters:
      out - the writer to export to
      options - If non-null and contains the key "list", get from that list (default "hosts.txt", NOT all lists) Key "search": return only those matching substring Key "startsWith": return only those starting with ("[0-9]" allowed) Key "beginWith": start here in the iteration
      Throws:
      IOException - if an I/O error occurs

    • getNames

      public Set<String> getNames(Properties options)
      Broken prior to 0.9.62, would only return one name.
      Overrides:
      getNames in class NamingService
      Parameters:
      options - If non-null and contains the key "list", get from that list (default "hosts.txt", NOT all lists) Key "skip": skip that many entries Key "limit": max number to return Key "search": return only those matching substring Key "startsWith": return only those starting with ("[0-9]" allowed) Key "beginWith": start here in the iteration Don't use both startsWith and beginWith. Search, startsWith, and beginWith values must be lower case.
      Returns:
      the hostnames matching the filters, never null

    • reverseLookup

      public String reverseLookup(Destination d, Properties options)
      Look up a hostname from a Destination's hash. Returns the first name found, or null.
      Overrides:
      reverseLookup in class NamingService
      Parameters:
      d - non-null destination to look up
      options - to configure lookup behavior (unused)
      Returns:
      hostname or null

    • reverseLookup

      public String reverseLookup(Hash h)
      Look up a hostname from a hash. Returns the first name found, or null.
      Overrides:
      reverseLookup in class NamingService
      Parameters:
      h - the hash to look up
      Returns:
      the first hostname, or null if none

    • reverseLookupAll

      public List<String> reverseLookupAll(Destination d, Properties options)
      Return all hostnames registered for this Destination's hash.
      Overrides:
      reverseLookupAll in class NamingService
      Parameters:
      d - non-null
      options - to configure lookup behavior (unused)
      Returns:
      a non-empty list of hostnames for this Destination, or null

    • reverseLookupAll

      public List<String> reverseLookupAll(Hash h)
      Return all hostnames registered for this hash.
      Overrides:
      reverseLookupAll in class NamingService
      Parameters:
      h - the hash to look up
      Returns:
      the list of hostnames, or null if none

    • size

      public int size(Properties options)
      Number of entries.
      Overrides:
      size in class NamingService
      Parameters:
      options - If non-null and contains the key "list", return the size of that list (default "hosts.txt", NOT all lists)
      Returns:
      number of entries (matching the options if non-null) or -1 if unknown

    • shutdown

      public void shutdown()
      Shut down the naming service and close the backing store.
      Overrides:
      shutdown in class NamingService

    • lookupAll

      public List<Destination> lookupAll(String hostname, Properties lookupOptions, List<Properties> storedOptions)
      Return all of the entries found in the first list found, or in the list specified in lookupOptions. Does not aggregate all destinations found in all lists.

      If storedOptions is non-null, it must be a List that supports null entries. If the returned value (the List of Destinations) is non-null, the same number of Properties objects will be added to storedOptions. If no properties were found for a given Destination, the corresponding entry in the storedOptions list will be null.

      Overrides:
      lookupAll in class NamingService
      Parameters:
      hostname - the host name to resolve, for which every known destination is returned
      lookupOptions - input parameter, NamingService-specific, may be null
      storedOptions - output parameter, NamingService-specific, any stored properties will be added if non-null
      Returns:
      non-empty List of Destinations, or null if nothing found

    • addDestination

      public boolean addDestination(String hostname, Destination d, Properties options)
      Add a Destination to an existing hostname's entry in the addressbook.

      This does not prevent adding b32. Caller must check.

      Overrides:
      addDestination in class NamingService
      Parameters:
      hostname - the host name whose existing entry is to be extended
      d - the destination to add
      options - NamingService-specific, may be null
      Returns:
      true if the destination was added to the entry

    • remove

      public boolean remove(String hostname, Destination d, Properties options)
      Remove a hostname's entry only if it contains the Destination d. If the NamingService supports multiple Destinations per hostname, and this is the only Destination, removes the entire entry. If aditional Destinations remain, it only removes the specified Destination from the entry.
      Overrides:
      remove in class NamingService
      Parameters:
      hostname - the host name whose entry holds the destination to remove
      d - the destination that must be present
      options - NamingService-specific, may be null
      Returns:
      true if entry containing d was successfully removed.

    • validate

      private boolean validate(String key, BlockfileNamingService.DestEntry de, String listname)
      Continuously validate anything we read in. Queue anything invalid to be removed at the end of the operation. Caller must sync!
      Parameters:
      key - hostname read from the skiplist, null or empty is invalid
      de - entry read from the skiplist, null is invalid
      listname - skiplist the entry came from, recorded for the cleanup
      Returns:
      true if the entry is well-formed and has a usable destination

    • deleteInvalid

      private void deleteInvalid()
      Remove and log all invalid entries queued by validate(String, DestEntry, String) while scanning in lookup(String, Properties, Properties) or getEntries(). We delete in the order detected, as an error may be corrupting later entries in the skiplist. Caller must sync!

    • close

      private void close()
      Closes the block file and stops the service.

    • logError

      private static void logError(String msg, Throwable t)
      For logging errors in the static serializers below.
      Parameters:
      msg - message logged at error level
      t - the failure being reported

    • writeProperties

      private static void writeProperties(ByteArrayOutputStream rawStream, Properties p) throws DataFormatException, IOException
      Same as DataHelper.writeProperties, UTF-8, unsorted, except that values may be up to 4K bytes.
      Parameters:
      rawStream - stream the length-prefixed mapping is written into
      p - source may be null
      Throws:
      DataFormatException - if any key string is over 255 bytes long, if any value string is over 4096 bytes long, or if the total length (not including the two length bytes) is greater than 65535 bytes.
      IOException - if the stream cannot be written

    • readProperties

      public static Properties readProperties(ByteArrayInputStream in) throws DataFormatException, IOException
      Same as DataHelper.readProperties, UTF-8, unsorted, except that values may be up to 4K bytes.
      Parameters:
      in - stream to read the mapping from
      Returns:
      the properties read, UTF-8 and unsorted
      Throws:
      DataFormatException - if the format is invalid, including a duplicate key
      IOException - if there is a problem reading the data

    • writeLongStringUTF8

      private static void writeLongStringUTF8(ByteArrayOutputStream out, String string) throws DataFormatException, IOException
      Same as DataHelper.writeStringUTF8, except that strings up to 4K bytes are allowed. Format is: one-byte length + data, or 0xff + two-byte length + data.
      Parameters:
      out - stream to write string
      string - to write out: null strings are valid, but strings of excess length will cause a DataFormatException to be thrown
      Throws:
      DataFormatException - if the string is not valid
      IOException - if there is an IO error writing the string

    • readLongString

      private static String readLongString(ByteArrayInputStream in) throws DataFormatException, IOException
      Same as DataHelper.readString, except that strings up to 4K bytes are allowed. Format is: one-byte length + data, or 0xff + two-byte length + data.
      Parameters:
      in - stream to read from
      Returns:
      UTF-8 string
      Throws:
      DataFormatException - if the stream doesn't contain a validly formatted string
      EOFException - if there aren't enough bytes to read the string
      IOException - if there is an IO error reading the string

    • main

      public static void main(String[] args)
      BlockfileNamingService [force] force = force writable.
      Parameters:
      args - when the first element is "force" the database is opened writable