java.lang.Object
net.metanotion.io.block.BlockFile
All Implemented Interfaces:
Closeable, AutoCloseable
public class BlockFileextends Object implements Closeable
Block-based file storage system with indexed access.

Provides paged file storage with skiplist indexes for efficient key-value data retrieval. Supports multiple named indexes within a single file.

On-disk format:

   Magic number (6 bytes)
   Version major/minor (2 bytes)
   file length (long)
   free list start (unsigned int)
   is mounted (unsigned short) 0 = no, 1 = yes
   span size (unsigned short)
   block size (unsigned int)

Metaindex skiplist is on page 2. Pages are 1 KB and are numbered starting from 1.

  • Field Summary

    Modifier and Type
    Field
    Description
    private boolean
    _isClosed
    True once the file is closed.
    private final boolean
    _wasMounted
    I2P was the file locked when we opened it?
    final RandomAccessInterface
    file
    The underlying file interface for this block file
    private long
    fileLen
    new BlockFile length, containing a superblock page and a metaindex page.
    private FreeListBlock
    flb
    cached list of free pages, only valid if freListStart > 0
    private int
    freeListStart
    First block of the free block list.
    final Log
    log
    Logger instance for this BlockFile
    private static final long
    MAGIC
    The magic number of the current format, version included.
    private static final long
    MAGIC_BASE
    The magic number with the version bits cleared.
    static final int
    MAGIC_CONT
    Magic number for continuation pages ("CONT")
    private long
    magicBytes
    The magic number read from or written to the file.
    private static final int
    MAJOR
    Major version of the file format written.
    private final BSkipList<String,Integer>
    metaIndex
    Index of the metadata records.
    static final int
    METAINDEX_PAGE
    Page number for the metaindex
    private static final int
    MIN_MAJOR
    Oldest major version that can be read.
    private static final int
    MIN_MINOR
    Oldest minor version that can be read.
    private static final int
    MINOR
    Minor version of the file format written.
    private int
    mounted
    True once the file header was read.
    static final long
    OFFSET_MOUNTED
    File offset where the mounted flag is stored
    private final HashMap<String, BSkipList>
    openIndices
    The indices kept open, by name.
    static final int
    PAGESIZE
    Page size in bytes for block file pages
    int
    spanSize
    Default span size for new skiplists

  • Constructor Summary

    Constructors
    Constructor
    Description
    BlockFile(File f, boolean init)
    Create a BlockFile from a File with optional initialization.
    BlockFile(RandomAccessFile raf)
    Create a BlockFile from a RandomAccessFile.
    BlockFile(RandomAccessFile raf, boolean init)
    Create a BlockFile from a RandomAccessFile with optional initialization.
    BlockFile(RandomAccessInterface rai)
    Use this constructor with a readonly RAI for a readonly blockfile.
    BlockFile(RandomAccessInterface rai, boolean init)
    Create a BlockFile from a RandomAccessInterface with optional initialization.

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    allocPage()
    Allocate a new page from the free list or extend the file.
    boolean
    bfck(boolean fix)
    Run an integrity check on the blockfile and all the skiplists in it.
    void
    close()
    Closes all open skiplists and then the blockfile itself.
    void
    closeIndex(String name)
    Close a skiplist if it is open.
    void
    delIndex(String name)
    Delete a skiplist if it exists.
    void
    freePage(int page)
    Add the page to the free list.
    <K extends Comparable<? super K>, V>
    BSkipList<K,V>
    getIndex(String name, Serializer<K> key, Serializer<V> val)
    Open a skiplist if it exists.
    static void
    main(String[] args)
    Run an integrity check on the blockfile and all the skiplists in it.
    <K extends Comparable<? super K>, V>
    BSkipList<K,V>
    makeIndex(String name, Serializer<K> key, Serializer<V> val)
    Create and open a new skiplist if it does not exist.
    private void
    mount()
    Reads the file header or writes a new one.
    static void
    pageSeek(RandomAccessInterface file, int page)
    Go to any page but the superblock.
    int
    readMultiPageData(byte[] arr, int page, int[] curPageOff, int[] nextPage)
    Read bytes from multiple pages.
    private void
    readSuperBlock()
    Reads and validates the file header block.
    <K extends Comparable<? super K>, V>
    void
    reformatIndex(String name, Serializer<K> oldKey, Serializer<V> oldVal, Serializer<K> newKey, Serializer<V> newVal)
    Reformat a skiplist with new Serializers if it exists.
    int
    skipMultiPageBytes(int length, int page, int[] curPageOff, int[] nextPage)
    Skip length bytes across multiple pages.
    boolean
    wasMounted()
    Check if the file was locked when opened.
    int
    writeMultiPageData(byte[] data, int page, int[] curPageOff, int[] nextPage)
    Write bytes to multiple pages.
    private void
    writeSuperBlock()
    Writes the file header block.

    Methods inherited from class Object

    clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • PAGESIZE

      public static final int PAGESIZE
      Page size in bytes for block file pages
      See Also:

    • OFFSET_MOUNTED

      public static final long OFFSET_MOUNTED
      File offset where the mounted flag is stored
      See Also:

    • log

      public final Log log
      Logger instance for this BlockFile

    • file

      public final RandomAccessInterface file
      The underlying file interface for this block file

    • MAJOR

      private static final int MAJOR
      Major version of the file format written.
      See Also:

    • MINOR

      private static final int MINOR
      Minor version of the file format written.
      See Also:

    • MIN_MAJOR

      private static final int MIN_MAJOR
      Oldest major version that can be read.
      See Also:

    • MIN_MINOR

      private static final int MIN_MINOR
      Oldest minor version that can be read.
      See Also:

    • MAGIC_BASE

      private static final long MAGIC_BASE
      The magic number with the version bits cleared. I2P changed the magic number and the format, the magic numbers are now on all pages.
      See Also:

    • MAGIC

      private static final long MAGIC
      The magic number of the current format, version included.
      See Also:

    • magicBytes

      private long magicBytes
      The magic number read from or written to the file.

    • MAGIC_CONT

      public static final int MAGIC_CONT
      Magic number for continuation pages ("CONT")
      See Also:

    • METAINDEX_PAGE

      public static final int METAINDEX_PAGE
      Page number for the metaindex
      See Also:

    • fileLen

      private long fileLen
      new BlockFile length, containing a superblock page and a metaindex page.

    • freeListStart

      private int freeListStart
      First block of the free block list.

    • mounted

      private int mounted
      True once the file header was read.

    • spanSize

      public int spanSize
      Default span size for new skiplists

    • _wasMounted

      private final boolean _wasMounted
      I2P was the file locked when we opened it?

    • metaIndex

      private final BSkipList<String,Integer> metaIndex
      Index of the metadata records.

    • _isClosed

      private boolean _isClosed
      True once the file is closed.

    • flb

      private FreeListBlock flb
      cached list of free pages, only valid if freListStart > 0

    • openIndices

      private final HashMap<String, BSkipList> openIndices
      The indices kept open, by name.

  • Constructor Details

    • BlockFile

      public BlockFile(RandomAccessInterface rai) throws IOException
      Use this constructor with a readonly RAI for a readonly blockfile.
      Parameters:
      rai - the RandomAccessInterface to use
      Throws:
      IOException - if an I/O error occurs

    • BlockFile

      public BlockFile(RandomAccessFile raf) throws IOException
      Create a BlockFile from a RandomAccessFile. RAF must be writable.
      Parameters:
      raf - the RandomAccessFile to use
      Throws:
      IOException - if an I/O error occurs

    • BlockFile

      public BlockFile(RandomAccessFile raf, boolean init) throws IOException
      Create a BlockFile from a RandomAccessFile with optional initialization. RAF must be writable.
      Parameters:
      raf - the RandomAccessFile to use
      init - if true, initialize the file with a new block structure
      Throws:
      IOException - if an I/O error occurs

    • BlockFile

      public BlockFile(File f, boolean init) throws IOException
      Create a BlockFile from a File with optional initialization. File must be writable.
      Parameters:
      f - the File to use
      init - if true, initialize the file with a new block structure
      Throws:
      IOException - if an I/O error occurs

    • BlockFile

      public BlockFile(RandomAccessInterface rai, boolean init) throws IOException
      Create a BlockFile from a RandomAccessInterface with optional initialization. Use this constructor with a readonly RAI and init = false for a readonly blockfile.
      Parameters:
      rai - the RandomAccessInterface to use
      init - if true, initialize the file with a new block structure
      Throws:
      IOException - if an I/O error occurs

  • Method Details

    • mount

      private void mount() throws IOException
      Reads the file header or writes a new one.
      Throws:
      IOException - if the header cannot be read

    • writeSuperBlock

      private void writeSuperBlock() throws IOException
      Writes the file header block.
      Throws:
      IOException - if the header cannot be written

    • readSuperBlock

      private void readSuperBlock() throws IOException
      Reads and validates the file header block.
      Throws:
      IOException - if the header is missing, short, or from a newer version

    • main

      public static void main(String[] args)
      Run an integrity check on the blockfile and all the skiplists in it. WARNING: This only works on skiplists using UTF8StringBytes as a key serializer, unless the exception has been coded in bfck below. Will CORRUPT other skiplists.
      Parameters:
      args - command line arguments (file path)

    • writeMultiPageData

      public int writeMultiPageData(byte[] data, int page, int[] curPageOff, int[] nextPage) throws IOException
      Write bytes to multiple pages. This will allocate additional continuation pages as necessary.
      Parameters:
      data - data to write
      page - index of the page to start writing at
      curPageOff - in (current) and out (new) parameter at index 0
      nextPage - in (current) and out (new) parameter at index 0
      Returns:
      index of the page the write finished on
      Throws:
      IOException - if an I/O error occurs

    • readMultiPageData

      public int readMultiPageData(byte[] arr, int page, int[] curPageOff, int[] nextPage) throws IOException
      Read bytes from multiple pages.
      Parameters:
      arr - fill this array fully with data
      page - index of the page to start reading at
      curPageOff - in (current) and out (new) parameter at index 0
      nextPage - in (current) and out (new) parameter at index 0
      Returns:
      index of the page the read finished on
      Throws:
      IOException - if an I/O error occurs or not enough pages to read

    • skipMultiPageBytes

      public int skipMultiPageBytes(int length, int page, int[] curPageOff, int[] nextPage) throws IOException
      Skip length bytes across multiple pages. The same as readMultiPageData(byte[], int, int[], int[]) without returning a result
      Parameters:
      length - number of bytes to skip
      page - index of the page the skip starts at
      curPageOff - in (current) and out (new) parameter at index 0
      nextPage - in (current) and out (new) parameter at index 0
      Returns:
      index of the page the skip finished on
      Throws:
      IOException - if an I/O error occurs or not enough pages to skip

    • wasMounted

      public boolean wasMounted()
      Check if the file was locked when opened.
      Returns:
      true if the file was mounted

    • pageSeek

      public static void pageSeek(RandomAccessInterface file, int page) throws IOException
      Go to any page but the superblock. Page 1 is the superblock, must use file.seek(0) to get there.
      Parameters:
      file - the RandomAccessInterface to seek in
      page - the page number (>= 2)
      Throws:
      IOException - if page is invalid

    • allocPage

      public int allocPage() throws IOException
      Allocate a new page from the free list or extend the file.
      Returns:
      the page number of the newly allocated page
      Throws:
      IOException - if an I/O error occurs

    • freePage

      public void freePage(int page)
      Add the page to the free list. The file is never shrunk. Does not throw exceptions; logs on failure.
      Parameters:
      page - the page number to free

    • getIndex

      public <K extends Comparable<? super K>, V>BSkipList<K,V> getIndex(String name, Serializer<K> key, Serializer<V> val) throws IOException
      Open a skiplist if it exists. Returns null if the skiplist does not exist. Empty skiplists are not preserved after close. If the file is writable, this runs an integrity check and repair on first open.
      Type Parameters:
      K - the type of keys in the skiplist
      V - the type of values in the skiplist
      Parameters:
      name - the skiplist's name within this blockfile
      key - the Serializer for keys
      val - the Serializer for values
      Returns:
      the BSkipList, or null if not found
      Throws:
      IOException - if an I/O error occurs

    • makeIndex

      public <K extends Comparable<? super K>, V>BSkipList<K,V> makeIndex(String name, Serializer<K> key, Serializer<V> val) throws IOException
      Create and open a new skiplist if it does not exist. Throws IOException if it already exists.
      Type Parameters:
      K - the type of keys in the skiplist
      V - the type of values in the skiplist
      Parameters:
      name - the skiplist's name within this blockfile
      key - the Serializer for keys
      val - the Serializer for values
      Returns:
      the newly created BSkipList
      Throws:
      IOException - if already exists or other errors

    • delIndex

      public void delIndex(String name) throws IOException
      Delete a skiplist if it exists. Must be open. Throws IOException if exists but is closed. Broken before 0.9.26.
      Parameters:
      name - the skiplist's name within this blockfile
      Throws:
      IOException - if it is closed.

    • closeIndex

      public void closeIndex(String name)
      Close a skiplist if it is open.
      Parameters:
      name - the skiplist's name within this blockfile

    • reformatIndex

      public <K extends Comparable<? super K>, V> void reformatIndex(String name, Serializer<K> oldKey, Serializer<V> oldVal, Serializer<K> newKey, Serializer<V> newVal) throws IOException
      Reformat a skiplist with new Serializers if it exists. The skiplist must be closed. Throws IOException if the skiplist is open. The skiplist will remain closed after completion.
      Type Parameters:
      K - the type of keys in the skiplist
      V - the type of values in the skiplist
      Parameters:
      name - the skiplist's name within this blockfile
      oldKey - the current Serializer for keys
      oldVal - the current Serializer for values
      newKey - the new Serializer for keys
      newVal - the new Serializer for values
      Throws:
      IOException - if it is open or on errors

    • close

      public void close() throws IOException
      Closes all open skiplists and then the blockfile itself. Note (I2P) Does NOT close the RAF / RAI.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • bfck

      public boolean bfck(boolean fix)
      Run an integrity check on the blockfile and all the skiplists in it.
      Parameters:
      fix - if true, attempt to fix any corruption found
      Returns:
      true if the levels were modified.