java.lang.Object
org.klomp.snark.Storage
All Implemented Interfaces:
Closeable, AutoCloseable
public class Storageextends Object implements Closeable
Storage manages torrent data files on disk for I2PSnark.

This class handles all aspects of storing and retrieving torrent pieces:

  • Creating and managing files/directories for torrent data
  • Reading and writing piece data to disk
  • Tracking downloaded pieces via BitField
  • Validating downloaded pieces against piece hashes
  • File pre-allocation for sparse file support
  • Priority-based file selection for partial downloads
  • File checking and rechecking for data integrity

Storage implements Closeable and should be properly closed after use. The storage must be checked via check() before reading or writing pieces.

Thread safety: This class uses synchronized methods and concurrent collections for thread-safe access to file handles and metadata.

See Also:

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    private final class 
    Storage.FileCursor
    Cursor over the torrent's files, mapping an absolute byte position in the torrent to the containing file and the offset within it.
    private class 
    Storage.TorrentFile
    A single file in a torrent.

  • Field Summary

    Modifier and Type
    Field
    Description
    private final AtomicLong
    _activity
    When storage was last read or written.
    private final AtomicInteger
    _allocateCount
    Files whose allocation is in progress.
    private final File
    _base
    Directory the torrent's files live under.
    private static final ByteCache
    _cache
    Pooled buffers for piece data.
    private final AtomicInteger
    _checkProgress
    Pieces verified by the running hash check.
    private static final Semaphore
    _checkSemaphore
    Limits how many torrents hash-check at once.
    private final Set<Storage.TorrentFile>
    _copying
    Files currently being copied from the staging directory to the data directory, to prevent concurrent duplicate copies (putPiece may be called concurrently).
    private List<String>
    _filesExcluded
    File names excluded from the download.
    private static final Map<String,String>
    _filterNameCache
    Cached sanitized file names.
    private static final boolean
    _isARM
    True when running on ARM.
    private boolean
    _isChecking
    Whether a hash check is running.
    private static final boolean
    _isWindows
    True when running on Windows.
    private static final Log
    _log
    Logger for file and piece operations.
    private final boolean
    _preserveFileNames
    Whether the torrent's file names are kept as-is.
    private boolean
    _probablyComplete
    Whether files may be opened read-only.
    private final File
    _stagingBase
    Directory where incomplete files are written while downloading, when the i2psnark.tempDir property is configured.
    private final List<Storage.TorrentFile>
    _torrentFiles
    The torrent's files, in piece order.
    private List<String>
    _unreadableFiles
    Files or folders that exist but could not be read (e.g.
    private final I2PSnarkUtil
    _util
    The webapp's configuration helper.
    private BitField
    bitfield
    Which pieces are verified as present.
    private static final int
    BUFSIZE
    Buffer size for piece reads and writes.
    private boolean
    changed
    Whether the verified-piece state changed since saving.
    private static final long
    COPY_RETRY_DELAY
    Delay in milliseconds between copy attempts.
    private static final int
    DEFAULT_PIECE_SIZE
    The default piece size for new torrents.
    private static final int
    DEFAULT_VERIFY_THREADS
    Number of worker threads that verify pieces in parallel during a full recheck; scaled to the CPU count, at least 4.
    private static final char[]
    ILLEGAL
    Characters illegal in file names.
    private final StorageListener
    listener
    Notified of progress and errors.
    private static final int
    MAX_CONCURRENT_CHECKS
    Cap on simultaneous storage checks across all torrents; prevents a disk I/O storm when many torrents start or are rechecked at the same time.
    private static final int
    MAX_COPY_RETRIES
    Number of attempts to copy a completed file out of the staging directory.
    static final int
    MAX_PIECE_SIZE
    Maximum permitted piece size.
    static final int
    MAX_PIECES
    The maximum number of pieces allowed in a single torrent.
    static final long
    MAX_TOTAL_SIZE
    Maximum total torrent size (MAX_PIECE_SIZE * MAX_PIECES).
    private final MetaInfo
    metainfo
    The torrent's metadata.
    private int
    needed
    Pieces still missing from the torrent.
    (package private) static final String
    PAD_DIR
    BEP 47 directory holding synthetic padding files; skipped when walking a source dir.
    private final int
    piece_size
    Piece size in bytes.
    private final int
    pieces
    Number of pieces in the torrent.
    static final int
    PRIORITY_NORMAL
    Default priority value for files.
    static final int
    PRIORITY_SKIP
    Priority value indicating a file should be skipped during download.
    private static final long
    RAF_CLOSE_DELAY
    A file handle is closed after 4 minutes idle.
    private final long
    total_length
    Total bytes across all files.
    private static final int
    VERIFY_BUFSIZE
    Cap on the buffer used for piece verification during a full recheck; checking never allocates a whole piece-sized buffer (piece_size can be 64MB+ on large-piece torrents).
    private static final String[]
    WIN_ILLEGAL
    Names Windows reserves for devices.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Storage(I2PSnarkUtil util, File baseFile, String announce, List<List<String>> announce_list, String created_by, boolean privateTorrent, List<String> url_list, String comment, StorageListener listener, List<TorrentCreateFilter> filters)
    Creates a storage from the existing file or directory.
    Storage(I2PSnarkUtil util, File baseFile, String announce, List<List<String>> announce_list, String created_by, boolean privateTorrent, StorageListener listener, List<TorrentCreateFilter> filters)
    Creates a storage from the existing file or directory.
    Storage(I2PSnarkUtil util, File baseFile, MetaInfo metainfo, StorageListener listener, boolean preserveFileNames)
    Creates a new storage based on the supplied MetaInfo.

  • Method Summary

    Modifier and Type
    Method
    Description
    private void
    addFiles(List<File> l, File f, List<TorrentCreateFilter> filters)
    Count the files in the torrent.
    private void
    addUnreadable(File f, String reason)
    Record a file or folder that exists but could not be read, and warn.
    private void
    allocateFile(Storage.TorrentFile tf)
    Extend one torrent file to its full length and report it to the listener, so that reads don't fail with an IOE before the piece arrives.
    private File
    buildFilePath(File base, List<String> names)
    Builds a file path under the given base from filtered path elements, without creating anything on disk.
    void
    check()
    Creates (and/or checks) all files from the metainfo file list.
    void
    check(long savedTime, BitField savedBitField)
    Creates (and/or checks) all files from the metainfo file list.
    private boolean
    checkCreateFiles(boolean recheck)
    This is called at the beginning, and at presumed completion, so we have to be careful about locking.
    private boolean
    checkCreateFiles(boolean recheck, boolean[] pieceTrusted)
    Variant that skips hashing pieces marked trusted, for resuming with a partially trusted saved state.
    private void
    checkFileCompletions()
    Moves any file whose pieces are all downloaded out of the staging directory into the data directory.
    private boolean
    checkPieceHash(int piece, byte[] buf, int length)
    Verify a piece's hash without a whole piece-sized buffer: reads the piece in windows no larger than the supplied buffer, feeding each window to the SHA1 digest before comparing against the stored hash.
    void
    cleanRAFs()
    Close unused RAFs - call periodically.
    (package private) void
    clearChanged()
    Clear the storage changed variable.
    void
    close()
    Closes the Storage and makes sure that all RandomAccessFiles are closed.
    private void
    closeAllRAFs()
    Close every open file handle, whatever its last-use time.
    boolean
    complete()
    Whether or not this storage contains all pieces if the MetaInfo.
    private boolean[]
    computeTrustedPieces(List<Boolean> fileTrusted)
    For a multi-file torrent with some files trusted from a saved state, marks the pieces that lie entirely within a trusted file.
    private boolean
    copyFile(File src, File dst)
    Copies a file's contents to the destination, truncating the destination first.
    private boolean
    copyOnce(Storage.TorrentFile tf)
    Copies the file from the staging directory to the data directory once.
    private void
    copyToFinal(Storage.TorrentFile tf)
    Copies the file to the data directory with a bounded number of retries in case the destination is temporarily unavailable.
    (package private) static int
    countNonPad(File[] files)
    Count the entries in a directory listing for the max files per torrent check, excluding padding directories.
    private File
    createFileFromNames(File base, List<String> names, boolean areFilesPublic)
    Note that filtering each path element individually may lead to things going in the wrong place if there are duplicates in intermediate path elements after filtering.
    void
    deleteStagingData()
    Removes all partially downloaded data from the staging directory for this torrent.
    private byte[]
    fast_digestCreate()
    Creates piece hashes for a new storage.
    private boolean
    fileComplete(long start, long end)
    Whether all pieces overlapping the byte range [start, end) are set in the bitfield.
    static String
    filterName(String name)
    Removes 'suspicious' characters from the given file name.
    long
    getActivity()
    The last activity timestamp.
    File
    getBase()
    The base file or directory.
    String
    getBaseName()
    The base file or directory name of the data, as specified in the .torrent file, but filtered to remove illegal characters.
    BitField
    getBitField()
    The BitField that tells which pieces this storage contains.
    double
    getCheckingProgress()
    If checking is in progress, return completion 0.0 ...
    SortedSet<File>
    getDirectories()
    Includes the base for a multi-file torrent.
    List<String>
    getExcludedFiles(File base)
    The excluded files.
    int
    getFileCount()
    Does not include directories.
    List<String>
    getFileNames()
    The torrent-relative paths of every file in the torrent, index-aligned with the arrays returned by remaining2().
    int[]
    getFilePriorities()
    The file priorities array.
    List<File>
    getFiles()
    Does not include directories.
    private List<Storage.TorrentFile>
    getFiles(File base, List<TorrentCreateFilter> filters)
    Lists the files on disk matching the torrent.
    private String
    getFolderHierarchy(Path path, Path base)
    Returns a file's path relative to the base directory.
    MetaInfo
    getMetaInfo()
    Returns the MetaInfo associated with this Storage.
    ByteArray
    getPiece(int piece, int off, int len)
    Returns a byte array containing a portion of the requested piece or null if the storage doesn't contain the piece yet.
    private int
    getPieceLength(int piece)
    This is a dup of MetaInfo.getPieceLength(int) but we need it before the MetaInfo is created in our second constructor.
    int[]
    getPiecePriorities()
    Call setPriority(int, int) for all changed files first, then call this.
    boolean
    getPreserveFileNames()
    Whether original file names are being preserved, false if filtered.
    int
    getPriority(int fileIndex)
    The priority of the file at the index.
    long
    getSkippedLength()
    Call setPriority() for all changed files first, then call this.
    (package private) static File
    getStagingBase(String tempDir, String baseName, MetaInfo metainfo)
    The staging directory for a torrent: the configured temp dir plus a subdirectory unique to the torrent.
    File
    getStagingDir()
    The directory this torrent's incomplete files are staged in while downloading, or null when the staging feature is disabled.
    private int
    getUncheckedPiece(int piece, byte[] bs)
    Reads a piece without checking its hash.
    private int
    getUncheckedPiece(int piece, byte[] bs, int off, int length)
    Reads part of a piece without checking its hash.
    private int
    getUncheckedPiece(int piece, byte[] bs, int off, int length, boolean closeAfterRead)
    Read a piece, or part of one, into the buffer.
    (package private) static int
    getVerifyThreads(I2PAppContext ctx)
    Worker threads for a parallel piece verification: the i2psnark.verifyThreads property when set, otherwise scaled to the CPU count (at least 4).
    int
    indexOf(File file)
    Index to pass to remaining(), getPriority(), setPriority().
    boolean
    isAllocating()
    Disk allocation (ballooning) in progress.
    boolean
    isChanged()
    Has the storage changed since instantiation?
    boolean
    isChecking()
    File checking in progress.
    private boolean
    isPaddingPiece(long pieceStart)
    BEP 47: true if the piece lies entirely within padding files, so it hashes as zeros.
    (package private) static boolean
    isPadDir(String name)
    Is this a padding directory (BEP 47 ".pad" or a parse-renamed "_pad")? Padding entries hold only synthetic zero files: they must not count toward the max files per torrent limit, nor be included as data when creating a torrent.
    File[]
    listMerged(String pathInTorrent)
    Lists a directory within the torrent, merging the staging tree while the download is incomplete, so completed files already moved to the data directory and files still staged in the temp dir both appear.
    private boolean
    locked_checkCreateFiles(boolean recheck, boolean[] pieceTrusted)
    Check the pieces, reporting whether anything changed.
    static void
    main(String[] args)
    Create a metainfo.
    int
    needed()
    How many pieces are still missing from this storage.
    (package private) static String
    noDataMessage(File base, List<String> unreadable)
    The message for a failed torrent creation with no readable data, so a permission problem is reported as such rather than as an empty folder.
    private String
    optFilterName(String name)
    Filter the name, but only if configured to do so.
    boolean
    putPiece(PartialPiece pp)
    Put the piece in the Storage if it is correct.
    boolean
    recheck()
    Blocking.
    private void
    recreateFile(Storage.TorrentFile tf)
    Recreates a missing file at its active location (the staging dir while incomplete, the data dir otherwise).
    long[]
    remaining()
    For efficiency, calculate remaining bytes for all files at once.
    long[][]
    remaining2()
    For efficiency, calculate remaining bytes for all files at once.
    void
    reopen()
    Doesn't really reopen the file descriptors for a restart.
    private void
    setActivity()
    Mark the activity timestamp as now.
    void
    setActivity(long time)
    Store the activity timestamp.
    (package private) void
    setFilePriorities(int[] p)
    The file priorities array.
    void
    setPriority(int fileIndex, int pri)
    Must call Snark.updatePiecePriorities() (which calls getPiecePriorities()) after calling this.
    private void
    sleep()
    Delays between copy attempts; returns immediately on interruption.
    private void
    startCopy(Storage.TorrentFile tf)
    Starts copying a completed file from the staging directory to the data directory on a background thread, unless a copy for the file is already running.
    private void
    writePiece(PartialPiece pp, int piece, boolean shouldPreallocate, boolean forceRW)
    Write a (hash-verified) partial piece to the data files.

    Methods inherited from class Object

    clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • metainfo

      private final MetaInfo metainfo
      The torrent's metadata.

    • _torrentFiles

      private final List<Storage.TorrentFile> _torrentFiles
      The torrent's files, in piece order.

    • _base

      private final File _base
      Directory the torrent's files live under.

    • listener

      private final StorageListener listener
      Notified of progress and errors.

    • _util

      private final I2PSnarkUtil _util
      The webapp's configuration helper.

    • _log

      private static final Log _log
      Logger for file and piece operations.

    • bitfield

      private BitField bitfield
      Which pieces are verified as present.

    • needed

      private int needed
      Pieces still missing from the torrent.

    • _probablyComplete

      private boolean _probablyComplete
      Whether files may be opened read-only.

    • piece_size

      private final int piece_size
      Piece size in bytes.

    • pieces

      private final int pieces
      Number of pieces in the torrent.

    • total_length

      private final long total_length
      Total bytes across all files.

    • _preserveFileNames

      private final boolean _preserveFileNames
      Whether the torrent's file names are kept as-is.

    • changed

      private boolean changed
      Whether the verified-piece state changed since saving.

    • _isChecking

      private volatile boolean _isChecking
      Whether a hash check is running.

    • _allocateCount

      private final AtomicInteger _allocateCount
      Files whose allocation is in progress.

    • _checkProgress

      private final AtomicInteger _checkProgress
      Pieces verified by the running hash check.

    • _activity

      private final AtomicLong _activity
      When storage was last read or written.

    • _filesExcluded

      private List<String> _filesExcluded
      File names excluded from the download.

    • _unreadableFiles

      private List<String> _unreadableFiles
      Files or folders that exist but could not be read (e.g. permissions).

    • _stagingBase

      private final File _stagingBase
      Directory where incomplete files are written while downloading, when the i2psnark.tempDir property is configured. Files are moved to the data directory only when complete. Null when the feature is disabled.

    • _copying

      private final Set<Storage.TorrentFile> _copying
      Files currently being copied from the staging directory to the data directory, to prevent concurrent duplicate copies (putPiece may be called concurrently).

    • PAD_DIR

      static final String PAD_DIR
      BEP 47 directory holding synthetic padding files; skipped when walking a source dir.
      See Also:

    • MAX_PIECE_SIZE

      public static final int MAX_PIECE_SIZE
      Maximum permitted piece size. Torrents with pieces larger than this will be rejected.
      See Also:

    • MAX_PIECES

      public static final int MAX_PIECES
      The maximum number of pieces allowed in a single torrent.
      See Also:

    • MAX_TOTAL_SIZE

      public static final long MAX_TOTAL_SIZE
      Maximum total torrent size (MAX_PIECE_SIZE * MAX_PIECES).
      See Also:

    • PRIORITY_SKIP

      public static final int PRIORITY_SKIP
      Priority value indicating a file should be skipped during download.
      See Also:

    • PRIORITY_NORMAL

      public static final int PRIORITY_NORMAL
      Default priority value for files.
      See Also:

    • _filterNameCache

      private static final Map<String,String> _filterNameCache
      Cached sanitized file names.

    • _isWindows

      private static final boolean _isWindows
      True when running on Windows.

    • _isARM

      private static final boolean _isARM
      True when running on ARM.

    • BUFSIZE

      private static final int BUFSIZE
      Buffer size for piece reads and writes.

    • _cache

      private static final ByteCache _cache
      Pooled buffers for piece data.

    • VERIFY_BUFSIZE

      private static final int VERIFY_BUFSIZE
      Cap on the buffer used for piece verification during a full recheck; checking never allocates a whole piece-sized buffer (piece_size can be 64MB+ on large-piece torrents).
      See Also:

    • DEFAULT_VERIFY_THREADS

      private static final int DEFAULT_VERIFY_THREADS
      Number of worker threads that verify pieces in parallel during a full recheck; scaled to the CPU count, at least 4. Override with the i2psnark.verifyThreads property.

    • MAX_CONCURRENT_CHECKS

      private static final int MAX_CONCURRENT_CHECKS
      Cap on simultaneous storage checks across all torrents; prevents a disk I/O storm when many torrents start or are rechecked at the same time. Override with the i2psnark.maxConcurrentChecks property (read once at class load; restart to change).

    • _checkSemaphore

      private static final Semaphore _checkSemaphore
      Limits how many torrents hash-check at once.

    • DEFAULT_PIECE_SIZE

      private static final int DEFAULT_PIECE_SIZE
      The default piece size for new torrents.
      See Also:

    • MAX_COPY_RETRIES

      private static final int MAX_COPY_RETRIES
      Number of attempts to copy a completed file out of the staging directory.
      See Also:

    • COPY_RETRY_DELAY

      private static final long COPY_RETRY_DELAY
      Delay in milliseconds between copy attempts.
      See Also:

    • ILLEGAL

      private static final char[] ILLEGAL
      Characters illegal in file names.

    • WIN_ILLEGAL

      private static final String[] WIN_ILLEGAL
      Names Windows reserves for devices. See also: Windows file naming

    • RAF_CLOSE_DELAY

      private static final long RAF_CLOSE_DELAY
      A file handle is closed after 4 minutes idle.
      See Also:

  • Constructor Details

    • Storage

      public Storage(I2PSnarkUtil util, File baseFile, MetaInfo metainfo, StorageListener listener, boolean preserveFileNames)
      Creates a new storage based on the supplied MetaInfo.

      Does not check storage. Caller MUST call check(), which will try to create and/or check all needed files in the MetaInfo.

      Parameters:
      util - the wrapper for general configuration and torrent filtering
      baseFile - the torrent data file or dir
      metainfo - the parsed metainfo describing the torrent's pieces and files
      listener - notified of storage events such as completion, may be null
      preserveFileNames - if true, do not remap names to a 'safe' charset

    • Storage

      public Storage(I2PSnarkUtil util, File baseFile, String announce, List<List<String>> announce_list, String created_by, boolean privateTorrent, StorageListener listener, List<TorrentCreateFilter> filters) throws IOException
      Creates a storage from the existing file or directory. Creates an in-memory metainfo but does not save it to a file, caller must do that.

      Creates the metainfo, this may take a LONG time. BLOCKING.

      Parameters:
      util - the wrapper for general configuration and torrent filtering
      baseFile - the torrent data file or dir
      announce - the announce URL to record in the metainfo, may be null
      announce_list - additional announce URLs, may be null
      created_by - the metainfo's "created by" string, may be null
      privateTorrent - if true, set the metainfo private flag
      listener - notified of storage events such as completion, may be null
      filters - the filters selecting which files are stored, may be null
      Throws:
      IOException - when creating and/or checking files fails.

    • Storage

      public Storage(I2PSnarkUtil util, File baseFile, String announce, List<List<String>> announce_list, String created_by, boolean privateTorrent, List<String> url_list, String comment, StorageListener listener, List<TorrentCreateFilter> filters) throws IOException
      Creates a storage from the existing file or directory. Creates an in-memory metainfo but does not save it to a file, caller must do that.

      Creates the metainfo, this may take a LONG time. BLOCKING.

      Parameters:
      util - the wrapper for general configuration and torrent filtering
      baseFile - the torrent data file or dir
      announce - the announce URL to record in the metainfo, may be null
      announce_list - additional announce URLs, may be null
      created_by - the metainfo's "created by" string, may be null
      privateTorrent - if true, set the metainfo private flag
      url_list - web seed URLs, may be null
      comment - the metainfo comment, may be null
      listener - notified of storage events such as completion, may be null
      filters - the filters selecting which files are stored, may be null
      Throws:
      IOException - when creating and/or checking files fails.

  • Method Details

    • getStagingBase

      static File getStagingBase(String tempDir, String baseName, MetaInfo metainfo)
      The staging directory for a torrent: the configured temp dir plus a subdirectory unique to the torrent. Null when the feature is disabled or the infohash is unavailable.
      Parameters:
      tempDir - the configured staging directory, or null
      baseName - the torrent's base name
      metainfo - the torrent supplying the info hash that names the subdirectory
      Returns:
      the per-torrent staging dir, or null

    • fast_digestCreate

      private byte[] fast_digestCreate() throws IOException
      Creates piece hashes for a new storage. This does NOT create the files, just the hashes. Also sets all the bitfield bits.

      Hashing walks the torrent from end to end, so a file handle is closed as soon as the piece that read it is done, rather than held until cleanRAFs(): leaving every handle open for the duration exhausts the process file descriptor limit on torrents with many files, and there is no periodic sweep running yet because the Storage has not been handed to a PeerCheckerTask.

      Returns:
      the concatenated 20-byte SHA-1 hash of every piece
      Throws:
      IOException - if a file cannot be read while hashing

    • closeAllRAFs

      private void closeAllRAFs()
      Close every open file handle, whatever its last-use time. Used by the creation-time hash pass, which runs before any periodic cleanRAFs() exists.

    • getFiles

      private List<Storage.TorrentFile> getFiles(File base, List<TorrentCreateFilter> filters) throws IOException
      Lists the files on disk matching the torrent.
      Parameters:
      base - the base directory
      filters - the filters selecting the files
      Returns:
      the matching files
      Throws:
      IOException - if the directory cannot be read

    • addUnreadable

      private void addUnreadable(File f, String reason)
      Record a file or folder that exists but could not be read, and warn. The creation proceeds with the readable data; if there is none, the caller fails with noDataMessage(File, List) naming the unreadable paths.
      Parameters:
      f - the unreadable file or folder
      reason - a short human-readable reason

    • noDataMessage

      static String noDataMessage(File base, List<String> unreadable)
      The message for a failed torrent creation with no readable data, so a permission problem is reported as such rather than as an empty folder.
      Parameters:
      base - the data folder or file
      unreadable - paths that exist but could not be read, may be empty
      Returns:
      a user-facing error string; names the unreadable paths when there are any

    • getExcludedFiles

      public List<String> getExcludedFiles(File base)
      The excluded files.
      Parameters:
      base - the torrent data dir the excluded paths are reported relative to
      Returns:
      comma-separated names of excluded files and directories

    • getFolderHierarchy

      private String getFolderHierarchy(Path path, Path base)
      Returns a file's path relative to the base directory.
      Parameters:
      path - the file path
      base - the base directory
      Returns:
      the folder path with forward slashes

    • isPadDir

      static boolean isPadDir(String name)
      Is this a padding directory (BEP 47 ".pad" or a parse-renamed "_pad")? Padding entries hold only synthetic zero files: they must not count toward the max files per torrent limit, nor be included as data when creating a torrent.
      Parameters:
      name - directory name
      Returns:
      true if it is a padding directory

    • countNonPad

      static int countNonPad(File[] files)
      Count the entries in a directory listing for the max files per torrent check, excluding padding directories.
      Parameters:
      files - directory listing, may be null
      Returns:
      the number of non-padding entries

    • addFiles

      private void addFiles(List<File> l, File f, List<TorrentCreateFilter> filters) throws IOException
      Count the files in the torrent.
      Parameters:
      l - the list the discovered files are added to
      f - the file or directory to walk
      filters - the torrent-create filters applied to each name
      Throws:
      IOException - if too many total files

    • getMetaInfo

      public MetaInfo getMetaInfo()
      Returns the MetaInfo associated with this Storage.
      Returns:
      the metainfo describing this torrent

    • needed

      public int needed()
      How many pieces are still missing from this storage.
      Returns:
      the count of pieces not yet present on disk

    • complete

      public boolean complete()
      Whether or not this storage contains all pieces if the MetaInfo.
      Returns:
      true if no pieces are still needed

    • isChanged

      public boolean isChanged()
      Has the storage changed since instantiation?
      Returns:
      true once any setter has flagged the storage as needing a save

    • clearChanged

      void clearChanged()
      Clear the storage changed variable.

    • getActivity

      public long getActivity()
      The last activity timestamp.
      Returns:
      timestamp in milliseconds since epoch, or 0 if never set

    • setActivity

      private void setActivity()
      Mark the activity timestamp as now.

    • setActivity

      public void setActivity(long time)
      Store the activity timestamp.
      Parameters:
      time - timestamp in milliseconds since epoch

    • isChecking

      public boolean isChecking()
      File checking in progress.
      Returns:
      true while checkCreateFiles(boolean) holds the check, cleared in a finally

    • getCheckingProgress

      public double getCheckingProgress()
      If checking is in progress, return completion 0.0 ... 1.0, else return 1.0.
      Returns:
      the checking progress

    • isAllocating

      public boolean isAllocating()
      Disk allocation (ballooning) in progress. Always false on Windows.
      Returns:
      true while at least one file is still ballooned

    • indexOf

      public int indexOf(File file)
      Index to pass to remaining(), getPriority(), setPriority().
      Parameters:
      file - non-canonical path (non-directory)
      Returns:
      internal index of file; -1 if unknown file

    • remaining

      public long[] remaining()
      For efficiency, calculate remaining bytes for all files at once.
      Returns:
      number of bytes remaining for each file, use indexOf(File) to get index for a file

    • getFileNames

      public List<String> getFileNames()
      The torrent-relative paths of every file in the torrent, index-aligned with the arrays returned by remaining2().

      BEP 47 padding files are included, as the synthetic name (a number, optionally "-n", under the .pad directory) the metainfo gives them, so that element i of this list describes the same file as element i of the remaining/preview arrays. Callers that must skip padding entries test the name for the .pad directory prefix, or use indexOf(File) and the metainfo attribute list.

      Returns:
      a new list with one entry per torrent file, never null

    • remaining2

      public long[][] remaining2()
      For efficiency, calculate remaining bytes for all files at once. Remaining bytes is rv[0]. Preview bytes is rv[1].
      Returns:
      number of bytes remaining and number of bytes available for a preview for each file, use indexOf() to get index for a file

    • getPriority

      public int getPriority(int fileIndex)
      The priority of the file at the index.
      Parameters:
      fileIndex - as obtained from indexOf
      Returns:
      a PRIORITY_* constant, or normal when the torrent is complete or the index is out of range

    • setPriority

      public void setPriority(int fileIndex, int pri)
      Must call Snark.updatePiecePriorities() (which calls getPiecePriorities()) after calling this.
      Parameters:
      fileIndex - as obtained from indexOf
      pri - default 0; <0 to disable

    • getFilePriorities

      public int[] getFilePriorities()
      The file priorities array.
      Returns:
      null on error, if complete, or if only one file

    • setFilePriorities

      void setFilePriorities(int[] p)
      The file priorities array. Only call this when stopped, but after check().
      Parameters:
      p - may be null

    • getPiecePriorities

      public int[] getPiecePriorities()
      Call setPriority(int, int) for all changed files first, then call this. Set the piece priority to the highest priority of all files spanning the piece. Caller must pass array to the PeerCoordinator.
      Returns:
      null on error, if complete, or if only one file and inOrder not set.

    • getSkippedLength

      public long getSkippedLength()
      Call setPriority() for all changed files first, then call this. The length of all the pieces that are not yet downloaded, and are set to skipped. This is not the same as the total of all skipped files, since pieces may span multiple files.
      Returns:
      0 on error, if complete, or if only one file

    • getBitField

      public BitField getBitField()
      The BitField that tells which pieces this storage contains. Do not change this since this is the current state of the storage.
      Returns:
      the bit field

    • getBaseName

      public String getBaseName()
      The base file or directory name of the data, as specified in the .torrent file, but filtered to remove illegal characters. This is where the data actually is, relative to the snark base dir.
      Returns:
      the base name

    • getPreserveFileNames

      public boolean getPreserveFileNames()
      Whether original file names are being preserved, false if filtered.
      Returns:
      true if original file names are being preserved, false if filtered

    • check

      public void check() throws IOException
      Creates (and/or checks) all files from the metainfo file list. Only call this once, and only after the constructor with the metainfo. Use recheck() to check again later.
      Throws:
      IllegalStateException - if called more than once
      IOException - if a required file or directory cannot be created or deleted

    • check

      public void check(long savedTime, BitField savedBitField) throws IOException
      Creates (and/or checks) all files from the metainfo file list. Use a saved bitfield and timestamp from the metadata file. Only call this once, and only after the constructor with the metainfo. Use recheck() to check again later.
      Parameters:
      savedTime - the creation time of the metadata file in milliseconds, or 0 to ignore
      savedBitField - the completion state loaded from the metadata file, or null to ignore
      Throws:
      IllegalStateException - if called more than once
      IOException - if a required file or directory cannot be created or deleted, or the on-disk file lengths do not sum to the length in the metainfo

    • computeTrustedPieces

      private boolean[] computeTrustedPieces(List<Boolean> fileTrusted)
      For a multi-file torrent with some files trusted from a saved state, marks the pieces that lie entirely within a trusted file. A piece overlapping any changed file must be re-verified.
      Parameters:
      fileTrusted - one entry per file, in metainfo order
      Returns:
      array indexed by piece, true if the piece may be trusted without re-hashing

    • reopen

      public void reopen() throws IOException
      Doesn't really reopen the file descriptors for a restart. Just does an existence check but no length check or data reverification.
      Throws:
      IOException - if the storage was never checked, or a vanished data file cannot be recreated

    • optFilterName

      private String optFilterName(String name)
      Filter the name, but only if configured to do so. We will do so on torrents received from others, but not on those we created ourselves, so we do not lose track of files.
      Parameters:
      name - one unfiltered path element
      Returns:
      the filtered name, or name unchanged when file-name filtering is off

    • filterName

      public static String filterName(String name)
      Removes 'suspicious' characters from the given file name. http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx Then replace chars not supported in the charset.

      This is called frequently and it can be pretty slow so cache the result. will blow up. Check at torrent creation?

      Parameters:
      name - the file name as listed in the torrent's metainfo
      Returns:
      the name with suspicious characters replaced by underscores

    • createFileFromNames

      private File createFileFromNames(File base, List<String> names, boolean areFilesPublic) throws IOException
      Note that filtering each path element individually may lead to things going in the wrong place if there are duplicates in intermediate path elements after filtering.
      Parameters:
      base - the directory the path is created under
      names - the unfiltered path elements, each passed through the name filter in turn
      areFilesPublic - whether to create plain files, readable by other users, rather than private ones
      Returns:
      the file at the end of the created path
      Throws:
      IOException - if a directory or file in the path cannot be created

    • buildFilePath

      private File buildFilePath(File base, List<String> names)
      Builds a file path under the given base from filtered path elements, without creating anything on disk.
      Parameters:
      base - the directory the path is built under
      names - the unfiltered path elements, each passed through the name filter in turn
      Returns:
      the composed File, which does not exist on disk

    • recreateFile

      private void recreateFile(Storage.TorrentFile tf) throws IOException
      Recreates a missing file at its active location (the staging dir while incomplete, the data dir otherwise).
      Parameters:
      tf - the file to recreate
      Throws:
      IOException - if the file cannot be created

    • getBase

      public File getBase()
      The base file or directory.
      Returns:
      the base this storage was opened with

    • getStagingDir

      public File getStagingDir()
      The directory this torrent's incomplete files are staged in while downloading, or null when the staging feature is disabled. Paths inside it mirror the data-directory layout of the torrent.
      Returns:
      the staging base directory, or null when staging is disabled

    • listMerged

      public File[] listMerged(String pathInTorrent)
      Lists a directory within the torrent, merging the staging tree while the download is incomplete, so completed files already moved to the data directory and files still staged in the temp dir both appear. Entries are deduplicated by name, data-directory entries first.
      Parameters:
      pathInTorrent - the directory path within the torrent, "/" or empty for the root
      Returns:
      the merged children, or null if neither tree contains the directory

    • getFiles

      public List<File> getFiles()
      Does not include directories. Unsorted.

      In staging mode, returns the final data-directory location of each file, even while the data is still being written to the staging dir.

      Returns:
      a new List

    • getFileCount

      public int getFileCount()
      Does not include directories.
      Returns:
      the file count

    • getDirectories

      public SortedSet<File> getDirectories()
      Includes the base for a multi-file torrent. Sorted bottom-up for easy deletion. Slow. Use for deletion only.
      Returns:
      a new Set or null for a single-file torrent

    • deleteStagingData

      public void deleteStagingData()
      Removes all partially downloaded data from the staging directory for this torrent. No-op when the staging feature is disabled.

    • recheck

      public boolean recheck() throws IOException
      Blocking. Holds lock. Recommend running only when stopped. Caller should thread. Calls listener.setWantedPieces() on completion if anything changed.
      Returns:
      true if anything changed, false otherwise
      Throws:
      IOException - if the recheck cannot read or create the torrent's files

    • checkCreateFiles

      private boolean checkCreateFiles(boolean recheck) throws IOException
      This is called at the beginning, and at presumed completion, so we have to be careful about locking.
      Parameters:
      recheck - if true, this is a check after we downloaded the last piece, and we don't modify the global bitfield unless the check fails.
      Returns:
      true if changed (only valid if recheck == true)
      Throws:
      IOException - if a file cannot be read, or the wait to check is interrupted

    • getVerifyThreads

      static int getVerifyThreads(I2PAppContext ctx)
      Worker threads for a parallel piece verification: the i2psnark.verifyThreads property when set, otherwise scaled to the CPU count (at least 4).
      Parameters:
      ctx - the context supplying the i2psnark.verifyThreads property
      Returns:
      the configured thread count, or the CPU-derived default when the property is unset

    • checkCreateFiles

      private boolean checkCreateFiles(boolean recheck, boolean[] pieceTrusted) throws IOException
      Variant that skips hashing pieces marked trusted, for resuming with a partially trusted saved state.
      Parameters:
      recheck - if true, this is a check after we downloaded the last piece, and we don't modify the global bitfield unless the check fails
      pieceTrusted - array indexed by piece, or null to hash everything
      Returns:
      true if changed (only valid if recheck == true)
      Throws:
      IOException - if a file cannot be read, or the wait to check is interrupted

    • locked_checkCreateFiles

      private boolean locked_checkCreateFiles(boolean recheck, boolean[] pieceTrusted) throws IOException
      Check the pieces, reporting whether anything changed.
      Parameters:
      recheck - if true, this is a check after we downloaded the last piece, and we don't modify the global bitfield unless the check fails
      pieceTrusted - array indexed by piece, or null to hash everything
      Returns:
      true if changed (only valid if recheck == true)
      Throws:
      IOException - if a file cannot be read while hashing

    • isPaddingPiece

      private boolean isPaddingPiece(long pieceStart)
      BEP 47: true if the piece lies entirely within padding files, so it hashes as zeros.
      Parameters:
      pieceStart - the byte offset the piece begins at
      Returns:
      true if the whole piece range is padding

    • allocateFile

      private void allocateFile(Storage.TorrentFile tf) throws IOException
      Extend one torrent file to its full length and report it to the listener, so that reads don't fail with an IOE before the piece arrives.
      Parameters:
      tf - the file to extend
      Throws:
      IOException - on allocation failure

    • close

      public void close() throws IOException
      Closes the Storage and makes sure that all RandomAccessFiles are closed. The Storage is unusable after this.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • getPiece

      public ByteArray getPiece(int piece, int off, int len) throws IOException
      Returns a byte array containing a portion of the requested piece or null if the storage doesn't contain the piece yet.
      Parameters:
      piece - the zero-based index of the piece
      off - the offset within the piece where the returned data starts, in bytes
      len - the number of piece bytes to return
      Returns:
      the requested bytes, or null when the piece is not held yet
      Throws:
      IOException - if the piece cannot be read from storage

    • writePiece

      private void writePiece(PartialPiece pp, int piece, boolean shouldPreallocate, boolean forceRW) throws IOException
      Write a (hash-verified) partial piece to the data files.
      Parameters:
      pp - the verified partial piece to write
      piece - the piece number
      shouldPreallocate - whether to balloon sparse files
      forceRW - if true, open the data files read-write even when the storage thinks they are complete, and try to make them writable if permissions deny it. Without this, Storage.TorrentFile.checkRAF() opens files read-only for a complete torrent, and the write (or the pre-allocation ballooning) then fails with EBADF on a healthy disk.
      Throws:
      IOException - when some storage related error occurs.

    • putPiece

      public boolean putPiece(PartialPiece pp) throws IOException
      Put the piece in the Storage if it is correct. Warning - takes a LONG time if complete as it.
      Parameters:
      pp - the received piece data, which is hashed against the metainfo to verify it
      Returns:
      true if the piece was correct (sha metainfo hash matches), otherwise false.
      Throws:
      IOException - when some storage related error occurs.

    • checkFileCompletions

      private void checkFileCompletions()
      Moves any file whose pieces are all downloaded out of the staging directory into the data directory. No-op when the staging feature is disabled.

      Called after every piece write and after checks/rechecks. The move itself happens on a background thread; this method only starts it.

    • fileComplete

      private boolean fileComplete(long start, long end)
      Whether all pieces overlapping the byte range [start, end) are set in the bitfield.
      Parameters:
      start - first byte offset of the file in the torrent
      end - one past the last byte of the file
      Returns:
      true if the file's pieces are all downloaded

    • startCopy

      private void startCopy(Storage.TorrentFile tf)
      Starts copying a completed file from the staging directory to the data directory on a background thread, unless a copy for the file is already running.

      The staging file is read with its open RAF handle; the handle is closed before the file is renamed away, so a concurrent write to the same file (which would be a hash failure anyway) is not a concern.

      Parameters:
      tf - the completed file, still located in the staging directory

    • copyToFinal

      private void copyToFinal(Storage.TorrentFile tf)
      Copies the file to the data directory with a bounded number of retries in case the destination is temporarily unavailable. Always runs on a daemon thread.
      Parameters:
      tf - the completed file

    • copyOnce

      private boolean copyOnce(Storage.TorrentFile tf)
      Copies the file from the staging directory to the data directory once.
      Parameters:
      tf - the completed file
      Returns:
      true if the file was moved, false if it must be retried

    • copyFile

      private boolean copyFile(File src, File dst)
      Copies a file's contents to the destination, truncating the destination first.
      Parameters:
      src - the file to read
      dst - the file overwritten with the copy
      Returns:
      true on success, false on transient failure (e.g. disk full)

    • sleep

      private void sleep()
      Delays between copy attempts; returns immediately on interruption.

    • getPieceLength

      private int getPieceLength(int piece)
      This is a dup of MetaInfo.getPieceLength(int) but we need it before the MetaInfo is created in our second constructor.
      Parameters:
      piece - the piece number whose length is wanted
      Returns:
      the number of bytes in that piece, less than piece_size for the last piece

    • getUncheckedPiece

      private int getUncheckedPiece(int piece, byte[] bs) throws IOException
      Reads a piece without checking its hash.
      Parameters:
      piece - the piece number
      bs - the array to fill
      Returns:
      the bytes read
      Throws:
      IOException - if a file read fails

    • checkPieceHash

      private boolean checkPieceHash(int piece, byte[] buf, int length) throws IOException
      Verify a piece's hash without a whole piece-sized buffer: reads the piece in windows no larger than the supplied buffer, feeding each window to the SHA1 digest before comparing against the stored hash. Padding regions hash as zeros (BEP 47) via getUncheckedPiece, which never touches disk for them.
      Parameters:
      piece - the piece index
      buf - the reusable read buffer, also the window size cap
      length - the piece length
      Returns:
      true if the computed SHA1 matches the stored piece hash
      Throws:
      IOException - on read failure

    • getUncheckedPiece

      private int getUncheckedPiece(int piece, byte[] bs, int off, int length) throws IOException
      Reads part of a piece without checking its hash.
      Parameters:
      piece - the piece number
      bs - the array to fill
      off - the offset in the array
      length - the bytes to read
      Returns:
      the bytes read
      Throws:
      IOException - if a file read fails

    • getUncheckedPiece

      private int getUncheckedPiece(int piece, byte[] bs, int off, int length, boolean closeAfterRead) throws IOException
      Read a piece, or part of one, into the buffer.
      Parameters:
      piece - the piece index
      bs - the destination the piece bytes are copied into
      off - the offset within the piece
      length - the number of bytes to read
      closeAfterRead - if true, close each file handle as soon as the read that used it completes. The creation-time hash pass walks the whole torrent and would otherwise hold one handle per file until it finished.
      Returns:
      the number of bytes read
      Throws:
      IOException - on read failure

    • cleanRAFs

      public void cleanRAFs()
      Close unused RAFs - call periodically.

    • main

      public static void main(String[] args)
      Create a metainfo. Used in the installer build process; do not comment out.
      Parameters:
      args - the command line, with -a announce, -c created-by, -m comment and -w web seed options