Method Details

• main

  public static void main(String[] args)

  Command-line entry point for EepGet.

  Usage: eepget [-p 127.0.0.1:4444 | -c] [-n #retries] [-e etag] [-o outputFile] [-t inactivityTimeout] [-u username] [-x password] [-h name=value] [-m markSize] [-l lineLen] url

  All timeouts are in seconds. The header timeout is not settable from the command line; it comes from the eepget.connectTimeout property, and the -t option sets the inactivity timeout instead (eepget.inactivityTimeout).

  As of 0.9.45, supports https and redirect to https.

  Parameters:

  args - command-line arguments, parsed by this method against the usage above

• suggestName

  public static String suggestName(String url)

  Parse URL for a viable filename.

  Parameters:

  url - a URL giving the location of an online resource

  Returns:

  a filename to save the resource as on local filesystem

• sanitize

  private static String sanitize(String name)

  Removes characters unsafe for filenames, replacing them with underscores. References: http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx

  Parameters:

  name - the raw name to sanitize

  Returns:

  a sanitized name safe for use as a filename

• usageText

  static String usageText()

  The command-line help text, listing every option accepted by main(String[]). The defaults shown must match the constants and properties read in main().

  Returns:

  the usage help text

• usage

  private static void usage()

  Print the command-line help to stdout.

  See Also:

  • usageText()

• addStatusListener

  public void addStatusListener(EepGet.StatusListener lsnr)

  Add a status listener.

  Parameters:

  lsnr - the listener to add

• stopFetching

  public void stopFetching()

  Stop the fetch in progress.

• fetch

  public boolean fetch()

  Blocking fetch, returning true if the URL was retrieved, false if all retries failed.

  Uses the timeouts configured when this instance was created: the header timeout defaults to DEFAULT_CONNECT_TIMEOUT (90 seconds), the total timeout to none, and the inactivity timeout to DEFAULT_INACTIVITY_TIMEOUT (5 minutes).

  Returns:

  whether the URL was retrieved; false if every attempt failed

• fetch

  public boolean fetch(long fetchHeaderTimeout)

  Blocking fetch, timing out individual attempts if the HTTP response headers don't come back in the time given. If the timeout is zero or less, this will wait indefinitely.

  Total timeout default none, inactivity timeout default DEFAULT_INACTIVITY_TIMEOUT (5 minutes).

  Parameters:

  fetchHeaderTimeout - timeout in ms

  Returns:

  whether the URL was retrieved; false if every attempt failed

• fetch

  public boolean fetch(long fetchHeaderTimeout, long totalTimeout, long inactivityTimeout)

  Blocking fetch.

  The three timeouts are independent: the header timeout covers the wait for the response status line and headers of each attempt, the total timeout covers the whole operation including retries and backoff, and the inactivity timeout covers the wait between bytes once the body has started.

  Parameters:

  fetchHeaderTimeout - <= 0 for none (proxy will timeout if none, none isn't recommended if no proxy) forced to totalTimeout if totalTimeout is set and this is <= 0

  totalTimeout - operation-wide deadline for the whole fetch, including retries and backoff between attempts; <= 0 for default none

  inactivityTimeout - <= 0 for default DEFAULT_INACTIVITY_TIMEOUT (5 minutes)

  Returns:

  whether the URL was retrieved; false if every attempt failed

• inactivityBudget

  private long inactivityBudget()

  The inactivity timeout in force for a fetch, the default when none was configured. Matches what doFetch() installs as the socket soTimeout.

  Returns:

  the inactivity budget in milliseconds

• failureReason

  static String failureReason(Throwable t, long inactivityMs)

  The failure reason for a transfer-failed log line. Socket exceptions frequently carry no message at all, and "-> null" tells the reader nothing. A bare read timeout is named together with the inactivity budget that expired, so the line says how long the peer stayed quiet; any other message-less exception falls back to its class name - or its full name, which an anonymous or local class leaves blank as a simple name.

  Parameters:

  t - the failure, non-null

  inactivityMs - the inactivity timeout in force on the failed read

  Returns:

  a non-empty reason for the log line

• seconds

  static String seconds(long ms)

  Render a millisecond duration in seconds for log output, keeping milliseconds only for sub-second or non-whole-second values.

  Parameters:

  ms - the duration in milliseconds

  Returns:

  the duration as "30s", or "1500ms" when it is not a whole number of seconds

• redactedUrl

  static String redactedUrl(String url)

  The request URL for log output. Magnet-style URLs carrying an info_hash are cut before the hash, or at an earlier first parameter when one precedes it, and the http:// prefix is dropped, so the hash never reaches the logs; other URLs pass through unchanged.

  Parameters:

  url - the request URL, not null

  Returns:

  the URL, redacted when it carries an info_hash

• isStalledButProgressing

  static boolean isStalledButProgressing(IOException ioe, boolean aborted, long attemptStartBytes, long bytesTransferred)

  Whether a failed attempt was a data-phase stall on an otherwise working path: the read timed out or our SocketTimeout watchdog aborted the transfer (which closes the socket first, surfacing as a raw SocketException, so the explicit aborted signal. Flag set or socket closed by the watchdog — not exception shape, identifies that path) but the attempt still appended bytes. Wrapped timeout causes are walked so a single hop of context cannot hide a socket timeout. Header-phase timeouts make no progress within the attempt and non-timeout failures return false, so callers only skip the default backoff for one immediate Range-resume retry.

  Parameters:

  ioe - failure raised by the attempt

  aborted - whether our SocketTimeout watchdog aborted this attempt

  attemptStartBytes - body bytes transferred when the attempt started; header and connect phases advance no body bytes

  bytesTransferred - body bytes transferred now, after the failure

  Returns:

  true when the attempt timed out after making progress

• isTotalBudgetExhausted

  static boolean isTotalBudgetExhausted(long totalTimeoutMs, long attemptStartMs, long nowMs)

  Whether an attempt has already consumed its total-timeout budget. The budget is operation-wide: SocketTimeout's total period and fetch()'s deadline both start at fetch() entry, so a failure at or past that deadline is the caller's overall timeout, not a transient stall. It must take the default retry backoff instead of an immediate Range-resume retry.

  Parameters:

  totalTimeoutMs - total timeout for the fetch, <= 0 for none

  attemptStartMs - when fetch() started the operation

  nowMs - when the failure was classified

  Returns:

  true if the total budget is exhausted

• hasTimeoutCause

  private static boolean hasTimeoutCause(Throwable t)

  Whether the exception (or any cause within a bounded depth) is a read timeout: SocketTimeoutException, or the canonical watchdog abort message. A bare InterruptedIOException is not classified; modern socket reads do not throw it on interrupt, so it cannot be attributed to our watchdog without the explicit abort signal.

  Parameters:

  t - exception or cause chain head, may be null

  Returns:

  true when a timeout cause is found

• doFetch

  protected void doFetch(SocketTimeout timeout) throws IOException

  Read response body for a single fetch. Call after sendRequest(SocketTimeout).

  Parameters:

  timeout - may be null

  Throws:

  IOException - on IO error

• readHeaders

  protected void readHeaders() throws IOException

  Read and parse HTTP response headers.

  Throws:

  IOException - on IO error

• shouldReadBody

  protected boolean shouldReadBody()

  Whether to read the response body.

  Returns:

  true always, overridden in EepHead

• readChunkLength

  protected long readChunkLength() throws IOException

  Parse a chunk length from the HTTP response stream. The chunk-size line is read whole, so any chunk extensions (RFC 9112 sec. 7.1.1, "chunk-ext") come along with it and are discarded rather than rejected. A trailer section after the terminating zero-length chunk is not consumed.

  Returns:

  the chunk length

  Throws:

  IOException - on IO error, or if the chunk-size line is not hex

• parseChunkLength

  static long parseChunkLength(String line) throws IOException

  Extract the chunk size from a chunk-size line, dropping any chunk extensions. Only the hex digits before the first ';' are significant, per RFC 9112 sec. 7.1, so a negative size, which is outside that grammar, is rejected rather than silently ending the transfer early.

  Parameters:

  line - the raw chunk-size line, without the trailing CRLF

  Returns:

  the chunk length, never negative

  Throws:

  IOException - if the chunk size is not a hex number

• fatalAbort

  static boolean fatalAbort(boolean aborted, boolean strictSize, long remaining, boolean sawEof)

  Decide whether a watchdog abort observed after the body loop must fail the attempt. A transfer whose declared body completed — fixed length reached, chunked terminator read, or EOF on an unknown-length body — carries all its bytes, so an abort landing between the last read and finalization must not report failure; anything less is a genuine abort.

  Parameters:

  aborted - the watchdog aborted this attempt

  strictSize - whether Content-Length framed the body

  remaining - declared bytes still unread after the loop

  sawEof - the body stream reached end of stream

  Returns:

  true when the abort must fail the attempt

• validateResumeResponse

  boolean validateResumeResponse() throws IOException

  Validate a resume-sensitive response once all headers are parsed. A 206 must carry a Content-Range whose start matches the start we requested — our resume offset or a caller-supplied Range — and whose span matches the declared Content-Length, or appending its body would corrupt the output. A 416 to a partial fetch is only a success when the server proves our offset equals the complete length; anything else fails without a retry, since the same Range would produce the same answer. Other responses are untouched here.

  Returns:

  true when the response body must be skipped

  Throws:

  IOException - when the response cannot safely be applied

• requestedRangeStart

  static long requestedRangeStart(long alreadyTransferred, List<String> extraHeaders)

  Determine the start offset this fetch requested, so a 206's Content-Range can be validated against what was actually asked for. A resume Range generated by this class (alreadyTransferred > 0) is written before any caller-supplied headers and wins; otherwise a caller-supplied "Range: bytes=N-..." header (webseed piece fetches) provides the offset; with no Range at all a 206 must start at zero. A present but unparseable Range yields -1, leaving the start check to the caller's own contract.

  Parameters:

  alreadyTransferred - resume offset; greater than zero when this class generated the Range header

  extraHeaders - caller headers as "Name: value", may be null

  Returns:

  the expected Content-Range start, or -1 when unknown

• parseRangeHeaderStart

  private static long parseRangeHeaderStart(String val)

  Parse the start of a Range header value. Only a simple single range ("bytes=start-end" or "bytes=start-") is usable here; multi-ranges and suffix ranges yield -1.

  Parameters:

  val - Range header value without the field name, may be null

  Returns:

  the requested start, or -1 if not a simple single range

• parseContentRange

  static EepGet.ContentRange parseContentRange(String val)

  Parse a Content-Range header value. Accepts the satisfiable form ("bytes start-end/total") and the unsatisfied form sent with 416 ("bytes star/total"), with the total optionally a star.

  Parameters:

  val - header value, may be null

  Returns:

  the parsed range, or null if missing or invalid

• parseDigits

  private static long parseDigits(String s)

  Parse an unsigned decimal byte count.

  Parameters:

  s - digits, non-null

  Returns:

  the value, or -1 if empty, non-digit, or too large

• handleStatus

  private int handleStatus(String line)

  Parse the first status line and extract the response code. Side effect: stores status text in _responseText.

  Parameters:

  line - the first line of the HTTP response

  Returns:

  HTTP response code (200, 206, other)

• handle

  private void handle(String key, String val)

  Handles one header from the response.

  Parameters:

  key - the header name

  val - the header value

• increment

  private static void increment(byte[] lookahead, int cur)

  Adds one to the counter in the lookahead buffer.

  Parameters:

  lookahead - the buffer holding the counter

  cur - the position of the counter

• isEndOfHeaders

  private static boolean isEndOfHeaders(byte[] lookahead)

  Checks whether the headers have ended.

  Parameters:

  lookahead - the buffer to check

  Returns:

  true if the blank line was seen

• followRedirect

  private void followRedirect(SocketTimeout timeout) throws IOException

  Handles a redirect or proxy-auth response, either transferring control to a new (possibly https) fetch or fixing up the URL and retrying.

  Also reached after a 407 proxy-auth response, to retry through the proxy.

  Parameters:

  timeout - the fetch timeout, may be null

  Throws:

  IOException - on invalid redirect or auth failure

• openConnection

  private void openConnection() throws IOException

  Opens the connection to the proxy or the destination host.

  Throws:

  IOException - on connect failure

• gzipMarkerFor

  private static File gzipMarkerFor(File outFile)

  The sidecar marker file recording gunzipped state for outFile.

  Parameters:

  outFile - output file, non-null

  Returns:

  the marker file

• gzipMarkerExists

  boolean gzipMarkerExists()

  Whether this fetch's output file was written by the transparent gunzipper, so its length is a decompressed size that must not be used as a Range resume offset.

  Returns:

  true when a gzip marker exists for the output file

• markGzipPartial

  void markGzipPartial(File outFile) throws IOException

  Record that the output file now holds decompressed bytes. Called before the first decompressed byte is written, so an interrupted fetch cannot leave decompressed content without its marker.

  Parameters:

  outFile - output file being written by the gunzipper, non-null

  Throws:

  IOException - if the marker cannot be persisted

• clearGzipMarker

  private static void clearGzipMarker(File outFile)

  Drop the gunzip marker; called when the output file is rewritten from scratch.

  Parameters:

  outFile - output file, non-null

• getResumeOffset

  protected long getResumeOffset(File outFile)

  Byte offset to resume from for a file-based fetch, accounting for the persisted gunzip state: a marker means the file holds decompressed bytes, whose length is not a valid offset into the compressed representation, so the fetch restarts instead of resuming.

  Parameters:

  outFile - output file, may be absent

  Returns:

  resume offset; 0 when the file is absent or must be restarted

• sendRequest

  protected void sendRequest(SocketTimeout timeout) throws IOException

  Open connection and send HTTP request.

  Parameters:

  timeout - may be null

  Throws:

  IOException - on IO error

• getRequest

  protected String getRequest() throws IOException

  Build the HTTP request string for the current URL and state.

  Returns:

  the HTTP request string

  Throws:

  IOException - on bad URL

• getEtag

  public String getEtag()

  After fetch, the received value from the server, or null if none. Before fetch, and after some errors, may be the value passed in the constructor.

  Returns:

  the ETag from the server's response, or null if none was sent

• getLastModified

  public String getLastModified()

  After fetch, the received value from the server, or null if none. Before fetch, and after some errors, may be the value passed in the constructor.

  Returns:

  the last modified date

• getNotModified

  public boolean getNotModified()

  Whether the server returned 304.

  Returns:

  true if the server returned 304

• getContentType

  public String getContentType()

  After fetch, the received value from the server, or null if none.

  Returns:

  the content type

• getServer

  public String getServer()

  Show the Server field.

  Returns:

  the Server header from the response, or null if none was sent

• getContentLanguage

  public String getContentLanguage()

  Show the Content-Language field.

  Returns:

  the content language

• getTransferEncoding

  public String getTransferEncoding()

  Show the Transfer-Encoding field.

  Returns:

  the transfer encoding

• getContentEncoding

  public String getContentEncoding()

  Show the Content-Encoding field.

  Returns:

  the content encoding

• getCacheControl

  public String getCacheControl()

  Show the Cache-Control field.

  Returns:

  the cache control

• getAcceptRanges

  public String getAcceptRanges()

  Show the Accept-Ranges field.

  Returns:

  the accept ranges

• getExpiryDate

  public String getExpiryDate()

  Show the Expires field.

  Returns:

  the expiry date

• getCookie

  public String getCookie()

  Show the Set-Cookie field.

  Returns:

  the Set-Cookie header from the response, or null if none was sent

• getReferrerPolicy

  public String getReferrerPolicy()

  Referrer-Policy header.

  Returns:

  the referrer policy, or null

• getVary

  public String getVary()

  Vary header.

  Returns:

  the vary value, or null

• getXframeOptions

  public String getXframeOptions()

  X-Frame-Options header.

  Returns:

  the frame options, or null

• getCSP

  public String getCSP()

  Content-Security-Policy header.

  Returns:

  the CSP value, or null

• getXSSProtection

  public String getXSSProtection()

  X-XSS-Protection header.

  Returns:

  the XSS protection value, or null

• getXContentTypeOptions

  public String getXContentTypeOptions()

  X-Content-Type-Options header.

  Returns:

  the content type options, or null

• getXPoweredBy

  public String getXPoweredBy()

  X-Powered-By header.

  Returns:

  the powered-by value, or null

• getStatusCode

  public int getStatusCode()

  The server response (200, etc).

  Returns:

  -1 if invalid, or if the proxy never responded, or if no proxy was used and the server never responded. If a non-proxied request partially succeeded (for example a redirect followed by a fail, or a partial fetch followed by a fail), this will be the last status code received. Note that fetch() may return false even if this returns 200.

• getStatusText

  public String getStatusText()

  The server text ("OK", "Not Found", etc). Note that the text may contain % encoding.

  Returns:

  null if invalid, or if the proxy never responded, or if no proxy was used and the server never responded. If a non-proxied request partially succeeded (for example a redirect followed by a fail, or a partial fetch followed by a fail), this will be the last status code received. Note that fetch() may return false even if this returns "OK".

• getStatus

  public String getStatus()

  Combined server status message e.g. "200 OK".

  Returns:

  the status string

• setWriteErrorToOutput

  public void setWriteErrorToOutput()

  If called (before calling fetch()), data from the server or proxy will be written to the output file or stream even on an error response code (4xx, 5xx, etc). The error data will only be written if no previous data was written on an earlier try. Caller must of course check getStatusCode() or the fetch() return value.

• addHeader

  public void addHeader(String name, String value)

  Add an extra header to the request. Must be called before fetch(). Not supported by EepHead. As of 0.9.10, If name is User-Agent, this will replace the default User-Agent header. As of 0.9.14, If name is If-None-Match or If-Modified-Since, this will replace the etag or last-modified value given in the constructor. Note that headers may be subsequently modified or removed in the I2PTunnel HTTP Client proxy.

  In proxied SSLEepGet, these headers are sent to the remote server, NOT the proxy.

  Parameters:

  name - header name

  value - header value

• addAuthorization

  public void addAuthorization(String userName, String password)

  Add basic authorization header for the proxy. Only added if the request is going through a proxy. Must be called before fetch().

  Parameters:

  userName - username offered in the proxy's Basic or Digest challenge

  password - the matching password, or null if the challenge offers none

• setPostData

  protected void setPostData(String contentType, String data)

  Set post data (string body). Must be called before fetch().

  Parameters:

  contentType - Content-Type header value

  data - the request body, sent verbatim after the headers

  Throws:

  IllegalStateException - if already set

• setPostData

  protected void setPostData(String contentType, File data)

  Set post data (file body). Must be called before fetch().

  Parameters:

  contentType - Content-Type header value

  data - file containing POST body

  Throws:

  IllegalStateException - if already set

• setPostData

  protected void setPostData(String contentType, byte[] data)

  Set post data (binary body). Must be called before fetch().

  Parameters:

  contentType - Content-Type header value

  data - POST body bytes

  Throws:

  IllegalStateException - if already set

• parseAuthArgs

  public static Map<String,String> parseAuthArgs(String args)

  Parse the args in an authentication header.

  Modified from LoadClientAppsJob. All keys are mapped to lower case. Double quotes around values are stripped. Ref: RFC 2617

  Public for I2PTunnelHTTPClientBase; use outside of tree at own risk, subject to change or removal.

  Parameters:

  args - non-null, starting after "Digest " or "Basic "

  Returns:

  parsed key-value pairs

• lc8hex

  private static String lc8hex(int nc)

  Low 8 hex digits of the nonce.

  Parameters:

  nc - the nonce value to format

  Returns:

  8 hex chars, lower case, e.g. 00000001