EepHead

java.lang.Object

net.i2p.util.EepGet

net.i2p.util.EepHead

public class EepHeadextends EepGet

This is a quick hack to get a working EepHead, primarily for the following usage:

EepHead foo = new EepHead(...);
if (foo.fetch()) {
    String lastmod = foo.getLastModified();
    if (lastmod != null) {
        parse the string...
        ...
    }
}

Other use cases (command line, listeners, etc...) lightly- or un-tested. Note that this follows redirects! This may not be what you want or expect.

Writing from scratch rather than extending EepGet would maybe have been less bloated memory-wise. This way gets us redirect handling, among other benefits.

Nested Class Summary
Nested classes/interfaces inherited from class EepGet
EepGet.AUTH_MODE, EepGet.AuthState, EepGet.CLIStatusListener, EepGet.ContentRange, EepGet.Gunzipper, EepGet.StatusListener
Modifier and Type
Class
Description
protected static enum
EepGet.AUTH_MODE
Authentication mode enumeration.
protected class
EepGet.AuthState
Manage the authentication parameters Ref: RFC 2617 Supports both Basic and Digest, however i2ptunnel HTTP proxy has migrated all previous Basic support to Digest.
protected class
EepGet.CLIStatusListener
Reports fetch progress to the command line, one mark per markSize bytes and a summary line every lineSize marks.
(package private) static final class
EepGet.ContentRange
Parsed Content-Range header value.
protected class
EepGet.Gunzipper
Decompressor thread.
static interface
EepGet.StatusListener
Callback interface for monitoring EepGet transfer progress.
Field Summary
Fields
Modifier and Type
Field
Description
private static final OutputStream
_dummyStream
EepGet needs either a non-null file or a stream...
Fields inherited from class EepGet
_aborted, _acceptRanges, _actualURL, _alreadyTransferred, _authState, _bytesRemaining, _bytesTransferred, _cacheControl, _contentEncoding, _contentLanguage, _contentRange, _contentType, _context, _cookie, _csp, _currentAttempt, _decompressException, _encodingChunked, _etag, _etagOrig, _expiryDate, _extraHeaders, _fetchHeaderTimeout, _fetchInactivityTimeout, _fetchTotalTimeout, _isGzippedResponse, _keepFetching, _lastModified, _lastModifiedOrig, _listeners, _log, _maxCompleteFails, _notModified, _numRetries, _out, _outputFile, _outputStream, _proxy, _proxyHost, _proxyIn, _proxyOut, _proxyPort, _redirectLocation, _redirects, _referrerPolicy, _responseCode, _responseText, _retryDelayMs, _server, _shouldProxy, _shouldWriteErrorToOutput, _status, _transferEncoding, _transferFailed, _url, _vary, _xContentTypeOptions, _xframeOptions, _xPoweredBy, _xssProtection, DEFAULT_CONNECT_TIMEOUT, DEFAULT_INACTIVITY_TIMEOUT, DEFAULT_MAX_COMPLETE_FAILS, DEFAULT_NUM_RETRIES, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_JITTER, GZIP_PARTIAL_MARKER_SUFFIX, PROP_CONNECT_TIMEOUT, PROP_DEFAULT_RETRIES, PROP_INACTIVITY_TIMEOUT, PROP_MAX_COMPLETE_FAILS, PROP_RETRY_DELAY, USER_AGENT
Modifier and Type
Field
Description
protected boolean
_aborted
Whether the fetch was aborted due to timeout or user request.
protected String
_acceptRanges
Accept-Ranges header value from the response.
protected String
_actualURL
The URL we actually fetch from (may differ from _url in case of redirect).
protected long
_alreadyTransferred
Total bytes transferred across all attempts, including resumed content.
protected EepGet.AuthState
_authState
Authentication state for proxy digest/basic auth; created via EepGet.addAuthorization(String, String).
protected long
_bytesRemaining
Remaining bytes on the current attempt, -1 if unknown (chunked or no Content-Length).
protected long
_bytesTransferred
Body bytes transferred in the current EepGet.fetch(), summed across attempts.
protected String
_cacheControl
Cache-Control header value from the response.
protected String
_contentEncoding
Content-Encoding header value from the response.
protected String
_contentLanguage
Content-Language header value from the response.
protected String
_contentRange
Content-Range header value from the response, for validating 206/416 resumes.
protected String
_contentType
Content-Type header value from the response.
protected final I2PAppContext
_context
I2P application context for configuration, logging, and random number generation.
protected String
_cookie
Set-Cookie header value from the response.
protected String
_csp
Content-Security-Policy header value from the response.
protected int
_currentAttempt
Zero-based attempt counter for the current fetch; read by the watchdog command.
protected IOException
_decompressException
IOException from the background decompressor thread, if any.
protected boolean
_encodingChunked
Whether the response is using chunked transfer encoding.
protected String
_etag
ETag received from the server (for conditional requests).
protected final String
_etagOrig
Original ETag passed in constructor, preserved across redirects.
protected String
_expiryDate
Expires header value from the response.
protected List<String>
_extraHeaders
Additional HTTP headers to include in the request.
protected int
_fetchHeaderTimeout
Timeout in ms for receiving the HTTP response headers.
protected int
_fetchInactivityTimeout
Inactivity timeout in ms between data packets.
protected int
_fetchTotalTimeout
Total timeout in ms for the entire fetch operation.
protected boolean
_isGzippedResponse
Whether the response body is gzip-compressed.
protected boolean
_keepFetching
Whether to continue fetching; set false to abort.
protected String
_lastModified
Last-Modified header received from the server.
protected final String
_lastModifiedOrig
Original Last-Modified passed in constructor, preserved across redirects.
protected final List<EepGet.StatusListener>
_listeners
Registered status listeners for progress callbacks.
protected final Log
_log
Logger for this instance.
protected int
_maxCompleteFails
Maximum number of complete (zero-byte) failures before giving up.
protected boolean
_notModified
Whether the server returned 304 Not Modified.
protected final int
_numRetries
Maximum number of retry attempts per fetch.
protected OutputStream
_out
Output stream for writing fetched data (file or pipe).
protected final String
_outputFile
Path to the output file for saving the fetched content.
protected final OutputStream
_outputStream
Output stream destination, overrides outputFile when non-null.
protected Socket
_proxy
The socket connection to the proxy or the target server (if not proxied).
protected final String
_proxyHost
Proxy hostname or IP address.
protected InputStream
_proxyIn
Input stream from the proxy/server socket.
protected OutputStream
_proxyOut
Output stream to the proxy/server socket.
protected final int
_proxyPort
Proxy port number.
protected String
_redirectLocation
Location header value from redirect or 407 proxy auth response.
protected AtomicInteger
_redirects
Counter tracking the number of HTTP redirects followed.
protected String
_referrerPolicy
Referrer-Policy header value from the response.
protected int
_responseCode
HTTP response status code, -1 if no response received.
protected String
_responseText
HTTP response status text (e.g.
protected final long
_retryDelayMs
Fixed retry delay in ms, -1 selects the jittered default @since 0.9.71+.
protected String
_server
Server header value from the response.
protected final boolean
_shouldProxy
Whether to route requests through an HTTP proxy.
protected boolean
_shouldWriteErrorToOutput
Whether to write error response bodies to the output.
protected String
_status
Status header value from the response.
protected String
_transferEncoding
Transfer-Encoding header value from the response.
protected boolean
_transferFailed
Whether the current transfer attempt failed (non-2xx response code).
protected final String
_url
The URL we were asked to fetch.
protected String
_vary
Vary header value from the response.
protected String
_xContentTypeOptions
X-Content-Type-Options header value from the response.
protected String
_xframeOptions
X-Frame-Options header value from the response.
protected String
_xPoweredBy
X-Powered-By header value from the response.
protected String
_xssProtection
X-XSS-Protection header value from the response.
protected static final int
DEFAULT_CONNECT_TIMEOUT
Default connect timeout in milliseconds (90 seconds).
protected static final int
DEFAULT_INACTIVITY_TIMEOUT
Default inactivity timeout in milliseconds (5 minutes).
protected static final int
DEFAULT_MAX_COMPLETE_FAILS
Maximum times to try without getting any data at all, even if numRetries is higher @since 0.7.14.
protected static final int
DEFAULT_NUM_RETRIES
Default number of retry attempts.
protected static final int
DEFAULT_RETRY_DELAY
Base retry delay in milliseconds when the jittered default is used @since 0.9.71+.
protected static final int
DEFAULT_RETRY_JITTER
Maximum additional jitter in milliseconds added to the base retry delay @since 0.9.71+.
(package private) static final String
GZIP_PARTIAL_MARKER_SUFFIX
Suffix of the sidecar marker file recording that the output file holds transparently-decompressed (gunzipped) bytes rather than raw response bytes, so its length must not be used as a Range resume offset.
(package private) static final String
PROP_CONNECT_TIMEOUT
Property name for connect timeout configuration.
(package private) static final String
PROP_DEFAULT_RETRIES
Property name for default retries configuration.
(package private) static final String
PROP_INACTIVITY_TIMEOUT
Property name for inactivity timeout configuration.
(package private) static final String
PROP_MAX_COMPLETE_FAILS
Property name for max complete fails configuration.
(package private) static final String
PROP_RETRY_DELAY
Property name for the fixed retry delay in milliseconds.
protected static final String
USER_AGENT
Default User-Agent header value.
Constructor Summary
Constructors
Constructor
Description
EepHead(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String url)
Create a new EepHead to fetch HTTP response headers from the given URL.
Method Summary
Modifier and Type
Method
Description
protected void
doFetch(SocketTimeout timeout)
Read the response headers and notify listeners.
long
getContentLength()
We don't decrement the variable (unlike in EepGet), so this is valid.
protected String
getRequest()
Build the HTTP HEAD request, including headers for Host, User-Agent, and proxy auth.
private static String
guessServer(String server, String cc, String cl, String lm, String ar, String et, String xf, String cto)
Try to identify the server software from the response headers.
static void
main(String[] args)
CLI entry point: fetch and display HTTP response headers for a URL.
private static void
printResponse(EepHead get, String url)
Print all available response headers.
protected boolean
shouldReadBody()
Should we read the body of the response?
private static String
usage()
Return the command-line usage text for eephead.
Methods inherited from class EepGet
addAuthorization, addHeader, addStatusListener, failureReason, fatalAbort, fetch, fetch, fetch, getAcceptRanges, getCacheControl, getContentEncoding, getContentLanguage, getContentType, getCookie, getCSP, getEtag, getExpiryDate, getLastModified, getNotModified, getReferrerPolicy, getResumeOffset, getServer, getStatus, getStatusCode, getStatusText, getTransferEncoding, getVary, getXContentTypeOptions, getXframeOptions, getXPoweredBy, getXSSProtection, gzipMarkerExists, isStalledButProgressing, isTotalBudgetExhausted, markGzipPartial, parseAuthArgs, parseChunkLength, parseContentRange, readChunkLength, readHeaders, redactedUrl, requestedRangeStart, seconds, sendRequest, setPostData, setPostData, setPostData, setWriteErrorToOutput, stopFetching, suggestName, usageText, validateResumeResponse
Modifier and Type
Method
Description
void
addAuthorization(String userName, String password)
Add basic authorization header for the proxy.
void
addHeader(String name, String value)
Add an extra header to the request.
void
addStatusListener(EepGet.StatusListener lsnr)
Add a status listener.
(package private) static String
failureReason(Throwable t, long inactivityMs)
The failure reason for a transfer-failed log line.
(package private) static boolean
fatalAbort(boolean aborted, boolean strictSize, long remaining, boolean sawEof)
Decide whether a watchdog abort observed after the body loop must fail the attempt.
boolean
fetch()
Blocking fetch, returning true if the URL was retrieved, false if all retries failed.
boolean
fetch(long fetchHeaderTimeout)
Blocking fetch, timing out individual attempts if the HTTP response headers don't come back in the time given.
boolean
fetch(long fetchHeaderTimeout, long totalTimeout, long inactivityTimeout)
Blocking fetch.
String
getAcceptRanges()
Show the Accept-Ranges field.
String
getCacheControl()
Show the Cache-Control field.
String
getContentEncoding()
Show the Content-Encoding field.
String
getContentLanguage()
Show the Content-Language field.
String
getContentType()
After fetch, the received value from the server, or null if none.
String
getCookie()
Show the Set-Cookie field.
String
getCSP()
Content-Security-Policy header.
String
getEtag()
After fetch, the received value from the server, or null if none.
String
getExpiryDate()
Show the Expires field.
String
getLastModified()
After fetch, the received value from the server, or null if none.
boolean
getNotModified()
Whether the server returned 304.
String
getReferrerPolicy()
Referrer-Policy header.
protected long
getResumeOffset(File outFile)
Byte offset to resume from for a file-based fetch, accounting for the persisted gunzip state: a marker means the file holds decompressed bytes, whose length is not a valid offset into the compressed representation, so the fetch restarts instead of resuming.
String
getServer()
Show the Server field.
String
getStatus()
Combined server status message e.g.
int
getStatusCode()
The server response (200, etc).
String
getStatusText()
The server text ("OK", "Not Found", etc).
String
getTransferEncoding()
Show the Transfer-Encoding field.
String
getVary()
Vary header.
String
getXContentTypeOptions()
X-Content-Type-Options header.
String
getXframeOptions()
X-Frame-Options header.
String
getXPoweredBy()
X-Powered-By header.
String
getXSSProtection()
X-XSS-Protection header.
(package private) boolean
gzipMarkerExists()
Whether this fetch's output file was written by the transparent gunzipper, so its length is a decompressed size that must not be used as a Range resume offset.
(package private) static boolean
isStalledButProgressing(IOException ioe, boolean aborted, long attemptStartBytes, long bytesTransferred)
Whether a failed attempt was a data-phase stall on an otherwise working path: the read timed out or our SocketTimeout watchdog aborted the transfer (which closes the socket first, surfacing as a raw SocketException, so the explicit aborted signal.
(package private) static boolean
isTotalBudgetExhausted(long totalTimeoutMs, long attemptStartMs, long nowMs)
Whether an attempt has already consumed its total-timeout budget.
(package private) void
markGzipPartial(File outFile)
Record that the output file now holds decompressed bytes.
static Map<String,String>
parseAuthArgs(String args)
Parse the args in an authentication header.
(package private) static long
parseChunkLength(String line)
Extract the chunk size from a chunk-size line, dropping any chunk extensions.
(package private) static EepGet.ContentRange
parseContentRange(String val)
Parse a Content-Range header value.
protected long
readChunkLength()
Parse a chunk length from the HTTP response stream.
protected void
readHeaders()
Read and parse HTTP response headers.
(package private) static String
redactedUrl(String url)
The request URL for log output.
(package private) static long
requestedRangeStart(long alreadyTransferred, List<String> extraHeaders)
Determine the start offset this fetch requested, so a 206's Content-Range can be validated against what was actually asked for.
(package private) static String
seconds(long ms)
Render a millisecond duration in seconds for log output, keeping milliseconds only for sub-second or non-whole-second values.
protected void
sendRequest(SocketTimeout timeout)
Open connection and send HTTP request.
protected void
setPostData(String contentType, byte[] data)
Set post data (binary body).
protected void
setPostData(String contentType, File data)
Set post data (file body).
protected void
setPostData(String contentType, String data)
Set post data (string body).
void
setWriteErrorToOutput()
If called (before calling fetch()), data from the server or proxy will be written to the output file or stream even on an error response code (4xx, 5xx, etc).
void
stopFetching()
Stop the fetch in progress.
static String
suggestName(String url)
Parse URL for a viable filename.
(package private) static String
usageText()
The command-line help text, listing every option accepted by EepGet.main(String[]).
(package private) boolean
validateResumeResponse()
Validate a resume-sensitive response once all headers are parsed.
Methods inherited from class Object
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details
- _dummyStream
  private static final OutputStream _dummyStream
  EepGet needs either a non-null file or a stream... shouldn't actually be written to...
Constructor Details
- EepHead
  public EepHead(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String url)
  Create a new EepHead to fetch HTTP response headers from the given URL.
  Parameters:
  ctx - I2P app context
  proxyHost - hostname of the eepproxy to fetch through; empty for no proxy
  proxyPort - eepproxy port, 0 when no proxy is used
  numRetries - number of retries on failure
  url - absolute URL to send the HEAD request to
Method Details
- main
  public static void main(String[] args)
  CLI entry point: fetch and display HTTP response headers for a URL. Usage: EepHead [-p 127.0.0.1:4444] [-n #retries] url.
  Parameters:
  args - the command line arguments, with options set by -p host:port, -n retries, -t seconds, -u user and -x password, followed by the URL whose headers are to be fetched
- printResponse
  private static void printResponse(EepHead get, String url)
  Print all available response headers.
  Parameters:
  get - the EepHead instance after fetch
  url - the original request URL
- guessServer
  private static String guessServer(String server, String cc, String cl, String lm, String ar, String et, String xf, String cto)
  Try to identify the server software from the response headers. Uses known header fingerprints when the Server header is absent.
  Parameters:
  server - Server header value (may be null)
  cc - Cache-Control header value
  cl - Content-Length as string
  lm - Last-Modified header value
  ar - Accept-Ranges header value
  et - ETag header value
  xf - X-Frame-Options header value
  cto - X-Content-Type-Options header value
  Returns:
  the identified server name, or "unknown"
- usage
  private static String usage()
  Return the command-line usage text for eephead.
  Returns:
  CLI usage string
- doFetch
  protected void doFetch(SocketTimeout timeout) throws IOException
  Read the response headers and notify listeners.
  Overrides:
  doFetch in class EepGet
  Parameters:
  timeout - may be null as of 0.9.49
  Throws:
  IOException - on IO error
- shouldReadBody
  protected boolean shouldReadBody()
  Should we read the body of the response?
  Overrides:
  shouldReadBody in class EepGet
  Returns:
  false always
- getRequest
  protected String getRequest() throws IOException
  Build the HTTP HEAD request, including headers for Host, User-Agent, and proxy auth.
  Overrides:
  getRequest in class EepGet
  Returns:
  the full HTTP 1.1 request string
  Throws:
  IOException - on bad URL
- getContentLength
  public long getContentLength()
  We don't decrement the variable (unlike in EepGet), so this is valid.
  Returns:
  the Content-Length reported by the server, in bytes

Nested Class Summary

Nested classes/interfaces inherited from class EepGet

Field Summary

Fields inherited from class EepGet

Constructor Summary

Method Summary

Methods inherited from class EepGet

Methods inherited from class Object

Field Details

_dummyStream

Constructor Details

Method Details

main

printResponse

guessServer

usage

doFetch

shouldReadBody

getRequest

getContentLength