Package net.i2p.util
Class EepGet
java.lang.Object
net.i2p.util.EepGet
- Direct Known Subclasses:
EepHead,I2PSocketEepGet,PartialEepGet,SSLEepGet
public class EepGet extends Object
EepGet [-p 127.0.0.1:4444]
[-n #retries]
[-o outputFile]
[-m markSize lineLen]
url
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected static classEepGet.AUTH_MODEprotected classEepGet.AuthStateManage the authentication parameters Ref: RFC 2617 Supports both Basic and Digest, however i2ptunnel HTTP proxy has migrated all previous Basic support to Digest.protected classEepGet.CLIStatusListenerprotected classEepGet.GunzipperDecompressor thread.static interfaceEepGet.StatusListenerCallback interface -
Field Summary
Fields Modifier and Type Field Description protected boolean_abortedprotected String_acceptRangesprotected String_actualURLthe URL we actually fetch from (may differ from the _url in case of redirect)protected long_alreadyTransferredprotected EepGet.AuthState_authStateprotected long_bytesRemainingprotected long_bytesTransferredprotected String_cacheControlprotected String_contentEncodingprotected String_contentLanguageprotected String_contentTypeprotected I2PAppContext_contextprotected String_cookieprotected String_cspprotected int_currentAttemptprotected IOException_decompressExceptionprotected boolean_encodingChunkedprotected String_etagprotected String_etagOrigprotected String_expiryDateprotected List<String>_extraHeadersprotected int_fetchHeaderTimeoutprotected int_fetchInactivityTimeoutprotected int_fetchTotalTimeoutprotected boolean_isGzippedResponseprotected boolean_keepFetchingprotected String_lastModifiedprotected String_lastModifiedOrigprotected List<EepGet.StatusListener>_listenersprotected Log_logprotected boolean_notModifiedprotected int_numRetriesprotected OutputStream_outprotected String_outputFileprotected OutputStream_outputStreamprotected Socket_proxyprotected String_proxyHostprotected InputStream_proxyInprotected OutputStream_proxyOutprotected int_proxyPortprotected String_redirectLocationprotected int_redirectsprotected String_referrerPolicyprotected int_responseCodeprotected String_responseTextprotected String_serverprotected boolean_shouldProxyprotected boolean_shouldWriteErrorToOutputprotected String_statusprotected String_transferEncodingprotected boolean_transferFailedprotected String_urlurl we were asked to fetchprotected String_varyprotected String_xContentTypeOptionsprotected String_xframeOptionsprotected String_xPoweredByprotected String_xssProtectionprotected static intCONNECT_TIMEOUTprotected static intINACTIVITY_TIMEOUTprotected static intMAX_COMPLETE_FAILSmaximum times to try without getting any data at all, even if numRetries is higher @since 0.7.14protected static StringUSER_AGENTthis will be replaced by the HTTP Proxy if we are using it -
Constructor Summary
Constructors Constructor Description EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, long minSize, long maxSize, String outputFile, OutputStream outputStream, String url, boolean allowCaching, String etag, String postData)EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, long minSize, long maxSize, String outputFile, OutputStream outputStream, String url, boolean allowCaching, String etag, String lastModified, String postData)EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url)EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching, String etag)EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching, String etag, String lastModified)EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, String postData)EepGet(I2PAppContext ctx, int numRetries, String outputFile, String url)EepGet(I2PAppContext ctx, int numRetries, String outputFile, String url, boolean allowCaching)EepGet(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String outputFile, String url)EepGet(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching) -
Method Summary
Modifier and Type Method Description voidaddAuthorization(String userName, String password)Add basic authorization header for the proxy.voidaddHeader(String name, String value)Add an extra header to the request.voidaddStatusListener(EepGet.StatusListener lsnr)protected voiddoFetch(SocketTimeout timeout)This reads the response to a single fetch.booleanfetch()Blocking fetch, returning true if the URL was retrieved, false if all retries failed.booleanfetch(long fetchHeaderTimeout)Blocking fetch, timing out individual attempts if the HTTP response headers don't come back in the time given.booleanfetch(long fetchHeaderTimeout, long totalTimeout, long inactivityTimeout)Blocking fetch.StringgetAcceptRanges()Show the Accept-Ranges fieldStringgetCacheControl()Show the Cache-Control fieldStringgetContentEncoding()Show the Content-Encoding fieldStringgetContentLanguage()Show the Content-Language fieldStringgetContentType()After fetch, the received value from the server, or null if none.StringgetCookie()Show the Set-Cookie fieldStringgetCSP()Show the Content-Security-Policy fieldStringgetEtag()After fetch, the received value from the server, or null if none.StringgetExpiryDate()Show the Expires fieldStringgetLastModified()After fetch, the received value from the server, or null if none.booleangetNotModified()StringgetReferrerPolicy()Show the Referrer-Policy fieldprotected StringgetRequest()StringgetServer()Show the Server fieldStringgetStatus()Show a combined server status message e.g.intgetStatusCode()The server response (200, etc).StringgetStatusText()The server text ("OK", "Not Found", etc).StringgetTransferEncoding()Show the Transfer-Encoding fieldStringgetVary()Show the Vary fieldStringgetXContentTypeOptions()Show the X-Content-Type-Options fieldStringgetXframeOptions()Show the X-frame-Options fieldStringgetXPoweredBy()Show the X-Powered-By fieldStringgetXSSProtection()Show the X-XSS-Protection fieldstatic voidmain(String[] args)EepGet [-p 127.0.0.1:4444] [-n #retries] [-e etag] [-o outputFile] [-m markSize lineLen] url As of 0.9.45, supports https and redirect to httpsstatic Map<String,String>parseAuthArgs(String args)Parse the args in an authentication header.protected longreadChunkLength()TODO this does not skip over chunk extensions (RFC 2616 sec.protected voidreadHeaders()protected voidsendRequest(SocketTimeout timeout)voidsetWriteErrorToOutput()If called (before calling fetch()), data from the server or proxy will be written to the output file or stream even on an error response code (4xx, 5xx, etc).protected booleanshouldReadBody()Should we read the body of the response?voidstopFetching()static StringsuggestName(String url)Parse URL for a viable filename.
-
Field Details
-
_context
-
_log
-
_shouldProxy
protected final boolean _shouldProxy -
_proxyHost
-
_proxyPort
protected final int _proxyPort -
_numRetries
protected final int _numRetries -
_outputFile
-
_outputStream
-
_url
url we were asked to fetch -
_actualURL
the URL we actually fetch from (may differ from the _url in case of redirect) -
_listeners
-
_extraHeaders
-
_keepFetching
protected boolean _keepFetching -
_proxy
-
_proxyOut
-
_proxyIn
-
_out
-
_alreadyTransferred
protected long _alreadyTransferred -
_bytesTransferred
protected long _bytesTransferred -
_bytesRemaining
protected long _bytesRemaining -
_currentAttempt
protected int _currentAttempt -
_responseCode
protected int _responseCode -
_responseText
-
_shouldWriteErrorToOutput
protected boolean _shouldWriteErrorToOutput -
_etag
-
_lastModified
-
_etagOrig
-
_lastModifiedOrig
-
_encodingChunked
protected boolean _encodingChunked -
_notModified
protected boolean _notModified -
_contentType
-
_server
-
_status
-
_transferEncoding
-
_contentEncoding
-
_contentLanguage
-
_cacheControl
-
_acceptRanges
-
_vary
-
_expiryDate
-
_cookie
-
_referrerPolicy
-
_xframeOptions
-
_csp
-
_xssProtection
-
_xContentTypeOptions
-
_xPoweredBy
-
_transferFailed
protected boolean _transferFailed -
_aborted
protected volatile boolean _aborted -
_fetchHeaderTimeout
protected int _fetchHeaderTimeout -
_fetchTotalTimeout
protected int _fetchTotalTimeout -
_fetchInactivityTimeout
protected int _fetchInactivityTimeout -
_redirects
protected int _redirects -
_redirectLocation
-
_isGzippedResponse
protected boolean _isGzippedResponse -
_decompressException
-
_authState
-
USER_AGENT
this will be replaced by the HTTP Proxy if we are using it- See Also:
- Constant Field Values
-
CONNECT_TIMEOUT
protected static final int CONNECT_TIMEOUT- See Also:
- Constant Field Values
-
INACTIVITY_TIMEOUT
protected static final int INACTIVITY_TIMEOUT- See Also:
- Constant Field Values
-
MAX_COMPLETE_FAILS
protected static final int MAX_COMPLETE_FAILSmaximum times to try without getting any data at all, even if numRetries is higher @since 0.7.14- See Also:
- Constant Field Values
-
-
Constructor Details
-
EepGet
public EepGet(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String outputFile, String url) -
EepGet
public EepGet(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching) -
EepGet
-
EepGet
public EepGet(I2PAppContext ctx, int numRetries, String outputFile, String url, boolean allowCaching) -
EepGet
public EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url) -
EepGet
public EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, String postData) -
EepGet
public EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching, String etag) -
EepGet
-
EepGet
public EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, long minSize, long maxSize, String outputFile, OutputStream outputStream, String url, boolean allowCaching, String etag, String postData)- Parameters:
outputFile- ignored if outputStream is non-nulloutputStream- takes precedence over outputFile
-
EepGet
public EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, long minSize, long maxSize, String outputFile, OutputStream outputStream, String url, boolean allowCaching, String etag, String lastModified, String postData)- Parameters:
outputFile- ignored if outputStream is non-nulloutputStream- takes precedence over outputFile
-
-
Method Details
-
main
EepGet [-p 127.0.0.1:4444] [-n #retries] [-e etag] [-o outputFile] [-m markSize lineLen] url As of 0.9.45, supports https and redirect to https -
suggestName
Parse URL for a viable filename.- Parameters:
url- a URL giving the location of an online resource- Returns:
- a filename to save the resource as on local filesystem
-
addStatusListener
-
stopFetching
public void stopFetching() -
fetch
public boolean fetch()Blocking fetch, returning true if the URL was retrieved, false if all retries failed. Header timeout default 45 sec, total timeout default none, inactivity timeout default 60 sec.- Returns:
- success
-
fetch
public boolean fetch(long fetchHeaderTimeout)Blocking fetch, timing out individual attempts if the HTTP response headers don't come back in the time given. If the timeout is zero or less, this will wait indefinitely. Total timeout default none, inactivity timeout default 60 sec.- Returns:
- success
-
fetch
public boolean fetch(long fetchHeaderTimeout, long totalTimeout, long inactivityTimeout)Blocking fetch.- Parameters:
fetchHeaderTimeout- <= 0 for none (proxy will timeout if none, none isn't recommended if no proxy)totalTimeout- <= 0 for default noneinactivityTimeout- <= 0 for default 60 sec- Returns:
- success
-
doFetch
This reads the response to a single fetch. Call after sendRequest()- Parameters:
timeout- may be null- Throws:
IOException
-
readHeaders
- Throws:
IOException
-
shouldReadBody
protected boolean shouldReadBody()Should we read the body of the response?- Returns:
- true always, overridden in EepHead
- Since:
- 0.9.50
-
readChunkLength
TODO this does not skip over chunk extensions (RFC 2616 sec. 3.6.1)- Throws:
IOException
-
sendRequest
- Parameters:
timeout- may be null- Throws:
IOException
-
getRequest
- Throws:
IOException
-
getEtag
After fetch, the received value from the server, or null if none. Before fetch, and after some errors, may be the value passed in the constructor. -
getLastModified
After fetch, the received value from the server, or null if none. Before fetch, and after some errors, may be the value passed in the constructor. -
getNotModified
public boolean getNotModified()- Returns:
- true if the server returned 304
-
getContentType
After fetch, the received value from the server, or null if none. -
getServer
Show the Server field- Since:
- I2P+ 0.9.47
-
getContentLanguage
Show the Content-Language field- Since:
- I2P+ 0.9.47
-
getTransferEncoding
Show the Transfer-Encoding field- Since:
- I2P+ 0.9.47
-
getContentEncoding
Show the Content-Encoding field- Since:
- I2P+ 0.9.47
-
getCacheControl
Show the Cache-Control field- Since:
- I2P+ 0.9.47
-
getAcceptRanges
Show the Accept-Ranges field- Since:
- I2P+ 0.9.47
-
getExpiryDate
Show the Expires field- Since:
- I2P+ 0.9.47
-
getCookie
Show the Set-Cookie field- Since:
- I2P+ 0.9.47
-
getReferrerPolicy
Show the Referrer-Policy field- Since:
- I2P+ 0.9.47
-
getVary
Show the Vary field- Since:
- I2P+ 0.9.47
-
getXframeOptions
Show the X-frame-Options field- Since:
- I2P+ 0.9.47
-
getCSP
Show the Content-Security-Policy field- Since:
- I2P+ 0.9.47
-
getXSSProtection
Show the X-XSS-Protection field- Since:
- I2P+ 0.9.47
-
getXContentTypeOptions
Show the X-Content-Type-Options field- Since:
- I2P+ 0.9.47
-
getXPoweredBy
Show the X-Powered-By field- Since:
- I2P+ 0.9.47
-
getStatusCode
public int getStatusCode()The server response (200, etc).- Returns:
- -1 if invalid, or if the proxy never responded, or if no proxy was used and the server never responded. If a non-proxied request partially succeeded (for example a redirect followed by a fail, or a partial fetch followed by a fail), this will be the last status code received. Note that fetch() may return false even if this returns 200.
- Since:
- 0.8.8
-
getStatusText
The server text ("OK", "Not Found", etc). Note that the text may contain % encoding.- Returns:
- null if invalid, or if the proxy never responded, or if no proxy was used and the server never responded. If a non-proxied request partially succeeded (for example a redirect followed by a fail, or a partial fetch followed by a fail), this will be the last status code received. Note that fetch() may return false even if this returns "OK".
- Since:
- 0.9.9
-
getStatus
Show a combined server status message e.g. "200 OK"- Since:
- I2P+ 0.9.47
-
setWriteErrorToOutput
public void setWriteErrorToOutput()If called (before calling fetch()), data from the server or proxy will be written to the output file or stream even on an error response code (4xx, 5xx, etc). The error data will only be written if no previous data was written on an earlier try. Caller must of course check getStatusCode() or the fetch() return value.- Since:
- 0.8.8
-
addHeader
Add an extra header to the request. Must be called before fetch(). Not supported by EepHead. As of 0.9.10, If name is User-Agent, this will replace the default User-Agent header. As of 0.9.14, If name is If-None-Match or If-Modified-Since, this will replace the etag or last-modified value given in the constructor. Note that headers may be subsequently modified or removed in the I2PTunnel HTTP Client proxy. In proxied SSLEepGet, these headers are sent to the remote server, NOT the proxy.- Since:
- 0.8.8
-
addAuthorization
Add basic authorization header for the proxy. Only added if the request is going through a proxy. Must be called before fetch().- Since:
- 0.8.9
-
parseAuthArgs
Parse the args in an authentication header. Modified from LoadClientAppsJob. All keys are mapped to lower case. Double quotes around values are stripped. Ref: RFC 2617 Public for I2PTunnelHTTPClientBase; use outside of tree at own risk, subject to change or removal- Parameters:
args- non-null, starting after "Digest " or "Basic "- Since:
- 0.9.4, moved from I2PTunnelHTTPClientBase in 0.9.12
-