HTTP Client Adapters¶

pyhaul works with the HTTP client you already use. Pass your session object directly to haul() or haul_async() — pyhaul auto-detects the library and wraps it internally.

Supported clients¶

Extra	Client type	Async	Minimum	Install
`httpx`	`httpx.Client` / `httpx.AsyncClient`	Yes	0.27	`pip install pyhaul[httpx]`
`niquests`	`niquests.Session` / `niquests.AsyncSession`	Yes	3.14	`pip install pyhaul[niquests]`
`aiohttp`	`aiohttp.ClientSession`	Async only	3.10	`pip install pyhaul[aiohttp]`
`wreq`	`wreq.blocking.Client` / `wreq.Client`	Yes	0.11	`pip install pyhaul[wreq]`
`requests`	`requests.Session`	Sync only	2.32	`pip install pyhaul[requests]`
`urllib3`	`urllib3.PoolManager`	Sync only	2.0	`pip install pyhaul[urllib3]`

Supported client versions¶

Each pyhaul release supports every client release published in the 24 months before that pyhaul release's date, and often older ones. A pyhaul version's window is fixed when it ships.

A minimum version rises only in a pyhaul minor release, and only to a client release published at least 24 months before that pyhaul release.
A security fix or a hard technical need can raise a minimum sooner. The changelog calls out every raise.
CI runs the full suite against each minimum on the oldest supported Python.

Your session, your config¶

pyhaul borrows your session. It never creates, configures, or closes it. This means all your existing configuration — auth headers, proxies, connection pools, TLS settings, timeouts — passes through unchanged.

import httpx
from pyhaul import haul

client = httpx.Client(
    headers={"Authorization": "Bearer sk-..."},
    proxy="http://proxy.corp:3128",
    timeout=30.0,
    limits=httpx.Limits(max_connections=10),
)
with client:
    result = haul("https://internal.cdn/asset.bin", client, dest="asset.bin")

pyhaul adds per-request headers for range negotiation (Range, If-Range, Accept-Encoding: identity, Cache-Control: no-transform) but does not modify your session. After haul() returns, the session is exactly as you left it.

You can also supply extra headers for that download via the optional headers= keyword on haul() / haul_async(). Those merge before structural headers; structural headers still win where they overlap. That is separate from session defaults (which many clients merge in on their own). To adjust headers inside the adapter layer — for example tests or telemetry — implement or wrap prepare_headers(); see Writing a Custom Adapter.

Redirects¶

pyhaul does not override your HTTP client's redirect policy. Library defaults differ: httpx.Client / httpx.AsyncClient use follow_redirects=False (httpx treats opt-in redirect following as conservative with respect to credentials on cross-host redirects), and wreq.blocking.Client / wreq.Client do not follow redirects either; requests.Session, niquests.Session, aiohttp.ClientSession, and urllib3 (via pyhaul's adapter) follow redirects in the usual configurations unless you turn that off. CDN and mirror URLs often redirect — configure the session or client you pass to pyhaul accordingly (for example httpx.Client(follow_redirects=True) or wreq.Client(redirect=wreq.redirect.Policy.limited())).

Per-client examples¶

httpx (sync)¶

import httpx
from pyhaul import haul

with httpx.Client() as client:
    result = haul("https://example.com/file.bin", client, dest="file.bin")

httpx (async)¶

import asyncio
import httpx
from pyhaul import haul_async


async def main():
    async with httpx.AsyncClient() as client:
        result = await haul_async("https://example.com/file.bin", client, dest="file.bin")


asyncio.run(main())

requests¶

import requests
from pyhaul import haul

with requests.Session() as session:
    result = haul("https://example.com/file.bin", session, dest="file.bin")

niquests (sync)¶

import niquests
from pyhaul import haul

with niquests.Session() as session:
    result = haul("https://example.com/file.bin", session, dest="file.bin")

niquests (async)¶

import asyncio
import niquests
from pyhaul import haul_async


async def main():
    async with niquests.AsyncSession() as session:
        result = await haul_async("https://example.com/file.bin", session, dest="file.bin")


asyncio.run(main())

aiohttp¶

import asyncio
import aiohttp
from pyhaul import haul_async


async def main():
    async with aiohttp.ClientSession() as session:
        result = await haul_async("https://example.com/file.bin", session, dest="file.bin")


asyncio.run(main())

wreq (sync)¶

import wreq.blocking
from pyhaul import haul

with wreq.blocking.Client() as client:
    result = haul("https://example.com/file.bin", client, dest="file.bin")

wreq (async)¶

import asyncio
import wreq
from pyhaul import haul_async


async def main():
    async with wreq.Client() as client:
        result = await haul_async("https://example.com/file.bin", client, dest="file.bin")


asyncio.run(main())

urllib3¶

import urllib3
from pyhaul import haul

http = urllib3.PoolManager()
result = haul("https://example.com/file.bin", http, dest="file.bin")

Auth and session integration¶

Since pyhaul uses your session as-is, authentication works exactly as it does in your application today.

Bearer tokens¶

client = httpx.Client(headers={"Authorization": "Bearer sk-..."})

Basic auth¶

httpxrequestsniquestsaiohttpurllib3wreq

client = httpx.Client(auth=("user", "password"))

session = requests.Session()
session.auth = ("user", "password")

session = niquests.Session()
session.auth = ("user", "password")

aiohttp 3.14 deprecates auth=aiohttp.BasicAuth(...). A default Authorization header works on every supported aiohttp version:

token = base64.b64encode(b"user:password").decode()
session = aiohttp.ClientSession(headers={"Authorization": f"Basic {token}"})

http = urllib3.PoolManager(headers=urllib3.make_headers(basic_auth="user:password"))

wreq.Client takes basic auth only per request, so set the header on the client:

token = base64.b64encode(b"user:password").decode()
client = wreq.Client(headers={"Authorization": f"Basic {token}"})

Custom auth flows¶

If your application uses a custom auth handler (e.g. OAuth token refresh), attach it to the session before passing it to pyhaul:

from httpx import Auth, Request, Response


class TokenRefreshAuth(Auth):
    def auth_flow(self, request: Request):
        request.headers["Authorization"] = f"Bearer {self.get_token()}"
        yield request

    def get_token(self) -> str: ...  # your token refresh logic


client = httpx.Client(auth=TokenRefreshAuth())
result = haul(url, client, dest="file.bin")

Proxies¶

httpxrequestsniquestsurllib3aiohttpwreq

client = httpx.Client(proxy="http://proxy.corp:3128")

session = requests.Session()
session.proxies = {
    "http": "http://proxy.corp:3128",
    "https": "http://proxy.corp:3128",
}

session = niquests.Session()
session.proxies = {
    "http": "http://proxy.corp:3128",
    "https": "http://proxy.corp:3128",
}

http = urllib3.ProxyManager("http://proxy.corp:3128")

Session-level proxy= needs aiohttp 3.11 or later. On aiohttp 3.10, pass trust_env=True and set HTTP_PROXY / HTTPS_PROXY in the environment.

session = aiohttp.ClientSession(proxy="http://proxy.corp:3128")

client = wreq.Client(proxies=[wreq.Proxy.all("http://proxy.corp:3128")])

Connection pooling¶

Your session's connection pool is shared across all haul() calls. pyhaul does not create its own connections.

client = httpx.Client(
    limits=httpx.Limits(
        max_connections=20,
        max_keepalive_connections=10,
    )
)

Adapter-specific notes¶

httpx¶

Both sync (httpx.Client) and async (httpx.AsyncClient) are supported.
pyhaul uses response.iter_raw() / response.aiter_raw() to bypass content decoding, ensuring byte-accurate resume.
Per-request follow_redirects can be forwarded via TransportRequestOptions when an adapter receives it (see Writing a Custom Adapter); otherwise httpx uses your client's default (constructor default is follow_redirects=False — see Redirects above).

requests¶

Not thread-safe. If you need concurrent sync downloads, create one requests.Session per thread or use niquests.Session instead.
pyhaul reads raw bytes via response.raw.stream(chunk_size, decode_content=False) to avoid decompression interference with range requests.

niquests¶

Thread-safe and supports HTTP/2 and HTTP/3.
Both sync and async sessions are supported.
Uses the same raw streaming approach as the requests adapter.

aiohttp¶

Async only — there is no sync aiohttp adapter.
pyhaul sets auto_decompress=False on the request to ensure raw bytes for accurate resume.
TLS certificate errors map to TransportTLSError.

wreq¶

Both sync (wreq.blocking.Client) and async (wreq.Client) are supported.
The sync client is thread-safe and releases the GIL during network I/O, so one client can serve a thread pool. Calls block until they return and Ctrl-C waits for them, so set a read timeout to bound a stalled server.
A default wreq client decompresses gzip, brotli, deflate, and zstd bodies; pyhaul turns that off on every request to keep raw bytes for accurate resume.
Responses close when pyhaul is done with them. On wreq 0.13+ a fully read response returns its connection to the pool; on 0.11 each download uses a fresh connection.
Body chunks are passed through as zero-copy memoryviews (wreq 0.13+).
TLS verification and connect timeouts are client settings; set them on the wreq client. TLS errors map to TransportTLSError; read failures and truncated bodies map to TransportConnectionError.

urllib3¶

Sync only. Pass a urllib3.PoolManager (or urllib3.ProxyManager for proxied requests).
The adapter streams raw bytes via the response's stream() method.

Error handling¶

Transport errors from your HTTP library pass through unwrapped. pyhaul does not wrap httpx.ReadTimeout in a pyhaul exception — you catch the types you already know:

import httpx
from pyhaul import haul, PartialHaulError

try:
    result = haul(url, client, dest="file.bin")
except PartialHaulError:
    # stream ended early — progress saved, safe to retry
    ...
except httpx.ReadTimeout:
    # your library's native timeout — handle as you normally would
    ...
except httpx.ConnectError:
    # connection failed — no progress to save
    ...

See Exceptions Reference for the full pyhaul exception hierarchy, including PartialHaulError.