OpenTelemetry urllib Instrumentation

This library allows tracing HTTP requests made by the urllib library.

Usage

from urllib import request
from opentelemetry.instrumentation.urllib import URLLibInstrumentor

# You can optionally pass a custom TracerProvider to
# URLLibInstrumentor().instrument()

URLLibInstrumentor().instrument()
req = request.Request('https://postman-echo.com/post', method="POST")
r = request.urlopen(req)

Configuration

Request/Response hooks

The urllib instrumentation supports extending tracing behavior with the help of request and response hooks. These are functions that are called back by the instrumentation right after a Span is created for a request and right before the span is finished processing a response respectively. The hooks can be configured as follows:

from http.client import HTTPResponse
from urllib.request import Request

from opentelemetry.instrumentation.urllib import URLLibInstrumentor
from opentelemetry.trace import Span


def request_hook(span: Span, request: Request):
    pass


def response_hook(span: Span, request: Request, response: HTTPResponse):
    pass


URLLibInstrumentor().instrument(
    request_hook=request_hook,
    response_hook=response_hook
)

Exclude lists

To exclude certain URLs from being tracked, set the environment variable OTEL_PYTHON_URLLIB_EXCLUDED_URLS (or OTEL_PYTHON_EXCLUDED_URLS as fallback) with comma delimited regexes representing which URLs to exclude.

For example,

export OTEL_PYTHON_URLLIB_EXCLUDED_URLS="client/.*/info,healthcheck"

will exclude requests such as https://site/client/123/info and https://site/xyz/healthcheck.

Capture HTTP request and response headers

You can configure the agent to capture specified HTTP headers as span attributes, according to the semantic conventions.

Request headers

To capture HTTP request headers as span attributes, set the environment variable OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST to a comma delimited list of HTTP header names.

For example using the environment variable,

export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST="content-type,custom_request_header"

will extract content-type and custom_request_header from the request headers and add them as span attributes.

Request header names in urllib are case-insensitive. So, giving the header name as CUStom-Header in the environment variable will capture the header named custom-header.

Regular expressions may also be used to match multiple headers that correspond to the given pattern. For example:

export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST="Accept.*,X-.*"

Would match all request headers that start with Accept and X-.

To capture all request headers, set OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST to ".*".

export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST=".*"

The name of the added span attribute will follow the format http.request.header.<header_name> where <header_name> is the normalized HTTP header name (lowercase, with - replaced by _). The value of the attribute will be a single item list containing all the header values.

For example: http.request.header.custom_request_header = ["<value1>", "<value2>"]

Note

Some headers are injected at a lower level by the http.client module and so are not captured by this instrumentation

Response headers

To capture HTTP response headers as span attributes, set the environment variable OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE to a comma delimited list of HTTP header names.

For example using the environment variable,

export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE="content-type,custom_response_header"

will extract content-type and custom_response_header from the response headers and add them as span attributes.

Response header names in urllib are case-insensitive. So, giving the header name as CUStom-Header in the environment variable will capture the header named custom-header.

Regular expressions may also be used to match multiple headers that correspond to the given pattern. For example:

export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE="Content.*,X-.*"

Would match all response headers that start with Content and X-.

To capture all response headers, set OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE to ".*".

export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE=".*"

The name of the added span attribute will follow the format http.response.header.<header_name> where <header_name> is the normalized HTTP header name (lowercase, with - replaced by _). The value of the attribute will be a list containing the header values.

For example: http.response.header.custom_response_header = ["<value1>", "<value2>"]

Sanitizing headers

In order to prevent storing sensitive data such as personally identifiable information (PII), session keys, passwords, etc, set the environment variable OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS to a comma delimited list of HTTP header names to be sanitized.

Regexes may be used, and all header names will be matched in a case-insensitive manner.

For example using the environment variable,

export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS=".*session.*,set-cookie"

will replace the value of headers such as session-id and set-cookie with [REDACTED] in the span.

Note

The environment variable names used to capture HTTP headers are still experimental, and thus are subject to change.

API

class opentelemetry.instrumentation.urllib.URLLibInstrumentor(*args, **kwargs)[source]

Bases: BaseInstrumentor

An instrumentor for urllib See BaseInstrumentor

instrumentation_dependencies()[source]

Return a list of python packages with versions that the will be instrumented.

The format should be the same as used in requirements.txt or pyproject.toml.

For example, if an instrumentation instruments requests 1.x, this method should look like: :rtype: Collection[str]

def instrumentation_dependencies(self) -> Collection[str]:
return [‘requests ~= 1.0’]

This will ensure that the instrumentation will only be used when the specified library is present in the environment.

uninstrument_opener(opener)[source]: uninstrument_opener a specific instance of urllib.request.OpenerDirector