LDIO HTTP Out

Ldio:HttpOut

The LDIO HTTP Out is a basic Http Client that will send the given Linked Data model to a target url.

Config

Property	Description	Required	Default	Example	Supported values
endpoint	Target url.	Yes	N/A	http://example.com/endpoint	HTTP and HTTPS urls
continueOnFail	Indicates if the pipeline continues when the target still answers with a non-2xx status after the retries.	No	true	false	true or false

Failed deliveries

When the target still answers with a non-2xx status after the configured retries (see the retry properties below), the output always logs the failure. What happens next depends on continueOnFail:

true (default): the member is dropped and the pipeline continues. The input is not informed, so for example Ldio:HttpIn still answers 202 and ldio_data_out_total still counts the member.
false: the output throws an exception. The member is not counted in ldio_data_out_total, and the input decides what happens:
- Ldio:HttpIn answers with an error status (500) instead of 202, so the caller can retry.
- Ldio:HttpInPoller logs the error; the next scheduled poll still runs. Members later in the same polled response are not processed in that poll.
- Ldio:LdesClient halts the pipeline. The failed member is still in the client state at that point, so with a persisted state it is offered again when the client restarts. Resuming the pipeline through the admin API does not restart the client; restart the LDIO instead.

Connection errors (for example an unreachable target) are always raised to the input, regardless of this setting.

RDF Writer Properties

Property	Description	Required	Default	Supported values	Example
rdf-writer.content-type	Target content type.	No	text/turtle	Any type supported by Apache Jena	application/ld+json
rdf-writer.frame	Additional JSON-LD Frame to format the outputted JSON-LD Object.	No	N/A	Any valid JSON Object that describes a JSON-LD Frame	See https://www.w3.org/TR/json-ld11-framing/#sample-library-frame

Example RDF Writer config

Format as N-Quads:

      config:
        rdf-writer:
          content-type: application/n-quads

Format as JSON-LD with given frame:

      config:
        rdf-writer:
          content-type: application/ld+json
          frame: |
            {
              "@context": {"@vocab": "http://example.org/"},
              "@type": "Library",
              "contains": {
                "@type": "Book",
                "contains": {
                  "@type": "Chapter"
                }
              }
            }

LDIO Http Requester properties

Authentication properties

Property	Description	Required	Default	Supported values	Example
auth.type	The type of authentication required by the LDES server	No	NO_AUTH	NO_AUTH, API_KEY or OAUTH2_CLIENT_CREDENTIALS	OAUTH2_CLIENT_CREDENTIALS
auth.api-key	The api key when using auth.type ‘API_KEY’	No	N/A	String	myKey
auth.api-key-header	The header for the api key when using auth.type ‘API_KEY’	No	X-API-KEY	String	X-API-KEY
auth.client-id	The client identifier when using auth.type ‘OAUTH2_CLIENT_CREDENTIALS’	No	N/A	String	myId
auth.client-secret	The client secret when using auth.type ‘OAUTH2_CLIENT_CREDENTIALS’	No	N/A	String	mySecret
auth.token-endpoint	The token endpoint when using auth.type ‘OAUTH2_CLIENT_CREDENTIALS’	No	N/A	HTTP and HTTPS urls	http://localhost:8000/token
auth.scope	The Oauth2 scope when using auth.type ‘OAUTH2_CLIENT_CREDENTIALS’	No	N/A	HTTP and HTTPS urls	http://localhost:8000/token

Retry properties

Property	Description	Required	Default	Supported values	Example
retries.enabled	Indicates if the http client should retry http requests when the server cannot be reached.	No	true	Boolean value	true
retries.max	Max number of retries the http client should do when retries.enabled = true	No	5	Integer	100
retries.statuses-to-retry	Custom comma seperated list of http status codes that can trigger a retry in the http client.	No	N/A	Comma seperated list of Integers	410,451

When retries are enabled, the following statuses are always retried, regardless of the configured statuses-to-retry:

429
5xx (500 and above)

Rate limit properties

Property	Description	Required	Default	Supported values	Example
rate-limit.enabled	Indicates if the http client should limit http requests when calling the server.	No	false	true or false	false
rate-limit.limit	Limit of requests per period, which is defined below, that the http client should do when `rate-limit.enabled = true`	No	500	Integer	100
rate-limit.period	Period in which the limit of requests, which is defined above, can be reached by the http client when `rate-limit.enabled = true`	No	PT1M	ISO 8601 Duration	PT1H

Timeout properties

Property	Description	Required	Default	Supported values	Example
timeouts.connect	Maximum time to establish the connection with the server.	No	PT30S	Positive ISO 8601 Duration	PT10S
timeouts.connection-request	Maximum time to wait for a free connection from the http client’s connection pool.	No	PT1M	Positive ISO 8601 Duration	PT30S
timeouts.socket	Maximum time of inactivity while waiting for data from the server, e.g. for the response of a slow query. Raise it for endpoints that legitimately need longer.	No	PT5M	Positive ISO 8601 Duration	PT15M

A request that exceeds one of these timeouts fails with an error and is retried like any other connection error when retries are enabled. Without a timeout, a server that accepts the connection but never answers would block the pipeline indefinitely.

Http headers

Property	Description	Required	Default	Supported values	Example
http.headers.[].key/value	A list of custom http headers can be added. A key and value has to be provided for every header.	No	N/A	String	role

Example Http Requester config

      config:
        http:
          headers:
            - key: role
              value: developer
            - key: alt-role
              value: programmer
        auth:
          type: API_KEY
          api-key: my-secret
          api-key-header: x-api-key
        retries:
          enabled: true
          max: 10
          statuses-to-retry: 410,451
        rate-limit:
          enabled: true
          period: P1D
          limit: 1000
        timeouts:
          connect: PT10S
          socket: PT15M