Data Plane

• 1: Data plane integration
• 2: Building a data plane

1 - Data plane integration

• 1. Overview
• 2. Data plane registration
• 3. Data plane selection
  • 3.1 Building the catalog
  • 3.2 Selecting a data plane for a transfer
• 4. Signaling
  • 4.1 Signals sent to the data plane
  • 4.2 Notifications from the data plane
  • 4.3 The data address
• 5. Authorization profiles

1. Overview

The control plane integrates data planes through the Data Plane Signaling (DPS) protocol, implemented by the data-plane-signaling modules (data-protocols/data-plane-signaling):

• data-plane-signaling-core: the registration API, the signaling client, the DataFlowController that drives the data planes for the transfer process state machine, and the Signaling API that receives the data planes’ notifications;
• data-plane-signaling-oauth2 and data-plane-signaling-oauth2-token-exchange: the authorization profiles oauth2_client_credentials and oauth2_token_exchange (see 5. Authorization profiles).

The control plane doesn’t move data and doesn’t create data addresses: it decides which data plane handles a transfer and when, the data plane decides how.

2. Data plane registration

The life cycle of a data plane is decoupled from the one of the control plane: data planes can be added, replaced or removed at any time. A data plane registers with the control plane through the Management API:

• PUT /v4/dataplanes and DELETE /v4/dataplanes/{dataplaneId};
• PUT /v5/participants/{participantContextId}/dataplanes and DELETE /v5/participants/{participantContextId}/dataplanes/{dataplaneId}, which register the data plane for one participant context and need the scope management-api:dataplanes:write.

The request body is a plain JSON object:

{
  "dataplaneId": "dataplane-1",
  "endpoint": "https://dataplane.internal/api/signaling",
  "transferTypes": [ "HttpData-PULL", "HttpData-PUSH" ],
  "labels": [ "eu" ],
  "authorization": {
    "type": "oauth2_client_credentials",
    "tokenEndpoint": "https://idp.internal/token",
    "clientId": "control-plane",
    "clientSecret": "...",
    "jwksUri": "https://idp.internal/jwks"
  }
}

• dataplaneId: the id of the data plane, unique per participant context. Registering the same id again replaces the registration, so a data plane can register on every start.
• endpoint: the base URL of the data plane’s DPS API, to which the control plane sends the signals.
• transferTypes: the transfer types (DPS profiles) the data plane supports.
• labels: free-form labels that assets and transfer requests can require of the data plane (see 3.2).
• authorization: optional, the authorization profile with which the control plane and the data plane authenticate to each other (see 5. Authorization profiles).

A registered data plane is a logical instance: replicas of a data plane share one registration, behind one endpoint. The control plane doesn’t check whether a data plane is available, that is the job of the infrastructure (e.g. replicas behind a load balancer). The registrations are persisted (edc.sql.store.dataplaneinstance.datasource for the SQL store), so that all the control plane replicas see them.

3. Data plane selection

3.1 Building the catalog

The transfer types offered for an asset in the catalog, as Distributions, come from the registered data planes of the participant context:

• if the asset’s dataplaneMetadata lists no profiles, the transfer types of all the registered data planes;
• otherwise, the transfer types of the data planes that support at least one of the listed profiles.

For example, if one data plane supports HttpData-PULL and another AmazonS3-PUSH, an asset without profiles is offered with both.

3.2 Selecting a data plane for a transfer

When a transfer process is prepared (consumer) or started (provider), the control plane selects a data plane that:

• is registered for the participant context of the transfer process;
• supports the transfer type of the transfer process;
• has all the labels of the transfer process’s dataplaneMetadata. The provider takes them from the asset, the consumer from the transfer request.

If several data planes match, the selection strategy picks one: random by default, set with edc.dataplane.client.selector.strategy. Extensions can add strategies through the SelectionStrategyRegistry. The id of the selected data plane is stored on the transfer process, and all later signals go to that data plane.

4. Signaling

The DataPlaneSignalingFlowController is the DataFlowController of the control plane: the transfer process state machine calls it at each step, and it sends the corresponding DPS message to the data plane with the DataPlaneSignalingClient. The messages and their payloads are defined by the DPS specification; the transfer type of the transfer process is sent as the profile, together with the labels and the metadata (properties) of its dataplaneMetadata.

4.1 Signals sent to the data plane

The flowId is the id of the transfer process.

A data plane answers prepare, start and resume either synchronously, with the final state of the data flow (e.g. STARTED) and possibly a data address, or asynchronously, with a pending state (e.g. STARTING). In the asynchronous case, the transfer process waits in PREPARATION_REQUESTED or STARTUP_REQUESTED until the data plane notifies the control plane.

4.2 Notifications from the data plane

The data plane notifies the control plane through the Signaling API, a dedicated API context (web.http.signaling.port, default 8182, and web.http.signaling.path, default /api/signaling):

• POST /transfers/{transferId}/dataflow/prepared: an asynchronous prepare is done, with the data address if any;
• POST /transfers/{transferId}/dataflow/started: an asynchronous start is done, with the data address if any;
• POST /transfers/{transferId}/dataflow/completed: the data flow is complete, the control plane completes the transfer process and notifies the counter-party;
• POST /transfers/{transferId}/dataflow/errored: the data flow failed. The control plane logs the error, the transfer process doesn’t change state.

The data plane must be configured with the URL of this API: the control plane doesn’t send it in the prepare and start messages.

4.3 The data address

The data address carries what the counter-party needs to access the data flow, e.g. an endpoint and a token. It is created and used only by the data planes; the control plane stores it and passes it on through the Dataspace Protocol:

• pull: the provider data plane returns the data address in its start response (or started notification). The provider control plane sends it to the consumer in the DSP TransferStartMessage, and the consumer control plane hands it to its data plane with started.
• push: the consumer data plane returns the destination as data address in its prepare response (or prepared notification). The consumer control plane sends it in the DSP TransferRequestMessage, and the provider control plane passes it to its data plane in the start message.

5. Authorization profiles

The authorization object of the registration selects how the control plane authenticates its signals to the data plane, and how it authenticates the data plane’s notifications:

• oauth2_client_credentials (data-plane-signaling-oauth2): the control plane gets a token with the client credentials (tokenEndpoint, clientId, clientSecret) for its signals, and validates the tokens of the notifications against jwksUri or jwks.
• oauth2_token_exchange (data-plane-signaling-oauth2-token-exchange): the control plane exchanges its workload credential (by default the Kubernetes service account token, edc.dps.oauth2.tokenexchange.subjecttokenpath) for a token at tokenExchangeEndpoint; the notifications are validated against the issuer and jwksUri or jwks of the profile.

Without an authorization object the signals and the notifications are not authenticated. The control plane logs a warning when such a data plane registers, see Securing the Signaling API. The authorization profiles are defined in the DPS profiles; further ones can be added by extensions through the SignalingAuthorizationRegistry.

2 - Building a data plane

• 1. Where to start
• 2. What a data plane implements
  • 2.1 Registration
  • 2.2 The DPS endpoints
  • 2.3 Data addresses and credentials
  • 2.4 Notifications
• 3. Testing against the EDC control plane

A data plane is any application that implements the Data Plane Signaling (DPS) protocol and registers with the control plane. It can be written in any language and run anywhere the control plane can reach it. This page describes what the EDC control plane expects of it; the protocol itself is defined by the specification.

1. Where to start

• Use an existing data plane. siglet, from the Data Plane Core project, is driven by DPS and issues, verifies and renews the access tokens of the data flows, so that existing APIs and data infrastructure can serve the data.
• Build one with a Data Plane SDK. The Data Plane SDKs implement the DPS endpoints, the data flow state machine and its persistence, and leave the transfer itself to your code: Java, Go, Rust.
• Implement the specification directly, if no SDK fits.

The transfer types a data plane supports are DPS profiles; the common ones are listed in the DPS profiles. A data plane can also define profiles of its own, as long as the participants of a dataspace agree on them.

2. What a data plane implements

2.1 Registration

At startup, the data plane registers with the control plane, with its id, its DPS endpoint, the transfer types it supports, its labels and its authorization profile. Registration is idempotent, so the data plane can register on every start; it deregisters only when it is removed for good. The API and the payload are described in Data plane registration.

All the replicas of a data plane register with the same id and endpoint: the control plane sends the signals of a data flow to that endpoint, and any replica must be able to handle them, so the data flows are kept in a store shared by the replicas.

2.2 The DPS endpoints

The control plane calls the endpoints defined by the specification, relative to the registered endpoint (see Signals sent to the data plane):

• prepare (consumer side) and start (provider side) create the data flow, whose id is the id of the transfer process. They receive the profile (the transfer type), the agreement and dataset ids, the participants, and the labels and metadata of the asset or transfer request; the metadata carries what the data plane needs to find the data, e.g. the URL of the backend of an HTTP asset.
• {flowId}/started (consumer side) hands over the provider’s data address of a pull transfer.
• {flowId}/suspend, {flowId}/resume, {flowId}/terminate and {flowId}/completed change the state of the data flow.

Every call can be repeated: the control plane retries a signal whose response it didn’t receive, so a data plane answers a repeated call with the current state of the data flow instead of an error.

A data plane can answer prepare, start and resume synchronously, with the final state, or asynchronously, with a pending state, and notify the control plane later (see 2.4 Notifications). An asynchronous answer fits work that takes time, e.g. provisioning a bucket or a queue for the transfer.

2.3 Data addresses and credentials

The data plane creates the data address of a transfer and the credentials in it, and validates them when the counter-party uses them; the control plane only passes them on:

• pull: the provider data plane returns, in its start answer, the endpoint and the token at which the consumer fetches the data, and accepts requests with that token until the data flow is suspended, terminated or completed.
• push: the consumer data plane returns, in its prepare answer, the destination the provider pushes the data to; the provider data plane receives it in the start message.

2.4 Notifications

The data plane reports the progress of a data flow to the control plane’s Signaling API: prepared and started after an asynchronous answer, completed when a finite transfer is done, errored when it failed (see Notifications from the data plane). The URL of the Signaling API is part of the data plane’s configuration. With an authorization profile, the data plane authenticates these calls, and validates the control plane’s signals, as the profile defines (see Authorization profiles).

3. Testing against the EDC control plane

The Connector’s end-to-end transfer tests use a data plane built with the Java SDK (system-tests/e2e-transfer-test/signaling-data-plane in the Connector repository): it shows a registration, the handling of the signals, and pull and push transfers between two control planes, and can serve as a starting point for tests of your own data plane.