Contributors Manual

• 1: Best practices and recommendations
• 2: Autodoc Gradle plugin
• 3: OpenApi spec
• 4: Data Persistence with PostgreSQL
• 5: Logging
• 6: Writing tests
• 7: Control Plane
• 7.1: Entities
• 7.2: Json LD
• 7.3: Policy Monitor
• 7.4: Programming Primitives
• 7.5: Protocol Extensions
• 7.6: Service Layers
• 7.7: Dependency Injection
• 7.8: Extension Model
• 8: Data Plane
• 8.1: Extensions
• 8.2: Data Plane Signaling interface
• 8.3: Custom Data Plane
• 9: Custom validation framework
• 10: Performance Tuning
• 11: Instrumentation with Micrometer
• 12: Contribution Guidelines
• 12.1: Pull Request Etiquette
• 12.2: Style Guide
• 12.3: PR Check List

@Extension(value = "Some supercool extension", categories = {"category1", "category2"})
public class SomeSupercoolExtension implements ServiceExtension {

  // default value -> not required
  @Setting(value = "Some string config property", type = "string", defaultValue = "foobar", required = false)
  public static final String SOME_STRING_CONFIG_PROPERTY = "edc.some.supercool.string";

  //no default value -> required
  @Setting(value = "Some numeric config", type = "integer", required = true)
  public static final String SOME_INT_CONFIG_PROPERTY = "edc.some.supercool.int";

  // ...
}

1 - Best practices and recommendations

• Best practices and recommendations
  • 1. Preface
  • 2. Security recommendations
    • 2.1 Exposing APIs to the internet
    • 2.2 Use only official TLS certificates/CAs
    • 2.3 Use appropriate network infrastructure
    • 2.4 A word on authentication and authorization
    • 2.5 Docker builds
    • 2.6 Use proper database security
    • 2.7 Store sensitive data in a vault
  • 3. General recommendations
    • 3.1 Use only official releases
    • 3.2 Dependency hygiene
    • 3.3 Use database persistence wherever possible
    • 3.4 Use proper Vault implementations
    • 3.4 Use UUIDs as object identifiers

1. Preface

This document aims at giving guidelines and recommendations to developers who want to use or extend EDC or EDC modules in their applications, to DevOps engineers who are tasked with packaging and operating EDC modules as runnable application images.

Please understand this document as a recommendation from the EDC project committers team that they compiled to the best of their knowledge. We realize that use case scenarios are plentiful and requirements vary, and not every best practice is applicable everywhere. You know your use case best.

This document is not an exhaustive list of prescribed steps, that will shield adopters from any conceivable harm or danger, but rather should serve as starting point for engineers to build upon.

Finally, it should go without saying that the software of the EDC project is distributed “as is” and committers of EDC take no responsibility or liability, direct or indirect, for any harm or damage caused by the us`e of it. This document does not change that.

2. Security recommendations

2.1 Exposing APIs to the internet

The EDC code base has several outward-facing APIs, exclusively implemented as HTTP/REST endpoints. These have different purposes, different intended consumers and thus different security implications.

As a general rule, APIs should not be exposed directly to the internet. That does not mean that they shouldn’t be accessible via the internet, obviously the connector and related components cannot work without a network connection. This only means that API endpoints should not be directly facing the internet, instead, there should be appropriate infrastructure in place.

It also means that we advise extreme caution when making APIs accessible via the internet - by default only the DSP API and the data plane’s public API should be accessible via the internet, the others (management API, signaling API,…) are intended only for local network access, e.g. within a Kubernetes cluster.

Corporate security policies might require that only HTTPS/TLS connections be used, even between pods in a Kubernetes cluster. While the EDC project makes no argument pro or contra, that is certainly an idea worth considering in high security environments.

The key take-away is that all of EDC’s APIs - if accessible outside the local network - should only be accessible through separate software components such as API gateways or load balancers. These are specialized tools with the sole purpose of performing authentication, authorization, rate limiting, IP blacklisting/whitelisting etc.

There is a plethora of ready-made components available, both commercial and open-source, therefor the EDC project will not provide that functionality. Feature requests and issues to that effect will be ignored.

In the particular case of the DSP API, the same principle holds, although with the exception of authentication and authorization. That is handled by the DSP protocol itself.

We have a rudimentary token-based API security module available, which can be used to secure the connection API gateway <-> connector if so desired. It should be noted that it is not designed to act as a ingress point!

TL;DR: don’t expose any APIs if you can help it, but if you must, use available tools to harden the ingress

2.2 Use only official TLS certificates/CAs

Typically, JVMs ship with trust stores that contain a number of widely accepted CAs. Any attempts to package additional CAs/certificates with runtime base images are discouraged, as that would be problematic because:

• scalability: in a heterogenous networks one cannot assume such a custom CA to be accepted by the counterparty
• maintainability: TLS certificates expire, so there is a chance that mandatory software rollouts become necessary because of expired certificates lest the network breaks down completely.
• security: there have been a number of issues with CAs (1, 2), so adding non-sanctioned ones brings a potential security weakness

2.3 Use appropriate network infrastructure

As discussed earlier, EDC does not (and will not) provide or implement tooling to harden network ingress, as that is an orthogonal concern, and there are tools better suited for that.

We encourage every connector deployment to plan and design their network layout and infrastructure right from the onset, before even writing code. Adding that later can be difficult and time-consuming.

For example, in Kubernetes deployments, which are the de-facto industry standard, networking can be taken on by ingress controllers and load balancers. Additional external infrastructure, such as API gateways are recommended to handle authentication, authorization and request throttling.

2.4 A word on authentication and authorization

EDC does not have a concept of a “user account” as many client-facing applications do. In terms of identity, the connector itself represents a participant in a dataspace, so that is the level of granularity the connector operates on. That means, that client-consumable APIs such as the Management API only have rudimentary security. This is by design and must be solved out-of-band.

The reasoning behind this is that requirements for authentication and authorization are so diverse and heterogeneous, that it is virtually impossible for the EDC project to satisfy them all, or even most of them. In addition, there is very mature software available that is designed for this very use case.

Therefore, adopters of EDC have two options to consider:

1. develop a custom AuthenticationService (or even a ContainerRequestFilter), that integrates with an IDP
2. use a dedicated API gateway (recommended)

Both these options are viable, and may have merit depending on the use case.

2.5 Docker builds

As Docker is a very popular method to build and ship applications, we put forward the following recommendations:

• use official Eclipse Temurin base images for Java
• use dedicated non-root users: in your Dockerfile, add the following lines

   ARG APP_USER=docker
   ARG APP_UID=10100
   RUN addgroup --system "$APP_USER"
   RUN adduser \
  shell /sbin/nologin \
  disabled-password \
  gecos "" \
  ingroup "$APP_USER" \
  no-create-home \
  uid "$APP_UID" \
  APP_USER"
  
   USER "$APP_USER"
  

2.6 Use proper database security

Database connections are secured with a username and a password. Please choose non-default users and strong passwords. In addition, database credentials should be stored in an HSM (vault).

Further, the roles of the technical user for the connector should be limited to SELECT, INSERT, UPDATE, and DELETE. There is no reason for that user to have permissions to modify databases, tables, permissions or execute other DDL statements.

2.7 Store sensitive data in a vault

While the default behaviour of EDC is that configuration values are taken either from environment variables, system properties or from configuration extensions, it is highly recommended to store sensitive data in a vault when developing EDC extensions.

Here is a (non-exhaustive) list of examples of such sensitive values:

• database credentials
• cryptographic keys, e.g. private keys in an asymmetric key pair
• symmetric keys
• API keys/tokens
• credentials for other third-party services, even if temporary

Sensitive values should not be passed through multiple layers of code. Instead, they should be referenced by their alias, and be resolved from the vault wherever they are used. Do not store sensitive data as class members but use local variables that are garbage-collected when leaving execution scope.

3. General recommendations

3.1 Use only official releases

We recommend using only official releases of our components. The latest version can be obtained from the project’s GitHub releases page and the modules are available from MavenCentral.

Snapshots are less stable, less tested and less reliable than release versions and they make for non-repeatable builds.

That said, we realize that sometimes living on the bleeding edge of technology is thrilling, or in some circumstances even necessary. EDC components publish a -SNAPSHOT build on every commit the main branch, so there could be several such builds per day, each overwriting the previous one. In addition, we publish nightly builds, that are versioned <VERSION>-<YYYYMMDD>-SNAPSHOT and those don’t get overwritten. For more information please refer to the respective documentation.

3.2 Dependency hygiene

It should be at the top of every software engineer’s todo list to keep application dependencies current, to avoid security issues, minimize technical debt and prevent difficult upgrade paths. We strongly recommend using a tool to keep dependencies up-to-date, or at least notify when a new version is out.

This is especially true for EDC versions. Since the project has not yet reached a state of equilibrium, where we can follow SemVer rules, major (potentially breaking) changes and incompatibilities are to be expected on every version increment.

Internally we use dependabot to maintain our dependencies, as it is well integrated with GitHub actions, but this is not an endorsement. Alternatives exist.

3.3 Use database persistence wherever possible

While the connector runtime provides in-memory persistence by default, it is recommended to use database persistence in production scenarios, if possible. Hosting the persistence of several modules (e.g. AssetIndex and PolicyDefinitionStore) in the same database is generally OK.

This is because although memory stores are fast and easy to use, they have certain drawbacks, for instance:

• clustered deployments: multiple replica don’t have the same data, thus they would operate on inconsistent data
• security: if an attacker is able to create a memdump of the pod, they gain access to all application data
• memory consumption: Kubernetes has no memory limits out-of-the-box, so depending on the amount of data that is stored by a connector, this could cause runtime problems when databases start to grow, especially on resource constrained deployments.

3.4 Use proper Vault implementations

Similar to the previous section, proper HSM (Vault) implementations should be used in all but the most basic test and demo scenarios. Vaults are used to store the most sensitive information, and by default EDC provides only an in-memory variant.

3.4 Use UUIDs as object identifiers

While we don’t enforce any particular shape or form for object identifiers, we recommend using UUIDs because they are reasonably unique, reasonably compact, and reasonably available on most tech stacks. Use the JDK UUID implementation. It’s good enough.

2 - Autodoc Gradle plugin

• 1. Introduction
• 2. Module structure
• 3. Usage
  • 3.1 Add the plugin to the buildscript block of your build.gradle.kts:
  • 3.2 Apply the plugin to the project:
  • 3.3 Configure the plugin [optional]
• 4. Merging the manifests
• 5. Rendering manifest files as Markdown or HTML
• 6. Using published manifest files (MavenCentral)

1. Introduction

In EDC, the autodoc plugin is intended to be used to generate metamodel manifests for every Gradle module, which can then transformed into Markdown or HTML files, and subsequently be rendered for publication in static web content.

The plugin code can be found in the GradlePlugins GitHub Repository.

The autodoc plugin hooks into the Java compiler task (compileJava) and generates a module manifest file that contains meta information about each module. For example, it exposes all required and provided dependencies of an EDC ServiceExtension.

2. Module structure

The autodoc plugin is located at plugins/autodoc and consists of four separate modules:

• autodoc-plugin: contains the actual Gradle Plugin and an Extension to configure the plugin. This module is published to MavenCentral.
• autodoc-processor: contains an AnnotationProcessor that hooks into the compilation process and builds the manifest file. Published to MavenCentral.
• autodoc-converters: used to convert JSON manifests to Markdown or HTML

3. Usage

In order to use the autodoc plugin we must follow a few simple steps. All examples use the Kotlin DSL.

3.1 Add the plugin to the buildscript block of your build.gradle.kts:

buildscript {
 repositories {
     maven {
         url = uri("https://oss.sonatype.org/content/repositories/snapshots/")
     }
 }
 dependencies {
     classpath("org.eclipse.edc.autodoc:org.eclipse.edc.autodoc.gradle.plugin:<VERSION>>")
 }
}

Please note that the repositories configuration can be omitted, if the release version of the plugin is used.

3.2 Apply the plugin to the project:

There are two options to apply a plugin. For multi-module builds this should be done at the root level.

1. via plugin block:

   plugins {
       id("org.eclipse.edc.autodoc")
   }
   

2. using the iterative approach, useful when applying to allprojects or subprojects:

   subprojects{
      apply(plugin = "org.eclipse.edc.autodoc")
   }
   

3.3 Configure the plugin [optional]

The autodoc plugin exposes the following configuration values:

1. the processorVersion: tells the plugin, which version of the annotation processor module to use. Set this value if the version of the plugin and of the annotation processor diverge. If this is omitted, the plugin will use its own version. Please enter just the SemVer-compliant version string, no groupId or artifactName are needed.

   configure<org.eclipse.edc.plugins.autodoc.AutodocExtension> {
       processorVersion.set("<VERSION>")
   }
   

   Typically, you do not need to configure this and can safely omit it.

The plugin will then generate an edc.json file for every module/gradle project.

4. Merging the manifests

There is a Gradle task readily available to merge all the manifests into one large manifest.json file. This comes in handy when the JSON manifest is to be converted into other formats, such as Markdown, HTML, etc.

To do that, execute the following command on a shell:

./gradlew mergeManifest

By default, the merged manifests are saved to <rootProject>/build/manifest.json. This destination file can be configured using a task property:

    // delete the merged manifest before the first merge task runs
tasks.withType<MergeManifestsTask> {
    destinationFile = YOUR_MANIFEST_FILE
}

Be aware that due to the multithreaded nature of the merger task, every subproject’s edc.json gets appended to the destination file, so it is a good idea to delete that file before running the mergeManifest task. Gradle can take care of that for you though:

// delete the merged manifest before the first merge task runs
rootProject.tasks.withType<MergeManifestsTask> {
    doFirst { YOUR_MANIFEST_FILE.delete() }
}

5. Rendering manifest files as Markdown or HTML

Manifests get created as JSON, which may not be ideal for end-user consumption. To convert them to HTML or Markdown, execute the following Gradle task:

./gradlew doc2md # or doc2html

this looks for manifest files and convert them all to either Markdown (doc2md) or static HTML (doc2html). Note that if merged the manifests before (mergeManifests), then the merged manifest file gets converted too.

The resulting *.md or *.html files are located next to the edc.json file in <module-path>/build/.

6. Using published manifest files (MavenCentral)

Manifest files (edc.json) are published alongside the binary jar files, sources jar and javadoc jar to MavenCentral for easy consumption by client projects. The manifest is published using type=json and classifier=manifest properties.

Client projects that want to download manifest files (e.g. for rendering static web content), simply define a Gradle dependency like this (kotlin DSL):

implementation("org.eclipse.edc:<ARTIFACT>:<VERSION>:manifest@json")

For example, for the :core:control-plane:control-plane-core module in version 0.4.2-SNAPSHOT, this would be:

implementation("org.eclipse.edc:control-plane-core:0.4.2-SNAPSHOT:manifest@json")

When the dependency gets resolved, the manifest file will get downloaded to the local gradle cache, typically located at .gradle/caches/modules-2/files-2.1. So in the example the manifest would get downloaded at ~/.gradle/caches/modules-2/files-2.1/org.eclipse.edc/control-plane-core/0.4.2-SNAPSHOT/<HASH>/control-plane-core-0.4.2-SNAPSHOT-manifest.json

3 - OpenApi spec

• Generate *.yaml files
• Gradle Plugins
  • Categorizing your API
• How to generate code

It is possible to generate an OpenApi spec in the form of a *.yaml file by invoking two simple Gradle tasks.

Generate *.yaml files

Every module (=subproject) that contains REST endpoints is scanned for Jakarta Annotations which are then used to generate a *.yaml specification for that particular module. This means that there is one *.yamlfile per module, resulting in several *.yaml files.

Those files are named MODULENAME.yaml, e.g. observability.yaml or control.yaml.

To re-generate those files, simply invoke

./gradlew clean resolve

This will generate all *.yaml files in the resources/openapi/yaml directory.

Gradle Plugins

We use the official Swagger Gradle plugins:

• "io.swagger.core.v3.swagger-gradle-plugin": used to generate a *.yaml file per module

So in order for a module to be picked up by the Swagger Gradle plugin, simply add it to the build.gradle.kts:

// in yourModule/build.gradle.kts

val rsApi: String by project

plugins {
    `java-library`
    id(libs.plugins.swagger.get().pluginId) //<-- add this
}

Categorizing your API

All APIs in EDC should be “categorized”, i.e. they should belong to a certain group of APIs. Please see this decision record for reference. In order to add your module to one of the categories, simply add this block to your module’s build.gradle.kts:

plugins {
    `java-library`
    id(libs.plugins.swagger.get().pluginId)
}

dependencies {
    // ...
}

// add this block:
edcBuild {
    swagger {
        apiGroup.set("management-api")
    }
}

This tells the build plugin how to categorize your API and SwaggerHub will list it accordingly. Note: currently we have categories for control-api and management-api

How to generate code

This feature does neither expose the generated files through a REST endpoint providing any sort of live try-out feature, nor does it generate any sort of client code. A visual documentation page for our APIs is served through SwaggerHub.

However, there is Gradle plugin capable of generating client code. Please refer to the official documentation.

4 - Data Persistence with PostgreSQL

• 1. Configuring DataSources
  • 1.2 Using custom datasource in stores
• 2. SQL Statement abstraction
• 3. Querying PostgreSQL databases
  • 3.1 The canonical form
  • 3.1 Translation Mappings
    • 3.1.1 Mapping primitive fields
    • 3.1.2 Mapping complex objects
    • Option 1: using foreign keys
    • Option 2a: encoding the object
    • Option 2b: encoding lists/arrays

By default, the in-memory stores are provided by the dependency injection, the sql implementations can be used by simply registering the relative extensions (e.g. asset-index-sql, contract-negotiation-store-sql, …).

1. Configuring DataSources

For using sql extensions, a DataSource is needed, and it should be registered on the DataSourceRegistry service.

The sql-pool-apache-commons extension is responsible for creating and registering pooled data sources starting from configuration. At least one data source named "default" is required.

edc.datasource.default.url=...
edc.datasource.default.name=...
edc.datasource.default.password=...

It is recommended to hold these values in the Vault rather than in configuration. The config key (e.g. edc.datasource.default.url) serves as secret alias. If no vault entries are found for these keys, they will be obtained from the configuration. This is unsafe and should be avoided!

Other datasources can be defined using the same settings structure:

edc.datasource.<datasource-name>.url=...
edc.datasource.<datasource-name>.name=...
edc.datasource.<datasource-name>.password=...

<datasource-name> is string that then can be used by the store’s configuration to use specific data sources.

1.2 Using custom datasource in stores

Using a custom datasource in a store can be done by configuring the setting:

edc.sql.store.<store-context>.datasource=<datasource-name>

Note that <store-context> can be an arbitrary string, but it is recommended to use a descriptive name. For example, the SqlPolicyStoreExtension defines a data source name as follows:

@Extension("SQL policy store")
public class SqlPolicyStoreExtension implements ServiceExtension {

    @Setting(value = "The datasource to be used", defaultValue = DataSourceRegistry.DEFAULT_DATASOURCE)
    public static final String DATASOURCE_NAME = "edc.sql.store.policy.datasource";

    @Override
    public void initialize(ServiceExtensionContext context) {
        var datasourceName = context.getConfig().getString(DATASOURCE_NAME, DataSourceRegistry.DEFAULT_DATASOURCE);
        //...
    }
}

2. SQL Statement abstraction

EDC does not use any sort of Object-Relation-Mapper (ORM), which would automatically translate Java object graphs to SQL statements. Instead, EDC uses pre-canned parameterized SQL statements.

We typically distinguish between literals such as table names or column names and “templates”, which are SQL statements such as INSERT.

Both are declared as getters in an interface that extends the SqlStatements interface, with literals being default methods and templates being implemented by a BaseSqlDialectStatements class.

A simple example could look like this:

public class BaseSqlDialectStatements implements SomeEntityStatements {

    @Override
    public String getDeleteByIdTemplate() {
        return executeStatement().delete(getSomeEntityTable(), getIdColumn());
    }

    @Override
    public String getUpdateTemplate() {
        return executeStatement()
                .column(getIdColumn())
                .column(getSomeStringFieldColumn())
                .column(getCreatedAtColumn())
                .update(getSomeEntityTable(), getIdColumn());
    }
    //...
}

Note that the example makes use of the SqlExecuteStatement utility class, which should be used to construct all SQL statements - except queries. Queries are special in that they have a highly dynamic aspect to them. For more information, please read on in this chapter.

As a general rule of thumb, issuing multiple statements (within one transaction) should be preferred over writing complex nested statements. It is very easy to inadvertently create an inefficient or wasteful statement that causes high resource load on the database server. The latency that is introduced by sending multiple statements to the DB server is likely negligible in comparison, especially because EDC is architected towards reliability rather than latency.

3. Querying PostgreSQL databases

Generally speaking, the basis for all queries is a QuerySpec object. This means, that at some point a QuerySpec must be translated into an SQL SELECT statement. The place to do this is the SqlStatements implementation often called BaseSqlDialectStatements:

@Override
public SqlQueryStatement createQuery(QuerySpec querySpec) {
    var select = "SELECT * FROM %s".formatted(getSomeEntityTable());
    return new SqlQueryStatement(select, querySpec, new SomeEntityMapping(this), operatorTranslator);
}

Now, there are a few things to unpack here:

• the SELECT statement serves as starting point for the query
• individual WHERE clauses get added by parsing the filterExpression property of the QuerySpec
• LIMIT and OFFSET clauses get appended based on QuerySpec#offset and QuerySpec#limit
• the SomeEntityMapping maps the canonical form onto the SQL literals
• the operatorTranslator is used to convert operators such as = or like into SQL operators

3.1 The canonical form

Theoretically it is possible to map every schema onto every other schema, given that they are of equal cardinality. To achieve that, EDC introduces the notion of a canonical form, which is our internal working schema for entities. In other words, this is the schema in which objects are represented internally. If we ever support a wider variety of translation and transformation paths, everything would have to be transformed into that canonical format first.

In actuality the canonical form of an object is defined by the Java class and its field names. For instance, a query for contract negotiations must be specified using the field names of a ContractNegotiation object:

public class ContractNegotiation {
    // ...
    private ContractAgreement contractAgreement;
    // ...
}

public class ContractAgreement {
    // ...
    private final String assetId;
}

Consequently, contractAgreement.assetId would be valid, whereas contract_agreement.asset_id would be invalid. Or, the left-hand operand looks like as if we were traversing the Java object graph. This is what we call the canonical form . Note the omission of the root object contractNegotiation!

3.1 Translation Mappings

Translation mappings are EDCs way to map a QuerySpec to SQL statements. At its core, it contains a Map that contains the Java entity field name and the related SQL column name.

In order to decouple the canonical form from the SQL schema (or any other database schema), a mapping scheme exists to map the canonical model onto the SQL model. This TranslationMapping is essentially a graph-like metamodel of the entities: every Java entity has a related mapping class that contains its field names and the associated SQL column names. The convention is to append *Mapping to the class name, e.g. PolicyDefinitionMapping.

3.1.1 Mapping primitive fields

Primitive fields are stored directly as columns in SQL tables. Thus, mapping primitive data types is trivial: a simple mapping from one onto the other is necessary, for example, ContractNegotiation.counterPartyAddress would be represented in the ContractNegotiationMappin as an entry

"counterPartyAddress"->"counterparty_address"

When constructing WHERE/AND clauses, the canonical property is simply be replaced by the respective SQL column name.

3.1.2 Mapping complex objects

For fields that are of complex type, such as the ContractNegotiation.contractAgreement field, it is necessary to accommodate this, depending on how the relational data model is defined. There are two basic variants we use:

Option 1: using foreign keys

In this case, the referenced object is stored in a separate table using a foreign key relation. Thus, the canonical property (contractAgreement) is mapped onto the SQL schema using another *Mapping class. Here, this would be the ContractAgreementMapping. When resolving a property in the canonical format (contractAgreement.assetId), this means we must recursively descend into the model graph and resolve the correct SQL expression.

Note: mapping one-to-many relations (= arrays/lists) with foreign keys is not implemented at this time.

Option 2a: encoding the object

Another popular way to store complex objects is to encode them in JSON and store them in a VARCHAR column. In PostgreSQL we use the specific JSON type instead of VARCHAR. For example, the TranferProcess is stored in a table called edc_transfer_process, its DataAddress property is encoded in JSON and stored in a JSON field.

Querying for TransferProcess objects: when mapping the filter expression contentDataAddress.properties.somekey=somevalue, the contentDataAddress is represented as JSON, therefore in the TransferProcessMapping the contentDataAddress field maps to a JsonFieldTranslator:

public TransferProcessMapping(TransferProcessStoreStatements statements) {
    // ...
    add(FIELD_CONTENTDATAADDRESS, new JsonFieldTranslator(statements.getContentDataAddressColumn()));
    // ...
}

which would then get translated to:

SELECT *
FROM edc_transfer_process
-- omit LEFT OUTER JOIN for readability
WHERE content_data_address -> 'properties' ->> 'somekey' = 'somevalue'

Note that JSON queries are specific to PostgreSQL and are not portable to other database technologies!

Option 2b: encoding lists/arrays

Like accessing objects, accessing lists/arrays of objects is possible using special JSON operators. In this case the special Postgres function json_array_elements() is used. Please refer to the official documentation.

For an example of how this is done, please look at how the TransferProcessMapping maps a ResourceManifest, which in turn contains a List<ResourceDefinition> using the ResourceManifestMapping.

5 - Logging

A comprehensive and consistent way of logging is a crucial pillar for operability. Therefore, the following rules should be followed:

Logging component

Logs must only be produced using the Monitor service, which offers 4 different log levels:

severe

Error events that might lead the application to abort or still allow it to continue running.

Used in case of an unexpected interruption of the flow or when something is broken, i.e. an operator has to take action. e.g. service crashes, database in illegal state, … even if there is chance of self recovery.

warning

Potentially harmful situations messages.

Used in case of an expected event that does not interrupt the flow but that should be taken into consideration.

info

Informational messages that highlight the progress of the application at coarse-grained level.

Used to describe the normal flow of the application.

debug

Fine-grained informational events that are most useful to debug an application.

Used to describe details of the normal flow that are not interesting for a production environment.

What should be logged

• every exception with severe or warning
• every Result object evaluated as failed:
  • with severe if this is something that interrupts the flow and someone should take care of immediately
  • with warning if this is something that doesn’t interrupt the flow but someone should take care of, because it could give worse results in the future
• every important message that’s not an error with info
• other informative events like incoming calls at the API layer or state changes with debug

What should be not logged

• secrets and any other potentially sensitive data, like the payload that is passed through the data-plane
• an exception that will be thrown in the same block
• not strictly necessary information, like “entering method X”, “leaving block Y”, “returning HTTP 200”

6 - Writing tests

• 1. Adding EDC test fixtures
• 2. Controlling test verbosity
• 3. Definition and distinction
• 4. Integration Tests
  • 4.1 TL;DR
  • 4.2 When to use them
  • 4.3 Coding Guidelines
  • 4.4 Running integration tests locally
  • 4.5 Running them in the CI pipeline
  • 4.6 Do’s and Don’ts
• 5. Running an EDC instance from a JUnit test (End2End tests)

1. Adding EDC test fixtures

To add EDC test utilities and test fixtures to downstream projects, simply add the following Gradle dependency:

  testImplementation("org.eclipse.edc:junit:<version>")

2. Controlling test verbosity

To run tests verbosely (displaying test events and output and error streams to the console), use the following system property:

./gradlew test -PverboseTest

3. Definition and distinction

• unit tests test one single class by stubbing or mocking dependencies.
• integration test tests one particular aspect of a software, which may involve external systems.
• system tests are end-to-end tests that rely on the entire system to be present.

4. Integration Tests

4.1 TL;DR

Use integration tests only when necessary, keep them concise, implement them in a defensive manner using timeouts and randomized names, use test containers for external systems wherever possible. This increases portability.

4.2 When to use them

Generally speaking developers should favor writing unit tests over integration tests, because they are simpler, more stable and typically run faster. Sometimes that is not (easily) possible, especially when an implementation relies on an external system that is not easily mocked or stubbed such as databases.

Therefore, in many cases writing unit tests is more involved that writing an integration test, for example say you want to test your implementation of a Postgres-backed database. You would have to mock the behaviour of the PostgreSQL database, which - while certainly possible - can get complicated pretty quickly. You might still choose to do that for simpler scenarios, but eventually you will probably want to write an integration test that uses an actual PostgreSQL instance.

4.3 Coding Guidelines

The EDC codebase has few annotations and these annotation focuses on two important aspects:

• Exclude integration tests by default from JUnit test runner, as these tests relies on external systems which might not be available during a local execution.
• Categorize integration tests with help of JUnit Tags.

Following are some available annotations:

• @IntegrationTest: Marks an integration test with IntegrationTest Junit tag. This is the default tag and can be used if you do not want to specify any other tags on your test to do further categorization.

Below annotations are used to categorize integration tests based on the runtime components that must be available for the test to run. All of these annotations are composite annotations and contains @IntegrationTest annotation as well.

• @ApiTest: marks an integration test that focuses on testing a REST API. To do that, a runtime the controller class with all its collaborators is spun up.
• @EndToEndTest: Marks an integration test with EndToEndTest Junit Tag. This should be used when entire system is
• involved in a test.
• @ComponentTest: Marks an integration test with ComponentTest Junit Tag. This should be used when the test does not use any external systems, but uses actual collaborator objects instead of mocks.
• there are other more specific tags for cloud-vendor specific environments, like @AzureStorageIntegrationTest or @AwsS3IntegrationTest. Some of those enviroments can be emulated (with test containers), others can’t.

We encourage you to use these available annotation but if your integration test does not fit in one of these available annotations, and you want to categorize them based on their technologies then feel free to create a new annotations but make sure to use composite annotations which contains @IntegrationTest. If you do not wish to categorize based on their technologies then you can use already available @IntegrationTest annotation.

• By default, JUnit test runner ignores all integration tests because in root build.gradle.kts file we have excluded all tests marked with IntegrationTest Junit tag.
• If your integration test does not rely on an external system then you may not want to use above-mentioned annotations.

All integration tests should specify annotation to categorize them and the "...IntegrationTest" postfix to distinguish them clearly from unit tests. They should reside in the same package as unit tests because all tests should maintain package consistency to their test subject.

Any credentials, secrets, passwords, etc. that are required by the integration tests should be passed in using environment variables. A good way to access them is ConfigurationFunctions.propOrEnv() because then the credentials can also be supplied via system properties.

There is no one-size-fits-all guideline whether to perform setup tasks in the @BeforeAll or @BeforeEach, it will depend on the concrete system you’re using. As a general rule of thumb long-running one-time setup should be done in the @BeforeAll so as not to extend the run-time of the test unnecessarily. In contrast, in most cases it is not advisable to deploy/provision the external system itself in either one of those methods. In other words, manually provisioning a cloud resource should generally be avoided, because it will introduce code that has nothing to do with the test and may cause security problems.

If possible all external system should be deployed using Testcontainers. Alternatively, in special situations there might be a dedicated test instance running continuously, e.g. a cloud-based database test instance. In the latter case please be careful to avoid conflicts (e.g. database names) when multiple test runners access that system simultaneously and to properly clean up any residue before and after the test.

4.4 Running integration tests locally

As mentioned above the JUnit runner won’t pick up integration tests unless a tag is provided. For example to run Azure CosmosDB integration tests pass includeTags parameter with tag value to the gradlew command:

./gradlew test -p path/to/module -DincludeTags="PostgresqlIntegrationTest"

running all tests (unit & integration) can be achieved by passing the runAllTests=true parameter to the gradlew command:

./gradlew test -DrunAllTests="true"

4.5 Running them in the CI pipeline

All integration tests should go into the verify.yaml workflow, every “technology” should have its own job, and technology specific tests can be targeted using Junit tags with -DincludeTags property as described above in document.

A GitHub composite action was created to encapsulate the tasks of setting up Java/Gradle and running tests.

For example let’s assume we’ve implemented a PostgreSQL-based store for SomeObject, and let’s assume that the verify.yaml already contains a “Postgres” job, then every module that contains a test class annotated with @PostgresqlIntegrationTest will be loaded and executed here. This tagging will be used by the CI pipeline step to target and execute the integration tests related to Postgres.

Let’s also make sure that the code is checked out before and integration tests only run on the upstream repo.

jobs:
  Postgres-Integration-Tests:
    # run only on upstream repo
    if: github.repository_owner == 'eclipse-edc'
    runs-on: ubuntu-latest

    # taken from https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers
    services:
      # Label used to access the service container
      postgres:
        # Docker Hub image
        image: postgres
        # Provide the password for postgres
        env:
          POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}

    steps:
      - uses: ./.github/actions/setup-build

      - name: Postgres Tests
        uses: ./.github/actions/run-tests
        with:
          command: ./gradlew test -DincludeTags="PostgresIntegrationTest"

  [ ... ]

4.6 Do’s and Don’ts

DO:

• aim to cover as many test cases with unit tests as possible
• use integration tests sparingly and only when unit tests are not practical
• deploy the external system test container if possible, or
• use a dedicated always-on test instance (esp. cloud resources)
• take into account that external systems might experience transient failures or have degraded performance, so test methods should have a timeout so as not to block the runner indefinitely.
• use randomized strings for things like database/table/bucket/container names, etc., especially when the external system does not get destroyed after the test.

DO NOT:

• try to cover everything with integration tests. It’s typically a code smell if there are no corresponding unit tests for an integration test.
• slip into a habit of testing the external system rather than your usage of it
• store secrets directly in the code. GitHub will warn about that.
• perform complex external system setup in @BeforeEach or @BeforeAll
• add production code that is only ever used from tests. A typical smell are protected or package-private methods.

5. Running an EDC instance from a JUnit test (End2End tests)

In some circumstances it is necessary to launch an EDC runtime and execute tests against it. This could be a fully-fledged connector runtime, replete with persistence and all bells and whistles, or this could be a partial runtime that contains lots of mocks and stubs. One prominent example of this is API tests. At some point, you’ll want to run REST requests using a HTTP client against the actual EDC runtime, using JSON-LD expansion, transformation etc. and real database infrastructure.

EDC provides a nifty way to launch any runtime from within the JUnit process, which makes it easy to configure and debug not only the actual test code, but also the system-under-test, i.e. the runtime.

To do that, two parts are needed:

• a runner: a module that contains the test logic
• one or several runtimes: one or more modules that define a standalone runtime (e.g. a runnable EDC definition)

The runner can load an EDC runtime by using the @RegisterExtension annotation:

@EndToEndTest
class YourEndToEndTest {

  @RegisterExtension
  private final RuntimeExtension controlPlane = new RuntimePerClassExtension(new EmbeddedRuntime(
          "control-plane", // the runtime's name, used for log output
          Map.of( // the runtime's configuration
                  "web.http.control.port", String.valueOf(getFreePort()),
                  "web.http.control.path", "/control"
                  //...
          ),
          // all modules to be put on the runtime classpath
          ":core:common:connector-core",
          ":core:control-plane:control-plane-core",
          ":core:data-plane-selector:data-plane-selector-core",
          ":extensions:control-plane:transfer:transfer-data-plane-signaling",
          ":extensions:common:iam:iam-mock",
          ":extensions:common:http",
          ":extensions:common:api:control-api-configuration"
          //...
  ));
}

This example will launch a runtime called "control-plane", add the listed Gradle modules to its classpath and pass the configuration as map to it. And it does that from within the JUnit process, so the "control-plane" runtime can be debugged from the IDE.

The example above will initialize and start the runtime once, before all tests run (hence the name “RuntimePerClassExtension”). Alternatively, there is the RuntimePerMethodExtension which will re-initialize and start the runtime before every test method.

In most use cases, RuntimePerClassExtension is preferable, because it avoids having to start the runtime on every test. There are cases, where the RuntimePerMethodExtension is useful, for example when the runtime is mutated during tests and cleaning up data stores is not practical. Be aware of the added test execution time penalty though.

To make sure that the runtime extensions are correctly built and available, they need to be set as dependency of the runner module as testCompileOnly.

This ensures proper dependency isolation between runtimes (very important the test need to run two different components like a control plane and a data plane).

Technically, the number of runtimes launched that way is not limited (other than by host system resource), so theoretically, an entire dataspace with N participants could be launched that way…

7 - Control Plane

7.1 - Entities

• 1. Assets
• 2. Policies
  • 2.1 Policy vs PolicyDefinition
  • 2.2 Policy scopes and bindings
  • 2.3 Policy evaluation functions
  • 2.4 Example: binding an evaluation function
  • 2.5 Advanced policy concepts
    • 2.5.1 Pre- and Post-Evaluators
    • 2.5.2 Dynamic functions
  • 2.6 Bundled policy functions
    • 2.6.1 Contract expiration function
• 3. Contract definitions
• 4. Contract negotiations
• 5. Contract agreements
• 6. Catalog
• 7 Transfer processes
  • 7.1 Transfer and data flows types
    • 7.1.1 Consumer Pull
    • 7.1.2 Provider Push
    • 7.1.2 Finite and Non-Finite Data
  • 7.2 About Data Destinations
  • 7.3 Transfer process callbacks
• 8 Endpoint Data References
• 9 Querying with QuerySpec and Criterion

1. Assets

Assets are containers for metadata, they do not contain the actual bits and bytes. Say you want to offer a file to the dataspace, that is physically located in an S3 bucket, then the corresponding Asset would contain metadata about it, such as the content type, file size, etc. In addition, it could contain private properties, for when you want to store properties on the asset, which you do not want to expose to the dataspace. Private properties will get ignored when serializing assets out over DSP.

A very simplistic Asset could look like this:

{
  "@context": {
    "edc": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
  "properties": {
    "somePublicProp": "a very interesting value"
  },
  "privateProperties": {
    "secretKey": "this is secret information, never tell it to the dataspace!"
  },
  "dataAddress": {
    "type": "HttpData",
    "baseUrl": "http://localhost:8080/test"
  }
}

The Asset also contains a DataAddress object, which can be understood as a “pointer into the physical world”. It contains information about where the asset is physically located. This could be a HTTP URL, or a complex object. In the S3 example, that DataAddress might contain the bucket name, region and potentially other information. Notice that the schema of the DataAddress will depend on where the data is physically located, for instance a HttpDataAddress has different properties from an S3 DataAddress. More precisely, Assets and DataAddresses are schemaless, so there is no schema enforcement beyond a very basic validation. Read this document to learn about plugging in custom validators.

A few things must be noted. First, while there isn’t a strict requirement for the @id to be a UUID, we highly recommend using the JDK UUID implementation.

Second, never store access credentials such as passwords, tokens, keys etc. in the dataAddress or even the privateProperties object. While the latter does not get serialized over DSP, both properties are persisted in the database. Always use a HSM to store the credential, and hold a reference to the secret in the DataAddress. Checkout the best practices for details.

By design, Assets are extensible, so users can store any metadata they want in it. For example, the properties object could contain a simple string value, or it could be a complex object, following some custom schema. Be aware, that unless specified otherwise, all properties are put under the edc namespace by default. There are some “well-known” properties in the edc namespace: id, description, version, name, contenttype.

Here is an example of how an Asset with a custom property following a custom namespace would look like:

{
  "@context": {
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "sw": "http://w3id.org/starwars/v0.0.1/ns/"
  },
  "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
  "properties": {
    "faction": "Galactic Imperium",
    "person": {
      "name": "Darth Vader",
      "webpage": "https://death.star"
    }
  }
}

(assuming the sw context contains appropriate definitions for faction and person).

Remember that upon ingress through the Management API, all JSON-LD objects get expanded, and the control plane only operates on expanded JSON-LD objects. The Asset above would look like this:

[
  {
    "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
    "https://w3id.org/edc/v0.0.1/ns/properties": [
      {
        "https://w3id.org/starwars/v0.0.1/ns/faction": [
          {
            "@value": "Galactic Imperium"
          }
        ],
        "http://w3id.org/starwars/v0.0.1/ns/person": [
          {
            "http://w3id.org/starwars/v0.0.1/ns/name": [
              {
                "@value": "Darth Vader"
              }
            ],
            "http://w3id.org/starwars/v0.0.1/ns/webpage": [
              {
                "@value": "https://death.star"
              }
            ]
          }
        ]
      }
    ]
  }
]

This is important to keep in mind, because it means that Assets get persisted in their expanded form, and operations performed on them (e.g. querying) in the control plane must also be done on the expanded form. For example, a query targeting the sw:faction field from the example above would look like this:

{
  "https://w3id.org/edc/v0.0.1/ns/filterExpression": [
    {
      "https://w3id.org/edc/v0.0.1/ns/operandLeft": [
        {
          "@value": "https://w3id.org/starwars/v0.0.1/ns/faction"
        }
      ],
      "https://w3id.org/edc/v0.0.1/ns/operator": [
        {
          "@value": "="
        }
      ],
      "https://w3id.org/edc/v0.0.1/ns/operandRight": [
        {
          "@value": "Galactic Imperium"
        }
      ]
    }
  ]
}

2. Policies

Policies are the EDC way of expressing that certain conditions may, must or must not be satisfied in certain situations. Policies are used to express what requirements a subject (e.g. a communication partner) must fulfill in order to be able to perform an action. For example, that the communication partner must be headquartered in the European Union.

Policies are ODRL serialized as JSON-LD. Thus, our previous example would look like this:

{
  "@context": {
    "edc": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type": "PolicyDefinition",
  "policy": {
    "@context": "http://www.w3.org/ns/odrl.jsonld",
    "@type": "Set",
    "duty": [
      {
        "target": "http://example.com/asset:12345",
        "action": "use",
        "constraint": {
          "leftOperand": "headquarter_location",
          "operator": "eq",
          "rightOperand": "EU"
        }
      }
    ]
  }
}

The duty object expresses the semantics of the constraint. It is a specialization of rule, which expresses either a MUST (duty), MAY (permission) or MUST NOT (prohibition) relation. The action expresses the type of action for which the rule is intended. Acceptable values for action are defined here, but in EDC you’ll exclusively encounter "use".

The constraint object expresses logical relationship of a key (leftOperand), the value (righOperand) and the operator. Multiple constraints can be linked with logical operators, see advanced policy concepts. The leftOperand and rightOperand are completely arbitrary, only the operator is limited to the following possible values: eq, neq, gt, geq, lt, leq, in, hasPart, isA, isAllOf, isAnyOf, isNoneOf.

Please note that not all operators are always allowed, for example headquarter_location lt EU is nonsensical and should result in an evaluation error, whereas headquarter_location isAnOf [EU, US] would be valid. Whether an operator is valid is solely defined by the policy evaluation function, supplying an invalid operator should raise an exception.

2.1 Policy vs PolicyDefinition

In EDC we have two general use cases under which we handle and persist policies:

1. for use in contract definitions
2. during contract negotiations

In the first case policies are ODRL objects and thus must have a uid property. They are typically used in contract definitions.

Side note: the ODRL context available at http://www.w3.org/ns/odrl.jsonld simply defines uid as an alias to the @id property. This means, whether we use uid or @id doesn’t matter, both expand to the same property @id.

However in the second case we are dealing with DCAT objects, that have no concept of Offers, Policies or Assets. Rather, their vocabulary includes Datasets, Dataservices etc. So when deserializing those DCAT objects there is no way to reconstruct Policy#uid, because the JSON-LD structure does not contain it.

To account for this, we defined the Policy class as value object that contains rules and other properties. In addition, we have a PolicyDefinition class, which contains a Policy and an id property, which makes it an entity.

2.2 Policy scopes and bindings

A policy scope is the “situation”, in which a policy is evaluated. For example, a policy may need to be evaluated when a contract negotiation is attempted. To do that, EDC defines certain points in the code called “scopes” to which policies are bound. These policy scopes (sometimes called policy evaluation points) are static, injecting/adding additional scopes is not possible. Currently, the following scopes are defined:

• contract.negotiation: evaluated upon initial contract offer. Ensures that the consumer fulfills the contract policy.
• transfer.process: evaluated before starting a transfer process to ensure that the policy of the contract agreement is fulfilled. One example would be contract expiry.
• catalog: evaluated when the catalog for a particular participant agent is generated. Decides whether the participant has the asset in their catalog.
• request.contract.negotiation: evaluated on every request during contract negotiation between two control plane runtimes. Not relevant for end users.
• request.transfer.process: evaluated on every request during transfer establishment between two control plane runtimes. Not relevant for end users.
• request.catalog: evaluated upon an incoming catalog request. Not relevant for end users.
• provision.manifest.verify: evaluated during the precondition check for resource provisioning. Only relevant in advanced use cases.

A policy scope is a string that is used for two purposes:

1. binding a scope to a rule type: implement filtering based on the action or the leftOperand of a policy. This determines for every rule inside a policy whether it should be evaluated in the given scope. In other words, it determines if a rule should be evaluated.
2. binding a policy evaluation function to a scope: if a policy is determined to be “in scope” by the previous step, the policy engine invokes the evaluation function that was bound to the scope to evaluate if the policy is fulfilled. In other words, it determines (implements) how a rule should be evaluated.

2.3 Policy evaluation functions

If policies are a formalized declaration of requirements, policy evaluation functions are the means to evaluate those requirements. They are pieces of Java code executed at runtime. A policy on its own only expresses the requirement, but in order to enforce it, we need to run policy evaluation functions.

Upon evaluation, they receive the operator, the rightOperand (or rightValue), the rule, and the PolicyContext. A simple evaluation function that asserts the headquarters policy mentioned in the example above could look similar to this:

import org.eclipse.edc.policy.engine.spi.AtomicConstraintFunction;

public class HeadquarterFunction implements AtomicConstraintFunction<Duty> {
    public boolean evaluate(Operator operator, Object rightValue, Permission rule, PolicyContext context) {
        if (!(rightValue instanceof String)) {
            context.reportProblem("Right-value expected to be String but was " + rightValue.getClass());
            return false;
        }
        if (operator != Operator.EQ) {
            context.reportProblem("Invalid operator, only EQ is allowed!");
            return false;
        }

        var participant = context.getContextData(ParticipantAgent.class);
        var participantLocation = extractLocationClaim(participant); // EU, US, etc.
        return participantLocation != null && rightValue.equalsIgnoreCase(participantLocation);
    }
}

This particular evaluation function only accepts eq as operator, and only accepts scalars as rightValue, no list types.

The ParticipantAgent is a representation of the communication counterparty that contains a set of verified claims. In the example, extractLocationClaim() would look for a claim that contains the location of the agent and return it as string. This can get quite complex, for example, the claim could contain geo-coordinates, and the evaluation function would have to perform inverse address geocoding.

Other policies may require other context data than the participant’s location, for example an exact timestamp, or may even need a lookup in some third party system such as a customer database.

The same policy can be evaluated by different evaluation functions, if they are meaningful in different contexts (scopes).

NB: to write evaluation code for policies, implement the org.eclipse.edc.policy.engine.spi.AtomicConstraintFunction interface. There is a second interface with the same name, but that is only used for internal use in the PolicyEvaluationEngine.

2.4 Example: binding an evaluation function

As we’ve learned, for a policy to be evaluated at certain points, we need to create a policy (duh!), bind the policy to a scope, create a policy evaluation function, and we need to bind the function to the same scope. The standard way of registering and binding policies is done in an extension. For example, here we configure our HeadquarterFunction so that it evaluates our headquarter_location function whenever someone tries to negotiate a contract:

public class HeadquarterPolicyExtension implements ServiceExtension {

    @Inject
    private RuleBindingRegistry ruleBindingRegistry;

    @Inject
    private PolicyEngine policyEngine;

    private static final String HEADQUARTER_POLICY_KEY = "headquarter_location";

    @Override
    public void initialize() {
        // bind the policy to the scope
        ruleBindingRegistry.bind(HEADQUARTER_POLICY_KEY, NEGOTIATION_SCOPE);
        // create the function object
        var function = new HeadquarterFunction();
        // bind the function to the scope
        policyEngine.registerFunction(NEGOTIATION_SCOPE, Duty.class, HEADQUARTER_POLICY_KEY, function);
    }
}

The code does two things: it binds the function key (= the leftOperand) to the negotiation scope, which means that the policy is “relevant” in that scope. Further, it binds the evaluation function to the same scope, which means the policy engine “finds” the function and executes it in the negotiation scope.

This example assumes, a policy object exists in the system, that has a leftOperand = headquarter_location. For details on how to create policies, please check out the OpenAPI documentation.

2.5 Advanced policy concepts

2.5.1 Pre- and Post-Evaluators

Pre- and post-validators are functions that are executed before and after the actual policy evaluation, respectively. They can be used to perform preliminary evaluation of a policy or to enrich the PolicyContext. For example, EDC uses pre-validators to inject DCP scope strings using dedicated ScopeExtractor objects.

2.5.2 Dynamic functions

These are very similar to AtomicConstraintFunctions, with one significant difference: they also receive the left-operand as function parameter. This is useful when the function cannot be bound to a left-operand of a policy, because the left-operand is not known in advance.

Let’s revisit our headquarter policy from earlier and change it a little:

{
  "@context": {
    "edc": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type": "PolicyDefinition",
  "policy": {
    "@context": "http://www.w3.org/ns/odrl.jsonld",
    "@type": "Set",
    "duty": [
      {
        "target": "http://example.com/asset:12345",
        "action": "use",
        "constraint": {
          "or": [
            {
              "leftOperand": "headquarter.location",
              "operator": "eq",
              "rightOperand": "EU"
            },
            {
              "leftOperand": "headerquarter.numEmployees",
              "operator": "gt",
              "rightOperand": 5000
            }
          ]
        }
      }
    ]
  }
}

This means two things. One, our policy has changed its semantics: now we require the headquarter to be in the EU, or to have more than 5000 employees.

2.6 Bundled policy functions

2.6.1 Contract expiration function

3. Contract definitions

Contract definitions are how assets and policies are linked together. It is EDC’s way of expressing which policies are in effect for an asset. So when an asset (or several assets) are offered in the dataspace, a contract definition is used to express under what conditions they are offered. Those conditions are comprised of a contract policy and an access policy. The access policy determines, whether a participant will even get the offer, and the contract policy determines whether they can negotiate a contract for it. Those policies are referenced by ID, but foreign-key constrainta are not enforced. This means that contract definitions can be created ahead of time.

It is important to note that contract definitions are implementation details (i.e. internal objects), which means they never leave the realm of the provider, and they are never sent to the consumer via DSP.

• access policy: determines whether a particular consumer is offered an asset when making a catalog request. For example, we may want to restrict certain assets such that only consumers within a particular geography can see them. Consumers outside that geography wouldn’t even have them in their catalog.
• contract policy: determines the conditions for initiating a contract negotiation for a particular asset. Note that this only guarantees the successful initiation of a contract negotiation, it does not automatically guarantee the successful conclusion of it!

Contract definitions also contain an assetsSelector. THat is a query expression that defines all the assets that are included in the definition, like an SQL SELECT statement. With that it is possible to configure the same set of conditions (= access policy and contract policy) for a multitude of assets.

Please note that creating an assetSelector may require knowledge about the shape of an Asset and can get complex fairly quickly, so be sure to read the chapter about querying.

Here is an example of a contract definition, that defines an access policy and a contract policy for assets id1, id2 and id3 that must contain the "foo" : "bar" property.

{
  "@context": {
    "edc": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type": "https://w3id.org/edc/v0.0.1/ns/ContractDefinition",
  "@id": "test-id",
  "edc:accessPolicyId": "access-policy-1234",
  "edc:contractPolicyId": "contract-policy-5678",
  "edc:assetsSelector": [
    {
      "@type": "https://w3id.org/edc/v0.0.1/ns/Criterion",
      "edc:operandLeft": "id",
      "edc:operator": "in",
      "edc:operandRight": [
        "id1",
        "id2",
        "id3"
      ]
    },
    {
      "@type": "https://w3id.org/edc/v0.0.1/ns/Criterion",
      "edc:operandLeft": "foo",
      "edc:operator": "=",
      "edc:operandRight": "bar"
    }
  ]
}

The sample expresses that a set of assets identified by their ID be made available under the access policy access-policy-1234 and contract policy contract-policy-5678, if they contain a property "foo" : "bar".

Note that asset selector expressions are always logically conjoined using an “AND” operation.

4. Contract negotiations

If a connector fulfills the contract policy, it may initiate the negotiation of a contract for a particular asset. During that negotiation, both parties can send offers and counter-offers that can contain altered terms (= policy) as any human would in a negotiation, and the counter-party may accept or reject them.

Contract negotiations have a few key aspects:

• they target one asset
• they take place between a provider and a consumer connector
• they cannot be changed by the user directly
• users can only decline, terminate or cancel them

As a side note it is also important to note that contract offers are ephemeral objects as they are generated on-the-fly for a particular participant, and they are never persisted in a database and thus cannot be queried through any API.

Contract negotiations are asynchronous in nature. That means after initiating them, they become (potentially long-running) stateful processes that are advanced by an internal state machine. The current state of the negotiation can be queried and altered through the management API.

Here’s a diagram of the state machine applied to contract negotiations:

A contract negotiation can be initiated from the consumer side by sending a ContractRequest to the connector management API.

{
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type": "ContractRequest",
  "counterPartyAddress": "http://provider-address",
  "protocol": "dataspace-protocol-http",
  "policy": {
    "@context": "http://www.w3.org/ns/odrl.jsonld",
    "@type": "odrl:Offer",
    "@id": "offer-id",
    "assigner": "providerId",
    "permission": [],
    "prohibition": [],
    "obligation": [],
    "target": "assetId"
  },
  "callbackAddresses": [
    {
      "transactional": false,
      "uri": "http://callback/url",
      "events": [
        "contract.negotiation"
      ],
      "authKey": "auth-key",
      "authCodeId": "auth-code-id"
    }
  ]
}

The counterPartyAddress is the address where to send the ContractRequestMessage via the specified protocol ( currently dataspace-protocol-http)

The policy should hold the same policy associated to the data offering chosen from the catalog, plus two additional properties:

• assigner the providers participantId
• target the asset (dataset) ID

In addition, the (optional) callbackAddresses array can be used to get notified about state changes of the negotiation. Read more on callbacks in the section about events and callbacks.

Note: if the policy sent by the consumer differs from the one expressed by the provider, the contract negotiation will fail and transition to a TERMINATED state.

5. Contract agreements

Once a contract negotiation is successfully concluded (i.e. it reaches the FINALIZED state), it “turns into” a contract agreement. It is always the provider connector that gives the final approval. Contract agreements are immutable objects that contain the final, agreed-on policy, the ID of the asset that the contract was negotiated for, the IDs of the negotiation parties and the exact signing date.

Note that in future iterations contracts will be cryptographically signed to further support the need for immutability and non-repudiation.

Like contract definitions, contract agreements are entities that only exist within the bounds of a connector.

About terminating contracts: once a contract negotiation has reached a terminal state TERMINATED or FINALIZED, it becomes immutable. This could be compared to not being able to scratch a signature off a physical paper contract. Cancelling or terminating a contract is therefor handled through other channels like eventing systems. The semantics of cancelling a contract are highly individual to each dataspace and may even bring legal side effects, so EDC cannot make an assumption here.

6. Catalog

The catalog contains the “data offerings” of a connector and one or multiple service endpoints to initiate a negotiation for those offerings.

Every data offering is represented by a Dataset object which contains a policy and one or multiple Distribution objects. A Distribution should be understood as a variant or representation of the Dataset. For instance, if a file is accessible via multiple transmission channels from a provider (HTTP and FTP), then each of those channels would be represented as a Distribution. Another example would be image assets that are available in different file formats (PNG, TIFF, JPEG).

A DataService object specifies the endpoint where contract negotiations and transfers are accepted by the provider. In practice, this will be the DSP endpoint of the connector.

The following example shows an HTTP response to a catalog request, that contains one offer that is available via two channels HttpData-PUSH and HttpData-PULL.

{
  "@id": "567bf428-81d0-442b-bdc8-437ed46592c9",
  "@type": "dcat:Catalog",
  "dcat:dataset": [
    {
      "@id": "asset-2",
      "@type": "dcat:Dataset",
      "odrl:hasPolicy": {
        "@id": "c2Vuc2l0aXZlLW9ubHktZGVm:YXNzZXQtMg==:MzhiYzZkNjctMDIyNi00OGJjLWFmNWYtZTQ2ZjAwYTQzOWI2",
        "@type": "odrl:Offer",
        "odrl:permission": [],
        "odrl:prohibition": [],
        "odrl:obligation": {
          "odrl:action": {
            "@id": "use"
          },
          "odrl:constraint": {
            "odrl:leftOperand": {
              "@id": "DataAccess.level"
            },
            "odrl:operator": {
              "@id": "odrl:eq"
            },
            "odrl:rightOperand": "sensitive"
          }
        }
      },
      "dcat:distribution": [
        {
          "@type": "dcat:Distribution",
          "dct:format": {
            "@id": "HttpData-PULL"
          },
          "dcat:accessService": {
            "@id": "a6c7f3a3-8340-41a7-8154-95c6b5585532",
            "@type": "dcat:DataService",
            "dcat:endpointDescription": "dspace:connector",
            "dcat:endpointUrl": "http://localhost:8192/api/dsp",
            "dct:terms": "dspace:connector",
            "dct:endpointUrl": "http://localhost:8192/api/dsp"
          }
        },
        {
          "@type": "dcat:Distribution",
          "dct:format": {
            "@id": "HttpData-PUSH"
          },
          "dcat:accessService": {
            "@id": "a6c7f3a3-8340-41a7-8154-95c6b5585532",
            "@type": "dcat:DataService",
            "dcat:endpointDescription": "dspace:connector",
            "dcat:endpointUrl": "http://localhost:8192/api/dsp",
            "dct:terms": "dspace:connector",
            "dct:endpointUrl": "http://localhost:8192/api/dsp"
          }
        }
      ],
      "description": "This asset requires Membership to view and SensitiveData credential to negotiate.",
      "id": "asset-2"
    }
  ],
  "dcat:distribution": [],
  "dcat:service": {
    "@id": "a6c7f3a3-8340-41a7-8154-95c6b5585532",
    "@type": "dcat:DataService",
    "dcat:endpointDescription": "dspace:connector",
    "dcat:endpointUrl": "http://localhost:8192/api/dsp",
    "dct:terms": "dspace:connector",
    "dct:endpointUrl": "http://localhost:8192/api/dsp"
  },
  "dspace:participantId": "did:web:localhost%3A7093",
  "participantId": "did:web:localhost%3A7093",
  "@context": {}
}

Catalogs are ephemeral objects, they are not persisted or cached on the provider side. Everytime a consumer participant makes a catalog request through DSP, the connector runtime has to evaluate the incoming request and build up the catalog specifically for that participant. The reason for this is that between two subsequent requests from the same participant, the contract definition or the claims or the participant could have changed.

The relevant component in EDC is the DatasetResolver, which resolves all contract definitions that are relevant to a participant filtering out those where the participant does not satisfy the access policy and collects all the assets therein.

In order to determine how an asset can be distributed, the resolver requires knowledge about the data planes that are available. It uses the Dataplane Signaling Protocol to query them and construct the list of Distributions for an asset.

For details about the FederatedCatalog, please refer to its documentation.

7 Transfer processes

A TransferProcess is a record of the data sharing procedure between a consumer and a provider. As they traverse through the system, they transition through several states (TransferProcessStates).

Once a contract is negotiated and an agreement is reached, the consumer connector may send a transfer initiate request to start the transfer. In the course of doing that, both parties may provision additional resources, for example deploying a temporary object store, where the provider should put the data. Similarly, the provider may need to take some preparatory steps, e.g. anonymizing the data before sending it out.

This is sometimes referred to as the provisioning phase. If no additional provisioning is needed, the transfer process simply transitions through the state with a NOOP.

Once that is done, the transfer begins in earnest. Data is transmitted according to the dataDestination, that was passed in the initiate-request.

Once the transmission has completed, the transfer process will transition to the COMPLETED state, or - if an error occurred - to the TERMINATED state.

The Management API provides several endpoints to manipulate data transfers.

Here is a diagram of the state machine applied to transfer processes on consumer side:

Here is a diagram of the state machine applied to transfer processes on provider side:

A transfer process can be initiated from the consumer side by sending a TransferRequest to the connector Management API:

{
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type": "https://w3id.org/edc/v0.0.1/ns/TransferRequest",
  "protocol": "dataspace-protocol-http",
  "counterPartyAddress": "http://provider-address",
  "contractId": "contract-id",
  "transferType": "transferType",
  "dataDestination": {
    "type": "data-destination-type"
  },
  "privateProperties": {
    "private-key": "private-value"
  },
  "callbackAddresses": [
    {
      "transactional": false,
      "uri": "http://callback/url",
      "events": [
        "contract.negotiation",
        "transfer.process"
      ],
      "authKey": "auth-key",
      "authCodeId": "auth-code-id"
    }
  ]
}

where:

• counterPartyAddress: the address where to send the TransferRequestMessage via the specified protocol ( currently dataspace-protocol-http)
• contractId: the ID of a previously negotiated contract agreement which is a result of the contract negotiation process.
• transferType and the dataDestinationdefine how and where the data transfer should happen.
• callbackAddresses custom hooks in order bo be notified about state transition of the transfer process.
• privateProperties: custom properties not shared with the counter party.

7.1 Transfer and data flows types

The transfer type defines the channel (Distribution) for the data transfer and it depends on the capabilities of the data plane if it can be fulfilled. The transferType available for a data offering is available in the dct:format of the Distribution when inspecting the catalog response.

Each transfer type also characterizes the type of the flow, which can be either pull or push and it’s data can be either finite or non-finite

7.1.1 Consumer Pull

A pull transfer is when the consumer receives information (in the form of a DataAddress) on how to retrieve data from the Provider. Then it’s up to the consumer to use this information for pulling the data.

Provider and consumer agree to a contract (not displayed in the diagram)

1. Consumer initiates the transfer process by sending a TransferRequestMessage
2. The Provider Control Plane retrieves the DataAddress of the actual data source and creates aDataFlowStartMessage.
3. The Provider Control Plane asks the selector which Data Plane instance can be used for this data transfer
4. The Selector returns an eligible Data Plane instance (if any)
5. Provider Control Plane sends the DataFlowStartMessage to the selected Data Plane instance through data plane signaling protocol.
6. The Provider DataPlaneManager validates the incoming request and delegates to the DataPlaneAuthorizationService the generation of DataAddress, containing the information on location and authorization for fetching the data
7. The Provider Data Plane acknowledges the Provider control plane and attach the DataAddress generated.
8. The Provider Control Plane notifies the start of the transfer attaching the DataAddress in the TransferStartMessage.
9. The Consumer Control plane receives the DataAddress and dispatch it accordingly to the configured runtime. Consumer can either decide to receive the DataAddress using the eventing system callbacks using the transfer.process.started type, or use the EDRs extensions for automatically store it on consumer control plane side.
10. With the informations in the DataAddress such as the endpointUrl and the Authorization data can be fetched.
11. The Provider Data plane validates and authenticates the incoming request and retrieves the source DataAddress.
12. The he provider data plane proxies the validated request to the configured backend in the source DataAddress.

7.1.2 Provider Push

A push transfer is when the Provider data plane initiates sending data to the destination specified by the consumer.

Provider and consumer agree to a contract (not displayed in the diagram)

1. The Consumer initiates the transfer process, i.e. sends TransferRequestMessage with a destination DataAddress
2. The Provider Control Plane retrieves the DataAddress of the actual data source and creates a DataFlowStartMessage with both source and destination DataAddress.
3. The Provider Control Plane asks the selector which Data Plane instance can be used for this data transfer
4. The Selector returns an eligible Data Plane instance (if any)
5. The Provider Control Plane sends the DataFlowStartMessage to the selected Data Plane instance through data plane signaling protocol.
6. The Provider Data Plane validates the incoming request
7. If request is valid, the Provider Data Plane returns acknowledgement
8. The DataPlaneManager of the the Provider Data Plane processes the request: it creates a DataSource/DataSinkpair based on the source/destination data addresses
9. The Provider Data Plane fetches data from the actual data source (see DataSource)
10. The Provider Data Plane pushes data to the consumer services (see DataSink)

7.1.2 Finite and Non-Finite Data

The charaterization of the data applies to either push and pull transfers. Finite data transfers cause the transfer process to transitition to the state COMPLETED, once the transmission has finished. For example a transfer of a single file that is hosted and transferred into a cloud storage system.

Non-finite data means that once the transfer process request has been accepted by the provider the transfer process is in the STARTED state until it gets terminated by the consumer or the provider. Exampes of Non-finite data are streams or API endpoins.

On the provider side transfer processes can also be terminated by the policy monitor that periodically watches over the on going transfer and checks if the associated contract agreement still fulfills the contract policy.

7.2 About Data Destinations

A data destination is a description of where the consumer expects to find the data after the transfer completes. In a " provider-push" scenario this could be an object storage container, a directory on a file system, etc. In a “consumer-pull” scenario this would be a placeholder, that does not contain any information about the destination, as the provider “decides” which endpoint he makes the data available on.

A data address is a schemaless object, and the provider and the consumer need to have a common understanding of the required fields. For example, if the provider is supposed to put the data into a file share, the DataAddress object representing the data destination will likely contain the host URL, a path and possibly a file name. So both connectors need to be “aware” of that.

The actual data transfer is handled by a data plane through extensions ( called “sources” and " sinks"). Thus, the way to establish that “understanding” is to make sure that both parties have matching sources and sinks. That means, if a consumer asks to put the data in a file share, the provider must have the appropriate data plane extensions to be able to perform that transfer.

If the provider connector does not have the appropriate extensions loaded at runtime, the transfer process will fail.

7.3 Transfer process callbacks

In order to get timely updates about status changes of a transfer process, we could simply poll the management API by firing a GET /v*/transferprocesses/{tp-id}/state request every X amount of time. That will not only put unnecessary load on the connector, you may also run into rate-limiting situations, if the connector is behind a load balancer of some sort. Thus, we recommend using event callbacks.

Callbacks must be specified when requesting to initiate the transfer:

{
  // ...
  "callbackAddresses": [
    {
      "transactional": false,
      "uri": "http://callback/url",
      "events": [
        "transfer.process"
      ],
      "authKey": "auth-key",
      "authCodeId": "auth-code-id"
    }
  ]
  //...
}

Currently, we support the following events:

• transfer.process.deprovisioned
• transfer.process.completed
• transfer.process.deprovisioningRequested
• transfer.process.initiated
• transfer.process.provisioned
• transfer.process.provisioning
• transfer.process.requested
• transfer.process.started
• transfer.process.terminated

The connector’s event dispatcher will send invoke the webhook specified in the uri field passing the event payload as JSON object.

More info about events and callbacks can be found here.

8 Endpoint Data References

9 Querying with QuerySpec and Criterion

Most of the entities can be queried with the QuerySpec object, which is a generic way of expressing limit, offset, sort and filters when querying a collection of objects managed by the EDC stores.

Here’s an example of how a QuerySpec object might look like when querying for Assets via management APIs:

{
  "@context": {
    "edc": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type": "QuerySpec",
  "limit": 1,
  "offset": 1,
  "sortField": "createdAt",
  "sortOrder": "DESC",
  "filterExpression": [
    {
      "operandLeft": "https://w3id.org/edc/v0.0.1/ns/description",
      "operator": "=",
      "operandRight": "This asset"
    }
  ]
}

which filters by the description custom property being equals to This asset. The query also paginates the result with limit and p set to 1. Additionally a sorting strategy is in place by createdAt property in descending order ( the default is ASC)

Note: Since custom properties are persisted in their expanded form, we have to use the expanded form also when querying.

The filterExpression property is a list of Criterion, which expresses a single filtering condition based on:

• operandLeft: the property to filter on
• operator: the operator to apply e.g. =
• operandRight: the value of the filtering

The supported operators are:

• Equal: =
• Not equal: !=
• In: in
• Like: like
• Ilike: ilike (same as like but ignoring case sensitive)
• Contains: contains

Note: multiple filtering expressions are always logically conjoined using an “AND” operation.

The properties that can be expressed in the operandLeft of a Criterion depend on the shape of the entity that we are want to query.

Note: nested properties are also supported using the dot notation.

QuerySpec can also be used when doing the catalog request using the querySpec property in the catalog request payload for filtering the datasets:

{
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "counterPartyAddress": "http://provider/api/dsp",
  "protocol": "dataspace-protocol-http",
  "counterPartyId": "providerId",
  "querySpec": {
    "filterExpression": [
      {
        "operandLeft": "https://w3id.org/edc/v0.0.1/ns/description",
        "operator": "=",
        "operandRight": "This asset"
      }
    ]
  }
}

Entities are backed by stores for doing CRUD operations. For each entity there is an associated store interface (SPI). Most of the stores SPI have a query like method which takes a QuerySpec type as input and returns the matched entities in a collection. Indivitual implementations are then responsible for translating the QuerySpec to a proper fetching strategy.

The description on how the translation and mapping works will be explained in each implementation. Currently EDC support out of the box:

• In-memory stores (default implementation).
• SQL stores provied as extensions for each store, mostly tailored for and tested with PostgreSQL.

For guaranteeing the highest compatibility between store implementations, a base tests suite is provided for each store that each technology implementors need to fulfill in order to have a minimum usable store implementation.

7.2 - Json LD

• JSON-LD in EDC
  • 1. JSON-LD in EDC
  • 1.1 Compact IRI
  • 1.2 Custom Remote Context
    • 1.1 JSON-LD Validation

Here is a simple example taken from json-ld.org

{
  "@context": "https://json-ld.org/contexts/person.jsonld",
  "@id": "http://dbpedia.org/resource/John_Lennon",
  "name": "John Lennon",
  "born": "1940-10-09",
  "spouse": "http://dbpedia.org/resource/Cynthia_Lennon"
}

It’s similar on how a Person would be represented in JSON, with additional known properties such as @context and @id.

The @id is used to uniquely identify an object.

The @context is used to define how terms should be interpreted and help expressing specific identifier with short-hand names instead of IRI.

Exhausting reserved keywords list and their meaning is available here

In the above example the @context is a remote one, but the @context can also be defined inline. Here is the same JSON-LD object using locally defined terms.

{
  "@context": {
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "name": "http://xmlns.com/foaf/0.1/name",
    "born": {
      "@id": "http://schema.org/birthDate",
      "@type": "xsd:date"
    },
    "spouse": {
      "@id": "http://schema.org/spouse",
      "@type": "@id"
    }
  },
  "@id": "http://dbpedia.org/resource/John_Lennon",
  "name": "John Lennon",
  "born": "1940-10-09",
  "spouse": "http://dbpedia.org/resource/Cynthia_Lennon"
}

which defines inline the name, born and spouse terms.

The two objects have the same meaning as Linked Data.

A JSON-LD document can be described in multiple forms and by applying certain transformations a document can change shape without changing the meaning.

Relevant forms in the realm of EDC are:

• Expanded document form
• Compacted document form

The examples above are in compacted form and by applying the expansion algorithm the output would look like this

[
  {
    "@id": "http://dbpedia.org/resource/John_Lennon",
    "http://schema.org/birthDate": [
      {
        "@type": "http://www.w3.org/2001/XMLSchema#date",
        "@value": "1940-10-09"
      }
    ],
    "http://xmlns.com/foaf/0.1/name": [
      {
        "@value": "John Lennon"
      }
    ],
    "http://schema.org/spouse": [
      {
        "@id": "http://dbpedia.org/resource/Cynthia_Lennon"
      }
    ]
  }
]

The expansion is the process of taking in input a JSON-LD document and applying the @context so that it is no longer necessary, as all the terms are resolved in their IRI representation.

The compaction is the inverse process. It takes in input a JSON-LD in expanded form and by applying the supplied @context, it creates the compacted form.

For playing around JSON-LD and processing algorithm the playground is a useful tool.

1. JSON-LD in EDC

EDC uses JSON-LD as primary serialization format at API layer and at runtime EDC manages the objects in their expanded form, for example when transforming JsonObject into EDC entities and and backwards in transformers or when validating input JsonObject at API level.

Extensible properties in entities are always stored expanded form.

To achieve that, EDC uses an interceptor (JerseyJsonLdInterceptor) that always expands in ingress and compacts in egress the JsonObject.

EDC uses JSON-LD for two main reasons:

Fist EDC embraces different protocols and standards such as:

• dsp
• odrl
• dcat

and they all rely on JSON-LD as serialization format.

The second reason is that EDC allows to extends entities like Asset with custom properties, and uses JSON-LD as the way to extend objects with custom namespaces.

EDC handles JSON-LD through the JsonLd SPI. It supports different operation and configuration for managing JSON-LD in the EDC runtime.

It supports expansion and compaction process:

  Result<JsonObject> expand(JsonObject json);

Result<JsonObject> compact(JsonObject json, String scope);

and allows the configuration of which @context and namespaces to use when processing the JSON-LD in a specific scope.

For example when using the JsonLd service in the management API the @context and namespaces configured might differs when using the same service in the dsp layer.

The JsonLd service also can configure cached contexts by allowing to have a local copy of the remote context. This limits the network request required when processing the JSON-LD and reduces the attack surface if the remote host of the context is compromised.

By default EDC make usage of @vocab for processing input/output JSON-LD document. This can provide a default vocabulary for extensible properties. An on-going initiative is available with

this extension

in order to provide a cached terms mapping (context) for EDC management API. The remote context definition is available here.

Implementors that need additional @context and namespaces to be supported in EDC runtime, should develop a custom extension that registers the required @context and namespace.

For example let’s say we want to support a custom namespace http://w3id.org/starwars/v0.0.1/ns/ in the extensible properties of an Asset.

The input JSON would look like this:

{
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "sw": "http://w3id.org/starwars/v0.0.1/ns/"
  },
  "@type": "Asset",
  "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
  "properties": {
    "sw:faction": "Galactic Imperium",
    "sw:person": {
      "sw:name": "Darth Vader",
      "sw:webpage": "https://death.star"
    }
  },
  "dataAddress": {
    "@type": "DataAddress",
    "type": "myType"
  }
}

Even if we don’t register a any additional @context or namespace prefix in the EDC runtime, the Asset will still be persisted correctly since the JSON-LD gets expanded correctly and stored in the expanded form.

But in the egress the JSON-LD document gets always compacted, and without additional configuration, it will look like this:

{
  "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
  "@type": "Asset",
  "properties": {
    "http://w3id.org/starwars/v0.0.1/ns/faction": "Galactic Imperium",
    "http://w3id.org/starwars/v0.0.1/ns/person": {
      "http://w3id.org/starwars/v0.0.1/ns/name": "Darth Vader",
      "http://w3id.org/starwars/v0.0.1/ns/webpage": "https://death.star"
    },
    "id": "79d9c360-476b-47e8-8925-0ffbeba5aec2"
  },
  "dataAddress": {
    "@type": "DataAddress",
    "type": "myType"
  },
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "odrl": "http://www.w3.org/ns/odrl/2/"
  }
}

That means that the IRIs are not shortened to terms or compact iri. This might be ok for some runtime and configuration. But if implementors want to achieve more usability and easy of usage, two main strategy can be applied:

• Compact IRI
• Custom remote context

1.1 Compact IRI

The first strategy is to register a namespace prefix in an extension:

public class MyExtension implements ServiceExtension {

    @Inject
    private JsonLd jsonLd;

    @Override
    public void initialize(ServiceExtensionContext context) {
        jsonLd.registerNamespace("sw", "http://w3id.org/starwars/v0.0.1/ns/", "MANAGEMENT_API");
    }
}

This will shorten the IRI to compact IRI when compacting the same JSON-LD:

{
  "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
  "@type": "Asset",
  "properties": {
    "sw:faction": "Galactic Imperium",
    "sw:person": {
      "sw:name": "Darth Vader",
      "sw:webpage": "https://death.star"
    },
    "id": "79d9c360-476b-47e8-8925-0ffbeba5aec2"
  },
  "dataAddress": {
    "@type": "DataAddress",
    "type": "myType"
  },
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "sw": "http://w3id.org/starwars/v0.0.1/ns/"
  }
}

1.2 Custom Remote Context

An improved version requires developers to draft a context (which should be resolvable with an URL), for example http://w3id.org/starwars/context.jsonld, that contains the terms definition.

An example of a definition might look like this:

{
  "@context": {
    "@version": 1.1,
    "sw": "http://w3id.org/starwars/v0.0.1/ns/",
    "person": "sw:person",
    "faction": "sw:faction",
    "name": "sw:name",
    "webpage": "sw:name"
  }
}

Then in a an extension the context URL should be registered in the desired scope and cached:

public class MyExtension implements ServiceExtension {

    @Inject
    private JsonLd jsonLd;

    @Override
    public void initialize(ServiceExtensionContext context) {
        jsonld.registerContext("http://w3id.org/starwars/context.jsonld", "MANAGEMENT_API");

        URI documentLocation = // load from filesystem or classpath
                jsonLdService.registerCachedDocument("http://w3id.org/starwars/context.jsonld", documentLocation)
    }
}

With this configuration the JSON-LD will be representend without the sw prefix, since the terms mapping is defined in the remote context http://w3id.org/starwars/context.jsonld:

{
  "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
  "@type": "Asset",
  "properties": {
    "faction": "Galactic Imperium",
    "person": {
      "name": "Darth Vader",
      "webpage": "https://death.star"
    },
    "id": "79d9c360-476b-47e8-8925-0ffbeba5aec2"
  },
  "dataAddress": {
    "@type": "DataAddress",
    "type": "myType"
  },
  "@context": [
    "http://w3id.org/starwars/context.jsonld",
    {
      "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
      "edc": "https://w3id.org/edc/v0.0.1/ns/",
      "odrl": "http://www.w3.org/ns/odrl/2/"
    }
  ]
}

In case of name clash in the terms definition, the JSON-LD processor should fallback to the compact URI representation.

1.1 JSON-LD Validation

EDC provides a mechanism to validate JSON-LD objects. The validation phase is typically handled at the network/controller layer. For each entity identified by it’s own @type, it is possible to register a custom Validator<JsonObject> using the registry JsonObjectValidatorRegistry. By default EDC provides validation for all the entities it manages like Asset, ContractDefinition ..etc.

For custom validator it is possible to either implements Validator<JsonObject> interface (not recommended) or or use the bundled JsonObjectValidator, which is a declarative way of configuring a validator for an object through the builder pattern. It also comes with a preset of validation rules such as id not empty, mandatory properties and many more.

An example of validator for a custom type Foo:

{
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@id": "79d9c360-476b-47e8-8925-0ffbeba5aec2",
  "@type": "Foo",
  "bar": "value"
}

might look like this:

public class FooValidator {

    public static JsonObjectValidator instance() {
        return JsonObjectValidator.newValidator()
                .verifyId(OptionalIdNotBlank::new)
                .verify("https://w3id.org/edc/v0.0.1/ns/bar")
                .build();
    }
}

and can be registered with the @Injectable JsonObjectValidatorRegistry:

public class MyExtension implements ServiceExtension {

    @Inject
    private JsonObjectValidatorRegistry validator;

    @Override
    public void initialize(ServiceExtensionContext context) {

        validator.register("https://w3id.org/edc/v0.0.1/ns/Foo", FooValidator.instance());
    }
}

When needed, it can be invoked like this:

public class MyController {

    private JsonObjectValidatorRegistry validator;

    @Override
    public void doSomething(JsonObject input) {
        validator.validate("https://w3id.org/edc/v0.0.1/ns/Foo", input)
                .orElseThrow(ValidationFailureException::new);
    }
}

7.3 - Policy Monitor

Some transfer types, once accepted by the provider, never reach the COMPLETED state. Streaming and HTTP transfers in consumer pull scenario are examples of this. In those scenarios the transfer will remain active (STARTED) until it gets terminated either manually by using the transfer processes management API, or automatically by the policy monitor, if it has been configured in the EDC runtime.

The policy monitor (PolicyMonitorManager) is a component that watches over on-going transfers on the provider side and ensures that the associated policies are still valid. The default implementation of the policy monitor tracks the monitored transfer processes in it’s own entity PolicyMonitorEntry stored in the PolicyMonitorStore.

Once a transfer process transition to the STARTED state on the provider side, the policy monitor gets notified through the eventing system of EDC and start tracking transfer process. For each monitored transfer process in the STARTED state the policy monitor retrieves the policy associated (through contract agreement) and runs the Policy Engine using the policy.monitor as scope. If the policy is no longer valid, the policy monitor marks the transfer process for termination (TERMINATING) and stops tracking it.

The data plane also gets notified through the data plane signaling protocol about the termination of the transfer process, and if accepted by the data plane, the data transfer terminates as well.

Note for implementors

Implementors that want a Policy function to be evaluated at the policy monitor layer need to bind such function to the policy.monitor scope.

Note that because the policy evaluation happens in the background, the PolicyContext does not contain ParticipantAgent as context data. This means that the Policy Monitor cannot evaluate policies that involve VerifiableCredentials.

Currently the only information published in the PolicyContext available for functions in the policy.monitor scope are the ContractAgreement, and the Instant at the time of the evaluation.

A bundled example of a Policy function that runs in the policy.monitor scope is the ContractExpiryCheckFunction which checks if the contract agreement is not expired.

7.4 - Programming Primitives

• 1 State machines
  • 1.1 Batch-size, sorting and tick-over timeout
  • 1.2 Database-level locking
• 2. Transformers
  • 2.1 Basic Serialization and Deserialization
  • 2.1 Transformer context
  • 2.2 Segmented transformer registries
  • 2.3 Reporting transformation errors
• 3. Token generation and decorators
• 4. Token validation and rules
  • 4.1 Public Key Resolvers
  • 4.2 Validation Rules
  • 4.3 Validation Rules Registry

1 State machines

EDC is asynchronous by design, which means that processes are processed in such a way that they don’t block neither the runtime nor the caller. For example starting a contract negotiation is a long-running process and every contract negotiation has to traverse a series of states, most of which involve sending remote messages to the counter party. These state transitions are not guaranteed to happen within a certain time frame, they could take hours or even days.

From that it follows that an EDC instance must be regarded as ephemeral (= they can’t hold state in memory), so the state (of a contract negotiation) must be held in persistent storage. This makes it possible to start and stop connector runtimes arbitrarily, and every replica picks up where the other left off, without causing conflicts or processing an entity twice.

The state machine itself is synchronous: in every iteration it processes a number of objects and then either goes back to sleep, if there was nothing to process, or continues right away.

At a high level this is implemented in the StateMachineManager, which uses a set of Processors. The StateMachineManager sequentially invokes each Processor, who then reports the number of processed entities. In EDC’s state machines, processors are functions who handle StatefulEntities in a particular state and are registered when the application starts up:

// ProviderContractNegotiationManagerImpl.java

@Override
protected StateMachineManager.Builder configureStateMachineManager(StateMachineManager.Builder builder) {
    return builder
            .processor(processNegotiationsInState(OFFERING, this::processOffering))
            .processor(processNegotiationsInState(REQUESTED, this::processRequested))
            .processor(processNegotiationsInState(ACCEPTED, this::processAccepted))
            .processor(processNegotiationsInState(AGREEING, this::processAgreeing))
            .processor(processNegotiationsInState(VERIFIED, this::processVerified))
            .processor(processNegotiationsInState(FINALIZING, this::processFinalizing))
            .processor(processNegotiationsInState(TERMINATING, this::processTerminating));
}

This instantiates a Processor that binds a given state to a callback function. For example AGREEING -> this::processAgreeing. When the StateMachineManager invokes this Processor, it loads all contract negotiations in that state (here: AGREEING) and passes each one to the processAgreeing method.

All processors are invoked sequentially, because it is possible that one single entity transitions to multiple states in the same iteration.

1.1 Batch-size, sorting and tick-over timeout

In every iteration the state machine loads multiple StatefulEntity objects from the database. To avoid overwhelming the state machine and to prevent entites from becoming stale, two main safeguards are in place:

• batch-size: this is the maximum amount of entities per state that are fetched from the database
• sorting: StatefulEntity objects are sorted based on when their state was last updated, oldest first.
• iteration timeout: if no StatefulEntities were processed, the statemachine simply yields for a configurable amount of time.

1.2 Database-level locking

In production deployments the control plane is typically replicated over several instances for performance and robustness. This must be considered when loading StatefulEntity objects from the database, because it is possible that two replicas attempt to load the same entity at the same time, which - without locks - would lead to a race condition, data inconsistencies, duplicated DSP messages and other problems.

To avoid this, EDC employs pessimistic exclusive locks on the database level for stateful entities, which are called Lease. These are entries in a database that indicate whether an entity is currently leased, whether the lease is expired and which replica leased the entity. Attempting to acquire a lease for an already-leased entity is only possible if the lease holder is the same.

Note that the value of the edc.runtime.id property is used to record the holder of a Lease. It is recommended not to configure this property in clustered environments so that randomized runtime IDs (= default) are used.

Generally the process is as follows:

• load N “leasable” entities and acquire a lease for each one. An entity is considered “leasable” if it is not already leased, or the current runtime already holds the lease, or the lease is expired.
• if the entity was processed, advance state, free the lease
• if the entity was not processed, free the lease

That way, each replica of the control plane holds an exclusive lock for a particular entity while it is trying to process and advance its state.

2. Transformers

EDC uses JSON-LD serialization on API ingress and egress. For information about this can be found in this chapter, but the TL;DR is that it is necessary because of extensible properties and namespaces on wire-level DTOs.

2.1 Basic Serialization and Deserialization

On API ingress and egress this means that conventional serialization and deserialization (“SerDes”) cannot be achieved with Jackson, because Jackson operates on a configurable, but ultimately rigid schema.

For that reason, EDC implements its own SerDes layer, called “transformers”. The common base class for all transformers is the AbstractJsonLdTransformer<I,O> and the naming convention is JsonObject[To|From]<Entity>Transformer for example JsonObjectToAssetTransformer. They typically come in pairs, to enable both serialization and deserialization.

Another rule is that the entity class must contain the fully-qualified (expanded) property names as constants and typical programming patterns are:

• deserialization: transformers contain a switch statement that parses the property names and populates the entity’s builder.
• serialization: transformers simply construct the JsonObject based on the properties of the entity using a JsonObjectBuilder

2.1 Transformer context

Many entities in EDC are complex objects that contain other complex objects. For example, a ContractDefinition contains the asset selector, which is a List<Criterion>. However, a Criterion is also used in a QuerySpec, so it makes sense to extract its deserialization into a dedicated transformer. So when the JsonObjectFromContractDefinitionTransformer encounters the asset selector property in the JSON structure, it delegates its deserialization back to the TransformerContext, which holds a global list of type transformers ( TypeTransformerRegistry).

As a general rule of thumb, a transformer should only deserialize first-order properties, and nested complex objects should be delegated back to the TransformerContext.

Every module that contains a type transformer should register it with the TypeTransformerRegistry in its accompanying extension:

@Inject
private TypeTransformerRegistry typeTransformerRegistry;

@Override
public void initialize(ServiceExtensionContext context) {
    typeTransformerRegistry.register(new JsonObjectToYourEntityTransformer());
}

2.2 Segmented transformer registries

One might encounter situations, where different serialization formats are required for the same entity, for example DataAddress objects are serialized differently on the Signaling API and the DSP API.

If we would simply register both transformers with the transformer registry, the second registration would overwrite the first, because both transformers have the same input and output types:

public class JsonObjectFromDataAddressTransformer extends AbstractJsonLdTransformer<DataAddress, JsonObject> {
    //...
}

public class JsonObjectFromDataAddressDspaceTransformer extends AbstractJsonLdTransformer<DataAddress, JsonObject> {
    //...
}

Consequently, all DataAddress objects would get serialized in the same way.

To overcome this limitation, EDC has the concept of segmented transformer registries, where the segment is defined by a string called a “context”:

@Inject
private TypeTransformerRegistry typeTransformerRegistry;

@Override
public void initialize(ServiceExtensionContext context) {
    var signalingApiRegistry = typeTransformerRegistry.forContext("signaling-api");
    signalingApiRegistry.register(new JsonObjectFromDataAddressDspaceTransformer(/*arguments*/));

    var dspRegistry = typeTransformerRegistry.forContext("dsp-api");
    dspRegistry.register(new JsonObjectToDataAddressTransformer());
}

Note that this example serves for illustration purposes only!

Usually, transformation happens in API controllers to deserialize input, process and serialize output, but controllers don’t use transformers directly because more than one transformer may be required to correctly deserialize an object. Rather, they have a reference to a TypeTransformerRegistry for this. For more information please refer to the chapter about service layers.

2.3 Reporting transformation errors

Generally speaking, input validation should be performed by validators. However, it is still possible that an object cannot be serialized/deserialized correctly, for example when a property has has the wrong type, wrong multiplicity, cannot be parsed, unknown property, etc. Those types of errors should be reported to the TransformerContext:

// JsonObjectToDataPlaneInstanceTransformer.java
private void transformProperties(String key, JsonValue jsonValue, DataPlaneInstance.Builder builder, TransformerContext context) {
    switch (key) {
        case URL -> {
            try {
                builder.url(new URL(Objects.requireNonNull(transformString(jsonValue, context))));
            } catch (MalformedURLException e) {
                context.reportProblem(e.getMessage());
            }
        }
        // other properties
    }
}

Transformers should report errors to the context instead of throwing exceptions. Please note that basic JSON validation should be performed by validators.

3. Token generation and decorators

A token is a datastructure that consists of a header and claims and that is signed with a private key. While EDC is able to create any type of tokens through extensions, in most use cases JSON Web Tokens (JWT) are a good option.

The TokenGenerationService offers a way to generate such a token by passing in a reference to a private key and a set of TokenDecorators. These are functions that mutate the parameters of a token, for example they could contribute claims and headers to JWTs:

TokenDecorator jtiDecorator = tokenParams -> tokenParams.claim("jti", UUID.randomUuid().toString());
TokenDecorator typeDecorator = tokenParams -> tokenParams.header("typ", "JWT");
var token = tokenGenerationService.generate("my-private-key-id", jtiDecorator, typeDecorator);

In the EDC code base the TokenGenerationService is not intended to be injectable, because client code typically should be opinionated with regards to the token technology.

4. Token validation and rules

When receiving a token, EDC makes use of the TokenValidationService facility to verify and validate the incoming token. Out-of-the-box JWTs are supported, but other token types could be supported through extensions. This section will be limited to validating JWT tokens.

Every JWT that is validated by EDC must have a kid header indicating the ID of the public key with which the token can be verified. In addition, a PublicKeyResolver implementation is required to download the public key.

4.1 Public Key Resolvers

PublicKeyResolvers are services that resolve public key material from public locations. It is common for organizations to publish their public keys as JSON Web Key Set (JWKS) or as verification method in a DID document. If operational circumstances require that multiple resolution strategies be supported at runtime, the recommended way to achieve this is to implement a PublicKeyResolver that dispatches to multiple sub-resolvers based on the shape of the key ID.

Sometimes it is necessary for the connector runtime to resolve its own public key, e.g. when validating a token that was sent out in a previous interaction. In these cases it is best to avoid a remote call to a DID document or a JWKS URL, but to resolve the public key locally.

4.2 Validation Rules

With the public key the validation service is able to verify the token’s signature, i.e. to assert its cryptographic integrity. Once that succeeds, the TokenValidationService parses the token string and applies all TokenValidationRules on the claims. We call this validation, since it asserts the correct (“valid”) structure of the token’s claims.

4.3 Validation Rules Registry

Usually, tokens are validated in different contexts, each of which brings its own validation rules. Currently, the following token validation contexts exist:

• "dcp-si": when validating Self-Issued ID tokens in the Decentralized Claims Protocol (DCP)
• "dcp-vc": when validating VerifiableCredentials that have an external proof in the form of a JWT (JWT-VCs)
• "dcp-vp": when validating VerifiablePresentations that have an external proof in the form of a JWT (JWT-VPs)
• "oauth2": when validating OAuth2 tokens
• "management-api": when validating external tokens in the Management API ingress (relevant when delegated authentication is used)

Using these contexts it is possible to register additional validation rules using extensions:

//YourSpecialExtension.java

@Inject
private TokenValidationRulesRegistry rulesRegistry;

@Override
public void initialize(ServiceExtensionContext context) {
    rulesRegistry.addRule(DCP_SELF_ISSUED_TOKEN_CONTEXT, (claimtoken, additional) -> {
        var checkResult = ...// perform rule check
        return checkResult;
    });
}

This is useful for example when certain dataspaces require additional rules to be satisfied or even private claims to be exchanged.

7.5 - Protocol Extensions

• Protocol Extensions
  • 1. RemoteMessage
    • 1.1 Delivering messages with RemoteMessageDispatcher
    • 1.2 Handling incoming messages with protocol services
  • 2. DSP protocol implementation
    • 2.1 RemoteMessage handlers
    • 2.2 HTTP endpoints
    • 2.2 RemoteMessage transformers

The EDC officially supports the Dataspace protocol using the HTTPs bindings, but since it is an extensible platform, multiple protocol implementations can be supported for inter-connectors communication. Each supported protocols is identified by a unique key used by EDC for dispatching a remote message.

1. RemoteMessage

At the heart of EDC message exchange mechanism lies the RemoteMessage interface, which describes the protocol, the counterPartyAddress and the counterPartyId used for a message delivery.

RemoteMessage extensions can be divided in three groups:

• Catalog (catalog-spi)
• Contract negotiation (contract-spi)
• Transfer process (transfer-spi)

Each RemoteMessage is:

• Delivered to the counter-party using a RemoteMessageDispatcher.
• Received by the counter-party through the protocol implementation (e.g HTTP) and handled by the protocol services layer.

1.1 Delivering messages with RemoteMessageDispatcher

Each protocol implements a RemoteMessageDispatcher:

public interface RemoteMessageDispatcher {

    String protocol();

    <T, M extends RemoteMessage> CompletableFuture<StatusResult<T>> dispatch(Class<T> responseType, M message);

}

and it is registered in the RemoteMessageDispatcherRegistry, where it gets associated to the protocol defined in RemoteMessageDispatcher#protocol.

Internally EDC uses the RemoteMessageDispatcherRegistry whenever it needs to deliver a RemoteMessage to the counter-party. The RemoteMessage then gets routed to the right RemoteMessageDispatcher based on the RemoteMessage#getProtocol property.

EDC also uses RemoteMessageDispatcherRegistry for non-protocol messages when dispatching event callbacks

1.2 Handling incoming messages with protocol services

On the ingress side, protocol implementations should be able to receive messages through the network (e.g. API Controllers), deserialize them into the corresponding RemoteMessages and then dispatching them to the right protocol service.

Protocol services are three:

• CatalogProtocolService
• ContractNegotiationProtocolService
• TransferProcessProtocolService

which handle respectively Catalog, ContractNegotiation and TransferProcess messages.

2. DSP protocol implementation

The Dataspace protocol protocol implementation is available under the data-protocol/dsp subfolder in the Connector repository and it is identified by the key dataspace-protocol-http.

It extends the RemoteMessageDispatcher with the interface DspHttpRemoteMessageDispatcher (dsp-spi), which adds an additional method for registering message handlers.

The implementation of the three DSP specifications:

• Catalog
• Contract Negotiation
• Transfer Process

is separated in multiple extension modules grouped by specification.

This allows for example to build a runtime that only serves a dsp catalog requests useful the Management Domains scenario.

Each specification implementation defines handlers, transformers for RemoteMessages and exposes HTTP endpoints.

The dsp implementation also provide HTTP endpoints for the DSP common functionalities.

2.1 RemoteMessage handlers

Handlers map a RemoteMessage to an HTTP Request and instruct the DspHttpRemoteMessageDispatcher how to extract the response body to a desired type.

2.2 HTTP endpoints

Each dsp-*-http-api module exposes its own API Controllers for serving the specification requests. Each request handler transforms the JSON-LD in input, if present, into a RemoteMessage and then calls the protocol service layer.

2.2 RemoteMessage transformers

Each dsp-*-transform module registers in the DSP API context Transformers for mapping JSON-LD objects from and to RemoteMessages.

7.6 - Service Layers

• 1. API controllers
  • 1.1 API contexts
  • 1.2 Registering controllers
  • 1.3 Registering other resources
  • 1.4 API Authentication
• 2. Validators
• 3. Transformers
• 4. Aggregate services
• 5. Data persistence
  • 5.1 In-Memory stores
• 6. Events and Callbacks
  • 6.1 Event vs EventEnvelope
  • 6.2 Registering for events (in-process)
  • 6.3 Registering for callbacks (webhooks)
  • 6.4 Emitting custom events
  • 6.5 Serialization and Deserialization of custom events

This document describes the EDC service layers.

1. API controllers

EDC uses JAX-RS/Jersey to expose REST endpoints, so our REST controllers look like this:

@Consumes({ MediaType.APPLICATION_JSON })
@Produces({ MediaType.APPLICATION_JSON })
@Path("/v1/foo/bar")
public class SomeApiController implements SomeApi {

    @POST
    @Override
    public JsonObject create(JsonObject someApiObject) {
        //perform logic
    }
}

it is worth noting that as a rule, EDC API controllers only carry JAX-RS annotations, where all other annotations, such as OpenApi should be put on the interface SomeApi.

In addition, EDC APIs accept their arguments as JsonObject due to the use of JSON-LD. This applies to internal APIs and external APIs alike.

API controllers should not contain any business logic other than validation, serialization and service invocation.

All API controllers perform JSON-LD expansion upon ingress and JSON-LD compaction upon egress.

1.1 API contexts

API controllers must be registered with the Jersey web server. To better separate the different API controllers and cluster them in coherent groups, EDC has the notion of “web contexts”. Technically, these are individual ServletContainer instances, each of which available at a separate port and URL path.

To register a new context, it needs to be configured first:

@Inject
private WebService webService;
@Inject
private WebServiceConfigurer configurer;
@Inject
private WebServer webServer;

@Override
public void initialize(ServiceExtensionContext context) {

    var defaultConfig = WebServiceSettings.Builder.newInstance()
            .apiConfigKey("web.http.yourcontext")
            .contextAlias("yourcontext")
            .defaultPath("/api/some")
            .defaultPort(10080)
            .useDefaultContext(false)
            .name("Some new API")
            .build();
    var config = context.getConfig("web.http.yourcontext"); //reads web.http.yourcontext.[port|path] from the configuration
    configurer.configure(config, webServer, defaultConfig);
}

1.2 Registering controllers

After the previous step, the "yourcontext" context is available with the web server and the API controller can be registered:

webservice.registerResource("yourcontext",new SomeApiController(/* arguments */)).

This makes the SomeApiController available at http://localhost:10080/api/some/v1/foo/bar. It is possible to register multiple controllers with the same context.

Note that the default port and path can be changed by configuring web.http.yourcontext.port and web.http.yourcontext.path.

1.3 Registering other resources

Any JAX-RS Resource (as per the JAX-RS Specification, Chapter 3. Resources) can be registered with the web server.

Examples of this in EDC are JSON-LD interceptors, that expand/compact JSON-LD on ingress and egress, respectively, and ContainerFilter instances that are used for request authentication.

1.4 API Authentication

In Jersey, one way to do request authentication is by implementing the ContainerRequestFilter interface. Usually, authentication and authorization information is communicated in the request header, so EDC defines the AuthenticationRequestFilter, which extracts the headers from the request, and forwards them to an AuthenticationService instance.