Middleware in design?

[ElectricCookie]

I'm really enjoying development with goa. ❤️

I wanted to move some repeating code to a middleware that is used by several methods. I was wondering why the middleware is not part of the design file? Currently the registration in the main.go feels a bit decoupled. Looking at the design or the implementation does not easily reveal which middleware is running for a specific endpoint.

My current workaround is just calling a method at the top of an endpoint, but this feels a bit against the "design first" philosophy.

[raphael]

That makes a lot of sense! A goal of the Goa design is to ensure that design code is never used at runtime, the DSL is only used for code generation. This makes it possible to ensure that the generated code follows an expected pattern and that in particular there is clear layer enforcements between transport and non-transport code.

So applying this to middlewares there could be a couple of non-mutually exclusive approaches:

There could be a new "Middleware" DSL that can be applied to non-transport or transport specific DSL, something like:

var _ = Service("calc", func() {
    Middleware("ErrorHandler", "The ErrorHandler middleware reports unexpected errors.")
})

This would cause Goa to generate hooks for every service method that the user provides the implementation to (something akin to how the security middlewares work today). the Middleware DSL could be applied at the API, Service or Method level and would cause hooks to be generated for all the methods in scope. The user provided function signature would be:

func ErrorHandler(goa.Endpoint) goa.Endpoint

Similarly the Middleware function could be applied at the transport level:

 var _ = Service("calc", func() {
    HTTP(func() {
        Middleware("ErrorHandler", "The ErrorHandler middleware reports unexpected errors.")
    })
    GRPC(func() {
        Middleware("ErrorHandler", "The ErrorHandler middleware reports unexpected errors.")
    })
})

In this case the hooks would be transport specific:

func HTTPErrorHandler(http.Handler) http.Handler
func GRPCErrorHandler() grpc.UnaryServerInterceptor // (or grpc.StreamServerInterceptor)

Alternatively (or complementary) there could be a set of "known" middlewares that have specific DSL, or a generic DSL that identifies the middleware:

var _ = Service("calc", func() {
    Middleware(ErrorHandler)
})

These specific middleware implementations might have different hook signatures, for example in this case:

func ErrorHandler(err error)

There might be other approaches, this is what comes to mind first :)

[ElectricCookie]

This is exactly what I had in mind - and now that you mention it the security middleware is already really close to this!

I like the idea of having several places to put the middleware (API, Service and Method).

While we are dreaming:

(1) I currently have some middleware that adds values to the context, which is sometimes error prone (forgetting the middleware leads to nils..) maybe we could design the middleware function to produce a result-type? This could then be passed to the method in a type safe manner?

var FooExtractor = Middleware("fooExtractor",func(){
       // Use a result to signal that the middleware has an output
        Result(func(){
          // Field numbers are probably not necessary, since the result is not exposed to GRPC directly
           Field(0,"extractedFoo",String)
       })
      Error("noFoo","No foo was found in the payload")
  }) 

Method("foo",func() {
    Middleware(FooExtractor)
})

which could result in

func (s *ExampleService) Foo(ctx context.Context, *p example.FooPayload) (res *example.FooResult, err error){
    extractedFoo := example.FooExtractorResult(ctx)
}

alternatively, for an even cleaner experience the result of the middleware could be merged into the payload type under the name of the middleware.

func (s *ExampleService) Foo(ctx context.Context, *p example.FooPayload) (res *example.FooResult, err error){
    extractedFoo :=  p.fooExtractor.extractedFoo
}

(2)If we wanted to go even further one could add a mechanism for making parameters in middlewares possible. This would require that a middleware can accept parameters (making it now very close in charachteristics to a method itself)

var FooExtractor = Middleware("fooExtractor",func(){
       Payload(func(){
           Attribute("fooSeed",String)
           Required("fooSeed")
       })
       // Use a result to signal that the middleware has an output
       Result(func(){
          // Field numbers are probably not necessary, since the result is not exposed to GRPC directly
          Attribute("extractedFoo",String)
       })
      Error("noFoo","No foo was found in the payload")
  }) 

This would then require a check that any method using the FooExtractor middleware has an attribute called fooSeed in its payload and that the types match.

This kind of feature would make many crud application a lot simpler because any endpoint that handles single item operations (GET /item/:id, PATCH /item/:id, DELETE /item/:id/ ...) could then use a middleware to fetch the item with the specified id using a middleware and provide it to the handler in a very convenient manner.

(3) Since now a middleware shares basically all mechanics of a method we could also use methods directly so instead of

var FooExtractor = Middleware(...)
var FooExtractor = Method(....)

and then implement the Middleware DSL to allow the user to add any method as middleware in another method definition.

The checks on type compatibility would still need to apply (meaning that the parameters of the middleware have to be a subset of the method using the middleware).

This would fit very nicely in the already existing DSL in my opinion 🥳 curious about your thoughts

[raphael]

This is great, considering only the "transport-agnostic" scenario for a moment the use cases that this would need to support are:

1. A middleware that modifies the request payload
2. A middleware that modifies the response result
3. A middleware that does neither
4. A middleware that does both

So we would need DSL that make it possible to:

• Read request payloads (potentially only reading certain fields)
• Modify request payloads
• Read response results
• Modify response results (potentially only setting certain fields)

The FooExtractor example above only supports use cases 1. and 3. Here an inspired but revised proposal that would support all the use cases:

The new RequestInterceptor DSL defines a middleware that reads and potentially modifies the method payload. It would support the following sub-DSL:

• Payload defines the the interceptor payload, that is what is read from the request payload. It would work mechanically like Body in HTTP, you could specify a list of request payload attributes, a single attribute by name or the entire payload type.
• Result indicates what the middleware modifies in request payload, and works similarly to the above: the interceptor could modify a single attribute, many attributes or the entire payload.

Similarly the new ResponseInterceptor DSL defines an interceptor that reads and potentially modifies the method result. It would also support a Payload and Result sub-DSL where Payload indicates what to read from the method result and Result what to send back to the client.

There could be both a RequestInterceptor and ResponseInterceptor defined on a method. The request interceptor could add data to the context that the response interceptor could use which allows for use cases where state needs to be communicated (e.g. measuring the time it took to run the method requires capturing time.Now() in the request interceptor and in the response interceptor then subtracting).

Putting it all together:

var SomeRequestInterceptor = RequestInterceptor("SomeRequestInterceptor", func() {
    Payload(func() {
        Attribute("a") // Read payload attribute `a` - must be a payload attribute
        Attribute("b") // Read payload attribute `b`
    })
    // OR
    // Payload("a")  // Read payload attribute `a` and uses that as interceptor argument (instead of object)
    // OR
    // Payload(PayloadType) // Read entire payload type - must be the same as the method payload type
    Result(func() {
        Attribute("c")  // Modifies payload attribute `c` - must be a payload attribute
    })
})

var SomeResponseInterceptor = ResponseInterceptor("SomeResponseInterceptor", func() {
    Payload(func() {
        Attribute("a") // Read result attribute `a` - must be a result attribute
        Attribute("b") // Read result attribute `b`
    })
    // OR
    // Payload("a")  // Read result attribute `a` and uses that as interceptor argument (instead of object)
    // OR
    // Payload(PayloadType) // Read entire result type - must be the same as the method result type
    Result(func() {
        Attribute("c")  // Modifies result attribute `c` - must be a result attribute
    })
})

The above would cause the following hooks to be generated:

func SomeRequestInterceptor(ctx context.Context, payload RequestInterceptorPayloadType) (context.Context, RequestInterceptorResultType, error)

Note that the interceptor returns a context that is then passed on to both the method and any response interceptor. If a request interceptor returns an error then the method is not invoked - instead the error is handled by Goa the same way a method error is (that is if it's a designed error then the corresponding status code is returned otherwise a 500 gets returned).

func SomeResponseInterceptor(ctx context.Context, payload ResponseInterceptorPayloadType) ( ResponseInterceptorResultType, error)

The response interceptor has a similar signature, there is just no context returned in this case.

Additionally there can be multiple request and/or response interceptors defined on a single method. They get run in the order in which they are defined.

[ElectricCookie]

This is looking really cool! Very good observation that my proposal was only covering a “slice” of what is considered middleware.

What do you think of some mechanism to allow “type safe” interaction with context?

Is there a need for interceptors that can pass errors to the following methods? Maybe one could have an error dsl that is non-critical that will not cancel the method from being called if the middleware fails.

[raphael]

What do you think of some mechanism to allow “type safe” interaction with context?

My knee-jerk reaction would be that this would be out-of-scope of Goa. Different use cases might call for different ways of doing this, in particular the actual data types are probably not generic and trying to build something that can cater to all use cases would add undue complexity (to Goa and to user code). But I could be missing something...

Is there a need for interceptors that can pass errors to the following methods? Maybe one could have an error dsl that is >non-critical that will not cancel the method from being called if the middleware fails.

At the end of the day the middleware is either able to transform the request or response or cannot process the request which would be an exception. That is an expected error case should be communicated via the transform (or non-transform) of the payload. In the extreme case the payload could have a field that indicates the outcome although I would expect the need for this to be fairly rare?

[ElectricCookie]

My knee-jerk reaction would be that this would be out-of-scope of Goa. Different use cases might call for different ways of doing this, in particular the actual data types are probably not generic and trying to build something that can cater to all use cases would add undue complexity (to Goa and to user code). But I could be missing something...

I was thinking of something like described above

func (s *ExampleService) Foo(ctx context.Context, *p example.FooPayload) (res *example.FooResult, err error){
    extractedFoo :=  p.fooExtractor.extractedFoo
}

where goa directly injects the result from the middleware into the payload object
or

func (s *ExampleService) Foo(ctx context.Context, *p example.FooPayload) (res *example.FooResult, err error){
    extractedFoo := example.FooExtractorResult(ctx)
}

where goa generates FooExtractorResult(ctx) which under the hood calls ctx.Value(...) with the correct types.

I would prefer the first option since then, a removal of a middleware or a change is directly visible in the implementation function (i.e. the property of the FooPayload does no longer exist)

At the end of the day the middleware is either able to transform the request or response or cannot process the request which would be an exception. That is an expected error case should be communicated via the transform (or non-transform) of the payload. In the extreme case the payload could have a field that indicates the outcome although I would expect the need for this to be fairly rare?

Fair point. I think error DSLs in middleware make sense (e.g. a permission checking middleware will often throw 403s), but maybe non critical errors can be passed via the result (e.g. a cache miss).

[ElectricCookie]

Coming back to this thread, to maybe flesh out the concept a little bit further.

I think there is a couple of different use cases / or ways people use middlware. Looking at the different things:

A middleware that modifies the request payload
A middleware that modifies the response result
A middleware that does neither
A middleware that does both

Looking at the the first two things I think they are actually 4 use-cases. Modifying the request payload/result could mean a run-time change of the values or a design-time change of the signature.

Both of these are, I think, equally interesting, as the ability to have a middleware definition extend or change a methods payload would be a cool feature to have that would make composability of methods a lot easier.

---

Programming in DSLs for generic methods?

While thinking about uses cases which I would use middleware for I thought about pagination.
Adding parameters, and types for pagination quickly gets old (skip, limit query parameters to the request and a total, items array to the response, errors for exceeding the list, etc.) and could lead to potential signature mismatches on different endpoints.

It might be that this is a different problem from most middleware, but I think its such a common functionality (in the end we all build CRUD services mostly...).

While trying to come up with way to generalize this behaviour, I was thinking about ways for a middleware to modify payloads and response types and found no really satisfying way to make clear how it interacts with the signatures of the methods.

It would be great if that was possible in a more dry way. One way I could imagine achieving this with current goa to at least get dry type definitions, is to simply writing a function in the design code that calls the respective DSL methods to "dynamically" generate a method with the correct signature. Not sure if this is an anti pattern, current examples (AFAIK) never do this. It sacrifices a bit of readability for consistency.

func PaginatedMethod(name string,resultType Type) {
  Method(name, func ()  {
    Payload(func() {
      Attribute("page", Int)
      Attribute("limit", Int)
    })
    Result(func() {
      Attribute("data", ArrayOf(resultType))
      Attribute("page", Int)
      Attribute("total", Int)
    })
  })
}

---

A common pattern in middleware functionality for frameworks is chaining, where the middleware has a common signature (req,res,next) the middleware is therefore able to read the request, write to the response and trigger the next execution step by calling next.

This pattern would work for all use cases (modifying payload, response, both or none) but has some caveats as it introduces some potential pitfals:

• The middleware could not call next
• If data is passed via a pointer, the middleware might modify payload or response data after or during calling next which could lead to unexpected concurrency issues

I don't think these are big issues, but maybe we can be clever in the design to avoid or minimize these risks

I think through goa's type system we are in a unique position to safely type middleware meaning that we can ensure the following:

1. Whenever a middleware expects a certain attribute to be present in the payload it is actually there
2. Whenever a middleware produces an error it is also captured in the api specification
3. Whenever a middleware modifies the response of the method it is also reflected in the api spec

Therefore if we wanted to use the chaining pattern we would need to have a way of defining when the middleware is ran in order to compute the types of the method correctly (and potentially adapter types)

Pre-Method middleware:

Say we have a WithUser middleware that expects a JWT token, extracts the sub claim and loads a user from persistent storage and injects the data into the flow.

Placing this middleware before the method, would extend the parameters passed to it by the output of the middleware.

Post-Method middleware:

A middleware that runs after the method, this could be adding some metadata to a response like an errorId.

Wrapping middleware

A middleware that runs before and after, basically wrapping the execution of the method. An example for this use case would a telemetry middleware that measures the execution time of a method.

This order should be somehow intuitive when writing the design file.

The first two would be quite simple. You could have a
RunBefore(...midlewares) and RunAfter(...middlewares) DSL that get added to a service or method definition. Where the order of the arguments to those DSL methods simply imply the order in which the middleware is running.

Adding wrapping middleware makes things considerably harder.

one way I can think of is using a func argument and a new Execute dsl.

The middleware and a WrappingMiddleware DSL now return a function that can be called from the method declaration. The function returned from WrappingMiddleware accepts a function that contains the calls that happen in between.
The Execute DSL signifies when the code of the method is run.

var PerfLogger = WrappingMiddleware(...)
var WithUser = Middleware(...)

Method("getFriends", func(){
  PerfLogger(func(){
     WithUser()
     Execute()
  })
})

In this case perf logger wrapps the execution of the WithUser and the implementation of the "getFriends" method. Calls are sequential meaning that the method has access to the result of WithUser.

As an aside: Calling execute in a method with no middlware has no effect.

Method("noMiddleware", func(){
   Execute()
})

---

Payload and Result of a middleware

While directly emitting types to the paramters and results of a method would be great (meaning that adding a middleware could automatically change the request and response types of a method). It would require some mechanisms to do type combinations, and also stable field numbering (for GRPC) would be nearly impossible.

I would therefore simply go with an approach where the middleware can expect certain parameters to be present in the payload.

If a middleware defines a result (like the WithUser) middleware it can return another parameter that is passed seperately from the response object. Meaning:

var WithUser = Middleware(func(){
    Payload(func(){
        Attribute("userId",String)
    })
    Result(User)
})

When adding this Middleware to a method goa will check whether the attribute userId is part of the payload to that method and whether the type matches at design time.

The middleware returns a User object which can then get passed to the method as well.

func (s *svg) WithUser(ctx context.Context, p *WithUserPayload) (*User,*WithUserResponse,error) {
....
} 

func (s *svc) GetFriends(ctx context.Context,withUserResult *User, p *GetFriendsPayload) (..., error) {
...
}

There's still some things that need to be fleshed out here... will think about this more.. 😄

[raphael]

Hello! Great conversation, I spent a bit of time going through various options and here is a concrete proposal that would fit well with the current design of Goa.

Goa Middleware and Interceptor Proposal

This proposal introduces a comprehensive approach to middleware in Goa that preserves type safety and explicit documentation while supporting both transport-level and business logic concerns. The goal is to provide clear mechanisms for adding cross-cutting functionality while maintaining Goa's core strengths of static typing and clear API contracts.

Background and Key Principles

1. Static Type Definitions

   • Goa's strength lies in static method signature definitions in the design
   • This enables clear contracts and automated generation of OpenAPI/Protobuf specs
   • Method payload and result types should remain in the Method DSL

2. Design Philosophy

   • Keep the DSL descriptive rather than dynamic
   • Avoid introducing function definitions that could make the DSL too dynamic

Middleware Use Cases

Services typically use middlewares in two scenarios:

1. Transport-Specific Behaviors

   • Handling observability
   • Managing security
   • Currently supported via HTTP middlewares and gRPC interceptors
   • Goa implementation: Uses generated Use method on HTTP servers
   • Current limitation: Not documented in the design

2. Cross-Cutting Concerns

   • Aspect-oriented logic that works across multiple endpoints
   • Currently supported via Goa endpoint middlewares
   • Implementation: Functions that transform Goa endpoints and can be mounted via the endpoints Use method
   • Current limitations:
     • Not documented in the design
     • Difficult to access specific payload/response fields due to type casting from any

Proposed Solution: A Dual Approach

The proposal introduces two complementary mechanisms that address different needs:

1. Middlewares: For broad, transport-level concerns
2. Interceptors: For type-safe, payload/result-specific operations

This separation allows us to maintain the benefits of both worlds:

• Transport-level middlewares retain their flexibility and broad applicability
• Interceptors provide type safety and explicit documentation of data dependencies
• Each mechanism is optimized for its specific use case

1. Enhanced Middleware Support

Middlewares are ideal for operations that:

• Need access to the full transport context
• Work with the raw request/response
• Apply transformations independently of payload/result types
• Handle cross-cutting concerns like logging, metrics, and authentication

Endpoint Middleware

Method("getFriends", func() {
    // Can be applied at API, service, or method level
    Middleware("CountRequests", "Count the number of times getFriends has been called.")
    // ... Method definition
})

Generated code in endpoints.go:

func NewEndpoints(s Service, countRequests func(goa.Endpoint) goa.Endpoint) {
    endpoints := // ... initialization
    endpoints.Use(countRequests)
    return endpoints
}

Transport-Specific Middleware

Method("getFriends", func() {
    HTTP(func() {
        Middleware("RefreshAuthToken", "Extends the auth token TTL.")
        // ... Method definition
    })
})

Generated code in http/server.go:

func New(
    e *service.Endpoints,
    mux goahttp.Muxer,
    refreshAuthToken func(http.Handler) http.Handler,
    decoder func(*http.Request) goahttp.Decoder,
    encoder func(context.Context, http.ResponseWriter) goahttp.Encoder,
    errhandler func(context.Context, http.ResponseWriter, error),
    formatter func(ctx context.Context, err error) goahttp.Statuser,
) *Server {
    server := // ... initialization
    server.Use(refreshAuthToken)
    return server
}

gRPC is also supported, the example is omitted for brevity.

2. Interceptor Framework

The interceptor framework complements middlewares by providing:

• Type-safe access to specific payload and result fields
• Explicit documentation of data dependencies
• Compile-time verification of field access
• Clear separation between request and response processing
• Generated OpenAPI specifications that accurately reflect data usage

This approach is particularly valuable when:

• You need to work with specific fields in the payload or result
• Type safety is crucial
• Documentation needs to reflect data access patterns

Interceptors make it possible to inject user code into the decoding and encoding process. For example combined with the struct:field:type metadata interceptors make it possible to define service level types that defined fields of any arbitrary type.

Design Example

package design

import . "goa.design/goa/v3/dsl"

var LoadTenant = RequestInterceptor("LoadTenant", func() {
    Description("Loads the tenant for the given friend ID, returns 404 if not found")
    Reads(func() {
        Attribute("Authorization")  // Authorization HTTP header
    })
    Writes(func() {
        Attribute("TenantID")    // Service payload field
    })
})

var DecodeULID = RequestInterceptor("DecodeULID", func() {
    Description("Decode a string into a ULID")
    Reads(func() {
        Attribute("ULID") // Transport payload field of type string
    })
    Writes(func() {
        Attribute("ULID") // Service payload field of type ulid.ULID
    })
})

var EncodeULID = ResponseInterceptor("EncodeULID", func() {
    Description("Encode a ULID into a string")
    Reads(func() {
        Attribute("ULID") // Service result field of type ulid.ULID
    })
    Writes(func() {
        Attribute("ULID") // Transport result field of type string
    })
})

var _ = Service("Friend", func() {
    Method("GetFriend", func() {
        RequestInterceptor(LoadTenant)
        RequestInterceptor(DecodeULID)
        ResponseInterceptor(EncodeULID)
        Payload(func() {
            Attribute("TenantID", Int, "Tenant ID from JWT token")
            Attribute("ULID", String, "Universally Unique Lexicographically Sortable Identifier", func() {
                Meta("struct:field:type", "ulid.ULID", "github.com/oklog/ulid") // Set service field type to ulid.ULID
            })
        })
        Result(Friend)
        Error("FriendNotFound")
        HTTP(func() {
            GET("/friends/{id}")
            Header("Authorization")
            Response("FriendNotFound", StatusNotFound)
        })
    })
})

Generated Code

In gen/friend/interceptors.go:

package friend

type (
    LoadTenantPayload struct {
        Authorization string
    }

    LoadTenantResult struct {
        TenantID int
    }

    DecodeULIDPayload struct {
        Ulid string
    }

    DecodeULIDResult struct {
        Ulid ulid.ULID
    }

    EncodeULIDPayload struct {
        Ulid ulid.ULID
    }

    EncodeULIDResult struct {
        Ulid string
    }
)

// LoadTenant loads tenant for given friend ID
func LoadTenant(context.Context, *LoadTenantPayload) (*LoadTenantResult, error)

// Decode a string into a ULID 
func DecodeULID(context.Context, *DecodeULIDPayload) (*DecodeULIDResult, error)

// Encode a ULID into a string
func EncodeULID(context.Context, *EncodeULIDPayload) (*EncodeULIDResult, error)

Implementation Notes

1. The generated HTTP server's New function will accept interceptors as arguments
2. Method handlers will be modified to:
   • Call request interceptors after request decoding
   • Call response interceptors before response encoding
3. Interceptors can be applied at API, Service, or Method level

Summary

The combination of middlewares and interceptors provides a complete solution:

• Middlewares for broad, transport-level concerns
• Interceptors for type-safe payload/result operations
• Clear separation of concerns between the two mechanisms

[ElectricCookie]

Another great angle.. I will get back to this soon. I didn't even think of the transport layers as a topic for this functionality.. :)

[tchssk]

#3484 (comment)

2. Interceptor Framework

• Generated OpenAPI specifications that accurately reflect data usage

How exactly do you plan to generate it?

[raphael]

The same way it gets generated today. The point being that the data structure used by the OpenAPI generator would correctly reflect the transport level layout.

[raphael]

I updated the proposal to illustrate a nice benefit of interceptors: they make is possible to inject user code in the marshalling/unmarshalling phase. This is pretty powerful as it has the potential to satisfy a number of use cases involving the need to use custom types at the service level. Specifically:

• Combined with the struct:field:type meta interceptors make it possible to serialize/deserialize fields of any type (not just the Goa built-in types)
• Interceptors also make it possible to initialize service level fields from transport data (e.g. extract info from HTTP headers)
• With a (hopefully) small tweak in how Goa behaves (no change to the DSL) interceptors would make it possible to have different fields on transport structs vs. service level structs. The tweak consists of making it possible to define additional fields in the transport specific sections that are only used to generate the transport types (Goa v1 had this ability). Goa currently validates that fields used in the transport also exist in the service level definition (Payload or Result) but that validation could be removed.

[tchssk]

I don't fully understand interceptor yet, but I have some concerns about middleware.

• It is not possible to guarantee that middleware specified in the DSL will be mounted.

  Method("get", func() {
      Middleware("Middleware1")
  })

  func NewEndpoints(s Service, middleware1 func(goa.Endpoint) goa.Endpoint) {
      endpoints := // ... initialization
      endpoints.Use(middleware1)
      return endpoints
  }

  fooEndpoints = foo.NewEndpoints(fooSvc, middleware2)

• The number of initializer arguments increases by the number of middleware.

  func NewEndpoints(s Service, middleware1, middleware2, middleware3, middleware4, middleware5, middleware6 func(goa.Endpoint) goa.Endpoint) {
      endpoints := // ... initialization
      endpoints.Use(middleware1)
      endpoints.Use(middleware2)
      endpoints.Use(middleware3)
      endpoints.Use(middleware4)
      endpoints.Use(middleware5)
      endpoints.Use(middleware6)
      return endpoints
  }

• Since middleware are all the same type, they may be passed in the wrong order.

  fooEndpoints = foo.NewEndpoints(fooSvc, middleware2, middleware1, middleware3, middleware4, middleware5, middleware6)

[raphael]

Completely agree with all these points @tchssk! Let's iterate on the proposal to tackle these limitations, I have a few ideas. Also I'm thinking it would be better to break out these two proposals into separate issues. I've created a new issue for the interceptor proposal after iterating a few times over it and implementing a proof of concept: #3614