feat(rest-middleware): Add Middleware handler capability for individual routes and globally (Router)

Issue number: #953 #1955 #1696
Summary
Adds middleware support to event_handlers for REST API Gateway resolvers (all types).  First pass POC implementation following a similar implementation style to Chalice and Starlette

need to add documentation
functional tests in place
Reference middleware implementations provided for CORSMiddleware, CacheControlMiddleware as an alternative to existing ResponseBuilder implementation - existing ResponseBuilder implementation unchanged

Changes

BaseRouter route functions have additional middleware parameter expecting a list of Callable middleware handlers
Added special case registered_api_middleware, which is added as the last middleware handler internally (maintain backward compatibility) - without this function, the signature for REST routes would change to match the middleware signature, which feels like a breaking change.  This special middleware also ensures all middleware will get a Response object as the response, which is essential for being able to write middleware (no type guessing).
Added example Core middleware as Class based middleware - CORSMiddleware, CacheControlMiddleware
Middleware can be any Python Callable
Capability for adding global middleware (all routes) and route-specific middleware.  Global middleware is always executed first since it represents cross-cutting concerns; anything else should be route specific.
No ordering gymnastics - Middleware is applied in Order of definition, router-middleware + route-middleware

User experience
Users can now add middleware either for a specific route or centrally for ALL routes:
import requests
from requests import Response

from aws_lambda_powertools.event_handler import ALBResolver, ApiGatewayResolver
from aws_lambda_powertools.utilities.typing import LambdaContext

app = ALBResolver()

def my_custom_route_middleware(app: ApiGatewayResolver, get_response: Callable, **kwargs):
    # Before handler Code goes here

    # Call next handler in chain
    result = get_response(app, injected_param="my-custom-value", **kwargs)

    # After handler code goes here

    # Return response here
    return result

def my_custom_router_middleware(app: ApiGatewayResolver, get_response: Callable, **kwargs):
    # Before handler Code goes here

    # Call next handler in chain
    result = get_response(app, router_param="my-router-param", **kwargs)

    # After handler code goes here

    # Return response here
    return result

# Add Middleware to ALL routes
app.use([my_custom_router_middleware])

# define a route and add specific middleware
@app.post("/todo", middleware=[my_custom_route_middleware])
def create_todo(router_param: str, injected_param: str):
    data: dict = app.current_event.json_body
    todo: Response = requests.post("https://jsonplaceholder.typicode.com/todos", data=data)

    # Returns the created todo object, with a HTTP 201 Created status
    return {"todo": todo.json()}, 201

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

Technical Debt
The following elements are listed as required technical debt to avoid breaking changes for existing REST API resolver REST users.

registered_api_adapter - this middleware adapter will ensure the existing API route functions will be called in the same way they are defined today, and it MUST be the last middleware handler in the Middleware call stack - LOTS of comments and context around where this happens when building the middleware call stack.
To avoid circular module references (because api_gateway.py module is full of self-references) unable to add proper Type Hinting to registered_api_adapter in aws_lambda_powertools/event_handler/middlewares/registered_api.py, but also wanted to place the middleware handler in the middleware folder where it belongs - could relocate this into api_gateway.py module (since is internal to powertools).
Return type of MiddlewareStackWrapper.__call__ is Union[Dict, Tuple, Response].  Ideally this would be consistent as Response - but not possible since API routes can return one of those 3 types - unsure if that could ever change but feel worth calling it out.

Checklist
If your change doesn't seem to apply, please leave them unchecked.

[x] Meet tenets criteria
[x] I have performed a self-review of this change
[x] Changes have been tested
[ ] Changes are documented
[ ] PR title follows conventional commit semantics

Is this a breaking change?
No, not a breaking change (I hope)
Checklist:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

Question:  I am thinking of relocating the internal registered_api_adapter into the api_gateway.py module and enforce typing of middleware to be inherited from the BaseMiddlewareHandler - I feel enforcing class usage for middleware allows for easier construction and configuration of middleware handlers.  Much simpler to construct and reason about IMO.

go for it!

@heitorlessa  have improved the BaseMiddlewareHandler class and implemented SchemaValidationMiddleware (based on the existing Powertools validate function).   Enforcing Class based middleware is not possible due to the cyclic nature of api_gateway.py and middleware dependencies - Leaving it as "Callable" will suit a wider audience of devs and is required for the registered_api_adapter to work.
Let me know of any updates required - The code is done and now the hard part - documentation

Thanks a lot!! I'll look into the changes tomorrow :)

Codecov Report
Patch coverage: 93.58% and project coverage change: -0.13% :warning:

Comparison is base (663c140) 96.41% compared to head (c8182e7) 96.29%.
Report is 1 commits behind head on develop.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2917      +/-   ##
===========================================
- Coverage    96.41%   96.29%   -0.13%
===========================================
  Files          169      172       +3
  Lines         7678     7741      +63
  Branches      1451      561     -890
===========================================
+ Hits          7403     7454      +51
- Misses         219      225       +6
- Partials        56       62       +6

Files Changed
Coverage Δ

aws_lambda_powertools/event_handler/api_gateway.py
98.61% <88.37%> (-1.39%)
:arrow_down:

...a_powertools/event_handler/middlewares/__init__.py
100.00% <100.00%> (ø)

...ambda_powertools/event_handler/middlewares/base.py
100.00% <100.00%> (ø)

...ols/event_handler/middlewares/schema_validation.py
100.00% <100.00%> (ø)

... and 5 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

thanks a lot for those changes! Managed to spend more quality time to debug and understand this now - made some initial asks to start to be ready for prod.

Let's tick off some of these items :-).

Adding tasks so we get this to the finish line - we can divide and conquer (let me know the ones you want to pick up)

[x]  Change docstrings to Numpy (our style)
[x]  Add logger.debug statements in key areas to improve observability
[x]  Include current middleware name and next in line in the call chain (suggested)
[x]  Include debugging for anyone printing get_response to find out the call chain
[x]  Address comments and/or suggestions
[x]  Add missing tests for middlewares with Router
[x]  Add missing tests for middlewares short circuiting via HTTP Errors
[x]  Improve early return test by having a third middleware to ensure it doesn't get called
[x]  Make attributes private unless we need to expose them (if so, docstrings)
[x]  Update docstrings with examples where relevant
[x]  Expand docstring for registered_api_adapter to provide an example on breaking change (hard to get full picture if glancing through without debugging)
[ ]  Update docstring on how middlewares stack is built with an ascii diagram (who takes precedence, etc. -- monodraw might be the best tool)
[ ]  Consider enriching exceptions with registered middlewares to ease debugging (which middlewares were registered? what was the call chain?) -- best effort
[ ]  Create diagram for happy path - Middleware at route level and router level (use())
[ ]  Create diagram for happy path - Middleware defined in both main and split routes
[ ]  Create diagram for early return - short circuiting
[ ]  Document patterns for authoring middlewares - being a good citizen
[ ]  Document anti-patterns (e.g., avoid discarding responses in first middleware unless intentional)
[ ]  Document design in alignment with Chalice

@heitorlessa lets discuss enriching exceptions techniques to assist with best-effort debugging
Started on some diagrams for the documentation side of things.

@heitorlessa last round of commits added:

additional debug in middleware code
array tracking frame processing
Additional code to support separate Router instances (via include_router)
Added source draw.io document (advise if source is okay) for current WIP diagrams for middleware docs
improved testing - Router, additional middleware in early-return tests, test for HTTPError return (exception handling)

Still to do:

[ ] Update docstring on how middlewares stack is built with an ascii diagram (who takes precedence, etc. -- monodraw might be the best tool)

[ ] Consider enriching exceptions with registered middlewares to ease debugging (which middlewares were registered? what was the call chain?) -- best effort

[ ] Create diagram for happy path - Middleware at route level and router level (use()) (@walmsles - WIP)

[ ] Create diagram for happy path - Middleware defined in both main and split routes (@walmsles - WIP)

[ ] Create diagram for early return - short circuiting (@walmsles - WIP)

[ ] Document patterns for authoring middlewares - being a good citizen

[ ] Document anti-patterns (e.g., avoid discarding responses in first middleware unless intentional)

[ ] Document design in alignment with Chalice

Unsure how to tackle docstring ascii diagram that makes sense in low-fi like that - happy for ideas

Need help with exception enrichment idea - happy for inputs here.

I feel a thorough review of the functionality is in order and feedback - need help with debug places - I added some, but there may be more I am missing.

I need input on Doc Strings for DX - this needs more eyes.

I plan to put together a starting point on documentation this week.

Quick notes through peer reviewing:

[ ] Correct _executed_middleware as it's not including the following middleware that might have returned early/raised exception
[ ] We no longer need the try/catch within middleware stack - bubble that up since we're catching all of it within API Processor
[ ] Let's stick with the alternative image in drawio for Middlewares
[ ] We need to add labels/legend to the boxes: Request, Response, Uncalled, Exception
[ ] Investigate how to provide the middleware stack info during an exception

Quick notes through peer reviewing:

[x]  Correct _executed_middleware as it's not including the following middleware that might have returned early/raised exception
Renamed to _push_middleware_stack_frame, related to MiddlewareFrame.__call__() method (where it needs to be).

[x]  We no longer need the try/catch within middleware stack - bubble that up since we're catching all of it within API Processor
Removed - less code is better code!

[ ]  Let's stick with the alternative image in drawio for Middlewares
[ ]  We need to add labels/legend to the boxes: Request, Response, Uncalled, Exception
[ ]  Investigate how to provide the middleware stack info during an exception

I'm planning to use this middleware to build automatic Pydantic validation based on route and the function params. In order for this to work, I need to have access to the Route that was matched during my middleware. Do you know if there's a way to do it the way this is implemented, or can we consider explosing that to the Middlewares?

I'm planning to use this middleware to build automatic Pydantic validation based on route and the function params. In order for this to work, I need to have access to the Route that was matched during my middleware. Do you know if there's a way to do it the way this is implemented, or can we consider explosing that to the Middlewares?

Hi @rubenfonseca - the only state passed between middleware is the Resolver (Router) object itself so middleware has access to the "app" instance created by the developer.  The actual "Route" that is being resolved is not currently referenced by the Resolver (which does seem unusual).
I feel the best way to achieve this would be to add a reference to the resolved Route object to the additional context - which is already managed between invocations and feels like the right place for this type of reference since it is Context related.
Have pushed a change adding "route" to the additional context - let me know your thoughts.

Looking at the docs <3

I'm planning to use this middleware to build automatic Pydantic validation based on route and the function params. In order for this to work, I need to have access to the Route that was matched during my middleware. Do you know if there's a way to do it the way this is implemented, or can we consider explosing that to the Middlewares?

Have pushed a change adding "powertools_route" to the additional context - let me know your thoughts.

@heitorlessa  and @rubenfonseca - I do not like the addition above.  Middleware should have the "Router" instance injected as one of the parameters instead, which is cleaner and more framework-centric.
Thinking something like this (stealing an example from the docs above):

from typing import Any, Callable

from aws_lambda_powertools import Logger
from aws_lambda_powertools.event_handler import Response, APIGatewayRestResolver
from aws_lambda_powertools.event_handler.api_gateway import BaseRouter

import requests

app = APIGatewayRestResolver()
logger = Logger()

# need a better first example, just to show completeness
def log_incoming_request(app: BaseRouter, route: Route, get_response: Callable[..., Any], **context_args) -> Response:
    # Do Before processing here
    logger.info("Received request...", path=app.current_event.path)  # (1)!

    # Get Next response
    result = get_response(app, **context_args)  # (2)

    # Do After processing here

    # return the response
    return result  # (3)

@app.get("/todos", middlewares=[log_incoming_request])
def get_todos():
    todos: Response = requests.get("https://jsonplaceholder.typicode.com/todos")
    todos.raise_for_status()

    # for brevity, we'll limit to the first 10 only
    return {"todos": todos.json()[:10]}

This is possible as the Route object when it is called for the first time, constructs the Middleware stack - each Middleware function handler or class instance is wrapped in the MiddlewareFrame, so the Route can be referenced by the MiddlewareFrame and injected through the python __call__ class function which calls the developer provided handler and can inject the Route reference at that point.
Reasons I Prefer this:

No "Magic" context var
Makes Route accessible in an intentional way
Developer-constructed middleware handlers do not need to pass this on to the next middleware
MidddlewareFrame wraps developer middleware handlers - so it will be completely transparent to users

This feels like a more natural and intuitive solution than the "hack" I added earlier (adding to context vars a magic value).
Let me know your thoughts - I have partially prototyped locally, but I am interested in your thoughts.

I'm planning to use this middleware to build automatic Pydantic validation based on route and the function params. In order for this to work, I need to have access to the Route that was matched during my middleware. Do you know if there's a way to do it the way this is implemented, or can we consider exposing that to the Middlewares?

Question time I think and maybe we need to do this as a separate issue to make sure we get this correct.  Exposing the Route class is not ideal IMO - it has been built as an internal object and doesn't contain a lot that is useful.
@rubenfonseca what do you need access to in the Route instance for your Pydantic use-case?

@walmsles I need access to the handler because I need to inspect it's signature and create automatic validators

Quick updates from docs (haven't seen previous comments so I can focus on docs).

[x] Start with an intro diagram
[x] Explain middleware function signature upfront
[x] Provide an initial middleware that can evolve with the examples (progressive)
[x] Add an output so customers can understand the middleware effect
[x] Create a global middleware section (introduce global middleware after route is explained)
[x] Create a returning early section (introduce short-circuiting now that combined middlewares are explained)
[ ] Create an exception handling section (introduce handled and unhandled exceptions)
[ ] Export diagrams as svg to improve quality and responsiveness
[ ] Move extra info under Staging area

[ ] Move Router middleware (this should be under Router subsection)

[ ] Create common practices/being a good citizen

@walmsles I need access to the handler because I need to inspect it's signature and create automatic validators

In that case adding to the actual resolver object will be cleaner and have lowest footprint.  I dislike the context args implementation it's a little "magical".  And means not an arg to the middleware which was confusing and in 90% of cases will never be referenced (linting issues).

To be sure I understood this thread, are you not inclined to have a typed key in the shared context dictionary?
matched_route: Route = app.context["_route"]  # customers use app.append_context(my_key=value) today

app.context exists today for sharing contextual data between routes, App and Router instances - it's underutilized by us today. It can be easily typed (TypedDict) to autocomplete keys we might need to add in the future, for example _route_handler now. It has the added benefit of API auto-docs generation - Logger example.

This was my first inclination - I was just terrible at naming it (naming is hard).  Also wanted to explore pro's/con's of other options - to make sure headed in right direction.  I like this option - if can be typed for completion even better - was not aware of this.

but there's another middleware context (route arguments + kwargs actually) which I think it's what both of you are referring to. If there was a way to not require this today in the function signature it could also remove this confusion (haven't thought much about it, since it'd have dynamic route arguments expanded).

Adding to these would possibly require the route instance to be added to function handlers or have typing errors.
Will go ahead with using the additional context (original thought with better naming) and make sure to add typing which resolves the main thing I was thinking of not liking.

Have made use of the Resolver.context adding in missing context that is not available anywhere but might be interesting to advanced middleware developers:
_path - that actual API path matched
_route - the internal Route instance matched and processed
Middleware Stack now also wraps to internal private var Route._call_stack, which is what is invoked so the Route.func remains as defined by developer code (was previously becoming the wrapped function which I feel is bad form).

Having some internal distractions and am unable to focus on this properly, I'll resume working on the docs this weekend

Having some internal distractions and am unable to focus on this properly, I'll resume working on the docs this weekend

👌.  I have updated images from draw.io for consistency in colours and made light/dark versions in media.  Added to docs where already used and also in staging area to save time

Having some internal distractions and am unable to focus on this properly, I'll resume working on the docs this weekend

👌. I have updated images from draw.io for consistency in colours and made light/dark versions in media. Added to docs where already used and also in staging area to save time

such a great attention to details ;) Thank you!!
I've just fixed a tiny issue with image's legend after that change, and just wrote the exception handling part using the newer images.

Made great progress in the docs, 70% done now. Just need to write about A) a middleware that accepts config (class-based only now AFAIK), B) how to use middlewares in Router.
I've got an ask.
Could you think of a way to not require **kwargs the way it is now as part of get_response(app, **kwargs) signature?
Why
As I was writing the Being a good citizen section and evolving code snippets, I've noticed the **kwargs is nothing more than a backward compatibility thing, so that the route handler is called with the dynamic path parameters expanded.
It'll improve readability and UX massively, and prevent common mistakes as I bumped into when writing example middlewares (forgot the **kwargs in the signature, then forgot to pass in to get_response).
My next tasks:

[ ] Add a section about setting up middlewares config (use SchemaValidationMiddleware example)
[ ] Add section for built-in middlewares
[ ] Allow use of a single middleware without passing a list app.use(my_middleware) to avoid UX surprises

After docs is done:

[ ] Review doc strings for typos, formatting, consistency, and whether it has any example (e.g., SchemaValidationMiddleware doesn't have an usage example)
[ ] Review whether we need to provide a sample on how to unit test (atm I think not)
[ ] Consider whether we stick with get_response to ease Chalice customers, or call_next like FastAPI. I'm indifferent, call_next better aligns with the diagrams created so far, and we use function arg so customers can change the name regarrdless.

Made great progress in the docs, 70% done now. Just need to write about A) a middleware that accepts config (class-based only now AFAIK), B) how to use middlewares in Router.
🤩

I've got an ask.
Could you think of a way to not require **kwargs the way it is now as part of get_response(app, **kwargs) signature?
Why
As I was writing the Being a good citizen section and evolving code snippets, I've noticed the **kwargs is nothing more than a backward compatibility thing, so that the route handler is called with the dynamic path parameters expanded.
It'll improve readability and UX massively, and prevent common mistakes as I bumped into when writing example middlewares (forgot the **kwargs in the signature, then forgot to pass in to get_response).

I am also not a fan of **kwargs - it's one thing I dislike about Python.  After our last discussion and under-utilisation of the additional context, I think adding a new internal context key of _route_args is workable.  The internal _registered_api_adapter middleware can take the _route_args and expand them into the Route handler call instead of **kwargs as it is now.  This means the route_args will never get lost and remain under Powertools library control.
We should expand the docs on this context and encourage this as the way forward for sharing data across Middlewares and the API route for any custom data to be passed to the API route.
Let me know your thoughts.

This means the route_args will never get lost and remain under Powertools library control.

YES! exactly what I was thinking - it also allows for a future Request object if we need to.
Let's go with the context - could you tackle that while I start documenting a new section using Class based middlewares, pretty please?
I'll also add a new test for class based middleware too.

This means the route_args will never get lost and remain under Powertools library control.

YES! exactly what I was thinking - it also allows for a future Request object if we need to.
Let's go with the context - could you tackle that while I start documenting a new section using Class based middlewares, pretty please?
I'll also add a new test for class based middleware too.

On it!  Am also renaming get_response to next_midleware I feel it will matchup everywhere better and we can worry about Chalice compatibility later

Awesome, I've got some errands this morning but should have the new section for extending middlewares with a class + unit tests.
Then all that is left docs wise is to merge the Router piece, and a thorough review before we merge and release this week ;)

Awesome, I've got some errands this morning but should have the new section for extending middlewares with a class + unit tests.
Then all that is left docs wise is to merge the Router piece, and a thorough review before we merge and release this week ;)

I have reworked middleware to user the resolver.context and renamed it to next_middleware universally.  Aligned typing across all modules and adjusted the "debug" failing test since there have been additions.
Let me know if there are other areas to assist I can find some time during my day tomorrow.

I can take care of the last bits now, the only thing I'd ask is to run the docs locally and confirm whether you're broadly in agreement. I've tried to avoid information overload as much as possible, so it doesn't become a page of Middleware by accident (one day we'll have a Powertools Hub where this can be more appropriate).
I'm gonna start working on class based middleware, built-in middlewares, and then the router. Hope I'm done with that by EOD.

@walmsles I think we have a bug but want to confirm in case I misunderstood the logic.
What's the expected processing order of middlewares when both Router and App have global middlewares?
I expected Router to be first, because App will depend on Router, hence Router logic will run first during import.

def test_api_gateway_middleware_order_with_both_app_router(app: EventHandlerInstance, event):
    # GIVEN two global middlewares: one for App and one for Router
    router = Router()

    def global_app_middleware(app: EventHandlerInstance, next_middleware):
        middleware_order: List[str] = app.context.get("middleware_order", [])
        middleware_order.append("app")
        return next_middleware(app)

    def global_router_middleware(app: EventHandlerInstance, next_middleware):
        app.append_context(middleware_order=["router"])
        return next_middleware(app)

    @router.get("/my/path")
    def dummy_route():
        middleware_order = app.context["middleware_order"]

        assert middleware_order[0] == "router"
        assert middleware_order[1] == "app"

        return Response(200, content_types.TEXT_PLAIN, "works!")

    # WHEN they are registered accordingly
    # and Router instance is included in App instance as expected

    # mimics import since Router logic will run first
    router.use(middlewares=[global_router_middleware])
    app.use(middlewares=[global_app_middleware])
    app.include_router(router)

    # THEN resolving a request should start processing global Router middlewares first
    # due to insertion order
    result = app(event, {})

    assert result["statusCode"] == 200

@walmsles I think we have a bug but want to confirm in case I misunderstood the logic.
What's the expected processing order of middlewares when both Router and App have global middlewares?
I expected Router to be first, because App will depend on Router, hence Router logic will run first during import. If so, we need to switch this part. I'm happy to do it, just need your confirmation to be sure that's expected or not expected behaviour.

Curious - the current implementation was my intent - unsure if I understood routers correctly so let's ensure we get this right.  I understood separate Router instances to be additive to the Main Resolver through the include_router mechanism.  To me, it made sense for the parent App Resolver global middlewares to come first, followed by each additional included router since they are "included".  I guess before with just routes order did not really matter.
My mental model is:
App() --> use Middleware() --> add routes()
Router() --> use Middleware() --> add routes()
App() --> include Router() (additive to step 1)
With the result: App-Global -> Router-Global --> Route Middleware() (route can only be in one place)

I can take care of the last bits now, the only thing I'd ask is to run the docs locally and confirm whether you're broadly in agreement. I've tried to avoid information overload as much as possible, so it doesn't become a page of Middleware by accident (one day we'll have a Powertools Hub where this can be more appropriate).
I'm gonna start working on class based middleware, built-in middlewares, and then the router. Hope I'm done with that by EOD.

Docs are 🥰.
I made 2 minor corrections of get_repsonse to next_middleware and removed reference to passing **context/kwargs.
Nice balance - I really like the docs flow - and it is just enough without, as you say, being too heavy.

@walmsles I think we have a bug but want to confirm in case I misunderstood the logic.
What's the expected processing order of middlewares when both Router and App have global middlewares?
I expected Router to be first, because App will depend on Router, hence Router logic will run first during import. If so, we need to switch this part. I'm happy to do it, just need your confirmation to be sure that's expected or not expected behaviour.

Curious - the current implementation was my intent - unsure if I understood routers correctly so let's ensure we get this right. I understood separate Router instances to be additive to the Main Resolver through the include_router mechanism. To me, it made sense for the parent App Resolver global middlewares to come first, followed by each additional included router since they are "included". I guess before with just routes order did not really matter.
My mental model is:
App() --> use Middleware() --> add routes() Router() --> use Middleware() --> add routes()
App() --> include Router() (additive to step 1)
With the result: App-Global -> Router-Global --> Route Middleware() (route can only be in one place)

That makes completely sense. In hindsight, I mixed up the issue customers had with having similar routes in App + Router, whereas .include_router always comes first, so those routes are registered first because of global scope execution order.
In this case, if a customer wants to have Router global middlewares to run execute first, they'd simply need to swap the order of .include and .use within an App instance.
I'll cover that edge case in the docs ;)

I made 2 minor corrections of get_repsonse to next_middleware and removed reference to passing **context/kwargs.

Thank you!!
I haven't worked on the Router section yet as I bumped into that edge case.

For the Native middlewares section, I'm leveraging the API docs. I've pushed changes to the Schema Validation middleware, and removed Schema Validation middleware from the __init__.py barrel export <- this would've been a breaking change since it'd require fastjsonschema validation library.
How the API docs looks like now for Schema Validation middleware

Two tests for the middleware order to ensure we don't introduce breaking change. I'll call out that in the docs now so a customer can decide whether to use the default or reverse order.
Appending to context logic is the same to ensure we are only testing the order in which .include_router is called.

@pytest.mark.parametrize(
    "app, event",
    [
        (ApiGatewayResolver(proxy_type=ProxyEventType.APIGatewayProxyEvent), API_REST_EVENT),
        (APIGatewayRestResolver(), API_REST_EVENT),
        (APIGatewayHttpResolver(), API_RESTV2_EVENT),
    ],
)
def test_api_gateway_middleware_order_with_include_router_last(app: EventHandlerInstance, event):
    # GIVEN two global middlewares: one for App and one for Router
    router = Router()

    def global_app_middleware(app: EventHandlerInstance, next_middleware: NextMiddleware):
        middleware_order: List[str] = router.context.get("middleware_order", [])
        middleware_order.append("app")

        app.append_context(middleware_order=middleware_order)
        return next_middleware(app)

    def global_router_middleware(router: EventHandlerInstance, next_middleware: NextMiddleware):
        middleware_order: List[str] = router.context.get("middleware_order", [])
        middleware_order.append("router")

        router.append_context(middleware_order=middleware_order)
        return next_middleware(app)

    @router.get("/my/path")
    def dummy_route():
        middleware_order = app.context["middleware_order"]

        assert middleware_order[0] == "app"
        assert middleware_order[1] == "router"

        return Response(status_code=200, body="works!")

    # WHEN App global middlewares are registered first
    # followed by include_router

    router.use([global_router_middleware])  # mimics App importing Router
    app.include_router(router)
    app.use([global_app_middleware])

    # THEN resolving a request should start processing global Router middlewares first
    # due to insertion order
    result = app(event, {})

    assert result["statusCode"] == 200

@pytest.mark.parametrize(
    "app, event",
    [
        (ApiGatewayResolver(proxy_type=ProxyEventType.APIGatewayProxyEvent), API_REST_EVENT),
        (APIGatewayRestResolver(), API_REST_EVENT),
        (APIGatewayHttpResolver(), API_RESTV2_EVENT),
    ],
)
def test_api_gateway_middleware_order_with_include_router_first(app: EventHandlerInstance, event):
    # GIVEN two global middlewares: one for App and one for Router
    router = Router()

    def global_app_middleware(app: EventHandlerInstance, next_middleware: NextMiddleware):
        middleware_order: List[str] = router.context.get("middleware_order", [])
        middleware_order.append("app")

        app.append_context(middleware_order=middleware_order)
        return next_middleware(app)

    def global_router_middleware(router: EventHandlerInstance, next_middleware: NextMiddleware):
        middleware_order: List[str] = router.context.get("middleware_order", [])
        middleware_order.append("router")

        router.append_context(middleware_order=middleware_order)
        return next_middleware(app)

    @router.get("/my/path")
    def dummy_route():
        middleware_order = app.context["middleware_order"]

        assert middleware_order[0] == "router"
        assert middleware_order[1] == "app"

        return Response(status_code=200, body="works!")

    # WHEN App include router middlewares first
    # followed by App global middlewares registration

    router.use([global_router_middleware])  # mimics App importing Router
    app.include_router(router)

    app.use([global_app_middleware])

    # THEN resolving a request should start processing global Router middlewares first
    # due to insertion order
    result = app(event, {})

    assert result["statusCode"] == 200

Two tests for the middleware order to ensure we don't introduce breaking change. I'll call out that in the docs now so a customer can decide whether to use the default or reverse order.

👌

Appending to context logic is the same to ensure we are only testing the order in which .include_router is called.

It is important to make it clear that the merging of global middleware happens when include_router is called.
For clarity, it may be easier to:

Store router middlewares during include_router in a separate included_router_middleware list internally
Actual merging then happens during _build_middleware_stack (which occurs just prior to processing the route for the first time).
Merge order would then be set:  Parent Router Globals + Included Router Globals (in order of include) + Route specific middleware

Is the above simpler to reason about? The current implementation does the merging when include_router is called, so if you have:

    router.use([global_router_middleware])  # mimics App importing Router

    app.use([global_app_middleware])
    app.include_router(router)
    app.use([global_app_middleware_two])

You end up with the middleware order: global_app_middleware, global_router_middleware, global_app_middleware_two
Basically - in the order they are used or included on the app instance.
The alternative I outlined above would result in the following order: global_app_middleware, global_app_middleware_two, global_router_middleware
It is simpler to reason about but also out of order regarding the process of constructing the app routes - there are definite pros/cons for each.

Merging after CI checks complete 🎉 Finally over @walmsles -- THANK YOU so much for all the time you've put into this to make it a reality.
I've cleaned up the last bits, added mentions of Middleware for Router, and am leaving this to another PR so I can get started on re:Invent stuff.

[ ] allow .use() to accept single middleware or a list of middleware
[ ] create a TypedDict for internal info we're adding to make it easier to access it (and document)


add constraints support

Current coverage is 42.36% (diff: 75.45%)

Merging #69 into master will decrease coverage by 0.25%

@@             master        #69   diff @@
==========================================
  Files            33         33
  Lines         10197      10262    +65
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           4345       4347     +2
- Misses         4835       4895    +60
- Partials       1017       1020     +3

Powered by Codecov. Last update fa41bf1...1105b7a

LGTM

+1


how to support tsrange[]

Do we support field parse function?

Can u please provide more detail please.

in postgres, how did this orm map range type to golang?
I didn't found any document related to this.

it is possible to specify column type in GORM.
`type Duration struct {
From time.Time
To   time.Time
}
func (d *Duration) Scan(src interface{}) error {
panic("implement me")
}
func (d Duration) Value() (driver.Value, error) {
return fmt.Sprintf(
"[%s, %s)",
d.From.Format("2006-01-02 15:04"),
d.To.Format("2006-01-02 15:04"),
), nil
}
type Booking struct {
ID        int       gorm:"primary_key"
Duration  Duration  gorm:"type:tsrange;column:duration;not null"
}
`

@rustjason fixed?


Bump major to 3.20

Needed to create stable3.19, out of which Minecraft will ship

Let's actually wait until tomorrow if only Minecraft is releasing


feat: release for version 2.5.0 of the spec

Refs https://github.com/asyncapi/spec/issues/830
Features

Server Variables can be now referenced from inside the Server Object (#776) (5deef1e), closes #775 by @char0n
Decorating Server Object with tags (#809) (b0c2f9c), closes #654 by @smoya
Added new Google Cloud Pub/Sub Bindings (#836) (babc3be) (binding definition) by @whitlockjc
Other enhancements

Clarified ChannelItem.$ref resolution rules (#779) (9a293c7), closes #612 by @char0n
Clarification around URL resolution rules (#782) (fc324ef), closes #674 by @char0n
Align wording with RFC 2119 (#818) (5580a9c) by @char0n

Fixes

Added missing IBM MQ Operation Binding (#840) (68d85d1) (binding definition), refs #605 by @magicmatatjahu

Resolves https://github.com/asyncapi/spec/issues/775
Resolves https://github.com/asyncapi/spec/issues/465
Resolves https://github.com/asyncapi/spec/issues/654

Here are the release notes for the 2.5.0 version of the spec that should be used for the GitHub Release.
Release notes article -> https://www.asyncapi.com/blog/release-notes-2.5.0

Features
---
- Server Variables can be now referenced from inside the Server Object (#776) ([5deef1e](https://github.com/asyncapi/spec/commit/5deef1e9148947689d53db3a1d013d49cd9df7d0)), closes #775 by @char0n
- Decorating Server Object with tags (#809) ([b0c2f9c](https://github.com/asyncapi/spec/commit/b0c2f9c086d940e8016ec1fa60107fe74cc070db)), closes #654 by @smoya
- Added new Google Cloud Pub/Sub Bindings (#836) ([babc3be](https://github.com/asyncapi/spec/commit/babc3bedb5d85375a305c7268e96257113c1933e)) ([binding definition](https://github.com/asyncapi/bindings/tree/master/googlepubsub)) by @whitlockjc
- Other enhancements
  - Clarified ChannelItem.$ref resolution rules (#779) ([9a293c7](https://github.com/asyncapi/spec/commit/9a293c77f6bf74a056a8845dabfbe20e73a6f320)), closes #612 by @char0n
  - Clarification around URL resolution rules (#782) ([fc324ef](https://github.com/asyncapi/spec/commit/fc324ef3aa92d313918a00ea9a5eb6c85720a36d)), closes #674 by @char0n
  - Align wording with [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119) (#818) ([5580a9c](https://github.com/asyncapi/spec/commit/5580a9c5fbef55e925e7514b986c95a0e4c85821)) by @char0n

Fixes
---
- Added missing IBM MQ Operation Binding (#840) ([68d85d1](https://github.com/asyncapi/spec/commit/68d85d19e5be12fa6e13d0fc5851993934dbdcde)) ([binding definition](https://github.com/asyncapi/bindings/tree/master/ibmmq#operation-binding-object)), refs [#605](https://github.com/asyncapi/spec/issues/605) by @magicmatatjahu

Full Changelog: https://github.com/asyncapi/spec/compare/v2.4.0...v2.5.0

/au

@fmvilas @smoya @derberg @delelane this is good to go. According to https://github.com/asyncapi/spec/blob/master/RELEASE_PROCESS.md#step-13---merge-the-release-branches, one of the code owner should merge this PR. Can I ask one of you to merge this PR?
Updated release notes to be used for GitHub release are in: https://github.com/asyncapi/spec/pull/842#issuecomment-1260785616
Thank you!

https://github.com/asyncapi/spec/releases/tag/v2.5.0 is out

Recent comments about the release from the bot were added by mistake. More details in https://github.com/asyncapi/spec/issues/899


Fix/shared lib load

Resolve https://github.com/abdeladim-s/pywhispercpp/issues/43

Thanks a lot for the time you put into this :)

Thanks for the PR, Looks great! One last thing before merging, can you build a wheel and check if the lib folder is included as well ?

$ ls
pywhispercpp-1.2.0-cp312-cp312-win_amd64  pywhispercpp-1.2.0-cp312-cp312-win_amd64.whl

$ ls pywhispercpp-1.2.0-cp312-cp312-win_amd64/pywhispercpp/lib/
_pywhispercpp.cp312-win_amd64.pyd  ggml.dll  whisper.dll

Thanks for the PR, Looks great! One last thing before merging, can you build a wheel and check if the lib folder is included as well ?

$ ls
pywhispercpp-1.2.0-cp312-cp312-win_amd64  pywhispercpp-1.2.0-cp312-cp312-win_amd64.whl

$ ls pywhispercpp-1.2.0-cp312-cp312-win_amd64/pywhispercpp/lib/
_pywhispercpp.cp312-win_amd64.pyd  ggml.dll  whisper.dll

Looks perfect.
Thanks!


Support Python UDF in SparkProcessor

What is the purpose of the change
This pull request supports using Python UDF in SparkProcessor.
Brief change log

Adds support for Python UDF when building DerivedFeatureView in SparkProcessor.
Add integration tests for the functionality of Python UDF in SparkProcessor.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Thanks for the PR. LGTM. Can you help rebase the PR?


URL.createObjectURL(stream) does nothing

Tried to ask this on forum: https://groups.google.com/forum/#!topic/cordova-plugin-iosrtc/-hSUCWPxa3Q but no response. I have a clean build (no errors) on cordova 6.4, cordova-ios 4.3, with cordova-plugin-iosrtc version 3.1.0 running on XCode 8.1.
Building for an iPad on iOS 10.1.1, permission is requested for my camera access using the permission descriptors as listed in the build.
I get the success callback for the stream, I can see this is a blob object and that there is a video and audio track.
When I attach it to the video tag in my page (coded in not dynamically added) nothing happens. I've tried listening the onloadedmetadata and oncanplay events and they aren't triggered. The video play button is disabled and nothing loads.
Any thoughts?

Do you see anything in the log files? Can you paste the reduced code you are using?

There's no errors other than warnings for threads. Attached are the index.js, index.html, and the xcode console log.

xcode-console.txt
index.js.txt
index.html.txt

Hum, I just ran into this, but I'm not sure it it has to do with me using webrtc-adapter... I'll debug it further soon.

Is your HTML element set to autoplay? I just fixed a related bug when using promises (which you don't seem to be using anyway).  Tentatively closing since 3.2.1 fixed it here.

Good catch at 9ada6b4262708dbb5870759d4592fb37e65b8d8e.

Still not working. I've removed ios platform removed cordova-plugin-iosrtc and re-added directly from master branch of GIT, confirming 3.2.1 version info. Same log output, same result. I see the blob and the attached video and audio tracks, but can't attach the stream to local video element.

Hum, that's weird.  How about an audio tag? Can you hear yourself over it? Do you have autoplay on the element? Did your application request access to the camera?

I tried with the autoplay included and not. I did get the request for the camera. I haven't tried just audio, but see that the stream object contains an audio track. I aslo confirmed that the video element is seen in the DOM by checking the value of the controls attribute and outputting it to the console. I have tried to set the URL object via window.URL = window.URL || window.webkitURL;
I'm not sure if there's a way to trap the error as the console in xCode isn't reporting anything wrong. The sequence of the code has the stream being attached via URL.createObjectURL(stream) before a notification is issued to indicate the stream has been received, so it isn't causing a break point in execution.
Are there additional dependencies for cordova plugins like device or camera that I need to include? Right now the only plugins that I have are whitelist, console, dialogs, and iosrtc.
What deployment target do you recommend? I've left the swift compatibility to 2.3 and am targeting 9.3 for deployment to my iPad iOS 10.11.1
I am using a Personal team for signing using an XCode managed profile and an older iPhone developer signing certificate. I'm waiting to upgrade the developer account for a business we are getting finalized with the D.U.N.S. registration.

Let me know if you'd like anything more in terms of code. It's a bare test project at this point.

Can you put the code up somewhere?   I cannot reproduce it locally but maybe I can spot something.

Sending the project via DropBox to your gmail account...

@yogaboy Received! I'll take a look as soon as I can.

Thank you and let me know if there are any other logs you might want.

Found it - was missing blob: in Content-Security-Policy, which now reads:
<meta http-equiv="Content-Security-Policy" content="default-src 'self' data: blob: gap: https://ssl.gstatic.com; style-src 'self' 'unsafe-inline'; media-src *">

Great! Sorry I couldn't come back to you earlier. I'm spending some time with my family this week.

Glad to hear the priority is family. Thanks for the offer of support.


Unreponsive monitor after first login bluefin 39

Issue
Monitor is unresponsive after logging on. A hard reset (unplug power) brings me back to the desktop.
Machine
Minisform HM80
AMD Ryzen 7 4800U with inbuilt Radeon Graphics (Renoir)
base
ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:39
Troubleshooting
The following was tried and was unsuccessful.

Tried different DisplayPort cables and ports
Tried different HDMI cables and ports
Disabled all extensions
Configured mutter with the following: gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer']"

Observations

Perviously was on Silverblue 38, ublue 38 and 39 and did not have this issue.
Issues started to occur on fresh installation via ISO and rebase from ublue:39.
Once the monitor has been power cycled on the initial login multiple login/logoff attemps/screen savers/etc do not exhibit this issue.

This works with all extensions enabled.

On reboot and opening another shell (Ctl+Alt+F5) and running startx after logging in does not exhibit this issue.
Issue appears to be isolated with the initial login process from a restart and in bluefin. Others have observed the same behavior in Bazzite.

Logs
rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:39
                   Digest: sha256:66226b60b7707098287df645b32218dde6e5962a9173ece9d7640eb7607d2573
                  Version: 39 (2023-11-15T19:01:14Z)
          LayeredPackages: langpacks-en

  ostree-unverified-image:docker://ghcr.io/ublue-os/silverblue-main:latest
                   Digest: sha256:6774ee72dcfa181f7656cd27353611bff50f84e981742fba20d2f431cc1f322d
                  Version: 39 (2023-11-14T15:06:07Z)
          LayeredPackages: langpacks-en

  ostree-unverified-image:docker://ghcr.io/ublue-os/silverblue-main:latest
                   Digest: sha256:72715b94adecba3e376046bf47f4f253998c1ac76dbd3d9c16460a040609e824
                  Version: 39 (2023-11-13T15:07:37Z)
          LayeredPackages: langpacks-en
                   Pinned: yes

rpm-ostree kargs
rd.luks.options=discard rhgb quiet root=UUID=5f0b7466-055f-405a-852c-64d092d89da1 rootflags=subvol=root rw ostree=/ostree/boot.1/default/aeb0521c78312096d7e2e55e731b564f2d16ae03f5261ea1a478651b52fe438f/0 iommu=pt

I have the same issue, which does not seems to appear when I rebase to ublue/silverblue-main.
The issue also appeared when my computer went in a level of seep mode, sadly I don't know which one as I've tried to change theses settings to fix the main issue.
GPU : Radeon™ RX 480

Same issue for me (on bluefin-nvidia:38/39), also not on ublue/silverblue-nvidia:38/39.
Lenovo ThinkPad P15s Gen 2i w/ GPU: Hybrid Intel® Xe Graphics (TGL GT2)/Nvidia T500

Ok let's rule out gnome vrr, can you do just gnome-vrr and disable it, you'll need to log out and back in.

Ok let's rule out gnome vrr, can you do just gnome-vrr and disable it, you'll need to log out and back in.

On my setup this did not solve the issue, neither after logging out nor after rebooting.

Because I have no clue what is the actual thing causing this behavior. It would be the easiest to strip down certain stuff and try if we can pinpoint what is causing this. I don't have the issue myself, so if somebody has the time to test it with me, please let me know in the bluefin discord and refer to this issue. Be ware this could take some time, as we need to rebase and build images back and forth.

I don't know if VRR is the cause for this situation, but for me, it causes frequent freezing on fullscreen apps.
VRR patch is in this state for like a month.

I've also noted the following scenarios where the screen would also lock up.

After monitor goes to sleep after x minutes
Shutting down the machine the monitor goes into that hung state and never enters sleep mode.

For me the issue was only on the bluefin image (38 and 39).
ublue-os/silverblue-main 38 and 39 were working fine.

For me the issue was only on the bluefin image (38 and 39). ublue-os/silverblue-main 38 and 39 were working fine.

I couldn't rebase to bluefin 38 because of some RPM Fusion key issues but the issue did not arive when I rebased to silverblue-main 39, and did not happen in 6 month with silverblue-main 38. Rebasing back to bluefin:latest brought back the issue.

OK let's rule out evdi. Can ya'll try disabling the service: systemctl stop displaylink and then systemctl disable displaylink.
And then sudo rmmod evdi to remove the kernel module.

Not sure if expected, evdi doesn't seems loaded
$ systemctl stop displaylink
$ systemctl disable displaylink
Removed "/etc/systemd/system/multi-user.target.wants/displaylink.service".
$ sudo rmmod evdi
rmmod: ERROR: Module evdi is not currently loaded

Applied, Rebooted and still an issue.
❯ systemctl status displaylink
○ displaylink.service - DisplayLink Manager Service
     Loaded: loaded (/usr/lib/systemd/system/displaylink.service; disabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)

I think I figured out what is causing this. If you want to try and let me know if using the image below fixes your problem
Bluefin image
rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bobslept/bluefin:39

Bluefin-dx image
rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bobslept/bluefin-dx:39

This change does indeed seems to fix the issue.

Confirming the image from @bobslept is working as expected. No screen freezes in any situation now. Can the changes be detailed so it can be merged with bluefin latest?
Also I rebased using bluefin-dx:39

Interesting! Does ddccontrol not have a service unit? I'm trying to figure out if we can ship the binaries for people who need it but not affect others.

Ok I've PRed the removal of these packages, nice teamwork ya'll!
There's an alternative tool in the archive named ddcutil that might cover the usecase, so if someone wants to layer that and see what happens we can always add that later.

Interesting! Does ddccontrol not have a service unit? I'm trying to figure out if we can ship the binaries for people who need it but not affect others.

Crazy right! Let me say it wasn't the first thing I tried to disable... :smile:
Disabling the service file is not really working because it is intended to be trigged by dbus. So you can't really disable it.
❯ sudo systemctl disable ddccontrol.service
The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.

Possible reasons for having this kind of units are:
• A unit may be statically enabled by being symlinked from another unit's
  .wants/ or .requires/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
  a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
  D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
  instance name specified.

@kookie @pyrignis Thanks for reporting, testing and confirming this issue.

Thanks everyone for the hard work getting this resolved. Now just waiting for the PR and a new build so I can rebase.

This change does indeed seems to fix the issue.

hey can u try layering ddcutil package and check if any issues occur ? ddcutil appears to be different compared to ddccontrol and yet it provides brightness control.


Deep Merge With Nested Identical Array Returns False

Please see: https://github.com/rtfeldman/seamless-immutable/issues/94.
I still think this is a valid issue -- I am came across it myself. Could we please look into it?

I still think this is a valid issue

I respectfully disagree. 🙂
From the original:

A .merge with {deep: true} of two identical objects that contain an array is returning a new object.

I expect that changing this behavior would be a serious performance regression. You would have to do comparisons at every single step of the merge, to check whether anything had changed. That is not free, and we should not act as though it were!


⚠ PostgreSQL has degraded performance

In 6275394, PostgreSQL (https://news.ycombinator.com) experienced degraded performance:

HTTP code: 200
Response time: 2290 ms

Resolved: PostgreSQL performance has improved in b38bc99 after 15 minutes.


feat: Added Bus to service.

This PR adds a Bus class to @platformatic/service for services to uses.
By default, @platformatic/runtime uses a Bus to broadcast services changes.

lockfile is broken

Should be fine now.


standardize consts

Please check below, if the PR fulfills these requirements:

[X] Commit(s) and code follow the repositories guidelines.
[X] Test(s) have been added or updated to support these change(s).
[X] Doc(s) have been added or updated to support these change(s).

Associated Links:

N/A

Types of Changes:

Feature

Proposed Changes:

Standardize references by consolidating all constants into the existing consts.go
Formatted/Commented all constants in the consts.go

Verification/Testing of Changes:

N/A

Additional Context:

N/A

please rebased due to merge of #352


fixed typo

Usually the commit messages should be "This commit will ".  It could be a bit more descriptive as well.


add link in TOC to sharedAPI conceptual

:warning: Validation status: warnings

File
Status
Preview URL
Details

docs/toc.yml
:warning:Warning

Details

docs/toc.json
:warning:Warning

Details

docs/add-ins/toc.yml
:white_check_mark:Succeeded

docs/toc.yml

[Warning] Invalid file link:(~/add-ins/office/dev/add-ins/overview/office-add-ins).

docs/toc.json

[Warning] Toc node with title: 'Office add-ins platform overview', relative path '', href 'add-ins/office/dev/add-ins/overview/office-add-ins?context=outlook/context' is invalid. Please make sure the node link does not exceed the docset. Reference error message: Illegal characters in path.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[doc] File is missing in the README.md

The prepare_bookcorpus.py file is missing on this README.md https://github.com/dmlc/gluon-nlp/tree/master/scripts/datasets/pretrain_corpus
(It should have been renamed prepared_gutenberg.py, and in which case the parameters need to be changed).
python3 prepare_bookcorpus.py --dataset gutenberg

@sxjscience

Thanks! A really nice catch!

@TristonC This is fixed via https://github.com/dmlc/gluon-nlp/pull/1378 so I'll close this.


No Loki metadata parsing with LokiJsonTextFormatter

Which version of Serilog.Sinks.Grafana.Loki are you using?
v7.1.0
Which version of .NET are you using?
net6.0
Describe the bug
When looking at the sample configuration defined here : https://github.com/serilog-contrib/serilog-sinks-grafana-loki/wiki/Using-LokiJsonTextFormatter-and-Grafana-Loki-json-parser the sample screen of Loki will show almost all metadata as detected fields.
When I try my output template configuration, almost no fields are detected.
Because of that, I'm now unable to link a field as internal link for Tempo from Loki logs.
My current config is as follows:
      {
        "Name": "GrafanaLoki",
        "Args": {
          "uri": "http://localhost:3100",
          "textFormatter": "Serilog.Sinks.Grafana.Loki.LokiJsonTextFormatter, Serilog.Sinks.Grafana.Loki",
          "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff}] [{Level}] [{SourceContext}] [{SpanId}] [{TraceId}] {Message:lj}{NewLine}{Exception}"
        }
      }

And I would like to link the SpanId with Tempo, but not sure how to get it so that Loki picks it up as a detected field.
Only way to get it working, is to revert to the regular non json textformatter and use an output template like:
"outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff}] [{Level}] [{SourceContext}] [{SpanId}] TraceId={TraceId} {Message:lj}{NewLine}{Exception}"
So that I then can use the Derived Fields feature on the Loki data source in grafana to search through a regex on TraceId=. https://grafana.com/docs/grafana/latest/datasources/loki/#derived-fields
But of course I would rather use the JSON formatter.
Any tips?
To Reproduce
https://github.com/Depechie/OpenTelemetryGrafana
Expected behavior
Would hope to get detected fields for spanid and traceid.
Log/SelfLog output or exception with stacktrace
No response
Application or code sample, which could be used to reproduce a bug
https://github.com/Depechie/OpenTelemetryGrafana
Additional context
No response
I have read the documentation

[X] I have read Common Problems section in project wiki

Hi @Depechie,
I don't see any problems with labels detecting, but I see that you didn't added a static label, which could be used for querying (you can't use | json pipeline without filter).
So I have downloaded your repo and changed config a little (added a static level and added filtering to ensure that none of properties except static won't be added)
      {
        "Name": "GrafanaLoki",
        "Args": {
          "uri": "http://localhost:3100",
          "textFormatter": "Serilog.Sinks.Grafana.Loki.LokiJsonTextFormatter, Serilog.Sinks.Grafana.Loki",
          "labels": [
            {
              "key": "app",
              "value": "web_app"
            }
          ],
          "filtrationMode": "Include",
          "filtrationLabels": [
            "app"
          ],
          "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff}] [{Level}] [{SourceContext}] [{SpanId}] [{TraceId}] {Message:lj}{NewLine}{Exception}"
        }
      }

And how I could do a query across Grafana - {app="web_app"} | json and next data is parsed by Loki:

So seems like all works as designed.
I could suggest that you had been confused by next screen:

Here are labels, which are appended to Loki as plain labels and not parsed. You need a json pipe to extract data. I strongly advice to give a look at this [article]https://grafana.com/blog/2020/10/28/loki-2.0-released-transform-logs-as-youre-querying-them-and-set-up-alerts-within-loki/), where described the way to work with this pipe

@mishamyte thx for the nice detailed info! I will take a look at the given documentation!

@Depechie hope it would help. Feel free to ask any questions if needed

Maybe while we are at it, did you already try to use the derived fields feature to link the trace id to tempo? While using your | json query?

No, we are using other system for tracing, but I will give a look how it could interact

@Depechie could be useful next link: Kubernetes Observability Using Loki, Cortex, Tempo and Grafana. Author uses autodetected labels

Ha found it!!
In the grafana datasources provisioning file I need to set
matcherRegex: "\u0022TraceId\u0022:\u0022(\w+)\u0022"
for the Loki derived field to link to Tempo while using your JSON config for log entries.
Thx again for the help.


Use the daemon for get

If we want resume, it would be easiest to have the daemon involved for get and get-ticket.
That is a breaking change, since currently get and get-ticket don't require a daemon. But I think it is the easiest way to make resume work.

There is now share, which is basically get but talking to the daemon. So closing this, but we still need to rethink the cli commands https://github.com/n0-computer/iroh/issues/1239 to make usage more intuitive.


Extending cookie_value_method

Hi, I have a question on implementation of extending the cookie_value_method. We have a requirement to have a znode id as the cookie value.
/> cat /service/server000
{"host": "<IP_ADDRESS>", "name": "service", "port": 12345}

Taking from this example above, I need to expose server000 as the cookie value. There are a few ways to expose this.
Since the discovery path is exposed in (https://github.com/airbnb/synapse/blob/master/lib/synapse/service_watcher/zookeeper.rb#L130) it could be simple to extend the node information and add the value of id?
Another solution is to resolve the backends IP. (https://github.com/jheung-r7/synapse/commit/730e8da480c4acd1102f9dde3d991b6e285f1775)
I have implemented both but was wondering which would be the best for a PR?
~ Joe

Closing due to requirement changes.


Need to set up linux boxes to auto-update to stable packages

I currently run sudo dpkg-reconfigure --priority=low unattended-upgrades on new Linux machines by hand (it has a GUI popup). Presumably there's a better way to set this up to happen automatically.

FYI, this is your third dupe of this issue :wink:
See #207 and #102.
I will say that I've never been a huge fan of automatic updates (see also: left-pad), preferring pinning + manual updating (possibly with automated rollout), but this is probably safe.

@aneeshusa HA! Great catch. I keep using different words and never see it.
I tend to open these when I log into a machine and see "147 security updates needed," which always freaks me out. I'd personally like to have at least something basic that will get us security updates, even if it means we might left-pad our CI infra one day rather than leave machines unpatched or rely on manual updating.

Something else that might help is auditing the packages that are installed; I just checked servo-master and I'm seeing around 800 packages. This is something that traditional CM systems like Salt aren't so good at. For us, a simple pkg.purged state with a long list of pkgs will do, but better solutions (longer-term) would be starting from a more minimal Ubuntu image,  or using NixOS or a similar declarative distribution, where only packages you explicitly list are installed (and packages you unlist get removed).

I definitely agree! The servo-master machine is particularly bad, and I'm glad (though also terrified!) that @edunham and I plan to burn it down and rebuild with salt from scratch.
I believe that it's extra bad because in the dark, dark early days we all did our testing and part of our CI development directly on that machine. It's also survived major transitions (e.g., from the "bors" autolander to "homu," completely different methods of github events vs. polling tools, etc.).
Is the pkg.purged just supposed to have a list of all of the approved packages?

Well, pkg.purged will ensure those packages don't exist, so we would need to list any packages that we don't want installed (e.g. extra packages in the base image we're using). It's not perfect, but it should do for now.
As I mentioned in #215, look into using Salt multimaster to manage the servo-master transition; I'd also recommend setting up the new gitfs based configuration on a new master and cutting over once it's confirmed working, instead of deploying it on the existing one.


Fix Odo dependency for 0.11.3 release

This should provide the basis for an 0.11.3 release containing critical bugfixes.

At the moment this seems to be breaking the build -- since this is happening across branches, and since odo is installed from Git for CI purposes, my hypothesis is that it's because of conflicting dask versions here vs transitively from odo.

Note that this explicitly excludes Odo 0.5.2 as a dependency until we figure out why the CI builds using master are failing, and fix them if possible.

This doesn't seem to pass CI no matter what, even when it's reduced to updating Dask dependencies (for the transitive odo dependency) -- so going ahead and closing this in favor of a more careful examination of upstream + errors down the road.


chore: refactor middleware setup in server

This PR aims to clean up the way the middlewares for the server are set up.
Previously, the Metrics, CORS and H2C Middlewares were each set up in different ways, which made that section of the code a bit hard to comprehend at first.
Therefore, i introduced a middleware package, including sub packages for the metrics, cors, and h2c middleware. Each of those middlewares implement an IMiddleware interface, that can be registered to the server using the newly introduced AddMiddleware function

Awesome work - this IMO is a long overdue refactoring to clean up.
I left two improvements of which one is essential and the other is a suggestion.


TestTaskRunTimeout: Reduce timeout 30s -> 1s.

Changes
There's no reason why this Task needs to run for a full 30s. Reduce it
to 1s so we can timeout ASAP.
Reduces test runtime from ~30s -> ~1s
Part of @sbwsg's efforts to deflake integration tests by reducing overall runtime to avoid test timeouts.

/kind flake
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[x] Docs included if any changes are user facing
[x] Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes

NONE

/cc @sbwsg

Nice! Drops the runtime to 10.32s.  I guess we can do the same for test/v1alpha1/timeout_test.go as well?

/lgtm

/approve


Is there a way to export RT-DETR rknn?

Hello! Thank you for your impressive work.
I’m trying to run RT-DETR on RK3588, but I’m having trouble converting the model to rknn format.
The exported .onnx model requires opset>=16, but rknn only supports opset<=12.
Do you know of any way to get around this issue?

Hi, I got the same issue while converting to TensorFlow. Did you solve it?


Add support to solve imaginary 2nd grade equations

The title it's pretty descriptive

Closed in #18


Added a Swift version check across all platforms

Issue Link :link:
No direct issue, but prompted by the discussion here: https://forums.swift.org/t/54715
Goals :soccer:
To reliably prevent the package from compiling when using Xcode 11 or Swift < 5.3 so the Swift Package Index can more accurately determine platform and Swift version compatibility. See the build success for watchOS with Xcode 11.6 here: https://swiftpackageindex.com/Alamofire/Alamofire/builds

Thanks Dave!

Thanks Jon. What a speedy merge!


evaluation results: lack of pair?

Hi,
I was comparing the test results to results pasted by in #132 by Marisa.
Her output ended like this:
"Evaluation pipeline:
Evaluating noisemesSad on /vagrant/data/VanDam-Daylong/BN32/test with respect to : accuracy
Pairs that have been found :
BN32_010007_test.rttm / noisemesSad_BN32_010007_test.rttm
accuracy report
detection accuracy true positive true negative false positive false negative
%
item
BN32_010007_test.rttm              11.24         30.80          3.11           0.20         267.57
TOTAL                              11.24         30.80          3.11           0.20         267.57
accuracy report
detection accuracy true positive true negative false positive false
%
item
BN32_010007_test.rttm              11.24         30.80          3.11           0.20         267.57
TOTAL                              11.24         30.80          3.11           0.20         267.57
Connection to <IP_ADDRESS> closed."
My test output end like this:
"Evaluation pipeline YOURS:
Evaluating noisemesSad on /vagrant/data/VanDam-Daylong/BN32/test with respect to : accuracy
Evaluation pipeline REFERENCE:
detection accuracy true positive true negative false positive false
%
item
BN32_010007_test.rttm              11.24         30.80          3.11           0.20         267.57
TOTAL                              11.24         30.80          3.11           0.20         267.57
Connection to <IP_ADDRESS> closed."
It lacks "Pairs that have been found" and one set of results.
I attach also a file with output of running vagrant up.
I work on MacOS Mojave 10.14.6, 8GB RAM, 256 GB memory;
I installed Vagrant 2.2.4 and Virtual Box 6.0.14
Vagrant_test.txt

Hello,
This is surprising, because in the latest version of the DiViME repo, the line responsible for logging pairs that have been found hasn't been commented out. Could you try to git pull the DiViMe repo to check if you're working with the latest version ?

I installed everything today so I guess DiViMe is up to date.
And git pull says that :)
Thus no reason for concern so far?

Yes, indeed. As long as the results are the same, everything's good ! I'll check but I think it might come from small log improvements to make the whole thing a bit clearer and more lightweight !
I'll close this issue for now :)


fix: remove ESBuildPlugin import.

The following warning is printed in the console with the current configuration of the plugin
[esbuild-loader] ESBuildPlugin is no longer required for usage and will be removed in the next major release. Please refer to the docs and release notes for more info.

This PR removes the ESBuildPlugin, since it will not be needed anymore and thus eliminates the warning. I furthermore have bumped the esbuild-loader version to the latest.

Thanks!


Disable tests on Windows for PRs, fix other workflows

Disable tests on Windows in the workflow running against PRs (too flaky, they slow down development; we run nightly jobs on the latest release and that's OK for now)
Give relevant names to workflows jobs (instead of just "build")
Add missing Autogluon dependency in nightly workflows (they have been failing for ~3 weeks due to this)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup

Interesting, changing the job names messes up with the approval rules, I'll disable-and-re-enable


Fix phpdoc for repositories

Q
A

Type
improvement

BC Break
yes/no

Fixed issues

Summary

Ref: https://github.com/doctrine/mongodb-odm/pull/2329
Removes ObjectRepository as return type since it will always return a more specific one (DocumentRepository, GridFSRepository or ViewRepository)

Dunno what's up with these 6 pending checks, all seems green from the "Checks" tab :)

Thanks @franmomu!

Dunno what's up with these 6 pending checks, all seems green from the "Checks" tab :)

I think that is configured in Settings > Branches > Branch protection rules, but no idea why those have showed up 😕

Yup, they were exactly there :)


Query time lookups need a tutorial

Query time lookups need a tutorial of the problems they solve and how they can be used. It is also worthwhile to add an entry in the TOC under "Operations" about QTL, described in a way that folks facing the problem can find and understand.

This is blocked by https://github.com/druid-io/druid/pull/1576

This is blocked by the implementation of a lookup manager as a druid component not as an extension.
First milestone will be the following PR [https://github.com/druid-io/druid/pull/2291]

I think it's not necessary, or even desirable, to wait until the implementation is complete before having user-facing documentation, at least in draft form.  It's a useful way to state what problem is being solved (what the requirements are).  It can also suggest integration tests.

@rasahner I think what you're describing is a proposal.The User docs are a major pain to keep in sync with the development docs as code reviews change things.

Removing milestone as this issue isn't getting much love


How to enable Shared Cache Mode in sqlite3

Hi Guys,
We are using sqlite3 as an analytics backend for our service, as a result we have very heavy writes on to the database. Now since sqlite3 has a database level locks. This is preventing us from utilizing the full write throughput of the sqlite3.
On some research i found out that sqlite3 has Shared Cache Support , ref:https://www.sqlite.org/sharedcache.html
This enables us to have more fine grained locking. I am unable to find the support for similar enable_shared_cache option while loading the database in node-sqlite3.
Can someone please let me know if its possible to enable the shared cache support in node-sqlite3 ?

+1

Hey! I'm definitely open to reviewing a PR for this 🙂

Upon review, it seems we already support this via the open flags:

Individual database connections created using sqlite3_open_v2() can choose to participate or not participate in shared cache mode by using the SQLITE_OPEN_SHAREDCACHE or SQLITE_OPEN_PRIVATECACHE flags the third parameter. The use of either of these flags overrides the global shared cache mode setting established by sqlite3_enable_shared_cache(). No more than one of the flags should be used; if both SQLITE_OPEN_SHAREDCACHE and SQLITE_OPEN_PRIVATECACHE flags are used in the third argument to sqlite3_open_v2() then the behavior is undefined.

https://www.sqlite.org/sharedcache.html#enabling_shared_cache_mode


Support multi-language in Dart Doc

This issue was originally filed by @kotofurumiya

Sometimes I need to create a document in both English and Japanese. But docgen supports single-language currently. Support multi-language output, please.
For example:
     /**
        This is a English doc comment.
     */
     /** {lang=ja}
        これは日本語のドキュメントコメントです。
     */
     void func() {}
Then,
     docgen . -> generates a default(English) document.
     docgen --lang=ja . -> generates a Japanese document.

This issue was moved to dart-lang/dartdoc#773


Fixes menu frame and unit test on the #2483.

Fixes #_____ - Include a terse summary of the change or which issue is fixed.
Pull Request checklist:

[ ] I've named my PR in the form of "Fixes #issue. Terse description."
[ ] My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
[ ] My code follows the Terminal.Gui library design guidelines
[ ] I ran dotnet test before commit
[ ] I have made corresponding changes to the API documentation (using /// style comments)
[ ] My changes generate no new warnings
[ ] I have checked my code and corrected any poor grammar or misspellings
[ ] I conducted basic QA to assure all features are working

@tig I broken the border around the listviews on the UICatalog, but I think you'll discovering that fast.

@tig On the ScrollView color is failing. Maybe not a big deal but I need to look in better. OnDrawContentComplete must run after the OnRenderLineCanvas. Otherwise all the work on the OnDrawContentComplete will be damaged by the OnRenderLineCanvas.

@tig the Driver.Clip must be restored before call the OnRenderLineCanvas and the OnDrawContentComplete must be called after the OnRenderLineCanvas, unless this was intentional done for testing.
@tig can you change the .editorconfig to set the tab = 2 and not force a space before a parentheses on methods as Method()? Thanks.

@tig can you change the .editorconfig to set the tab = 2 and not force a space before a parentheses on methods as Method()? Thanks.

Both are a Miguel/Xamarin thing. I don't think we should change it.


feat(web-modeler): add websockets health endpoint

Which problem does the PR fix?

Closes https://github.com/camunda/camunda-laravel-websockets/issues/186
Uses the new health endpoint for websockets probes instead of tcp socket, see https://github.com/camunda/camunda-laravel-websockets/issues/186
Merge only after Web Modeler's 8.6.0-alpha2/8.5.2/8.4.6/ 8.3.8/8.2.15 releases but before the corresponding Helm chart releases.
What's in this PR?

Additionally adds usage of the probes scheme configuration parameter which was previously missing for all web modeler components.
Checklist
Please make sure to follow our Contributing Guide.

Before opening the PR:

[x] In the repo's root dir, run make go.update-golden-only.
[x] There is no other open pull request for the same update/change.
[ ] Tests for charts are added (if needed).
[ ] In-repo documentation are updated (if needed).

After opening the PR:

[ ] Did you sign our CLA (Contributor License Agreement)? It will show once you open the PR.
[ ] Did all checks/tests pass in the PR?

@camunda/distribution I'm very sorry for the short notice - it would be nice to have this small change in the upcoming helm chart releases so we can close https://github.com/camunda/camunda-laravel-websockets/issues/186 and the corresponding support issue.
The PR is introducing a proper health endpoint GET /up for websockets instead of just using a tcp socket probe.


how much font data is needed?

awesome job!
I'm wondering how many kinds of fonts, characters are needed for training this model?
meanwhile, could you please share the dataset? thank you.

The details could be found in the blog: https://kaonashi-tyc.github.io/2017/04/06/zi2zi.html
As to the dataset, I am afraid I cannot help due to copyrights issue.

okay, thank you~


How to change stream track?

I would like to change the stream track dynamically without losing any message. What's the best method for changing the track of a stream?

You must stop and restart the stream. If you want to do this in a  script, using stream.running = False then start another filter stream.
https://michaelcurrin.github.io/python-twitter-guide/#/code_snippets?id=update-stream

Twitter's API does not include a way to change tracks dynamically for the same standard stream.
As of right now, you'll have to use the enterprise PowerTrack API if you want to use a non-Labs endpoint to do so.
Otherwise, you'll have to create a new stream with the new track, as explained.
I'd recommend using stream.disconnect rather than manually modifying stream.running though.
See also the Missing data from disconnections section of the documentation for consuming streaming data.


create ToU

Update: put in a request with TrustLaw for legal services to help draft an appropriate ToU. Phone call scheduled for Fri, Mar. 23 (a mandatory step before the project opens up to lawyers). Our lawyers we worked with previously said they can definitely help us draft one and will accept the project request when it becomes available.
FYI @mjamei

Our project request will go up this week and we'll hopefully have some options for lawyers shortly after.

@viviandien update?

No response yet from lawyers. Just pinged them again.

Received a response from lawyers-- will be setting up an intro call over the next 1-2 weeks to proceed.


Lengthen list of "Most searched features"

On the home page of caniuse.com there's a list of the "Most searched features".

It's very interesting data. And I'd love to see a longer list.
Perhaps the list on the home page can contain 10 items. Or perhaps a "more" link could lead to another page that has the full list in order (or top 50 or something).
There are many occasions when browser engineering teams or web standards folks are setting priorities. And knowing which features (beyond 5) that everyday web developers are looking up as they code sites could be helpful to see.
It can also be helpful for the people making content for web developers to know what people are looking up interop data for. Seeing "CSS Grid" high on the list seems like a signal people are looking up support as they start to more seriously considering using Grid, for example.

Unfortunately generating this list is currently a bit of a manual process based on the site's search analytics so not an easy thing to update a large list on a regular basis at this time. That said I agree it would be valuable so could look into automating the process.


Specification clarifications

Clarifications and suggestions from David Kruckemeyer:

Clarify the severity of information errors.
Section 2.4.1 Reword "overwrites" to "updates any field of" in description of rdip clearing.
Clarify that already asserted signals are unaffected by clearing else to 0
Add guidance for class/priority specific SW handlers and SW guidance on reconfiguration of signals
Clarify it is the first overflow, i.e. the one that sets the ceco bit, that leads to the signal being generated.
Section 2.5: Clarify that statement "new error is recorded" includes writes and overwrites.

These clarifications and suggestions were included in PR #49


Use a callback to identify worker/process for stream handlers

We shouldn't leave it to the user to decide whether the process is a worker or a supervisor. They are depending on what stream handlers are enabled.

OK looking into the changes, I'm not convinced. If the user doesn't use cowboy_stream_h, it's up to them to set connection_type right now. Let's not complicate the stream handlers interface for just a few special cases.


Return key from traverse_util.Traversal.iterate()

Currently traverse_util.Traversal.iterate() only returns the traversed values. In some cases we need access to the traversed keys as well, for example when checking that two ModelParamTraversal do not overlap in their keyspace (#1135).
Thus, the interface should be updated to
  @abc.abstractmethod
  def iterate(self, inputs):
    """Iterate over the values selected by this `Traversal`.

    Args:
      inputs: the object that should be traversed.
    Returns:
      An iterator over the traversed ``(key, value)``.
    """
    pass

And all uses of traverse_util.Traversal.iterate() updated accordingly.

Can I work on this?

@Dsantra92 sure, feel free to give it a try. We would have to run it against all our internal tests as well, but we can do that once all our public tests pass.

@marcvanzee To the best of my knowledge, the use of flax.traverese_utils.Traversal is now decrypted for flax.optim. So is there still the need of updating the flax.traverese_utils API? If yes, I would like to work on it.

I've never used that API so it's hard to comment. @andsteing WDYT?

@andsteing I have recently started exploring JAX and its ecosystem like flax. I am interested to contribute in this system. I would like to start with this issue. Can I take up this issue if no one else is working on it? Also, is there anything you want to suggest before I start working on this issue?


Add hex colour codes to README

I needed tooltips which had the same colours as the markers, so for the
CSS I needed to know the colours inside and outside of the markers.
It would have been helpful for me to have had the colours listed so I
could use them. I have now determined these colour and put them in the
README in the hope that they may be useful to someone else with similar
requirements.
I determined these colours by using a colour picker on the icons, so
they may not be 100% accurate, but they're good enough for my
application.

thx


macOS support

Rather than a trail of comments across Facebook, I thought I'd keep track of the macOS status here.
The good news is that, as of 1.7.35, Topo starts and runs on macOS, and all the reports seem to work.
The bad news is that there are a lot of steps to jump through.
Here is a basic set of commands that have to be run at terminal level, assuming Homebrew is already installed.
brew install mono-libgdiplus wget
mkdir topo-v1.XXX.YYY
cd topo-v1.XXX.YYY
wget https://github.com/NomisNostab/Topo/releases/download/v1.XXX.YYY/topo-osx.zip
unzip topo-osx.zip
./Topo

The main issues are:

You need to have  mono-libgdiplus installed - this is easy if you have Homebrew installed, just brew install mono-libgdiplus, but most people won't have Homebrew.
Because the executable isn't signed and notarised, Gatekeeper won't allow the executable to run; if you bypass this by holding down Alt when launching Topo, it still hits Gatekeeper when the SQLite module gets loaded and the only way around that is to dig through System Preferences. You can bypass Gatekeeper by downloading the zip with wget from Homebrew as above, or wipe the quarantine flag with xattr -d com.apple.quarantine Topo, but again this adds a lot of difficulty for inexpert users.

To fix the first, it might be possible to bundle mono-libgdiplus, using something like macdylibbundler, or to migrate to something which doesn't depend on GDI+.
To fix the second is easier (sign and notarise the package) but costs money - a certificate requires Apple Developer Program membership which costs A$149/year. Savvy users can wipe the quarantine flag or download the zip using wget, but getting the average user to do so is challenging.
I don't know whether macOS support is something you want to definitely consider - I have stopped doing Mac distribution for business-focused projects, but a lot of volunteers use Macs.
Also, ideally the build process needs to generate an app bundle. The current profile generates a single executable, but all the resources in wwwroot and Reports aren't included. Something like https://github.com/egramtel/dotnet-bundle might help.
I haven't played around with many of the things listed above, and building .NET 6 applications on macOS currently requires a pre-release version of Visual Studio.

Hi David,
Thank you for putting in that effort to get Topo going on a Mac.
I agree that it is not for the faint hearted and it is not really something that I want to spend much time on.
This really needs to be in a hosted web site so there is no installation and people just use the site, but that has its issues as well.
Still it's nice to know that it will work.
Simon.

Hi, I have an Apple Developer Program membership, can I assist?

Hi ,
Please feel free to do whatever you can to get this working on a Mac.
I suspect the Syncfusion library will be the main pain point.
Simon.


Modal: Update heading size for alert modals + fix docs

Design asked us to update the heading size for alert modals (make it smaller)
After:

Before:

Deploy preview for gestalt ready!
Built with commit 34e083526a279f35dd6e1589d2f684e2f0831125
https://deploy-preview-114--gestalt.netlify.com


[Regression] Rust code can no longer contain use renames, even though we don't try to parse them

Any Rust code that contains an import rename will cause source_graph.rs to panic. This is bad, since imports are not currently used by codegen and so shouldn't matter.
I'm working on a fix for this, but just leaving this issue here in case someone gets to releasing before I open a PR.

Sure. I will not release a new version before this is solved.


[feature request] echoing Access-Control-Request-Headers to Access-Control-Allow-Headers

It will be convinient if git-cors has ability to fill a response header "Access-Control-Allow-Headers" with content of a request header "Access-Control-Request-Headers".

What is this the intended use-case?
If it is to make development easier, how about adding a development flag which allows everything and logs the headers set via the log package?

Sorry for the late answer. Yes, the primary use-case is to make development more comfortable. The flag would be a good solution.

I'm open to a PR for this. Just don't have the time to do it myself.

Due to the lack of anyone interested in picking this up. I'm going to close.


[v2.9] Add util-linux ca-certificates-mozilla

Both packages needed to be added after the transition from bci-base to bci-micro (#44348). The former contains key Linux utilities such as mount and umount, which are needed by upstream dependencies. The latter contain static certificate files which are needed for the update-ca-certificates operation that takes place at entrypoint.sh.
Fixes https://github.com/rancher/rancher/issues/45199.
Potentially linked to https://github.com/rancher/rancher/issues/45199.

The Verify failure is orthogonal to the changes, as it is pre-existing on the target branch.

Thanks for verifying @ericpromislow. I rebased to fix the CI issue.


Module parse failed: Unexpected token react-scripts build fail

11:42:52.802 | > react-scripts build
11:42:52.802 |
11:42:55.044 | Creating an optimized production build...
11:45:50.165 | Failed to compile.
11:45:50.165 |
11:45:50.165 | Module parse failed: Unexpected token (10116:106)
11:45:50.166 | File was processed with these loaders:
11:45:50.166 | * ./node_modules/babel-loader/lib/index.js
11:45:50.166 | * ./node_modules/source-map-loader/dist/cjs.js
11:45:50.166 | You may need an additional loader to handle the result of these loaders.
11:45:50.166 | \|  */
11:45:50.166 | \|
11:45:50.166 | > function gen_bitlen(s, desc) /*    deflate_state *s;*/ /*    tree_desc *desc;    /* the tree descriptor */*/{
11:45:50.166 | \|   var tree = desc.dyn_tree;
11:45:50.167 | \|   var max_code = desc.max_code;
11:45:50.167 |
11:45:50.167 |
11:45:50.387 | npm ERR! code ELIFECYCLE
11:45:50.387 | npm ERR! errno 1
11:45:50.390 | npm ERR<EMAIL_ADDRESS>build: `react-scripts build`
11:45:50.390 | npm ERR! Exit status 1
11:45:50.390 | npm ERR!
11:45:50.390 | npm ERR! Failed at the<EMAIL_ADDRESS>build script.
11:45:50.391 | npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
11:45:50.401

"react": "^17.0.2"
"react-dom": "^17.0.2"
"react-scripts": "^5.0.1"
node v14.16.1
npm v6.14.12

how about solution ?

how about solution ?

https://github.com/babel/babel-loader/issues/972
maybe, this is a babel-loader issue.

this issue has been fixed in @babel/generator 7.20.3


KubeJS failing on first run

When starting up the game, KubeJS runs into a null pointer exception (stacktrace) in the StampingRecipe.getResultItem function.
Running a /reload command causes things to work properly.
Using:
Embers Rekindled 1.20.1-1.1.0
NeoForge 47.1.84
KubeJS 2001.6.4-build.114

If this is happening before a world is loaded then kubejs is trying to get the result of a recipe before tags are loaded, I can handle that a little more gracefully to prevent the log from being spammed but it still won't work properly.

I have 100 some odd mods loaded, but after reloading the world all embers machine recipes disappear, I don't know if that will help track down the bug or not

You should be able to lower the priority from kube recipes and make kube load all the tags first, in theory that should fix it


serializing learning network machines

It doesn't seem to be possible to directly serialize DeterministicSurrogate machines, is this intended?  One gets
ERROR: ArgumentError: You must specify at least one operation, as in `predict=<some node>`.
Stacktrace:
 [1] machine(::DeterministicSurrogate, ::MLJBase.Source, ::Vararg{MLJBase.Source}; pair_itr::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:cache,), Tuple{Bool}}})
   @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/composition/learning_networks/machines.jl:173
 [2] serializable(mach::Machine{DeterministicSurrogate, false})
   @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:909
 [3] save(file::String, mach::Machine{DeterministicSurrogate, false})
   @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:1004
 [4] top-level scope
   @ REPL[37]:1
 [5] top-level scope
   @ ~/.julia/packages/Infiltrator/qHXTS/src/Infiltrator.jl:680

It's coming from the constructor that is used for copying which lacks a predict argument.

Thanks for raising this and the work at #845.
I never imagined serialization of surrogate machines. They were introduced as a way to improve on an older way of "exporting" learning networks as new standalone model types. However, I have never felt great about them, as they feel like an unnecessary abstraction. See more at #831.
At #841 I am proposing a simpler way of exporting learning networks, that gets rid of the surrogates and learning network machines altogether, and has other advantages detailed at #831. My plan was to ultimately deprecate all other export options. So, I am curious, what is your use case for using the learning network machines?

If I understand correctly these networks are the most general types of machine objects and are a superset of linear pipelines and ensemble models.  Most of the time when I use them, I could (and perhaps should) use linear pipelines instead.  In some of the more complicated cases I decide on one of multiple inner machines based on the value of a categorical value or transform outputs.  Another example is cases in which I require positive-definite output and it's far simpler to do max.(y\hat, 0) and the real model output is already "positive enough" that I'm not going to lose sleep over not having trained it that way.
TL;DR: My use cases usually could use pipeline instead.
On the other hand, I agree that these constructions can be awkward in a number of ways.  Your proposal in #841 looks good to me, though I don't think I fully understand the advantages and disadvantages of the different approaches.
Do you already support serializing the composite models there or would that remain to be added?  Having that ability would resolve this issue in my mind.

Ah, I see you had that in your TODO and already have it checked 👍

I guess I was not clear in my question. For me, the raison d'etre for learning networks is to define new composite model types. The fact that they can be used without being "exported" is nice, because you can test them and play with them to test your understanding of them, but at the end of the day, they exist to define new composite model types. For example both Pipeline and Stack are implemented by exporting a learning network.
I guess my question is: Do you have a reason for using a learning network that is not ultimately exported as a new type (which you will be able to serialise, and whose reports and fitted_params give you access to essential parts of the underlying component machines)?

Do you have a reason for using a learning network that is not ultimately exported as a new type (which you will be able to serialise, and whose reports and fitted_params give you access to essential parts of the underlying component machines)?

No, I don't think so.  This was simply how I thought I was supposed to be doing this.  I'm not entirely sure if you are saying I was already able to do this or I would be able to do this after your PR.

No, one has always been able to export learning networks as standalone composite model types.  PR #841 just makes it easier.
I will shortly post a doc PR at MLJ which will (hopefully) make this easier for people to digest. Will copy you in for your comment.
So I will proceed with my plan to deprecate the surrogate business, but this will be removed until the next breaking release after the coming one.  Does that sound okay?

Yeah, sounds great.  Now I'm just confused about what was the intended way to do this in the first place, but I'm content to wait for #841 and your updated docs, thanks.


It would be better to remove the checkbox 'Also delete blob snapshots' when deleting one blob version

Storage Explorer Version: 1.32.0-dev (95)
Build Number: 20231021.1
Branch: rel/1.32.0
Platform/OS: Windows 10/Linux Ubuntu 20.04/MacOS Sonoma 14.0(Apple M1 Pro)
Architecture: x64/x64/arm64
How Found: From running test cases
Regression From: Not a regression
Steps to Reproduce

Expand one storage account -> Blob Containers. (make sure the blob versioning is enabled)
Create a blob container -> Upload one blob.
Modify the blob properties to create a blob version.
Right click the blob -> Click 'Manage History-> Manage Versions'.
Select the blob version -> Click 'Delete'.
Check there is no checkbox 'Also delete blob snapshots' on the dialog.

Expected Experience
It would be better to remove the checkbox 'Also delete blob snapshots'.
Actual Experience
There is a checkbox 'Also delete blob snapshots' on the dialog.

Additional Context
Fail to delete the blob version with the checkbox 'Also delete blob snapshots' checked.

Verified this issue on the rel build 20231024.2. Fixed.