State machine refactor

Currently, the code is a whole bunch of "megafunctions" that make it really hard to reason about. It also forces writers of other Providers into similar coding patterns. We are currently in efforts to refactor things into an actual state machine where each state can be handled as its own unit of work

cc @kflansburg, who has been driving this work, but I can't assign the issue to for some reason


Changes needed for constitution

Pre-submit checklist:

Branch

[ ] Tests are provided (if possible)
[ ] Commit sequence broadly makes sense
[ ] Key commits have useful messages
[ ] Changelog fragments have been written (if appropriate)
[ ] Relevant tickets are mentioned in commit messages
[ ] Formatting, PNG optimization, etc. are updated

PR

[ ] (For external contributions) Corresponding issue exists and is linked in the description
[ ] Targeting master unless this is a cherry-pick backport
[ ] Self-reviewed the diff
[ ] Useful pull request description
[ ] Reviewer requested

Everything succeeded, not sure why the checks are red.


✨ 179 graphical calendar

Issues #179

Current view

Things should that should be done in another issue:

Make the size of the calendar flexible (or at least more flexible ... small, medium or large as a prop)
Determine the days of week via calculation (dependent on the locales)
aria features ...

I improved the animation-part slightly with https://github.com/it-at-m/muc-patternlab-vue/pull/214/commits/1691937aa5de019078e388d32f035026e5b5bbe1
Making the size variable now only depends on fixing the day-grid to be flexible and use the whole height. But fixing this in a seperate issue would be ok for me, too.

Before Merge new components should be refactored to use the new reactive prop destructuring and useTemplateRef composable if applicable. See https://blog.vuejs.org/posts/vue-3-5#reactive-props-destructure and https://blog.vuejs.org/posts/vue-3-5#usetemplateref


Program comparisons empty

Hi all - Pretty much zero priority on this, but thought I'd log it.
In all my model runs, the program_comparisons.csv comes out empty. Haven't poked around with the code at all to see what the issue is.
Thank you!

Bug is  due to incorrect reference of the base directory (batch_reporting.py, function: batch_report() )


pkg_call_plugins put includefile and supported plugins in the normal …

…config.xml so there is no need to parse the package xml for them. this improves performance significantly for several pages like such as 'ipsec overview' and 'openvpn server edit page' which use certificates and gatewaygroups which acquire some information from plugins.

@PiBa-NL please rebase your fork

It can cause a problem when system files are updated but package don't, so these items won't be registered in config.xml. I'll merge it but keep it 2.4 only. When people migrate from < 2.4 all packages will be reinstalled for sure


[Brotli] Fix emscripten

Fixes issue reported in https://github.com/microsoft/vcpkg/pull/33665#issuecomment-1739094306
Related to: WerWolv/ImHex#1299

[X] Changes comply with the maintainer guide
[X] SHA512s are updated for each updated download
[X] The "supports" clause reflects platforms that may be fixed by this new version
[X] Any fixed CI baseline entries are removed from that file.
[X] Any patches that are no longer applied are deleted from the port's directory.
[X] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[X] Only one version is added to each modified port's versions file.

@WerWolv Could you try these changes out and let me know if its resolved for you? I've tested locally and it seems ok now.

Hey ! I just tried it, and confirm that we can now install freetype with emscripten, and our project compiles fine with it
Thanks a lot for your patch

@DownerCase, looks this PR is ready for review.


Deep Rock Galactic

Just wanting to open this issue up to see if fellow DXVK afiliarnos can chime in on getting this title working under DXVK. Its based off Unreal4 like ARK is and has no COOKED content.
You get a message that shows up saying global shader cache file missing GlobalShaderCache-GLSL_150.bin and the application is built to load COOKED content but that is not provided. I tried using -game argument and the error message goes away, however title sits at 16% CPU usage indefinitely so perhaps it can't run uncooked (-game) mode.
If anyone has some ideas please chime in I would like to test this one out a bit more. I was wondering if we could provide the GLSL shader files manually but maybe that won't work. I think ARK has these files cooked already.
EDIT: Game claims to be DX11 but I'm not so certain its running in that mode because DXVK logs are not made, could be running DX10 or something else?  (USE -d3d11 argument on game exe)
Software information
Deep Rock Galactic
System information

GPU: 1080Ti
Driver: 396.18
Wine version: 3.7staging (FO4/SSE patch)
DXVK version: lastest master

Apitrace file(s)

coming soon

Log files

d3d11.log: CrashReportClient_d3d11.log
dxgi.log: CrashReportClient_dxgi.log

That seems quite odd, they advertise as DirectX11, what on earth could be going on here? Maybe its a matter of getting it to run correctly under DX11. It's Unreal Engine 4 so I don't see why that isn't the default.
Hopefully someone else who has it can chime in as perhaps my test setup is wrong somehow.

I discovered something with this game, under windows if I go -opengl argument it pops up with the EXACT same error, so for some reason it looks like to be switching to opengl under wine by default instead of dx11. Not sure if there is a DX11  flag yet, testing required.

Wine errors on launch that MAY be of some interest (could be nothing)
Some general WINE errors:
[0503/112247:ERROR:network_change_notifier_win.cc(170)] WSALookupServiceBegin failed with: 8
006a:err:dc:CreateDCW no driver found for L"\\.\DISPLAY2\Monitor0"
003c:err:winediag:wined3d_dll_init Setting multithreaded command stream to 0.
003c:err:ole:CoGetClassObject class {294935ce-f637-4e7c-a41b-ab255460b862} not registered
003c:err:ole:CoGetClassObject class {294935ce-f637-4e7c-a41b-ab255460b862} not registered
003c:err:ole:create_server class {294935ce-f637-4e7c-a41b-ab255460b862} not registered
003c:err:ole:CoGetClassObject no class object {294935ce-f637-4e7c-a41b-ab255460b862} could be created for context 0x17
0075:err:winediag:wined3d_dll_init Setting multithreaded command stream to 0.
Assert( Assertion Failed: Attempting to use a Windows-Vista+ only path root on an older version of Windows for game 379430 ):c:\buildslave\steam_rel_client_win32\build\src\clientdll\userremotestorage.cpp:2270
DXVK related errors: Likely cause of crash.
0105:err:d3dcompiler:compile_shader HLSL shader parsing failed.
0105:err:d3dcompiler:compile_shader HLSL shader parsing failed.

Those errors are not DXVK-related. Make sure to use native d3dcompiler DLLs (not sure which version, if them game is older then you probably need _42 and _43).

ok, I just assumed they were DXVK related because they appeared directly in the middle of the DXVK debug read out.

Ok I have this game working 60+ fps now, turns out it doesn't like __GL_YIELD (NVIDIA) so be sure to not set that (I had NOTHING set). You do need dx 42/43 files and probably -d3d11 argument set.
I run at 4k as usual and have settings on HIGH-EPIC with low FSAA and get 60-80fps area (its quite variable), when I set the MAX fps cap to anything I notice frame skipping so keep that in mind.
If anyone else has this game please feel free to comment, it be great to hear from AMD users about its performance.

Can you record an apitrace?

I will certainly look into it. But the issue isn't normally present at the start of a mission, starts happening as you more further down, and when you return back to the drop pod to exit the mission, the issues start subsiding etc... In fact sometimes when you look in a certain direction the artifacts go away.
I just updated to 396.24 Nvidia drivers and will do some more testing, might also try the new compiler and see what happens. If the problems still persist I will look into exactly howto do a apitrace.,

You might want to keep an eye on #267 in that case, there's some discussion going on about the new shader compiler. I'll close this since there seem to be no bugs here, although I would like you to submit a trace for be to be able to test the game on different GPUs and drivers.

Just got a question about apitrace doitsujin, you say not to run it with DXVK, are you asking to have a apitrace ran with default wined3d11 wrapper?
and for games that have a issue at particular points (like 30mins into a mission), does it require me to play to thought said mission with wined3d11 to get the apitrace or am I misunderstanding what apitrace does?

I recorded a trace file of 1 mission where the graphics glitching occurs, and well its 14gb...

This no longer appears to work properly. Get 1-4fps now. Shame

Hi. Not sure if I'm in entirely the right place, but maybe somebody can help. Disclaimer: I'm pretty new to running games using DXVK, I only just discovered that it comes bundled with CrossOver 20.
I have a M1 MacBook Air and CrossOver 20.0.4 (which appears to bundle dxvk-v1.5.1-8-g7443ead4_x86_64.tar.bz2). After setting FSD-Win64-Shipping.exe to use »native« versions of d3d11.dll and dxgi.dll I was able to launch the game and move around with more than ~1 FPS, and use the settings menu properly.
However, everything is really dark. There appears to be a lighting source missing (?). I turned the Gamma value all the way up to be able to see anything at all. See screenshots, I don't really know how to describe it.
Any pointers to how this could be resolved? The logs appear not to contain anything out of the ordinary, I've attached them here. Am I in the right place anyway? Should I go complain to CrossOver instead? Is this a M1 limitation?
I'm happy to help trace this if you're interested. Let me know what I can do :)

FSD-Win64-Shipping_d3d11.log
FSD-Win64-Shipping_dxgi.log

I have a M1 MacBook Air and CrossOver 20.0.4 (which appears to bundle dxvk-v1.5.1-8-g7443ead4_x86_64.tar.bz2). After setting FSD-Win64-Shipping.exe to use »native« versions of d3d11.dll and dxgi.dll I was able to launch the game and move around with more than ~1 FPS, and use the settings menu properly.

Sorry, but this is the wrong place. That version of DXVK was modified. If you're having issues with CrossOver, I encourage you to file a bug and/or support ticket with CodeWeavers. Also, I'm pretty sure neither Phillip nor Joshua provide support for Mac OS at all, let alone Apple Silicon.

Thanks, I'll try my luck with CodeWeavers.

Ok, ahora tengo este juego funcionando a más de 60 fps. Resulta que no le gusta __GL_YIELD (NVIDIA), así que asegúrate de no configurarlo (no tenía NADA configurado). Necesitas archivos dx 42/43 y probablemente el argumento -d3d11 configurado.
Corro a 4k como siempre y tengo configuraciones en HIGH-EPIC con FSAA bajo y obtengo un área de 60-80 fps (es bastante variable), cuando configuro el límite de fps MÁXIMO en cualquier cosa, noto saltos de cuadros, así que tenlo en cuenta.
Si alguien más tiene este juego, no dude en comentarlo. Sería genial escuchar a los usuarios de AMD hablar sobre su rendimiento.
PD: Aún no he probado ninguna partida en línea ni la estabilidad a largo plazo.
ACTUALIZACIÓN: El juego funciona razonablemente bien, excepto que hay una falla gráfica que ocurre en algunos de los biomas.
Incluí una imagen de lo que sucede. Básicamente, parece que se está produciendo un estiramiento de polígonos. No estoy seguro de cómo obtener un registro de ningún tipo para este tipo de error, pero los problemas más comunes ocurren en misiones de tipo cristal y solo cuando desciendes un poco comienza a intensificarse.

y otro ejemplo

Ok wait, how do I do all that? or rather what did you do to get to that point, I have tried to solve that problem, I really don't know much about programming and I would like to check if it can work for me too, by the way I am trying it in lethal company but nothing at all


Potentially fixing #12

I basically had the same problem than #12, so I tried to find out why. Turns out, by analyzing the requests done towards the device, it would do "pixoo.push()" twice if there was text and images in the same page, which probably made my pixoo crash. This could fix #12, but that would require further testing from his part, but for me, this fixed it.
(I know the patch might be a little bit dirty, but I'm not certain what to do otherwise)

okay that actually looks quite ok. i'll still test it first. Many thanks for all your support!

What the hell!? I did not what do close this!?!? One second, sorry about this..

It works great thank u for support


[Template Request] Template for typescript + graphql-shield

Graphql-shield is a permission layer for graphql servers: https://github.com/maticzav/graphql-shield
It's configured by defining a schema middleware:
permissions/index.ts:
import { shield } from 'graphql-shield';
import * as rules from './rules';

const schemaPermissions = {
  Query: {
    me: rules.isAuthenticated,
  },
  Mutation: {
    editProfile: rules.isAuthenticated,
  }
};

export const permissions = shield(schemaPermissions);

permissions/rules.ts:
import { rule } from 'graphql-shield';
import { GQLContext } from '@core/gql/context';

export const isAuthenticated = rule()(async (parent, args, ctx: GQLContext, info) => {
  return !!ctx.me;
});

As seen on the example, generated type definitions for schemaPermissions would mostly be a copy of Resolvers' type definitions. The only difference is that instead of me?: MeResolver<UserAccount | null, any, Context>;, it should have something like me?: Rule.

Hi @alexk111, thanks for reporting it.
It's a very specific use case, any chance you want to create a template for it? or modify the existing template and create a PR?

Hi @alexk111
We got a new resolvers template now, that let's you override the parent and context types. I think it might be useful for this use-case :)
Please re-open if you think we should support graphql-shield in a different way.

I was able to make it work 90% of the time with the following config:
generates:
  src/generated/shared.ts:
    plugins:
      - 'typescript'
    config:
      enumsAsConst: true
      useTypeImports: true
  src/generated/permissions.ts:
    preset: import-types
    presetConfig:
      typesPath: ./shared
    plugins:
      - add:
          content: "import type { ShieldRule } from 'graphql-shield/dist/types';"
      - add:
          content: "import type * as Types from './shared';"
      - 'typescript-resolvers'
    config:
      useTypeImports: true
      customResolverFn: 'ShieldRule'
      resolverTypeSuffix: Permissions
      allResolversTypeName: Permissions
      allowParentTypeOverride: true
      makeResolverTypeCallable: true
      wrapFieldDefinitions: false
      skipTypename: true
  src/generated/resolvers.ts:
    preset: import-types
    presetConfig:
      typesPath: ./shared
    plugins:
      - add:
          content: "import type * as Types from './shared';"
      - 'typescript-resolvers'
    config:
      useTypeImports: true
      contextType: '~/context#Context'
      mapperTypeSuffix: Model
      mappers:
        MyType: '~/mytypes#MyType'

The only thing that doesn't work is the '*' since I was not able to inject it in the fields of the Resolvers objects.

I had the same need and I wrote this type to convert the Resolvers type we get from graphql-code-generator to something that can be used with graphql-shield.
It seems to work well so far, although I haven't done too much that's complicated with it yet. I can't promise it supports every use case, and it may need some tweaking, but hopefully it's useful to someone.
type ConvertResolversToShield<T extends Record<string, any>> = {
  [key in keyof T]: T[key] extends (args: any) => any
    ? Rule | RuleTrue | LogicRule
    : // Make all resolvers on Mutation and Query type required so we have to define them
    key extends "Mutation" | "Query"
    ? Required<ConvertResolversToShield<T[key]>> & { "*": RuleFalse }
    : // All other types can be undefined because they will allow by default
      ConvertResolversToShield<T[key]>;
};

type ResolverShield = ConvertResolversToShield<Resolvers>;

export const schemaShield: ResolverShield = {
  Mutation: {
    "*": deny,
   ...
  },
  Query: {
    "*": deny,
    ...
  },
};

I made all the queries and mutations required, as I wanted to force their definition, and you have to set a wildcard to deny all by default. But, you could easily remove that if you don't want that functionality.


[HTTP Cache] Support Server Cache-Control headers

Summary
Hello,
I am experiencing problems using the http cache. We currently configure the http cache to use the CACHE_FIRST policy.
Based on the documentation, apollo will retrieve data from the cache if it exists and is not expired.
My server is returning the response to Cache control max-age 60. But the cache is never expiring, causing a problem in the application to never update the data again.
Version
We are using version 1.3.0
ApolloClient
            .builder()
            .serverUrl(settings.environment())
            .useHttpGetMethodForQueries(true)
            .useHttpGetMethodForPersistedQueries(true)
            .enableAutoPersistedQueries(true)
            .addCustomTypeAdapter(CustomType.DATE, DateTypeAdapter())
            .defaultHttpCachePolicy(
                HttpCachePolicy.CACHE_FIRST
            )
            .httpCache(
                ApolloHttpCache(
                    DiskLruHttpCacheStore(File(settings.cacheDir(), CACHE_APOLLO), CACHE_SIZE)
                )
            )
            .defaultResponseFetcher(ApolloResponseFetchers.CACHE_FIRST)
            .normalizedCache(
                LruNormalizedCacheFactory(
                    EvictionPolicy.builder().maxSizeBytes(
                        CACHE_SIZE
                    ).build()
                )
            )
            .okHttpClient(
                OkHttpClient()
                    .newBuilder()
                    .connectTimeout(settings.timeout(), TimeUnit.SECONDS)
                    .readTimeout(settings.timeout(), TimeUnit.SECONDS)
                    .writeTimeout(settings.timeout(), TimeUnit.SECONDS)
                    .callTimeout(settings.timeout(), TimeUnit.SECONDS)
                    .addNetworkInterceptor(
                        HttpLoggingInterceptor().setLevel(
                            if (DEBUG) HttpLoggingInterceptor.Level.BODY else HttpLoggingInterceptor.Level.NONE
                        )
                    )
                    .addInterceptor(providerHeaderInterceptor())
                    .build()
            )
            .build()

My server is returning the response to Cache control max-age 60

Right now Apollo doesn't support server cache policy, only explicitly set by client is supported.
com.apollographql.apollo.api.cache.http.HttpCachePolicy.ExpirePolicy#expireAfter

Is there any way currently to always request the data and if the request fails or backend out, return the data from the cache?

@halysongoncalves what you need sounds very familiar with this feature we have: https://github.com/apollographql/apollo-android/blob/master/apollo-runtime/src/main/java/com/apollographql/apollo/fetcher/ApolloResponseFetchers.java#L46
Can you have a look?

I tried to update but it does not respect the cache control directives

+1

Closing this for now since this is already supported with responseFetcher API. Please open an issue with reproducible case if that is not working.

In my setup ApolloResponseFetchers.CACHE_FIRST (apollo-runtime 2.5.4) also does not pay heed to the Cache-Control headers, queries always reach the network.
These are example headers:
Cache-Control: max-age=30, must-revalidate Last-Modified: Thu, 25 Feb 2021 16:40:52 GMT Expires: Thu, 25 Feb 2021 16:41:22 GMT X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: DENY vary: accept-encoding Content-Encoding: gzip Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Thu, 25 Feb 2021 16:40:52 GMT Keep-Alive: timeout=60 Connection: keep-alive
So requests with the header above should be cached for 30 seconds.
I can set the cache TTL via
httpCachePolicy(HttpCachePolicy.CACHE_FIRST.expireAfter(...
but I would prefer if the server controls it via the Cache-Control header it sends out.

Reopened and changed the title to make it explicit that this issue is about server-side cache control. I'm curious how much of that could maybe be achieved through OkHttp caching? https://square.github.io/okhttp/caching/

AFAIK OkHttp-Caching only works for GET-Requests; I also use a separate OkHttp-Cache layer apart from the Apollo-Cache and the former does not affect the Apollo queries.

@semaphore3000 Can you try something like this to read the HTTP headers?
val okHttpResponse = apolloResponse.executionContext[OkHttpExecutionContext]
val value = okHttpResponse.header("Cache-Control")

IIRC this was added for exactly that purpose

Thank you, @martinbonnin. This looks like if it could work for my use case. I will try it out later.

@semaphore3000 anything else I can help with here?


Collapse sidebar on narrow screens

Description
When opening iD in a device with a narrow screen, the sidebar appears automatically even though it is empty, occupying precious real estate:

IMO it should automatically be loaded in the collapsed state when the container is narrow:

In contrast, the toolbar buttons automatically get a more compact display by omitting the labels and showing just the icons, which IMO is sensible behavior:

I would gladly contribute a PR if there's agreement that this is desirable, and I can get some guidance on how to do it — I suppose one needs to do something like, say selection.classed('collapsed', containerWidth < 600); (or whatever pixel width we consider narrow enough) somewhere when initializing the sidebar?
Screenshots
No response

Hello !!
I think my PR will be able to solve the issue !!


The current PR viewed in the modal changes if a different PR is voted upon

@jex441 @ramirc5, can I integrate this PR? See your comments below:

Not sure why the list of files changed is so long
Those are all changes already in master. Only files changed should be inject.js, ModalVote.js, and VoteStatusButton.js


New formula: solr4

Now that the solr formula in the master homebrew repos is up to version 5 it will be useful to have a solr4 version around.

Merged! Thanks for your contribution to Homebrew, we appreciate it :smiley_cat:. In future tests should be a little more thorough, but I let it go this time because Java stuff is a complete nightmare to test.


pyquil.quilbase in qvm_wavefunction.py cannot find Instr module

I recently installed referenceqvm via pip for python 3.6.3.  When I try to run the example on your website, I get the following error:
>>> import referenceqvm.api as api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gpfs0/home/gdhpcgroup/aps003/local/pip-installs/python-3.6.3/lib/python3.6/site-packages/referenceqvm/api.py", line 5, in <module>
    from referenceqvm.qvm_wavefunction import QVM_Wavefunction
  File "/gpfs0/home/gdhpcgroup/aps003/local/pip-installs/python-3.6.3/lib/python3.6/site-packages/referenceqvm/qvm_wavefunction.py", line 45, in <module>
    from pyquil.quilbase import (Instr,
ImportError: cannot import name 'Instr'

It seems that the ‘Instr’ does not exist.  Advice?

Seems like the reference-qvm on pip is not up to date. Install the reference-qvm from source by
cloning this repository, cd into it, and running
pip install -r requirements.txt -e .

Hmm.. I'm using a globally installed version of python on a computing cluster.  It appears that reference-qvm (through dependancies) requires packaging 17.1, but my python install has v16.8.  The pip process fails because it tries to uninstall the globally installed version of packaging, which is owned by root.
E.g.
pip install --prefix ~/local/pip-installs/python-3.6.3/ -r requirements.txt -e .
.
.
.
.
  Found existing installation: packaging 16.8
    Uninstalling packaging-16.8:
Exception:
Traceback (most recent call last):
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/shutil.py", line 544, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/packaging' -> '/tmp/pip-61h9k522-uninstall/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/packaging'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/pip/req/req_install.py", line 754, in uninst
all
    paths_to_remove.remove(auto_confirm)
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/pip/req/req_uninstall.py", line 115, in remo
ve
    renames(path, new_path)
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/shutil.py", line 556, in move
    rmtree(src)
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/gpfs0/export/opt/anaconda3-5.0.1/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'requirements.py'

Try updating to packaging 17.1: https://anaconda.org/conda-forge/packaging

@astrophys not sure when that particular package was added as a dep.  I can hunt around to look for where it is referenced.  In the meantime you can probably download anaconda (which installs a non-rooted version of python in your local directory).  As long as your bashrc or bash_profile is pointing towards the local python you can updated packages at will.
I will continue to hunt around for when this package is used.

@vtomole going with a fresh install of anaconda python 3.6.5 and the installing directly from the git hub repo (as suggested above), seemed to provide a work around solution.  Thanks.

@astrophys You are welcome!


Update repositories.toml

Added the Hedging of financial derivative project.
ℹ Repository information
A Hedging is a market neutral trading strategy enabling traders to profit from virtually any market conditions: uptrend, downtrend, or sideways movement. This strategy is categorized as a statistical arbitrage and convergence trading strategy.
To explain what our strategy exactly does and how is it working, we have taken the example of the Indian Market, we have taken a few securities over a specified time interval. Then we have found all the cointegrated pairs of stocks from all the possible pairs of securities by considering all the stocks having p-value less than a certain cut-off. For further analysis, we have identified the security pair with minimum p-value.
Next we have calculated the spread of the two series. In order to actually calculate the spread, we use a linear regression to get the coefficient for the linear combination to construct between our two securities. Since, the absolute spread isn't very useful in statistical terms. It is more helpful to normalize our signal by treating it as a z-score. A Z-score is nothing but a numerical measurement that describes a value's relationship to the mean of a group of values. Z-score is measured in terms of standard deviations from the mean.
Next we define our strategy and generate the trading signals during backtesting on another time period which we have defined-
Go "Long" the spread whenever the z-score is below -1.0
Go "Short" the spread when the z-score is above 1.0
Exit positions when the z-score approaches zero 4)Next we have created calculated the returns of our portfolio on the basis of strategy
The repository has:

[x] At least three issues with the good first issue label.
[x] At least 10 contributors.
[x] Detailed setup instructions for the project.
[x] CONTRIBUTING.md
[x] Actively maintained.

Hey @Akshat111111 — the repository does not meet the criterion of at least 10 contributors. Please try adding it to this project at a later time. Thanks!


Device simulator dependence on recursive description snapshots

I have found that recursive description snapshots produce different outcomes when tests run on different iOS simulators. Is that possible to make them repetitive no matter which device simulator I use?
I didn't find any solution on that yet. Maybe adding some configuration like on the image (i.e. .image(on: .iPhoneX)) will be useful here as well?

I have noticed something that maybe related to this issue, I still have limited understanding of when the declaration of layout constraints took place, and when the view is being snapshot, which could affect the result of the snapshot output.
I noticed that if I declare a constraint that's dependent on a view.bounds property at runtime, this will cause the snapshot output to be dependent on which Simulator with the corresponding screen size that we're running on.
For example, the following code will only produce a snapshot output that took the value of view.bounds.width in reference to the simulator it's running on, instead of the ViewConfig of the snapshot image. As a result the width is smaller when it's displayed on a larger view configuration.
override func viewDidLoad() {
    super.viewDidLoad()

    let offset: CGFloat = 20.0
    textField.widthAnchor.constraint(equalTo: view.bounds.width - 2*offset)

    // This text field is off screen initially and it doesn't have a right anchor,
    // but instead constrained by its width anchor
    textField.leftAnchor.constraint(equalTo: view.leftAnchor, constant: view.bounds.width + offset).isActive = true
}

I realized this is an unconventional layout declaration, so let me visualize it:

The initial layout
offscreen    visible     offscreen
[View] - [TextField]

On user action, text field will be animated to the left, and then layout will be updated
offscreen    visible       offscreen
[View] - [TextField]

I'm not sure if the I should have find a more stable declaration of width anchor of the text field, or whether should I dig deeper into the Snapshot testing process, to resolve this.

Also, I have tried to improve the layout constraints declaration, to rely on the widthAnchor of the view of a view controller. Which surprisingly works at runtime. I thought I have to set a widthAnchor explicitly to a value before using it. But, even in this case, snapshot still produced unexpected output if I update the initial layout to a runtime value of a view.bounds, and then take snapshot on different ViewImageConfig.

I discovered that to maintain consistency on various image snapshot of layout update on taking snapshot (updating means to modify the constant value of NSLayoutConstraint), a different instance of each view controller is needed.
For example, instead of this:
assertSnapshot(matching: vc, as: .image(on: .iPhoneSe))
assertSnapshot(matching: vc, as: .image(on: .iPhone8))
assertSnapshot(matching: vc, as: .image(on: .iPhoneX))

A separate instance need to be used:

func viewController() -> UIViewController {
    return UIViewController()
}

assertSnapshot(matching: viewController(), as: .image(on: .iPhoneSe))
assertSnapshot(matching: viewController(), as: .image(on: .iPhone8))
assertSnapshot(matching: viewController(), as: .image(on: .iPhoneX))


Refactor: Move all logging into middleware

We should move all logging into middleware.
This Issue is blocked on #18, #16 and #6.

Also blocked on #25

Closed by #31


Support of method return values and exceptions

If a method in the server return a value or simply throws an exception, the client will not be aware of the result
You should manage message type 3 and send them to the callback methods of the promise

fixed.


Analytics Webhook

Hi,
It will be great if you add and analytics webhook to the AppCenter
Thanks,
Jassim

@jrahma Can you please explain how you'd use this? What triggers/events are you expecting that would fire this web hook?

@jwargo this will allow to have own dashboards, notifications and alerts instantly. It should have the details of users and sessions in rows

@jrahma I'm sorry, but I don't understand how this relates to webhooks. I'm asking what events in Analytics you want to trigger a webhook.


test: use ApiContext::createFromUrl in E2E tests

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


[Bug]: Sticky Table of Contents active link state is inconsistent

What happened?
When using the 'sticky' variant of the table of contents component, the 'active' state of the link to show which section is in view, doesn't always work. It seems to only work when the subsequent section is also in the viewport.
What was the expected outcome?
The active state should always show which section you are reading
How is this problem impacting your service?
Affecting DAC proto for May 2022
What list of steps can be followed to reproduce the bug or issue?

Build a page using the sticky variant of the Table of Contents component
Create sections that are longer than the viewport
Observe the link list for active state bold styling when the relating section hits the top of the viewport.

Example
https://user-images.githubusercontent.com/43346934/161253018-ae38a6aa-541e-4e37-bd06-d8061aa07905.mp4
What version of the ONS Design System are you using?
47.0.0
What devices are you seeing the problem on?

[X] Desktop
[ ] Mobile
[ ] Tablet

Device details
No response
What operating systems are you seeing the problem on?

[ ] Windows
[X] macOS
[ ] Android
[ ] iOS
[ ] Other

Operating system details
No response
What browsers are you seeing the problem on?

[ ] Firefox
[X] Chrome
[ ] Safari
[ ] Microsoft Edge
[ ] Microsoft Internet Explorer
[ ] Other

Browser details
No response

With this one I think it might be worth fetching all heading elements with document.querySelector and then enumerating over them in reverse order to pick the one closest to the top pixel of the view area.
It might be necessary to cap the rate at which this takes place for performance reasons... perhaps setTimeout for 100 if a timeout hasn't already been scheduled.


Code re-organizations prepping for KCFG refactors

Part of #471, #413
As I was working towards the goal of better BMC prover visualization, it's clear we need:

Proof specific viewers,
Fixing the BMC checker off-by-one-error, and
Trimming of the structure stored in the KCFG (removing the notion of target, expanded, and stuck).

This makes the first least controversial steps towards those changes, including:

Method KCFG.replace_node now replaces the node in place, so that the node identifier doesn't change. This is possible now because the node identifier is not tied to the contents of the node, and makes tracking things like init/target/terminal/bounded sets externally in the *Proof classes more feasible even if nodes are being simplified for example.
The notion of a unique target and init is added to the APRProof class, in preparation for removing the target notion from KCFG.

Add fields target and init to APRProof state, which explicitly call out the unique init/target states.
Move KCFGExplore.target_subsume => APRProver._check_subsumed, which makes use of the explicitly called out init/target states.
Moves KCFG.path_constraints => APRProof.path_constraints, because it relies on a unique target node.

The notion of stuck nodes is moved into APRProof state as terminal nodes.

The field _terminal_nodes is added to APRProof state.
APRProver._check_terminal directly manipulates this set instead of stuck/expanded set.
The notion of APRProof.terminal is added, which returns the set of nodes in the APRProof._terminal_nodes set.
The notion of APRProof.pending  is added, which returns the set of KCFG.frontier nodes that are not in the proofs terminal_nodes set.
The APRProof.terminal and APRProof.pending sets are used for determining APRProof.status.
The APRBMCProof.bounded set is added, which similarly tracks the bounded set of nodes directly. This is used in computing APRBMCProof.pending and APRBMCProof.status.

Methods APRProof.from_claim and APRBMCProof.from_claim_with_bmc_depth are added for reading these proof types directly from claims (and tests are updated appropriately).
Method APRBMCProof.dict is refactored to use APRProof.dict directly first, then add the needed information to it.
Methods KCFG.node_attrs => KCFGShow.node_attrs and KCFG.node_short_info => KCFGShow.node_short_info are moved to localize them to use site.
Adds Proof.json as shortcut for json.dumps(Proof.dict).
For each of the integration/kcfg tests that do APRProofs, adds logging at info level for the generated KCFG, to make debugging them easier by just throwing a flag.

The notion of stuck nodes is moved into APRProof state as terminal nodes

Do you also plan to remove stuck from KCFG? It is reachability information that I think should be stored there. Proof classes can interpret that information accordingly.

The notion of a unique target and init is added to the APRProof class, in preparation for removing the target notion from KCFG.

Subsumption into the target will still be retained in the KCFG though, right?

After discussion, we'll be removing the notion of expanded, in favor of an explicit stuck set (to mark that the backend has attempted symbolic execution from that node and it will not be able to make progress). But target will be removed.


The drawing coordinates origin point is different depending the target

Hello,
while drawing on a polygon by using the graphics API (on a Sprite) the origin of the "draw" is inconsistent between platforms when the first action triggered is lineTo.
For example, in html5 and neko the first point on the circuit will be set to the first given one by "lineTo" but on the iOS target the starting point is preset to the already given coordinates x, y of the sprite allocating the graphics.
That means that a regular pentagon drawn inside of a circle which only gives instructions lineTo to the following the vertices, will work as expected in Neko, Flash and html5 but in iOS will create a funny irregular hexagon having an additional vertex in the coordinates of the coordinates x and y of the sprite.
A workaround is possible by just setting up the origin using "moveTo" to the first point.
An example showing the issue is given here; in order to show the issue remove the line 15 which has the moveTo workaround.

Are you using OpenFL legacy on iOS? Perhaps that is the difference? You could try #if openfl_legacy around the workaround, and see if that's the only case that needs it

I can't compile the app unless I activate the legacy flag (independently of the workaround):
haxelib run lime test project.xml ios -j16 -simulator
.
.
.
** BUILD SUCCEEDED **
Launch app on "com.apple.CoreSimulator.SimDeviceType.iPhone-6, 9.0"
2015-09-28 20:54:16.287 ios-sim[31073:6601106] Timeout: 60.000000 second(s)
2015-09-28 20:54:16.346 ios-sim[31073:6601106] [MT] DVTAssertions: Warning in /Library/Caches/com.apple.xbs/Sources/DVTFrameworks/DVTFrameworks-8201/DVTFoundation/PlugInArchitecture/PlugInManager/DVTPlugInManager.m:257
Details:  Requested but did not find extension point with identifier Xcode.DVTFoundation.DevicePlatformMapping
Object:   <DVTPlugInManager: 0x7f9753440ce0>
Method:   -extensionPointWithIdentifier:
Thread:   <NSThread: 0x7f9753410990>{number = 1, name = main}
Please file a bug at http://bugreport.apple.com with this warning message and any useful information you can provide.
Sep 28 20:54:20  Graphics[31143] : CGContextSaveGState: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Sep 28 20:54:20  Graphics[31143] : CGContextTranslateCTM: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Sep 28 20:54:20  Graphics[31143] : CGContextRestoreGState: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Primitive not found : lime_application_create__1
Primitive not found : lime_application_event_manager_register__2
Primitive not found : lime_application_exec__1
Primitive not found : lime_application_set_frame_rate__2
Primitive not found : lime_gamepad_event_manager_register__2
Primitive not found : lime_key_event_manager_register__2
Primitive not found : lime_mouse_event_manager_register__2
Primitive not found : lime_render_event_manager_register__2
Primitive not found : lime_text_event_manager_register__2
Primitive not found : lime_touch_event_manager_register__2
Primitive not found : lime_window_event_manager_register__2
Primitive not found : lime_mouse_set_cursor__1
Primitive not found : lime_renderer_create__1
Primitive not found : lime_renderer_flip__1
Primitive not found : lime_renderer_get_type__1
Primitive not found : lime_renderer_lock__1
Primitive not found : lime_renderer_make_current__1
Primitive not found : lime_renderer_unlock__1
Primitive not found : lime_window_close__1
Primitive not found : lime_window_create__5
Primitive not found : lime_window_get_height__1
Primitive not found : lime_window_get_id__1
Primitive not found : lime_window_get_width__1
Primitive not found : lime_window_get_x__1
Primitive not found : lime_window_get_y__1
Primitive not found : lime_window_set_enable_text_events__2
Primitive not found : lime_alc_close_device__1
Primitive not found : lime_alc_create_context__2
Primitive not found : lime_alc_destroy_context__1
Primitive not found : lime_alc_get_contexts_device__1
Primitive not found : lime_alc_get_current_context__0
Primitive not found : lime_alc_make_context_current__1
Primitive not found : lime_alc_open_device__1
Primitive not found : lime_alc_process_context__1
Primitive not found : lime_cairo_arc__MULT
Primitive not found : lime_cairo_clip__1
Primitive not found : lime_cairo_close_path__1
Primitive not found : lime_cairo_create__1
Primitive not found : lime_cairo_curve_to__MULT
Primitive not found : lime_cairo_destroy__1
Primitive not found : lime_cairo_fill__1
Primitive not found : lime_cairo_fill_preserve__1
Primitive not found : lime_cairo_get_current_point__1
Primitive not found : lime_cairo_get_target__1
Primitive not found : lime_cairo_has_current_point__1
Primitive not found : lime_cairo_identity_matrix__1
Primitive not found : lime_cairo_in_fill__3
Primitive not found : lime_cairo_in_stroke__3
Primitive not found : lime_cairo_line_to__3
Primitive not found : lime_cairo_move_to__3
Primitive not found : lime_cairo_new_path__1
Primitive not found : lime_cairo_paint__1
Primitive not found : lime_cairo_paint_with_alpha__2
Primitive not found : lime_cairo_pop_group_to_source__1
Primitive not found : lime_cairo_push_group__1
Primitive not found : lime_cairo_rectangle__5
Primitive not found : lime_cairo_restore__1
Primitive not found : lime_cairo_rotate__2
Primitive not found : lime_cairo_save__1
Primitive not found : lime_cairo_scale__3
Primitive not found : lime_cairo_set_antialias__2
Primitive not found : lime_cairo_set_fill_rule__2
Primitive not found : lime_cairo_set_font_face__2
Primitive not found : lime_cairo_set_font_options__2
Primitive not found : lime_cairo_set_font_size__2
Primitive not found : lime_cairo_set_line_cap__2
Primitive not found : lime_cairo_set_line_join__2
Primitive not found : lime_cairo_set_line_width__2
Primitive not found : lime_cairo_set_matrix__2
Primitive not found : lime_cairo_set_miter_limit__2
Primitive not found : lime_cairo_set_operator__2
Primitive not found : lime_cairo_set_source__2
Primitive not found : lime_cairo_set_source_rgb__4
Primitive not found : lime_cairo_set_source_rgba__5
Primitive not found : lime_cairo_set_source_surface__4
Primitive not found : lime_cairo_show_text__2
Primitive not found : lime_cairo_stroke__1
Primitive not found : lime_cairo_stroke_preserve__1
Primitive not found : lime_cairo_transform__2
Primitive not found : lime_cairo_translate__3
Primitive not found : lime_cairo_ft_font_face_create__2
Primitive not found : lime_cairo_font_face_destroy__1
Primitive not found : lime_cairo_font_options_create__0
Primitive not found : lime_cairo_font_options_set_antialias__2
Primitive not found : lime_cairo_font_options_set_hint_metrics__2
Primitive not found : lime_cairo_font_options_set_hint_style__2
Primitive not found : lime_bytes_get_data_pointer__1
Primitive not found : lime_cairo_image_surface_create_for_data__5
Primitive not found : lime_cairo_image_surface_get_height__1
Primitive not found : lime_cairo_image_surface_get_width__1
Primitive not found : lime_cairo_pattern_add_color_stop_rgba__MULT
Primitive not found : lime_cairo_pattern_create_for_surface__1
Primitive not found : lime_cairo_pattern_create_linear__4
Primitive not found : lime_cairo_pattern_create_radial__MULT
Primitive not found : lime_cairo_pattern_create_rgb__3
Primitive not found : lime_cairo_pattern_create_rgba__4
Primitive not found : lime_cairo_pattern_destroy__1
Primitive not found : lime_cairo_pattern_get_matrix__1
Primitive not found : lime_cairo_pattern_set_extend__2
Primitive not found : lime_cairo_pattern_set_matrix__2
Primitive not found : lime_cairo_surface_destroy__1
Primitive not found : lime_cairo_surface_flush__1
Primitive not found : lime_gl_active_texture__1
Primitive not found : lime_gl_attach_shader__2
Primitive not found : lime_gl_bind_buffer__2
Primitive not found : lime_gl_bind_framebuffer__2
Primitive not found : lime_gl_bind_texture__2
Primitive not found : lime_gl_blend_equation__1
Primitive not found : lime_gl_blend_func__2
Primitive not found : lime_gl_buffer_data__5
Primitive not found : lime_gl_buffer_sub_data__5
Primitive not found : lime_gl_clear__1
Primitive not found : lime_gl_clear_color__4
Primitive not found : lime_gl_color_mask__4
Primitive not found : lime_gl_compile_shader__1
Primitive not found : lime_gl_create_buffer__0
Primitive not found : lime_gl_create_program__0
Primitive not found : lime_gl_create_shader__1
Primitive not found : lime_gl_create_texture__0
Primitive not found : lime_gl_disable__1
Primitive not found : lime_gl_disable_vertex_attrib_array__1
Primitive not found : lime_gl_draw_arrays__3
Primitive not found : lime_gl_draw_elements__4
Primitive not found : lime_gl_enable__1
Primitive not found : lime_gl_enable_vertex_attrib_array__1
Primitive not found : lime_gl_get_attrib_location__2
Primitive not found : lime_gl_get_parameter__1
Primitive not found : lime_gl_get_program_parameter__2
Primitive not found : lime_gl_get_shader_info_log__1
Primitive not found : lime_gl_get_shader_parameter__2
Primitive not found : lime_gl_get_uniform_location__2
Primitive not found : lime_gl_link_program__1
Primitive not found : lime_gl_scissor__4
Primitive not found : lime_gl_shader_source__2
Primitive not found : lime_gl_stencil_func__3
Primitive not found : lime_gl_stencil_mask__1
Primitive not found : lime_gl_stencil_op__3
Primitive not found : lime_gl_tex_image_2d__MULT
Primitive not found : lime_gl_tex_parameteri__3
Primitive not found : lime_gl_uniform1f__2
Primitive not found : lime_gl_uniform1i__2
Primitive not found : lime_gl_uniform2f__3
Primitive not found : lime_gl_uniform4f__5
Primitive not found : lime_gl_uniform4fv__2
Primitive not found : lime_gl_uniform_matrix__4
Primitive not found : lime_gl_use_program__1
Primitive not found : lime_gl_vertex_attrib1fv__2
Primitive not found : lime_gl_vertex_attrib2fv__2
Primitive not found : lime_gl_vertex_attrib3fv__2
Primitive not found : lime_gl_vertex_attrib4fv__2
Primitive not found : lime_gl_vertex_attrib_pointer__MULT
Primitive not found : lime_gl_viewport__4
Primitive not found : lime_image_data_util_fill_rect__4
Primitive not found : lime_image_data_util_multiply_alpha__1
Primitive not found : lime_image_data_util_unmultiply_alpha__1
Primitive not found : lime_clipboard_get_text__0
Primitive not found : lime_clipboard_set_text__1
Primitive not found : lime_system_get_directory__3
Primitive not found : lime_system_get_timer__0
Primitive not found : lime_font_get_ascender__1
Primitive not found : lime_font_get_descender__1
Primitive not found : lime_font_get_family_name__1
Primitive not found : lime_font_get_units_per_em__1
Primitive not found : lime_font_load__1
Primitive not found : lime_text_layout_create__3
Primitive not found : lime_text_layout_position__5
Primitive not found : lime_gamepad_get_device_name__1
Primitive not found : lime_bytes_read_file__1
Error Null Function Pointer

Could you check if you have the latest version of OpenFL and Lime? Thanks :smile:

I think so, at least from haxelib:
openfl: 3.0.1 3.0.2 3.0.6 3.0.8 3.3.1 [3.3.3]
lime: 2.3.1 2.3.2 2.4.3 [2.6.2]

we're at 3.3.6 for openfl, and 2.6.6 for lime :)

Wow, thank you for the fast answer. I just updated to the latest versions, and unfortunately something didn't go fine...

Without the workaround I can compile the example but the mentioned bug is still there while using legacy; see screenshot attached:

Now I can compile as well without use -Dlegacy, and the polygons are well formed without the need of the mentioned workaround but something else is broken in relation to coordinates of the ngon sprite and the mouse position, it requires more debug in order to discover what is happening under the cover. A screenshot doesn't show the issue but you can observe it here by yourself by using this simple example which creates random ngons from class I linked before that manifests the issue. Now when touching the screen (in simulator) the mouse coordinates and the place where the center where the ngon appears is not matching anymore (but with legacy it works as it should); is not in the coordinates 0,0 of the created polygon (so I assume the coordinates is moved a bit down). This latest issue doesn't happen in html5 or flash, just in iOS without legacy flag.

So I think the legacy + workaround is still necessary because of the new introduced bug in the non-legacy implementation. Do you want to report it in a separate bug?

I verify that there is a separate bug for native (ios at laeast) when compiling without legacy; some additional offset is given to the Y coordianes of the sprite; I open it as a separate issue.


Update GeneratedLibraries.proj and ClientGenerator

Change in apis-client-generator:
https://github.com/mmdriley/apis-client-generator/commit/c1f5340257966671ea6c380ec6a98aee2e510b96

:shipit:


Allow setting a destination for log messages from wasm plugins

Title: Allow setting a destination for log messages from wasm plugins
Description:
Currently, logging from wasm plugins all go to envoy's app logs. However, there are use cases for separating out log files for specific domains - for example, a WAF filter will often need to have a separate audit log which is used for actioning on suspicious events.
One approach could be for the config to define log destinations
message LogDestination {
  oneof {
    string file_path = 1;

    string remote_log_service = 2;
  }
}

message VmConfig {
  ...
  map<string, LogDestination> log_destination

And the log ABI would be extended to accept a log destination string which would be mapped with that config.
[optional Relevant Links:]
https://github.com/envoyproxy/envoy/issues/22557#issuecomment-1205785100

@PiotrSikora Would be great to hear your thoughts on this.
/cc @mathetake

This looks fine.
One thing to keep in mind is that the current Proxy-Wasm ABI and various trivial integration (e.g. fwrite(stdout, ...) in C++ and logging macros in Rust) rely on having a default log destination, so it might be a good idea to add something like LogDestination default_log_destination (defaults to Envoy logs) to cover both: default output and a catch-all fallback for logging to an missing/unknown log_destination key.

@kyessenov could you reopen this as well? Thanks!

CC https://github.com/envoyproxy/envoy/issues/31148


Let rust and node handshake

The handshake does not work at the moment. See this issue in datrs/hypercore for a high-level overview.
This issue is only concerned with the handshake. I managed to track down where exactly the handshake fails:
It's always the client (initiator) that crashes. And both in Rust and in Node it happens at the same place. It happens when receiving the S token and then calling into the SymetricState and its DecryptAndHash function. There, the cipher's DecryptWithAd function is called, and this decryption fails.

In rust, the error occurs here in symmetricstate.rs when called from handshakestate.rs when receiving the S token
In node, the error occurs here in symmetric-state.js when called from handshake-state.js when receiving the S token

So - either the input parameters to the decrypt function are different, or the XChaCha20 impls differ.

I managed to get this to work :sweat_smile:
still not with the released versions, but it works!

This was fixed quite a while ago


E2E testing

This PR is for CI about the new Juno e2e test.
Close #317

@the-frey, could you take a look on this PR ? This PR is a bit big so need your review !

Okay, this is a big PR. Will take me a little while to digest haha.
Is there anything in here you're particularly worried about or think is spicy?

oh crap

🤦 I will look later and shout if there is something of concern.
Hopefully as it is just testing scaffolding there is nothing that could be an issue, but it is a big PR.


User "system:serviceaccount:dsaas-preview:platformdeployerbot" cannot "patch" "services" with name "bayesian-pgbouncer" in project "bayesian-preview"

Is there a different service account that I could use to deploy to bayesian-preview project on rh-idev?
+ saasherder --environment staging update hash fabric8-analytics-pgbouncer 7fa39b22d4709f6832a9a3c5c82c3c923a0d2e37
INFO:saasherder.saasherder:Current context: bay
INFO:saasherder.saasherder:Service: fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Downloading: https://raw.githubusercontent.com/fabric8-analytics/fabric8-analytics-pgbouncer/7fa39b22d4709f6832a9a3c5c82c3c923a0d2e37/openshift/template.yaml
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com
INFO:saasherder.saasherder:Services written to file bay-services/pgbouncer.yaml.
+ saasherder --environment staging pull fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Current context: bay
INFO:saasherder.saasherder:Service: fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Downloading: https://raw.githubusercontent.com/fabric8-analytics/fabric8-analytics-pgbouncer/7fa39b22d4709f6832a9a3c5c82c3c923a0d2e37/openshift/template.yaml
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com
INFO:saasherder.saasherder:Template written to bay-templates/fabric8-analytics-pgbouncer.yaml
+ saasherder --environment staging template --output-dir fabric8-analytics-pgbouncer-processed tag fabric8-analytics-pgbouncer
INFO:saasherder.saasherder:Current context: bay
INFO:saasherder.saasherder:oc process  --output yaml -f bay-templates/fabric8-analytics-pgbouncer.yaml IMAGE_TAG=7fa39b2 > fabric8-analytics-pgbouncer-processed/fabric8-analytics-pgbouncer.yaml
+ oc apply -f fabric8-analytics-pgbouncer-processed/fabric8-analytics-pgbouncer.yaml -n bayesian-preview
Error from server: User "system:serviceaccount:dsaas-preview:platformdeployerbot" cannot "patch" "deploymentconfigs" with name "bayesian-pgbouncer" in project "bayesian-preview"
Error from server: User "system:serviceaccount:dsaas-preview:platformdeployerbot" cannot "patch" "services" with name "bayesian-pgbouncer" in project "bayesian-preview"
Build step 'Execute shell' marked build as failure
Finished: FAILURE

Sould be fixed now

Works! Thanks 😉


Define tuples by large elimination (cont)

Continuation of #96

One disadvantage of defining tuples by large elimination is that it makes instance search less powerful on the side of Agda, i.e. Agda might think that there is overlap between instances in some cases. I don't think it's a big problem in this case as long as your instances follow the general pattern of having an instance for the empty tuple and one for the extended tuple, but in general it's something to be aware of. Probably this is still worth merging.


DCOS-19401: refactor(externalVolumes): add external volumes logic to volumes

Based on #2559 (MERGED)
This PR introduces the externalVolumes logic to the now called volumes reducers.
This is the second of three PR's which will merge external and local volumes

Alfred test this

@Poltergeist Note: this issue is not happening on #2561 👏

Yes sorry forgot to note this here. As there are two ways to handle external volumes in this PR the form will not work as expected, this is also why the test does fail. But this was impossible to avoid by keeping the PR as small as possible. So as soon as you enter the JSON editor and do something chaos breaks loose. As the parser will parse a volume and a externalVolume from one volume.

Alfred retest this please


[backport | master] devices: fix go test warning in manager_test.go

Create "class" and "config" file in temporary device BDF dir,
and remove dir created by ioutil.TempDir() when test finished.
Backport  from  kata 2.0 https://github.com/kata-containers/kata-containers/pull/747
Fixes: #2967
Signed-off-by: zhanghj<EMAIL_ADDRESS>

@dgibson, mind to take a look at this one?

/test

/test

rebase to latest master

/test


[META] Leverage protobuf for serializing select node-to-node objects

Please describe the end goal of this project
Provide a framework and migrate some key node-to-node requests to serialize via protobuf rather than the current native stream writing (i.e. writeTo(StreamOutput out)). Starting with node-to-node transport messages maintains api compatibility while introducing protobuf implementations which have several benefits:

Out of the box backwards compatibility/versioning
Speculative performance enhancements in some areas
Supports future gRPC work

Supporting References

Search API POC - flame graph + OSB numbers
Issue for API leveraging of gRPC + protobuf
gRPC-based Search API

Issues

[ ] Feature flag to enable protobuf transport protocol
[ ] Support handlers for non-native transport protocols
[ ] Initial protobuf implementation for types - FetchSearchResult, SearchHits, SearchHit

Related component
Search:Performance

Having spent way too much time diving into the byte-level (or bit-level if you consider 7-bit Vints) details of these protocols, I want to make sure we're focusing on the correct things here.
Protobuf isn't some magical thing that brings about 30% improvement the way the POC implemented it, and I don' think the existing POCs tested it fully vs. the existing capabilities of the transport protocol if used properly.
In summary:

I think most of the performance gain in the POC was based on reducing the number of bytes used to transmit information
Existing StreamInput/StreamOutput already contains the capability to use variable-length ints/longs, they're just underused
Broadly speaking, the existing Transport Protocol using Vint/Vlong is marginally (but likely not significant) more efficient (in bandwidth terms) than protobuf when dealing with unsigned values, and less efficient when dealing with signed values.
Broadly speaking, protobuf can be vastly more efficient in the case where there are a lot of "null" or "empty" values.

Protobuf's primary benefit is in backwards compatibility.
The performance impacts are very situation specific and shouldn't be assumed; it's possible a change in streaming to variable-length integers/longs can achieve similar gains.

Hi @dbwiddis, thanks for the feedback!

I think most of the performance gain in the POC was based on reducing the number of bytes used to transmit information

I am noticing protobuf objects I implement write more bytes to stream than the native "writeTo" serialization. Small protobuf objects are about ~20% larger but the difference shrinks as request content grows. So far I've been focusing on FetchSearchResult and it's members. If i'm understanding correctly the best use case for protobuf (only considering performance) would be something like a stats endpoint which contains a lot of ints/longs, particularly those which are not UUIDs/hashes and could be squished into a single byte representation.
As I learn more about protobuf I'm understanding any potential performance improvements just from making a change to serialization is gong to be speculative and I'll update this issue with benchmarks as I run them. Currently the concrete expected benefits would be out of the box backwards compatibility as well as providing a stepping stone for gRPC support.

Initial numbers for protobuf vs native serializers.
Run with 5 data nodes (r5.xlarge) per cluster against OS 2.16.
So far there is no apparent speedup and protobuf is as performant as the native protocol in this case. Vector workload in progress.
OSB Big 5
2.16 min distribution
|                                        90th percentile latency |                                  desc_sort_timestamp |     76.5407 |      ms |
|                                        90th percentile latency |                                                 term |     69.2465 |      ms |
|                                        90th percentile latency |                                  multi_terms-keyword |     71.7096 |      ms |
|                                        90th percentile latency |                                               scroll |      503024 |      ms |
|                                        90th percentile latency |                              query-string-on-message |     76.0508 |      ms |

FetchSearchResults as protobuf - Branch
|                                        90th percentile latency |                                  desc_sort_timestamp |     78.7184 |      ms |
|                                        90th percentile latency |                                                 term |     69.6506 |      ms |
|                                        90th percentile latency |                                  multi_terms-keyword |     69.0668 |      ms |
|                                        90th percentile latency |                                               scroll |      560605 |      ms |
|                                        90th percentile latency |                              query-string-on-message |     74.4521 |      ms |

Notes

Several FetchSearchResults fields could be further decomposed into protobuf. For example SearchHits.collapseFields is still encoded with it’s native writeTo implementation.
Initial POC work (diff viewable here) created protobuf exclusive copies of structures within the search path. This may have benefited performance by allowing inlining or reducing vtable lookups when handling transport layer responses. An interesting read on inlining potentially relevant to our use of  writeTo  in our native serialization.
Search results payload may not be large enough to see any performance difference in the above workloads. Workloads with larger search results may show a more substantial performance difference.
As noted above a key area where we expect benefit from protobuf is automatic variable length encoding for numeric values. FetchSearchResult is not particularly suited to this case. Regardless changes to the native protocol would provide the same benefit.
This implementation deserializes the message immediately. With a slightly different implementation some message types could benefit from protobuf's lazy loading functioncality.

Let's keep in mind that while node-to-node we may not see as much benefit, client-to-server switching from JSON to protobuf should. There's also a significant improvement in developer experience if we can replace the native implementation with protobuf.

Some additional benchmarks for a vector search workload.
Again no noticeable difference in performance.
5 data nodes (r5.xlarge) per cluster. 10000 queries against sift-128-euclidean.hdf5 data set.
FetchSearchResults as Protobuf
|                                                 Min Throughput |         prod-queries |        1.82 |  ops/s |
|                                                Mean Throughput |         prod-queries |       13.69 |  ops/s |
|                                              Median Throughput |         prod-queries |       13.84 |  ops/s |
|                                                 Max Throughput |         prod-queries |       13.93 |  ops/s |
|                                        50th percentile latency |         prod-queries |     70.1863 |     ms |
|                                        90th percentile latency |         prod-queries |     71.6104 |     ms |
|                                        99th percentile latency |         prod-queries |     73.9335 |     ms |
|                                      99.9th percentile latency |         prod-queries |     79.8549 |     ms |
|                                     99.99th percentile latency |         prod-queries |      273.26 |     ms |
|                                       100th percentile latency |         prod-queries |      546.72 |     ms |
|                                   50th percentile service time |         prod-queries |     70.1863 |     ms |
|                                   90th percentile service time |         prod-queries |     71.6104 |     ms |
|                                   99th percentile service time |         prod-queries |     73.9335 |     ms |
|                                 99.9th percentile service time |         prod-queries |     79.8549 |     ms |
|                                99.99th percentile service time |         prod-queries |      273.26 |     ms |
|                                  100th percentile service time |         prod-queries |      546.72 |     ms |
|                                                     error rate |         prod-queries |           0 |      % |
|                                                  Mean recall@k |         prod-queries |        0.97 |        |
|                                                  Mean recall@1 |         prod-queries |        0.99 |        |

2.16 Native serialization
|                                                 Min Throughput |         prod-queries |        1.63 |  ops/s |
|                                                Mean Throughput |         prod-queries |       13.49 |  ops/s |
|                                              Median Throughput |         prod-queries |       13.73 |  ops/s |
|                                                 Max Throughput |         prod-queries |       13.84 |  ops/s |
|                                        50th percentile latency |         prod-queries |      70.525 |     ms |
|                                        90th percentile latency |         prod-queries |     71.8059 |     ms |
|                                        99th percentile latency |         prod-queries |     77.1621 |     ms |
|                                      99.9th percentile latency |         prod-queries |      83.521 |     ms |
|                                     99.99th percentile latency |         prod-queries |     446.145 |     ms |
|                                       100th percentile latency |         prod-queries |     610.186 |     ms |
|                                   50th percentile service time |         prod-queries |      70.525 |     ms |
|                                   90th percentile service time |         prod-queries |     71.8059 |     ms |
|                                   99th percentile service time |         prod-queries |     77.1621 |     ms |
|                                 99.9th percentile service time |         prod-queries |      83.521 |     ms |
|                                99.99th percentile service time |         prod-queries |     446.145 |     ms |
|                                  100th percentile service time |         prod-queries |     610.186 |     ms |
|                                                     error rate |         prod-queries |           0 |      % |
|                                                  Mean recall@k |         prod-queries |        0.97 |        |
|                                                  Mean recall@1 |         prod-queries |        0.99 |        |

@finnegancarroll can we run vector search workload with client-to-server switching? Thanks.

Experimenting with a previous POC with diff here: https://github.com/finnegancarroll/OpenSearch/pull/2/files
Benchmarking locally with 5 nodes in separate docker containers with 20% big5 workload.
POC diff:
|                                                Mean Throughput | default |        2.01 |  ops/s |
|                                              Median Throughput | default |        2.01 |  ops/s |
|                                                 Max Throughput | default |        2.01 |  ops/s |
|                                        50th percentile latency | default |      5.2044 |     ms |
|                                        90th percentile latency | default |     5.78371 |     ms |
|                                        99th percentile latency | default |     6.11664 |     ms |
|                                       100th percentile latency | default |     6.24282 |     ms |
|                                   50th percentile service time | default |     3.73635 |     ms |
|                                   90th percentile service time | default |     4.09466 |     ms |
|                                   99th percentile service time | default |     4.55763 |     ms |
|                                  100th percentile service time | default |     4.74975 |     ms |
|                                                     error rate | default |           0 |      % |
|                                                 Min Throughput |   range |        2.01 |  ops/s |
|                                                Mean Throughput |   range |        2.01 |  ops/s |
|                                              Median Throughput |   range |        2.01 |  ops/s |
|                                                 Max Throughput |   range |        2.01 |  ops/s |
|                                        50th percentile latency |   range |     33.3813 |     ms |
|                                        90th percentile latency |   range |     35.1394 |     ms |
|                                        99th percentile latency |   range |     38.2579 |     ms |
|                                       100th percentile latency |   range |     39.8031 |     ms |
|                                   50th percentile service time |   range |     31.8209 |     ms |
|                                   90th percentile service time |   range |     33.8545 |     ms |
|                                   99th percentile service time |   range |     36.5908 |     ms |
|                                  100th percentile service time |   range |     38.0003 |     ms |

OpenSearch 2.17:
|                                                 Min Throughput | default |           2 |  ops/s |
|                                                Mean Throughput | default |           2 |  ops/s |
|                                              Median Throughput | default |           2 |  ops/s |
|                                                 Max Throughput | default |           2 |  ops/s |
|                                        50th percentile latency | default |     7.04535 |     ms |
|                                        90th percentile latency | default |     7.87684 |     ms |
|                                        99th percentile latency | default |     11.5237 |     ms |
|                                       100th percentile latency | default |     17.7137 |     ms |
|                                   50th percentile service time | default |     5.49358 |     ms |
|                                   90th percentile service time | default |     6.13559 |     ms |
|                                   99th percentile service time | default |     10.3286 |     ms |
|                                  100th percentile service time | default |     16.1667 |     ms |
|                                                     error rate | default |           0 |      % |
|                                                 Min Throughput |   range |           2 |  ops/s |
|                                                Mean Throughput |   range |           2 |  ops/s |
|                                              Median Throughput |   range |           2 |  ops/s |
|                                                 Max Throughput |   range |        2.01 |  ops/s |
|                                        50th percentile latency |   range |     15.2952 |     ms |
|                                        90th percentile latency |   range |     15.9011 |     ms |
|                                        99th percentile latency |   range |     20.6263 |     ms |
|                                       100th percentile latency |   range |     24.5943 |     ms |
|                                   50th percentile service time |   range |      13.646 |     ms |
|                                   90th percentile service time |   range |     14.3388 |     ms |
|                                   99th percentile service time |   range |     19.2499 |     ms |
|                                  100th percentile service time |   range |     23.1727 |     ms |
|                                                     error rate |   range |           0 |      % |

Protobuf increases our payload size around 5% based on term queries run on the big5 dataset. On average the following query run on the big5 dataset produces a ~9.5mb response with a native serializer and ~10mb with protobuf.
curl -X POST "http://localhost:9200/_search" \
-H "Content-Type: application/json" \
-d '{
  "size": 10000,
  "query": {
    "term": {
      "log.file.path": {
        "value": "/var/log/messages/birdknight"
      }
    }
  }
}'

Without trying to deconstruct the protobuf binary format one explanation could be protobuf losses some space efficiency by requiring each field  in a protobuf object have its position marked with a "field number". In contrast the OpenSearch native node-to-node protocol simply expects to read fields in the exact pre-determined order in which they are written.

Some micro benchmarks with flame graphs comparing protobuf serialization to native transport & client/server JSON serialization. Test branch can be found here: https://github.com/finnegancarroll/OpenSearch/tree/serde-micro-bench
Note: Test data is generated with SearchHitsTests::createTestItem so the content is randomized and may not accurately reflect real use cases.
While microbenchmarks are not particularly reliable this does seem to suggest protobuf is more performant in both cases. Much more so on client/server than transport layer. Serialization is likely not a large enough slice of the total latency to make an impact in OSB benchmarks.
protoWriteTo    - SearchHitsProtobufBenchmark.writeToBench  avgt    4  3.460  ms/op
nativeWriteTo   - SearchHitsProtobufBenchmark.writeToBench  avgt    2  5.991  ms/op
protoXCont      - SearchHitsProtobufBenchmark.toXContBench  avgt    2  19.393 ms/op
nativeXCont     - SearchHitsProtobufBenchmark.toXContBench  avgt    2  46.680 ms/op

flame.zip

End to end benchmarks with protobuf on transport layer as well as the REST response. Environment is identical to the previous tests here (with 2.17 as baseline): https://github.com/opensearch-project/OpenSearch/issues/15308#issuecomment-2334816295
|                                                 Min Throughput |         prod-queries |        1.48 |  ops/s |
|                                                Mean Throughput |         prod-queries |       13.57 |  ops/s |
|                                              Median Throughput |         prod-queries |       13.76 |  ops/s |
|                                                 Max Throughput |         prod-queries |       13.87 |  ops/s |
|                                        50th percentile latency |         prod-queries |     70.0255 |     ms |
|                                        90th percentile latency |         prod-queries |      70.999 |     ms |
|                                        99th percentile latency |         prod-queries |     73.3903 |     ms |
|                                      99.9th percentile latency |         prod-queries |     79.4551 |     ms |
|                                     99.99th percentile latency |         prod-queries |     457.676 |     ms |
|                                       100th percentile latency |         prod-queries |     673.036 |     ms |
|                                   50th percentile service time |         prod-queries |     70.0255 |     ms |
|                                   90th percentile service time |         prod-queries |      70.999 |     ms |
|                                   99th percentile service time |         prod-queries |     73.3903 |     ms |
|                                 99.9th percentile service time |         prod-queries |     79.4551 |     ms |
|                                99.99th percentile service time |         prod-queries |     457.676 |     ms |
|                                  100th percentile service time |         prod-queries |     673.036 |     ms |

The change in serialization does not appear to impact latencies in this case. Generated some flame graphs using OS 2.17 on a local 3 node cluster for this workload to estimate how much serialization impacts latency.
Examining the coordinator node for the cluster:
FetchSearchResults.<init> - 1.27 % of cpu w/ 61 samples
SearchHits.toXContent - 1.27% w/ 61 samples

On average REST response from the coordinator is a relatively small ~13kb.
VecWorkload-2.17.zip

Hi @reta, i've cleaned up SearchHitsProtobufBenchmark to make it easier to reproduce. To provide the micro benchmarks with test items i've added a small unit test which generates and writes instances of 'SearchHits' to a temporary directory. The whole micro benchmark test suite can be run with the following two commands:
./gradlew server:test --tests "org.opensearch.search.SearchHitsTests.testMicroBenchmarkHackGenerateTestFiles" -Dtests.security.manager=false
./gradlew -p benchmarks run --args 'SearchHitsProtobufBenchmark'

I agree it is surprising that SearchHitsProtobufBenchmark.writeToNativeBench is no faster than writing the protobuf object in this case, although they are very close. Messing with the test params and changing machines I see similar results. With more test files and iterations:
SearchHitsProtobufBenchmark.toXContNativeBench  avgt       26.244          ms/op
SearchHitsProtobufBenchmark.toXContProtoBench   avgt       13.193          ms/op
SearchHitsProtobufBenchmark.writeToNativeBench  avgt       16.033          ms/op
SearchHitsProtobufBenchmark.writeToProtoBench   avgt       12.170          ms/op

Looking through the flame graphs of SearchHitsProtobufBenchmark.writeToProtoBench and SearchHitsProtobufBenchmark.writeToNativeBench i'm wondering if the difference comes down to how SearchHit.documentFields and SearchHit.metaFields are handled. SearchHitsProtobufBenchmark.writeToNativeBench spends almost 40% of its time in StreamOutput::writeGenericValue but I don't see the same kind of time spent writing to stream in the protobuf implementation.

Thanks @finnegancarroll

SearchHitsProtobufBenchmark.writeToNativeBench spends almost 40% of its time in StreamOutput::writeGenericValue but I don't see the same kind of time spent writing to stream in the protobuf implementation.

Very true, I came to the same conclusions, I will try to briefly look what could be the issue there (without spending too much time). Thanks a lot for cleaning up the benchmarks!

While investigating areas we might expect to benefit from protobuf I spent some time reproducing this previous POC: https://github.com/opensearch-project/OpenSearch/issues/10684#issuecomment-1876077885
Below benchmarks were run locally with 5 nodes, gradle run, and the nyc_taxis workload which is as close an environment as I can manage to the previous POC benchmarks. There is a noticeable difference in latency between these results and the original OSB numbers which could be attributed to using more of the nyc_taxis dataset or a differences in hardware.
POC branch
OSB-poc-search-protobuf.zip
./poc-search-protobufDefault1.txt:|                                        90th percentile latency | default |     12.0153 |     ms |
./poc-search-protobufDefault2.txt:|                                        90th percentile latency | default |     10.2515 |     ms |
./poc-search-protobufDefault3.txt:|                                        90th percentile latency | default |     9.61354 |     ms |
./poc-search-protobufDefault4.txt:|                                        90th percentile latency | default |     9.62332 |     ms |

./poc-search-protobufRange1.txt:|                                        90th percentile latency |  range |      240.02 |     ms |
./poc-search-protobufRange2.txt:|                                        90th percentile latency |  range |     236.208 |     ms |
./poc-search-protobufRange3.txt:|                                        90th percentile latency |  range |     240.254 |     ms |
./poc-search-protobufRange4.txt:|                                        90th percentile latency |  range |     231.929 |     ms |

Head of main for the above POC branch (commit d916f9c1027f5b2ccff971a66921b7c26db1688f)
OSB-d916f9c.zip
./d916f9cRange2.txt:|                                        90th percentile latency |  range |     236.776 |     ms |
./d916f9cRange1.txt:|                                        90th percentile latency |  range |     243.967 |     ms |

./d916f9cDefault2.txt:|                                        90th percentile latency | default |     9.56527 |     ms |
./d916f9cDefault1.txt:|                                        90th percentile latency | default |     7.67412 |     ms |

Inspecting flame graphs for the poc-search-protobuf I see nearly all cpu cycles are consumed by lucene processing the search and little time is spent on serialization.
poc-search-protobuf5NodeClusterDefaultFlameGraphs.zip
poc-search-protobuf5NodeClusterRangeFlameGraphs.zip

@finnegancarroll This is great seeing benchmarking and deep diving into the details here!

Without trying to deconstruct the protobuf binary format one explanation could be protobuf losses some space efficiency by requiring each field in a protobuf object have its position marked with a "field number". In contrast the OpenSearch native node-to-node protocol simply expects to read fields in the exact pre-determined order in which they are written.

Having deconstructed the protobuf binary format, I'll confirm this observation. :)
In short:  protobuf uses a byte for each field that exists (type and index info) and 0 bytes for nonexistent fields.  The binary transport protocol doesn't mark each field but assumes all fields are present in a specific order, and requires the use of a byte for a boolean "optional".   So the takeaway from a "bandwidth" perspective is that if you tend to have a lot of optional information, you can save space with protobuf.
But as your tests seem to show, this marginal difference in bandwidth seems to be less significant, and maybe not the thing to focus on....

Thanks @finnegancarroll , my apologies for the delay, just completed may part. I didn't spend too much time on native serialization but it looks like (with the benchmarks) StreamOutput::getGenericType (that is being used by metaFields and documentFields) is the bottleneck. I think we could use some clever tricks to make it faster for collections (one such example would be to capture the type of the first element in the collection since I suspect those are homogeneous but have different types). I could spend more time on that if you think it is beneficial.

But as your tests seem to show, this marginal difference in bandwidth seems to be less significant, and maybe not the thing to focus on....

This is a very valid point but we also should not forget the maintenance costs. Changes in native protocol are difficult now and incur a burden on everyone (update main, backport + change version, forward port to main, ...). Protobuf takes care of that (as far as schema is properly evolved), I think that would bring more benefits than marginal latency improvements.

Hi @reta, I spent some time exploring your above suggestion and I'm seeing benefits but only some limited artificial cases for the moment. StreamOutput::getGenericType looks to inline getGenericType and getWriter but running benchmarks with the below gives about ~20% of our time is spent on StreamOutput::getGenericType.
./gradlew -p benchmarks run --args ' SearchHitsProtobufBenchmark.writeToNativeBench -jvmArgs "-XX:CompileCommand=dontinline,org.opensearch.core.common.io.stream.StreamOutput::getGenericType -XX:CompileCommand=dontinline,org.opensearch.core.common.io.stream.StreamOutput::getWriter"'

(StreamOutput::getGenericType in purple)
writeGenericNoInline.zip

Stepping through with a debugger it looks like we do save on some calls to StreamOutput::getGenericType but in the case of test items taken from the unit test suite and OSB datasets i'm having difficulty finding DocumentFields that are specifically large collections.
To quickly hack in a "best case" scenario dataset I modified the unit tests such that DocumentFields could only be 100 element long integer lists. With the optimized writing implemented like this.
This gives pretty substantial speedup in microbenchmarks, especially considering the cpu now spends nearly all of it's time writing specifically DocumentFields.
// Control
SearchHitsProtobufBenchmark.writeToNativeBench  avgt    2  2205.636          ms/op
// Skip type checking for homogeneous collections
SearchHitsProtobufBenchmark.writeToNativeBench  avgt    4  658.330 ± 255.170  ms/op

Control

Skip type checking for homogeneous collections

100IntListDocFieldsFlameGraphs.zip

Thanks a lot @finnegancarroll , it looks like we are at crossroads now: should we move forward with protobuf (since the latency wins are moderate)  or (may be) move forward with optimizing the native protocol?

Additional benchmarks comparing protobuf to native protocol with the same inflated DocumentFields size as used here.
SearchHitsProtobufBenchmark.toXContNativeBench  avgt    2  1787.097          ms/op
SearchHitsProtobufBenchmark.toXContProtoBench   avgt    2   832.177          ms/op
SearchHitsProtobufBenchmark.writeToNativeBench  avgt    2   783.957          ms/op
SearchHitsProtobufBenchmark.writeToProtoBench   avgt    2   807.185          ms/op

Additional benchmarks comparing protobuf to native protocol with the same inflated DocumentFields size as used here.

Thanks @finnegancarroll , writeToNativeBench   & writeToProtoBench  are really close, I think XContent ones we could discard (at the moment) since node-to-node does not use this serialization format (only native serialization).

Thanks a lot @finnegancarroll , it looks like we are at crossroads now: should we move forward with protobuf (since the latency wins are moderate) or (may be) move forward with optimizing the native protocol?

I think we should do both. Switching to an industry standard is a long term bet on an entire community optimizing the binary protocol for us.

Closing this issue to move forward with client facing gRPC server - #16556.


Add Hyper-V fixes, and Enhanced Guest support option

Description of changes
This change makes various changes supporting NixOS usage as a Hyper-V guest.
Hyper-V Guest changes:
NixOS on Hyper-V didn't work "out of the box" for me with any WM. With some digging I found https://github.com/NixOS/nixpkgs/issues/57290#issuecomment-1956812230. I found that adding blacklistedKernelModules = [ "hyperv_fb" ]; to use hyperv_drm instead was the only thing that was needed to solve this issue.
I then added support for Hyper-V's Enhanced Guest sessions, which is needed for audio/clipboard/etc. These are hidden behind the enhancedSession option.
xrdp changes:
Since Hyper-V uses RDP for its Enhanced Guest sessions, changes were made to the xrdp.nix package to support this.
Primarily, the --port=3389 flag needed to be set to --port=vsock://-1:3389. The current implementation just supplies the --port flag, which also overrides the config file's option. Thus a code change here was needed.
See the xrdp.ini comments for more information about possible ways to set this flag.
A transportMode and address option was added to be able to set these values, as well as support other xrdp setups. If they are set to null, then it follows the previous behaviour. The transportMode values were taken from the possible options that xrdp_listen_pp handles.
See also https://github.com/neutrinolabs/xrdp/pull/1441 for information about vsock://-1:3389.
Nix's xrdp package currently only supports pulseaudio (using pulseaudio-module-xrdp). So an assertion was added with enhancedSession to ensure it's enabled. However there's a pipewire-module-xrdp that doesn't have a nix package yet? So a future change could add that pipewire support with xrdp by default.
Previous work that I've referenced:
https://github.com/NixOS/nixpkgs/pull/75581
https://github.com/NixOS/nixpkgs/issues/57290#issuecomment-1956812230
https://github.com/NixOS/nixos-hardware/pull/252
I'm not very experienced in the Nix ecosystem, I just really wanted to get it working with Hyper-V. It currently didn't work at all "out of the box". Please let me know about any issues or best practices I've missed.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Tried this with GNOME, but it seems like it requires that pipewire is enabled, which conflicts with the assertion to enable pulseaudio instead.

After we investigated a bit, seems like it just works if you use gnome-session instead of gnome-remote-desktop (or grdctl). as suggested by the wiki here https://wiki.nixos.org/wiki/Remote_Desktop#GNOME. It may be possible to modify gnome-remote-desktop to be compatible, I'm not very familiar with it myself


fix(deps): major bump sdr

What does this PR do?
sdr7
What issues does this PR fix or reference?
@W-11590389@

built these locally (SDR => STL and STL/SDR => plugin-source) without compile failures.
Looks like a result of the major version conflicts.


⚠ Bofrost.lu has degraded performance

In 959d7b1, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1380 ms

Resolved: Bofrost.lu performance has improved in 24837e9 after 31 minutes.


⚠ Bofrost.at has degraded performance

In 8ae6458, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2764 ms

Resolved: Bofrost.at performance has improved in b59fbdf.


Qualify types to avoid compile error

Add eapis::intel_x64:: qualifier to method return types to fix compile error on clang 6.0.0

@JaredWright when you get a chance, can you add some documentation for the ept code? doxygen .doxygen.txt is the command to run to check for warnings.