[RFC]: Refactor tool parsers to eliminate coding errors and allow more efficient implementations.

Motivation.
Currently the tool parsers are buggy when used and are quite messy in terms of code, especially in the implementations of extract_tool_calls_streaming. Moreover, in the long term, maintaining the entire output string in the chat streaming server and parsing the entire output over and over again for each generated token will become very expensive. This will soon become a performance bottleneck in long tool calls.
Many of the implemented tool parsers aren't carefully written either in terms of correctness nor in terms of efficiency causing a lot of issues in this repository. A complete refactor of this part of the frontend will be required sooner of later.
So now is probably the best opportunity to refactor things before more tool calling support is added.
From the architectural perspective, clearly it's should be the tool parser's job to maintain states that it needs, so if they need the complete output, they should maintain them with delta_text and delta_token_ids in their class instead of relying on the server. On the other hand, this encourages the tool parser to use custom, efficient alternative implementations to generate delta messages based on e.g. a internal JSON parser state without needing to parse the full output string every time anymore.
No capability is lost. Performance gains are good and this must be done sooner or later anyway.
No changes to user interface is required and only a few lines of modification is required for the chat server (Only remove the output caching part). The major refactoring should happen in the tool parsers. However, as stated above, we can just move the string cache into the parsers first and refactor them gradually.
Proposed Change.
This RFC is actually pretty simple so it barely falls into the criteria of 'Large Changes' but I still propose it here.

[ ] Add internal output cache to existing tool parsers and modify tool parser interface.
[ ] Fix bugs/refactor parts of the parsing logic in existing tool parsers.
[ ] Add appropriate tests for the tool parsers.

Feedback Period.
This shouldn't be hard so 1-2 weeks.
CC List.
@mgoin @simon-mo @WoosukKwon @petersalas @comaniac @SolitaryThinker @ywang96 @DarkLight1337
Any Other Things.
No response
Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

@cedonley Maybe you can provide some insights on this?

Adding @K-Mistele as well.
I’ve only looked at the streaming implementation, but I agree that coordination between the server and the tool parsers in the current implementation makes tool parsing very fragile. Also agree that refactoring would offer opportunities to optimize.
That said, there are enough commonalities between how the parsing happens that having the tool parsers extend a BaseToolParser that provides a basic state tracking and tool token parsing would be helpful in preventing each parser from being a free-for-all. In fact, this is already happening — if you look at the llama, mistral, and qwen parsers, there are whole gobs of code that are practically copy and paste, but with slight variations. When I was fixing one, I had to make practically the same fix in all the others.
On the parsers themselves, I think we should remove all of the JSON loads/dumps from the parsers. Ideally we have access to the locations in the raw tool string where each start/end token occurs and the position in the string we’ve already returned. Then you just need to return the delta from whatever that is. This also allows us to remove most of the index/rindex and regex that happens inside the tool parser streaming loop.
Agree that this isn’t a huge project. The real work beyond design is just refactoring all of the existing parsers and testing them…but given that the number of future parsers is likely to go up, I agree it’s better to do this before things get completely out of hand…

I've written a draft for this idea refactoring the hermes parser. Take a look at #11554. With a streaming json parsers it's actually easier to write correct code and improve performance.

As the original author of the tool calling features, @K-Mistele, we definitely want to hear your ideas on this as well.

Thanks for opening this! Yes it has become increasingly obvious that a more stable strategy is needed for tool parsing due to the large number of un-anticipated edge & corner cases.
To my mind, the right approach seems to be to use a finite state machine to model streaming parsing, since it's a very discrete and easy-to-model process; this keeps things relatively atomic and avoids the spaghetti code, and also provides more straightforward ways to manage failure modes.
I am not familiar with what's available for this in the python ecosystem, but something like xstate (JS ecosystem) probably be a good solution. Here's how I would break it down:
States

generating text
generating function name
generating function arguments
tool call complete
Error state - when an event happens that is not supported for the current state, this should be targeted and should result in a defined error mode e.g. an error delta (not sure if openAI supports this? but it's not possible to send a 5XX once streaming has already started)
(additional states pertinent to the model, e.g. generating reasoning tokens)

Events + transitions:

tool call start - defined on "generating text", targets "generating function name"; defined on "tool call completed" and targets "generating function name" for parallel tool calls; on other states should result in an error mode
tool call name generated - defined on "generating function function name", targets "generating function arguments"
tool call arguments completed - defined on "Generating function arguments" for when the generated arguments are fully JSON-parsable without JSON autcompletion, targets "tool call completed"

Probably there needs to be a way to support transitioning from "Tool call completed" back to "Generating text" since some models will interleave tool calls and text generation e.g. the hermes models, possibly other OSS reasoning models.
Curious to hear your feedback on this - it's been floating around in my head for a while and would love to collaborate on this if it seems like a good idea.
cc @mgoin

That said, there are enough commonalities between how the parsing happens that having the tool parsers extend a BaseToolParser that provides a basic state tracking and tool token parsing would be helpful in preventing each parser from being a free-for-all.

Indeed, I think we could construct two types of basic tool parsers:

A parser that accepts text before a BOT token and followed by a JSON tool call. More BOT tokens & tool calls could follow in parallel calls.
A parser that parses JSON wrapped by BOT tokens and EOT tokens as tool calls, and taking the rest as text response.

To my mind, the right approach seems to be to use a finite state machine to model streaming parsing, since it's a very discrete and easy-to-model process; this keeps things relatively atomic and avoids the spaghetti code, and also provides more straightforward ways to manage failure modes.

I agree, we need a reliable way to record the required states of the parser. In some cases however the language of the tool call might not be one that can be handled by a FSM(or equivalently, a regular expression), e.g. JSON. I think it's valid in these cases to use a more complex, potentially external library based parser. Most of the tool calling languages probably fall into LL(k) and can be handled with ease.
That said, I think it's a good idea to use a well tested stream parsing library like ijson to avoid hand writing some of this code.

Hello everyone,
I’m very interested in this RFC as I’ve also noticed somewhat slow performance on an in-house tool called “Benchmark” that I’m running. Occasionally, I observe that some calls take around 1200 ms to complete (using LLaMA 3.3-70B on 4 H100 GPUs), while other calls unexpectedly take over 100,000 ms to finish. From a non-streaming perspective, I don’t fully understand how we can avoid using regex to accommodate different tool response formats.
Additionally, I’d like to propose a new parser for LLaMA that improves the accuracy of tool calls. However, it’s currently just a “copy-paste” of the existing design and would likely not include the cool stuff you are mentioning about FSM etc. Should I open a PR anyway?


Allow for installing the template in the content folder (?)

Not sure if we want to support it but:
It's not possible atm to install the template inside the content folder. This will result in the following esbuild error, even though we have .flowershow specified in contentlayer config's contentDirExclude.
ready - started server on <IP_ADDRESS>:3000, url: http://localhost:3000
Error: Found 1 problems in 1 documents.

 └── Encountered unexpected errors while processing of 1 documents. This is possibly a bug in Contentlayer. Please open an issue.

     • "index.md": UnexpectedMDXError: Error: Build failed with 1 error:
     _mdx_bundler_entry_point-8f9ad85f-55d1-40f7-81d2-a6e3869c4584.mdx:0:0: error: [plugin: @mdx-js/esbuild] Error: ELOOP: too many symbolic links encountered, stat '/home/ola/Projects/lifeitself/my-site/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content/.flowershow/content'
         at Object.statSync (node:fs:1583:3)
         at walkDir (/home/ola/Projects/lifeitself/my-site/content/.flowershow/node_modules/remark-wiki-link-plus/dist/index.cjs.js:404:13)
         at /home/ola/Projects/lifeitself/my-site/content/.flowershow/node_modules/remark-wiki-link-plus/dist/index.cjs.js:385:16
         at Array.map (<anonymous>)
         at /home/ola/Projects/lifeitself/my-site/content/.flowershow/node_modules/remark-wiki-link-plus/dist/index.cjs.js:399:18
         at /home/ola/Projects/lifeitself/my-site/content/.flowershow/node_modules/remark-wiki-link-plus/dist/index.cjs.js:386:14
         at Array.reduce (<anonymous>)
         at /home/ola/Projects/lifeitself/my-site/content/.flowershow/node_modules/remark-wiki-link-plus/dist/index.cjs.js:385:16

WONTFIX / DUPLICATE. Whilst i think it is reasonable to do this i think this relates to a bigger discussion about UX and behavior of self-publishing experience. I've moved the content of this to related discussion https://github.com/flowershow/flowershow/discussions/323


Generated @hrefs

Hey, I'm new to the concepts of micro-api though they look very attractive to me. I have few questions:

Is it by design that hrefs are generated hashes instead of human-readable resource names:

"@links": {
      "user": {
          "@href": "/dXNlcnM"
      }
 }

I'm using mongodb-adapter. Is there some doc where I can check all possible schema fields or is it just type? What about unique fields and mongo indexes etc.? As I understand all fields in defineType are required by default on POST/PUT?

Is there built-in logger?

Thanks!

It's by design that the Micro API serializer emits URIs that are opaque. The purpose is to prevent hard-coded client URIs, or to extract any information from the URI. This is in line with the opacity axiom of URIs.
With regards to the MongoDB adapter, there's not yet support for unique fields and indexes yet, but such a feature would be easy to add. It's the intention that each adapter will implement the base adapter class, and have extensions that are specific to the data store. MongoDB supports many other data types beyond just the few types that are required by Fortune to implement, it's also just a matter of extending the adapter. How it would be done is by using a Symbol as the type.
There's no built-in logger, but the dispatch and http functions return promises. For example, I could have an http server like so:
import fortune from 'fortune'
import http from 'http'

const store = fortune.create()
const listener = fortune.net.http(store)
const server = http.createServer(function () {
  listener(...arguments)
  .then(response => /* log successful response */)
  .catch(response => /* log error response */)
})

store.connect().then(() => server.listen(1337))

@0x8890 thanks, your answers are helpful! I will give it a try and probably come up with additional questions if they'll appear.

@0x8890 for example news ones :)

I would like to have POST method to some /resource but with some payload data I need to go to third-party API, is app.transform the right place to make such requests?
Is there a way to versionify API? I would like to be the root endpoint /v1.

Yes, to be more precise, transformInput.
Fortune is agnostic about API versioning, although it is highly advisable that you do not do so.

If you really wanted to have a /v1 route, here's how you could do it with express:
import express from 'express'
import fortune from 'fortune'

const app = express()
const store = fortune.create({
  serializers: [ {
    type: fortune.serializers.microApi,
    options: {
      prefix: '/v1'
    }
  } ]
})
const listener = fortune.net.http(store)

app.use('/v1', (request, response, next) =>
  listener(request, response).then(next, next))

app.listen(1337)

@0x8890 nice, I was also thinking about possibility mixing fortune with express, your example looks clear :+1:


Unit converter doesn't work with bigger numbers

For input:<PHONE_NUMBER>00671016345385209
both eth unit converter and token unit converter are broken.

Do we have to have a maximum value?


[MM-15388] Add GPO files to Desktop installation directory

Adding GPO definition files to the MSI installer for installation into the Desktop application’s install directory.

This issue has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!
/cc @jasonblais @hanzei

Closing in favour of a single branch based off of release-test-msi for all MSI related updates to make further updates more manageable. New PR pending.


BUG: scipy.linalg.fractional_matrix_power behaves differently with SciPy 1.14 on macOS

Describe your issue.
scipy.linalg.fractional_matrix_power behaves differently with SciPy 1.14 on macOS (CPU: Apple M1 Max) compared to outcomes with SciPy 1.13 or  Linux.
I guess it is related to the Accelerate library introduced by 1.14 for macOS.
Reproducing Code Example
import scipy
print("scipy", scipy.__version__)
mat = [[0, -1j], [1j, 0]]
print(scipy.linalg.fractional_matrix_power(mat, 0.5))

Error message
# macOS 15.0.1, Python 3.12.7, SciPy 1.14.1
scipy 1.14.1
[[ 0.5-0.5j  0.5-0.5j]
 [-0.5+0.5j  0.5-0.5j]]

# macOS 15.0.1, Python 3.12.7, SciPy 1.13.1
scipy 1.13.1
[[ 0.5+0.5j -0.5-0.5j]
 [ 0.5+0.5j  0.5+0.5j]]

# RedHat 9.4, Python 3.12.7, SciPy 1.14.1
scipy 1.14.1
[[ 0.5+0.5j -0.5-0.5j]
 [ 0.5+0.5j  0.5+0.5j]]

# RedHat 9.4, Python 3.12.7, SciPy 1.13.1
scipy 1.13.1
[[ 0.5+0.5j -0.5-0.5j]
 [ 0.5+0.5j  0.5+0.5j]]

SciPy/NumPy/Python version and system information
1.14.1 2.1.2 sys.version_info(major=3, minor=12, micro=7, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: extraframeworks
    found: true
    include directory: unknown
    lib directory: unknown
    name: accelerate
    openblas configuration: unknown
    pc file directory: unknown
    version: unknown
  lapack:
    detection method: extraframeworks
    found: true
    include directory: unknown
    lib directory: unknown
    name: accelerate
    openblas configuration: unknown
    pc file directory: unknown
    version: unknown
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.12.0
Compilers:
  c:
    commands: cc
    linker: ld64
    name: clang
    version: 15.0.0
  c++:
    commands: c++
    linker: ld64
    name: clang
    version: 15.0.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.11
  fortran:
    commands: gfortran
    linker: ld64
    name: gcc
    version: 13.3.0
  pythran:
    include directory: ../../../../../../private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/pip-build-env-bjescfad/overlay/lib/python3.12/site-packages/pythran
    version: 0.16.1
Machine Information:
  build:
    cpu: aarch64
    endian: little
    family: aarch64
    system: darwin
  cross-compiled: false
  host:
    cpu: aarch64
    endian: little
    family: aarch64
    system: darwin
Python Information:
  path: /private/var/folders/hw/1f0gcr8d6kn9ms0_wn0_57qc0000gn/T/cibw-run-oogm4_8g/cp312-macosx_arm64/build/venv/bin/python
  version: '3.12'

@t-imamichi can you confirm a few things:

You are installing SciPy using pip/PyPI wheels, right?
You've claimed different behavior, but not necessarily incorrect behavior--are the new results incorrect or just a different way of representing a similar result? If the new results are incorrect, as might be implied by the BUG label, one might expect our testsuite to fail.

I should cc @ilayn -- are these valid permutations of the same result, or are some of the results categorically incorrect here? My linear algebra foo is not strong enough.

Yes in this case they are permutations of a square root. When an array has an eigenvalue on the negative real line, then fractional powers can fall to either side of 0, depending what that architecture decides.
For dense square roots I'd recommend using sqrtm from linalg module though probably the problem is simplified to make a point. The plan is to rewrite sqrtm such that the algorithm tries its best to stay in the real domain as much as possible. So it might give a bit more stability but the problem itself is inherently difficult to tackle in floating point number system.

@tylerjereddy Thank you for your response.

Yes, I installed scipy 1.13.1 and 1.14.1 using pip
Exactly. They are not incorrect behavior. The resulting matrices are different as I showed, but they are both root square matrices. I attach a script below to confirm it.

@ilayn Thank you for your suggestion of sqrtm. I tried it and found that it behaves same as fractional_matrix_power for this matrix. I understand the stability issue due to floating number system is not easy. So, I'm fine to close this issue because it returns a correct square root matrix though it may vary depending on blas library.
import scipy
import numpy as np

print("scipy", scipy.__version__)
mat = [[0, -1j], [1j, 0]]

sqrt = scipy.linalg.fractional_matrix_power(mat, 0.5)
print("fractional_matrix_power")
print(sqrt)
print("sqrt?", np.allclose(sqrt @ sqrt, mat), end="\n\n")
print("eigvals", np.linalg.eigvals(sqrt))

sqrt = scipy.linalg.sqrtm(mat)
print("sqrtm")
print(sqrt)
print("sqrt?", np.allclose(sqrt @ sqrt, mat), end="\n\n")
print("eigvals", np.linalg.eigvals(sqrt))

# macOS 15.0.1, Python 3.12.7, SciPy 1.14.1
scipy 1.14.1
fractional_matrix_power
[[ 0.5-0.5j  0.5-0.5j]
 [-0.5+0.5j  0.5-0.5j]]
sqrt? True
eigvals [-1.11022302e-16-1.00000000e+00j  1.00000000e+00-8.32667268e-17j]

sqrtm
[[ 0.5-0.5j  0.5-0.5j]
 [-0.5+0.5j  0.5-0.5j]]
sqrt? True
eigvals [0.-1.00000000e+00j 1.-2.77555756e-17j]

# macOS 15.0.1, Python 3.12.7, SciPy 1.13.1
scipy 1.13.1
fractional_matrix_power
[[ 0.5+0.5j -0.5-0.5j]
 [ 0.5+0.5j  0.5+0.5j]]
sqrt? True
eigvals [8.32667268e-16+1.0000000e+00j 1.00000000e+00-1.2490009e-16j]

sqrtm
[[ 0.5+0.5j -0.5-0.5j]
 [ 0.5+0.5j  0.5+0.5j]]
sqrt? True
eigvals [-5.55111512e-17+1.00000000e+00j  1.00000000e+00+4.16333634e-17j]


Dockerfile clean up

What's changed

Remove duplicate pip env install in 22.04 docker
Set default LOGURU_LEVEL to INFO; my experience running tests locally requires that this env variable be set, otherwise I see an error

Checklist

[ ] Post commit CI passes

can you post a screenshot of loguru level failing?


Migrate from tendermint-proto to cometbft-proto depdency

Work in tandem with:

https://github.com/cosmos/ibc-rs/issues/1073
https://github.com/informalsystems/hermes/issues/3820

As described in https://github.com/cosmos/ibc-rs/issues/1073#issuecomment-1934724440, there is a somewhat hacky way forward that does not need waiting for upstream protos to convert to buf.build/cometbft/cometbft:

Fix #194 by adjusting the upstream commits.
Remap the tendermint.* packages to versioned module paths in cometbft_proto using the extern_path option of prost-build.


Updating all Java versions to v21

Updating all JAVA versions to Java v21.
The main reason for this is, that Gradle does not work anymore, if you have set Gradle to v21 locally.
Pushing all Java version to 21 might be an overkill, on the other side it increases consistency across the project.

Sorry @ToBeHH, we can't accept this due to the reasons mentioned by @mnonnenmacher over here. Also see related PRs https://github.com/oss-review-toolkit/ort/pull/4948 and https://github.com/oss-review-toolkit/ort/pull/6311.


feat: :bookmark: 1.5.0

:tada: This PR is included in version 1.5.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.6.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add metadata to WorkloadEntry

(This is used to request new product features, please visit https://discuss.istio.io for questions on using Istio)
Describe the feature request
Envoy supports  metadata on LbEndpoint, but Istio's API seems has no correspending config.
This feature is needed because we are considering using Envoy's Subset Load Balancing.
Describe alternatives you've considered
No alternatives have been thought of so far
Affected product area (please put an X in all that apply)
[ ] Ambient
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context

Not sure how you register WorkloadEntry, using istio auto registration or manually by some other tools.
If it is by istio auto, you can set different subset labels with different WorkloadGroup. For manually registered, it is possible to set different labels too.

@hzxuzhonghu  manually registered. Seems WorkloadEntry labels will not transfrom to endpoint metadata.

For example, I want config endpoint metadata as below used for envoy load balancer

If you mean these metadata, why donot you specify subsets with destinationrule

If you mean these metadata, why donot you specify subsets with destinationrule

We need Envoy's Subset Load Balancing, see  https://github.com/envoyproxy/envoy/blob/main/source/docs/subset_load_balancer.md
In short, if we use destinationrule, we have to keep changing destinationrule. @hzxuzhonghu

kind of remember there was already an issue discussing this. If so, please not open duplicate issues. Move talking there

I found this issue but alreay closed: https://github.com/istio/istio/issues/33896

Even if it is not used to support Subset Load Balancing, I think it is useful to add metadata to WorkloadEntry, because Envoy has many extended features that can be implemented based on metadata. Some configurations can be patched through Istio EnvoyFilter, but  it is impossible to patch each endpoint the metadata.


Improve visual contrast in TOC headers

The existing theme makes it hard to read top-level headers in the table of contents:

This pull request changes the color to this

I did not observe other side effects from this change, but it's worth someone else combing through the pages as well to check it didn't render other page elements hard to read.

Leaving my humble opinion as I was considering a pull request for this matter I would suggest a more contrasting header color, the contrast is too subtle right now and this doesn't suggest otherwise, and it really helps the user identify separate "chapters".
Here is what I did with the previous version, locally to help me read better the docs...

@tostasqb I see what you mean about making it stand out more. One thing which has changed since that old theme is the addition of the menu bar "Products...Solutions...etc."

We noticed that the green box surrounding the search box looks a little intense and out of place now that this menu and logo serve as a page heading.
We can explore a greater use of green in headers to make things more fun, but I'm thinking we can merge this more modest pull request first.
If you want to send pull requests to experiment with colors or adjust other style issues please feel free.

@begriffs :shipit:


Function name must be callable

Guzzle version(s) affected: 6.x
Description

My PhpStorm throws an error in CurlFactory.php line 519
This is the if wrapping it:
if ($onHeaders) {
    try {
        $onHeaders($easy->response);
    } catch (\Exception $e) {
        // Associate the exception with the handle and trigger
        // a curl header write error by returning 0.
        $easy->onHeadersException = $e;
        return -1;
    }
}

The error is thrown on  $onHeaders($easy->response);.

How to reproduce

View the file with PhpStorm (I think)... I never configured anything...

Try to update to master and check if the error still there.
Also do you get an error/exception when you run your script? Or your concerns are just about PHPStorm highlighting the line above?

Concerns is a big word :) I just noticed it and thought I'd report it.
I didn't notice any errors while using it (yet)...

This is just a false positive PhpStorm warning. I fixed that in https://github.com/guzzle/guzzle/commit/49086392c15e3bfe26be148618aee64332158284 as far as I remember.
You seem to use a very old Guzzle version btw.


Using MSAL to get tokens without using MSAL to sign in

Portal capabilities for Dynamics 365 provides the ability to sign in using Azure AD B2C. This is a feature of the product over which I have limited control, as far as I know, this is feature is implemented without MSAL.
Within the context of the portal I am attempting to use MSAL to acquire a token, acquireTokenSilent.
With a user signed into the portal - using the native features of the portal and not MSAL, e.g. loginPopup - using acquireTokenSilent results in user_login_error:User login is required.
Is this a scenario that MSAL actually supports? The ability to recognise the user is signed into Azure AD B2C but not via MSAL. Is there a way this could be supported?

@sirjameswood This is a feature which is not yet supported. We have plans to add this feature but in order for msal to acquire tokens for the user not signed in through msal, we would require login_hint and domain_hint of the user.
Do you have this information available for the user? login_hint would be the preferred_username field in the id_token and domain_hint would be either consumers/organizations.

Yes I think so, we configure AD, so we should have that information available.

@sirjameswood  were you able to resolve this? You need to parse the id_token that you got during login and check if it has preferred_username or upn field. You can use these fields as login_hint. For domain_hint, if it's a AAD account, you need to pass organizations and if it's a MSA account(consumer's account like live.com , hotmail.com etc) you will pass Consumer. Hope this was helpful. I will also document this on our wiki.

@nehaagrawal Sort of, I built a seperate connection without using MSAL.

@sirjameswood were you able to resolve this? You need to parse the id_token that you got during login and check if it has preferred_username or upn field. You can use these fields as login_hint. For domain_hint, if it's a AAD account, you need to pass organizations and if it's a MSA account(consumer's account like live.com , hotmail.com etc) you will pass consumers. Hope this was helpful. I will also document this on our wiki.

Is there an example on how to do this?

consumers

Is this supported yet?

@sirjameswood were you able to resolve this? You need to parse the id_token that you got during login and check if it has preferred_username or upn field. You can use these fields as login_hint. For domain_hint, if it's a AAD account, you need to pass organizations and if it's a MSA account(consumer's account like live.com , hotmail.com etc) you will pass consumers. Hope this was helpful. I will also document this on our wiki.

Is this supported now? If it is, do you know where I can find the documentation?

@ashleywb @jhossy The information for this scenario is documented towards end of this Wiki page:
https://github.com/AzureAD/microsoft-authentication-library-for-js/wiki/Sso


Request: remove or configure opt-out for including CWD/hostname from query

Hello, thanks for a neat program!
I am frequently launching vscode in a terminal, with variants of: code ~/code/my/project
Searching for that history entry, competes with all other entries made with "code/my/project" and I never get that suggestion, when I search for "code". To work around this, I played around with:
https://github.com/chelmertz/hishtory/commit/2e7e02751e327ba0e372fd59465b70a8ccd2e8e4
Bonus question: I tried running tests after changing this in my "fork", but the tests worked with the actual ~/.zsh_history et al. files, so I quickly abandoned that. It would be nicer to have a developer guide for installing things without going through the python script -> releases -> figure out what each release contains, just when trying out a small change; but I can imagine the current flow suits your flow of development :)

Removing this feature would kill the point of using hishtory for me. I have several machines, but they do different things and without CWD and hostname I would have a problem comparing results and data.

@roland-5 could you explain more? I'm neither suggesting "don't show CWD/hostname in search window" (there's config for that) nor "don't store CWD/hostname with history data".
I'm suggesting "don't match on CWD/hostname by default, without any way of opt-ing out". There is already a way to opt-in for the behavior, quoting hishtory help query:
Example queries:
'hishtory query apt-get'  		# Find shell commands containing 'apt-get'
'hishtory query apt-get install'  	# Find shell commands containing 'apt-get' and 'install'
'hishtory query curl cwd:/tmp/'  	# Find shell commands containing 'curl' run in '/tmp/'
'hishtory query curl user:david'	# Find shell commands containing 'curl' run by 'david'
'hishtory query curl host:x1'		# Find shell commands containing 'curl' run on 'x1'
'hishtory query exit_code:1'		# Find shell commands that exited with status code 1
'hishtory query before:2022-02-01'	# Find shell commands run before 2022-02-01

See the line with cwd:/tmp/.

Are you aware of any way for me to resolve my use case?

Thank you for all the detailed thoughts and feedback @chelmertz! Your use case makes sense to me, though I'll say I do often find myself searching based on the cwd such that I do like having it included in the query by default. But making this configurable makes perfect sense to me. I can work on a PR for this and will tag you on it.
Though if you are motivated to make the change yourself in the meantime (2e7e027 seems like it is on the right track for that), check out the contributing docs for info on how to run tests.


where is the anchor file or generated code?

Issue Type
Support
OS
Windows, Mac OS
OS architecture
aarch64
Programming Language
C++
Framework
ONNX
Model name and Weights/Checkpoints URL
palm_detection_inf_post_128x128.onnx
Description
i test your palm_detect_top2_post.onnx model, and found that result seems worse on mobile device than PC platform. so i want to test the palm_detect_inf.onnx, and do prostprocessing myself. However i cant find the anchor file (1 x 2016 size).
Relevant Log Output
No response
URL or source code for simple inference testing code
No response

I don't know exactly which file you are referring to since the file palm_detect_inf.onnx does not exist.
I am not sure what anchor refers to.
What does prostprocessing refer to?
I don't know what you want to do. If you simply want to adjust the parameters of the NMS, you can do so by following the steps below.
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/307_YOLOv7/post_process_gen_tools#how-to-change-nms-parameters
If you want to change all the processes after Sigmoid, you can use the following tools to break up the model and generate it again by combining the OPs you want.
https://github.com/PINTO0309/simple-onnx-processing-tools

I don't know exactly which file you are referring to since the file palm_detect_inf.onnx does not exist.
I am not sure what anchor refers to.
What does prostprocessing refer to?
I don't know what you want to do. If you simply want to adjust the parameters of the NMS, you can do so by following the steps below.
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/307_YOLOv7/post_process_gen_tools#how-to-change-nms-parameters
If you want to change all the processes after Sigmoid, you can use the following tools to break up the model and generate it again by combining the OPs you want.
https://github.com/PINTO0309/simple-onnx-processing-tools

I can only read what you are thinking from the information in the text. Therefore, there is too little information to read your written statement to understand exactly what you want to do.

sorry for my bad description. My problen is i use the palm_detection_lite_top2_post_192x192.mnn and convert it to MNN model in order to use on mobile device. Before that i test the MNN model on the PC platform, and the result seems correct. However, the result on mobile device seems worse than on PC. So i want tot debug where the problem is. I use the palm_detection_lite_192x192.onnx model and found that needs the anchors info to postprocess the output  with sizeof  1x2016x8. But I cannot find the anchor info in your project.

I don't know exactly which file you are referring to since the file palm_detect_inf.onnx does not exist.
I am not sure what anchor refers to.
What does prostprocessing refer to?
I don't know what you want to do. If you simply want to adjust the parameters of the NMS, you can do so by following the steps below.
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/307_YOLOv7/post_process_gen_tools#how-to-change-nms-parameters
If you want to change all the processes after Sigmoid, you can use the following tools to break up the model and generate it again by combining the OPs you want.
https://github.com/PINTO0309/simple-onnx-processing-tools

I can only read what you are thinking from the information in the text. Therefore, there is too little information to read your written statement to understand exactly what you want to do.

by the way, i test the model from https://github.com/zmurez/MediaPipePyTorch, which has 256x256 input size. the result  is correct on mobile phone. This model use anchor with  size of 2944.

Frankly, as long as the same model is used, I don't see any difference in accuracy whether it is run on a mobile phone or a PC.
https://github.com/PINTO0309/hand-gesture-recognition-using-onnx
https://user-images.githubusercontent.com/33194443/196896969-a9bbc274-d963-487b-b590-090b15d5e0c2.mp4
https://github.com/PINTO0309/MoveNet-Python-Example
https://user-images.githubusercontent.com/33194443/196897429-051715b5-a33e-4ad9-8935-6db02080c6c9.mp4
And the anchors are embedded in the model pre-computed from the beginning, which I really dislike about PyTorch's implementation.

I don't know exactly which file you are referring to since the file palm_detect_inf.onnx does not exist.
I am not sure what anchor refers to.
What does prostprocessing refer to?
I don't know what you want to do. If you simply want to adjust the parameters of the NMS, you can do so by following the steps below.
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/307_YOLOv7/post_process_gen_tools#how-to-change-nms-parameters
If you want to change all the processes after Sigmoid, you can use the following tools to break up the model and generate it again by combining the OPs you want.
https://github.com/PINTO0309/simple-onnx-processing-tools

I can only read what you are thinking from the information in the text. Therefore, there is too little information to read your written statement to understand exactly what you want to do.

i read the palm_detection_lite_top2_post_192x192.onnx  model structure, found these is no anchor info just NMS process. So the network regress the final image coordinate directly ? what i mean is the output box size and kp0-kp7 are in range[0, 192]? and so does palm_detection_lite_192x192.onnx ?

i read the palm_detection_lite_top2_post_192x192.onnx model structure, found these is no anchor info just NMS process. So the network regress the final image coordinate directly ? what i mean is the output box size and kp0-kp7 are in range[0, 192]? and so does palm_detection_lite_192x192.onnx ?

All cumbersome post-processing is already implemented in ONNX.

Calc anchor

cx, cy, w, h -> x1, y1, x2, y2

Frankly, as long as the same model is used, I don't see any difference in accuracy whether it is run on a mobile phone or a PC. https://github.com/PINTO0309/hand-gesture-recognition-using-onnx
189632510-1823cd54-bc36-4889-ac14-adc16deba9b8.1.mp4
https://github.com/PINTO0309/MoveNet-Python-Example
192566926-30454082-70ed-4f75-8dfd-076f2038a16d.mp4
And the anchors are embedded in the model pre-computed from the beginning, which I really dislike about PyTorch's implementation.

thanks for you reply. so the  model output is image coordinate.
as for my problem., the result on mobile device is not stable. What i mean is the result is correct or not both happens. I thought maybe because of the anchor process or NMS  process in ONNX model before. In fact, in my opinion, the result should be same on PC and mobile device, because both use the same processsing codel and converted MNN model.
by the way, hand_landmark_lite_1x3x224x224.onnx works well on mobile device.


Add Index to QueueElement

This is rather pointless, because:

adding "Position" field would require to enumerate through all QueueElements when moving any single element to rebuild indexes;
we are using IList type, which is ordered by design.

Having Position field in controller output may be desired, however this is done using a Dto (data transfer object, see #17 for more).
Also, it may be useful to implement Linked List (#18) to make sure that elements order is handled correctly using different database systems (classic SQL databases like MariaDB should persist order of records, but with other solutions it might be not the case)


goNextPageAutomatic doesn't work if 'html' type of question exists on page

Steps to reproduce:

Create a survey with 2 more pages where each page has 1 multiple choice radio button question.
On the first page, above its question, add a question of 'html' type (which is just a label, essentially).
Set goNextPageAutomatic: true.
Try to answer the first page's question, and it won't proceed.

You can see a sample survey JSON here.
{
            goNextPageAutomatic: true,
            requiredText: "",
            showQuestionNumbers: "off",
            showNavigationButtons: false,
            showProgressBar: "bottom",
            onValueChanged: surveyValueChanged,
            pages: [
                {
                    name: 1,
                    questions: [
                        {
                            type: "html",
                            html: "This quick quiz will help you cure your back pain (dynamically adapting based on your responses).",
                            name: "question1"
                        },
                        {
                            type: "radiogroup",
                            choices: [
                                "5: I have the torso of an Olympic athlete.",
                                "4: It feels fine (I don't even think about it).",
                                "3: It bothers me occasionally but usually isn't on my mind.",
                                "2: It often hampers me during various activities that I'd otherwise enjoy.",
                                "1: My back pain prohibits me from MANY activities and seriously disrupts my everyday life.",
                                "0: It's ruining my life."
                            ],
                            isRequired: true,
                            name: "backComfortRating",
                            title: "[From 0 to 5] How strong, flexible, comfortable, and resilient does your back feel nowadays?"
                        }
                    ]
                },
                {
                    name: 2,
                    questions: [
                        {
                            type: "radiogroup",
                            choices: [
                                "Yes",
                                "No"
                            ],
                            isRequired: true,
                            name: "visitedDoctor",
                            title: "And have you been examined by a licensed health professional?"
                        }
                    ]
                },
                {
                    name: 3,
                    questions: [
                        {
                            title: "When you followed their recommendations, what happened?",
                            name: 'x',
                            type: "radiogroup",
                            visibleIf: "{visitedDoctor}='Yes'",
                            choices: [
                                "one",
                                {
                                    value: "two",
                                    text: "second value"
                                },
                                {
                                    value: "three",
                                    text: "third value"
                                }
                            ],
                        },
                        {
                            title: "Why not?",
                            name: "dd",
                            visibleIf: "{visitedDoctor}='No'",
                            type: "radiogroup",
                            choices: [
                                "one",
                                {
                                    value: "two",
                                    text: "second value"
                                },
                                {
                                    value: "three",
                                    text: "third value"
                                }
                            ],
                            hasOther: true,
                            otherText: "Other (specify here)",
                        }
                    ]
                },
                {
                    name: 4,
                    questions: [
                        {
                            type: "radiogroup",
                            choices: [
                                "Yes",
                                "No"
                            ],
                            isRequired: true,
                            name: "xxx",
                            title: "(OPTIONAL) One last optional question before I give you something wonderful. What have your physical symptoms been like, and what have they prevented you from achieving in your life? (What will you enjoy doing once you feel better than ever?)"
                        }
                    ]
                },
            ],
        }

@ryancwalsh Yes, it doesn't work correctly. A new unit test is added and the bug is fixed.
Thank you,
Andrew


Data

Can you please provide your train_test_id.pickle, %s.h5 and %s_attribute_all.h5 files? And also say how data files and masks should be structured.

Hi Irek,
Thanks for your interest in my repo. I’m traveling and will upload the
pickle file and provide more instructions after I get back.
Thanks
Eric
On Fri, Apr 12, 2019 at 9:23 AM Irek<EMAIL_ADDRESS>wrote:

Can you please provide your pickle file? And also say how data files and
masks should be structured.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/chvlyl/ISIC2018/issues/1, or mute the thread
https://github.com/notifications/unsubscribe-auth/ACTUNDXXILSVZ54MCCYGXYDPQBBJRANCNFSM4HFOLUKA
.

--
Sent from Gmail Mobile

Yes, it will be great

I have the same problem as you,can you share a solution my email<EMAIL_ADDRESS>

Hi Irek, Thanks for your interest in my repo. I’m traveling and will upload the pickle file and provide more instructions after I get back. Thanks Eric
On Fri, Apr 12, 2019 at 9:23 AM Irek @.***> wrote: Can you please provide your pickle file? And also say how data files and masks should be structured. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or mute the thread https://github.com/notifications/unsubscribe-auth/ACTUNDXXILSVZ54MCCYGXYDPQBBJRANCNFSM4HFOLUKA .
-- Sent from Gmail Mobile

Can you please provide your train_test_id.pickle, %s.h5 and %s_attribute_all.h5 files? And also say how data files and masks should be structured.

I have the same problem as you,can you share a solution my email<EMAIL_ADDRESS>

Hi Irek, Thanks for your interest in my repo. I’m traveling and will upload the pickle file and provide more instructions after I get back. Thanks Eric
On Fri, Apr 12, 2019 at 9:23 AM Irek @.***> wrote: Can you please provide your pickle file? And also say how data files and masks should be structured. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or mute the thread https://github.com/notifications/unsubscribe-auth/ACTUNDXXILSVZ54MCCYGXYDPQBBJRANCNFSM4HFOLUKA .
-- Sent from Gmail Mobile

The link can’t be opened, can you post it again? Or send me email<EMAIL_ADDRESS>

I tested the link and it worked fine. It’s shared through Google drive.
Please make sure you can access Google drive.
On Tue, Mar 2, 2021 at 10:20 PM mmmxl6<EMAIL_ADDRESS>wrote:

Hi Irek, Thanks for your interest in my repo. I’m traveling and will
upload the pickle file and provide more instructions after I get back.
Thanks Eric
On Fri, Apr 12, 2019 at 9:23 AM Irek @.***> wrote: Can you please
provide your pickle file? And also say how data files and masks should be
structured. — You are receiving this because you are subscribed to this
thread. Reply to this email directly, view it on GitHub <#1
https://github.com/chvlyl/ISIC2018/issues/1>, or mute the thread
https://github.com/notifications/unsubscribe-auth/ACTUNDXXILSVZ54MCCYGXYDPQBBJRANCNFSM4HFOLUKA
.
-- Sent from Gmail Mobile
The link can’t be opened, can you post it again? Or send me email
<EMAIL_ADDRESS>—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/chvlyl/ISIC2018/issues/1#issuecomment-789396983, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACTUNDUF7KMNO2EMZ2XEF23TBWTHPANCNFSM4HFOLUKA
.

--
Sent from Gmail Mobile


RigidBodies go through walls

You can see it when you throw one of the balls into the monitor at maximum power. The monitor will go through the wall. I'm not a physicist, but I'm preeetty sure it's not how real world works.

This is likely because Godot's CCD has been broken for ages.

This is likely because Godot's CCD has been broken for ages.


fixing duplicate name errors on the pipeline

Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
Change description
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

dependency check failures now,
You'll also need to make some changes to the module for postgresql_tf branch to work now private endpoint support is in there:
https://github.com/hmcts/cnp-module-postgres/blob/postgresql_tf/example/main.tf


⚠ Science Web Services - QState has degraded performance

In f14fb47, Science Web Services - QState (https://sws.labbricks.com/qstate/) experienced degraded performance:

HTTP code: 523
Response time: 1512 ms

Resolved: Science Web Services - QState performance has improved in 4ad0472 after 18 minutes.


Allow gedmo:slug to be overrided by attributeOverride

The similar issue like in #1631 just for slugs

thanks ;)

prašom :)

@l3pp4rd can you add tag 2.4.17?


Keys problem with latest packages

Problem
Keys of newly-created atsigns become unusable upon app restart when using latest libraries
To reproduce

build iot_receiver app in mwc_demo repo
run it, create a new atSign. Remember to back up the atKeys file

should work correctly
^ restart app, choose the atSign you just created
you will be presented  with a screen asking that you type in your PIN

Workaround

onboard the same atSign into the store version of AtmospherePro, by importing the backed-up keys file
should work fine
restart the iot_receiver app, choose the same atSign again
^ should work fine

Observations

only interaction between iot_receiver and AtmosoherePro apps is via the keychain
problem is not the keys file

Hypothesis

New packages in combination result in keys being written incorrectly to the keychain

Encountered by @nickelskevin  and @cconstab
Reproduced by @tinashe404   and workaround identified. The fact that this workaround works points to where the problem is
@sonle-geekyants please can you investigate urgently as we have a broken collection of published packages right now

I've checked it and I see that serverEncryptionPublicKey from BE returns null => result returns ACTIVATE status => app navigate to OTP Screen. Please ask to BE checks Staging environment

https://user-images.githubusercontent.com/88494375/194239108-4e670fd8-2802-4dad-9559-39a5911e91dd.mov

@sonle-geekyants this is happening in production
What are you asking to be checked? Why do you need it be checked? Sorry, your message isn't clear to me.
Have you followed the instructions I gave in the bug description as to how to reproduce the issue?

More context: Issue was encountered by Scott, then by @cconstab and @nickelskevin. Reproduced by @tinashe404 yesterday using instructions given above.

Because I wasn't added to the iot_receiver app in the mwc_demo repo, I couldn't follow your instructions.
I see that when I call API transmission publickey to server, BE returns the error AT0015, could you check this error?

@sonle-geekyants The repo is public. The app code is here: https://github.com/atsign-foundation/mwc_demo/tree/trunk/flutter/iot_receiver
You will fail to find public:publickey@atsign if onboarding has not been completed for that atSign. Did you reset the atSign at some point?

I tried reverting back a few packages to use the previous versions and it started working for me, here's the branch https://github.com/atsign-foundation/mwc_demo/tree/fix/new_onboarding_atsigns, and here's the commit https://github.com/atsign-foundation/mwc_demo/commit/1472997d3747fb6b782eb70a7ac692ec4f66d3b9.
Will try to figure out if it is happening because of any one of the packages.

Thank you @nitesh2599 !

I see when we use at_client: 3.0.38 it is working but if upgrade to version 3.0.39 (or 3.0.40) it will work failure

Thank you @sonle-geekyants!
@VJag can you/team investigate what changed in at_persistence_secondary_server  between 3.0.34 and 3.0.36 please, and see if you can diagnose the problem?

Sorry, I've investigated but haven't found the cause yet. Could you ask someone who knows this library to investigate with me? @gkc

Because I wasn't added to the iot_receiver app in the mwc_demo repo, I couldn't follow your instructions. I see that when I call API transmission publickey to server, BE returns the error AT0015, could you check this error?

Investigated the issue and suspect the following changes might be causing the issue: https://github.com/atsign-foundation/at_server/pull/893
Reasoning:
The signing_privatekey and signing_publickey are being skipped from sync and the changes are available on the client side, but the corresponding changes on the server are NOT released to production yet. Hence server tries to send the signing keys to the client (to sync); client receives the keys but does not update the local commit id (since on client side, skip signing keys are in-place) which leads to sync run into a loop. As a result, the encryption public key does not sync to the remote secondary and hence throws public:publicekey@ not found the key store.
Currently, reverting to the changes related to "skip signing keys" and testing in local to assert the above reasoning.
Possible fixes (If above mentioned results to be root cause of the issue):

Revert the skip commit log changes on at_persistence_secondary_server
Promote the v3.0.24 version to production to make the "skip signing_key on sync" available on the server side.

Basing on discussion between Jagan, Murali and myself, the approach-1 seems feasible.
Also the initial thought to skip signing keys is to resolve the decryption error in the sync conflict resolution ( because the data in the  server is not encrypted which we are trying to decrypt in client side)
Later we added a check 'isEncrypted ' before decrypting the data in sync conflict resolution. Hence skip signing keys is no longer needed.
Please share your thoughts on this?

Have you tested to prove that "skip signing keys" is no longer necessary?

@gkc  :
Tested the below scenario's on the canary servers and staging server

On canary server, on-boarded atmosphere pro app and initial sync was successful.
On canary server, able to send and receiver files successfully.

On Staging atsign's:

Created two new atsign's and able to onboard the app with back-up keys file successfully.
With atmosphere pro app, able to send and receive files successfully.

Tested the issue on a staging atsign - "symbolicparody18" and upon reset, able to onboard the app successfully. Attaching the video recording.
@symbolicparody18
27e986a7-e3fc-5bb0-baa6-1c8024960b47.staging0001.atsign.zone:1237

https://user-images.githubusercontent.com/16645698/194898967-4fd1f7f0-71f7-4743-83d6-0701a48d2244.mp4

New client  sends sync:-1 to the server
Server sends the sync response which includes signing private key and signing public key
On the client side, the entries gets updated into the keystore but not into the commit log
(Due to changes in at_persistence_secondary_server v3.0.36).
Now, the localCommitId will always be less than the serverCommitId leading the sync process to get into infinite loop
Hence the public encryption key is not synced to the cloud secondary which leads to onboarding process failure

As an immediate solution, we are reverting the changes related to the skip sync of signing keys and statsNotificationId and publishing the at_persistence_secondary_server and at_client packages.
We will investigate possible approaches to accommodate skipping of certain keys on client/server. Also we have to revisit the condition that result in sync loop.

Got it. And to be specific, the condition that never becomes false is in _syncFromServer, while (serverCommitId > localCommitId) which means we never get to the part of the sync process which syncs from the client

Reopening as we have not yet published the new package nor pushed the new canary server release

Fixes the issues in at_persistence_secondary_server with 3.0.39 and at_client 3.0.41

Complete. Turned out to be a lot of work!


Add Pagination and Filtering

Problem Statement
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

I have a fix but it wont allow me to push the code up
git push --set-upstream origin pagination-and-filtering
git: 'credential-osxkeychain' is not a git command. See 'git --help'.
remote: Permission to jazzband/django-simple-history.git denied to RumitAP.
fatal: unable to access 'https://github.com/jazzband/django-simple-history.git/': The requested URL returned error: 403

@RumitAP Just create a fork of this repo, push to a branch there, and open a PR in this repo 🙂


Multiple is always applied to nested groups

I have something like this:
Group
panel
panel
SubGroup
subPanel
subPanel
For the subPanels the multiple attribute is always applied. At least they do not close when opening another subPanel.

Sorry my bad. I used angularjs and added same component ids with ng-repeat


Accessing new stage variables with Lua

Describe your problem here.
Basically, I wish to know if you can access the new stage variables with Lua (hide_girlfriend, camera_boyfriend, camera_opponent, camera_girlfriend and camera_speed). If yes, how do I access them with Lua?
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
Nothing

They are properties, you can use setProperty and getProperty to access them

I tried setProperty('camera_boyfriend.x', -300), but it doesn't work.

Bumping this issue because I still can't figure it out.

(getProperty('boyfriendCameraOffset')[1]) --x offset
(getProperty('boyfriendCameraOffset')[2]) --y offset

i don't think you can change them right now

Alright, I'll try this out later, thanks.

Tried this, sorry for the late reply. but it doesn't work.  Although, I used a slightly modified version of it, plus I was using setProperty. It just crashes the game if you run it.

I've already tested them myself so I can confirm they work, and I already said it may not be possible to change them

Alright, I see. Whatever, I might just sub in with the Camera Move Pos event for now.


[Question] AdaSeq for Sequence Labeling, How are the metrics calculated? What are the available models?

What is your question?
Hello, I am working on the SemEval2023 MultiCoNER-II task.
First of all thank you for sharing this amazing repo, its saving me a lot of time and effort.
With regards to the metrics, I was training an xlm-robert-laarge model on the English dataset and noticed the F1 score was low but accuracy was high (F1=0.38 and accuracy = ~0.8).
If you have taken a look at the English dataset for the MultiCoNER-II you'll see that the Other tag (aka 'O') is more frequent in the dataset than any other tag by a large margin. Hence its possible that the model(s) may overfit and just start predicting the tag O for most tokens in a sequence.
My question is, When calculating the metrics, do you take into account the O tags into the calculation? in other words, do you mask the tokens in the target sequence whose gold tag is O when calculating the loss/accuracy/F1?
My next question has to do with the possible configurations we can control.
What are the models (transformers) that we can use?
What have you tried?
multiconer2-en-exp#1.zip
The attached file contains the configuration I used.
The model was early stopped at epoch=7
Code (if necessary)
No response
What's your environment?

AdaSeq Version (e.g., 1.0 or master): 0.5.0
ModelScope Version (e.g., 1.0 or master): 1.1.1
PyTorch Version (e.g., 1.12.1): 1.12.1+cu102
OS (e.g., Ubuntu 20.04): Linux-5.10.147+-x86_64-with-glibc2.27 in Google colab
Python version: 3.8.16
CUDA/cuDNN version: 11.2
GPU models and configuration: Tesla T4 15109MiB
Any other relevant information: Used on Google Colab

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hello, @iR00i , we are very happy that you use AdaSeq.
For your 1st question:
For sequence labeling tasks, we just use seqeval to calculate the metrics: acc, p, r, mi-f1, ma-f1. Its evaluation method is the same as conlleval.
So when calculating accuracy, O is taken into account. However, when calculating F1, O is not.
For your 2nd question:
You can use basically ALL transformers from huggingface modelhub. Just copy the model_id to model.embedder.model_name_or_path in the configuration file.
e.g.
model:
  type: sequence-labeling-model
  embedder:
    model_name_or_path: dslim/bert-base-NER
  dropout: 0.1
  use_crf: true

Finally, some suggestions for you to get higher F1:
We found that without further information, it's hard to train a stable model with the original datasets as the context is short and the entity classes are fine-grained. You need to tune very very carefully.
You can try the retrieval-augmented dataset we provided, which can produce a much better result.

I have the similar question when simply using the code like 'python scripts/train.py -c examples/bert_crf/configs/wnut17.yaml
', but test result showed that F1 was really low while accuracy was high. I have tried a few yamls, it's the same problem. I think maybe it's the model. Why is this？really appreciate it, thanks！

I have the similar question when simply using the code like 'python scripts/train.py -c examples/bert_crf/configs/conllpp.yaml ', but test result showed that F1 was really low while accuracy was high. I have tried a few yamls, it's the same problem. Why is this？really appreciate it, thanks！

Hello, @t2413 .
I just ran the same training code on conllpp with master branch, python=3.7 torch=1.12.1 modelscope=1.3.0.
The results seemed reasonable:
test: {
  "precision": 0.9450856942987058,
  "recall": 0.947737635917222,
  "f1": 0.9464098073555166,
  "accuracy": 0.9869494993000969
}

I have no idea what the reason is for your bad performance.
Maybe you can check if your requirements are installed correctly?


Fixing release version links

Small change to fix the links with the correct versions in the README.

I did the change in the UI, that's why DCO is complaining.

@tsurdilo Looks good to me .


Work around webpack-dev-server hostname checking requirement

To fix security things, webpack-dev-server added a check for hostname in https://github.com/webpack/webpack-dev-server/issues/887.
Unfortunately, they also used that host check as the address to bind to, with only <IP_ADDRESS> and localhost as exceptions. This breaks docker containers or proxies.
The workaround is to set up a host entry in the container for buildkite.dev to point to <IP_ADDRESS>. This then still has the hostname check, but it binds to the correct interface in the container.

Insta-merging given I'm the only consumer of these docker containers.


feature: introduce a new XProtocol interface GracefulShutdown for graceful stop existing connections.

Also implemented the GoAway method for HTTP2 downstream connection.
TODO

[x] move the changes in vendor to mosn/api repo.
[x] tests
[x] rebase master

@jayantxie  pls take a look.

@dunjut

ref: https://github.com/sofastack/sofa-bolt/issues/278


Memory allocation error - incorrect checksum for freed object

Description
Memory allocation errors occur when launching 100 servers that do not appear when launching 10 servers in the same configuration.

OS: Mac OS 10.14
Node: Homebrew v10.15.2

Code to reproduce
https://github.com/bunchtogether/braid-server/blob/master/tests/peers-ring.test.js
(Apologies, I have been unable to reproduce in a simpler example.)
Output
node(65675,0x1091595c0) malloc: Incorrect checksum for freed object 0x104819200: probably modified after being freed.
Corrupt value: 0x1baffed00baffedf
node(65675,0x1091595c0) malloc: *** set a breakpoint in malloc_error_break to debug

Extended Description
The Braid application syncs key-value pairs between a small network of peers and a larger network of read-only subscribers. All communication between the peer and subscriber processes occurs over WebSockets.
In this test 100 uWebSocket servers are created and linked in a ring using ws, i.e. peer A opens a connection to peer B, peer B opens a connection to peer C, peer C opens a connection to peer A.
When the test is reduced to 10 uWebSocket servers the error does not occur.
Thanks
Thanks for the great work and let me know if there is any additional information I can provide.

Could you try out 15.9.0?

Still present in 15.9.0:
node(92178,0x10af8e5c0) malloc: Incorrect checksum for freed object 0x10517ac00: probably modified after being freed.
Corrupt value: 0x1baffed00baffedf
node(92178,0x10af8e5c0) malloc: *** set a breakpoint in malloc_error_break to debug

You have to strip down your code and provide instructions. I don't know what to do with it.

To run the tests and trigger the error you can “yarn install” then “yarn test” from the linked repo - but I wasn’t able to create an example that reproduces the error building from the bottom up. (See the previous code.)
I’ll work on

I figured npm install npm test but I got error when npm installing. So yarn install yarn test should work better?

I've updated the repo to work with npm, npm install followed by npm run test:js should work.
git clone https://github.com/bunchtogether/braid-server.git
cd braid-server/
npm install
npm run test:js

Also, while I can still reproduce the error with the test suite, it does seem to be less frequent with 15.9.0, happening ~30% of the time instead of 90% with 15.8.0.

How can I run that test without any other process or mid-step? Only vanilla nodejs, no test runner or anything. These files don't contain JavaScript, it's some kind of typed thing.
I need to run only nodejs and only that one script that triggers the issue.

These files use Flow, which is like Typescript. I’ll work on a version that triggers the error without the test framework or type checking.

Did you make any progress? I would like to run AddressSanitizer on the nodejs process that runs this test. But I don't know how to do that with all of these test runner parent processes spawning things to the left and right.

Even if you don't see any error, how can I run without spawning any new process?

To run the example without spawning:
node tests/malloc-example.js

Perfect, will try this out

Chiming in with something sort of relevant: I started watching this issue because it's similar to an error I'm coming across. I have a C addon that I wrote for Node, but I'm only using Mocha for my test framework. I also only get the error when running my tests. Practice samples seem to work fine.
I'm starting to think that it may be a problem related to mocha with native addons. Mocha is a dependency of Jest, so could be relevant?

I cannot reproduce this on Linux and I own no Apple products. It may be something about forking the process on non-Linux systems, as it really only has been tested on Linux.
Really, spawning multiple processes on non-Linux is completely untested.
You say it only happens when spawning processes? Can you try it on Linux instead?


#34692 Fix color rendering for shapes

Added functionality to render shapes in their assigned color if a 'color' column is present in the result set.
If no 'color' column is available, shapes are rendered based on their index in the row.

The first case

The second case

Verified


where is slack_export.py ?

thank you for special project!!
BTW,
after pip "install slacker",and "pick"
I cannot run slack_export command, because of missing of slack_export.py
Shall I download it .py code and  put to accurate place , ( and also path)??

Its in the root of the project. Make sure to open your console or change directory to there to run this.


Delete post tool tip

We need a tool tip that pops up so users can delete and edit their posts. For now if someone wants to create the layout and pop up , I will add in the functionality to the API.

@divanov11 I tried to make the layout, this what I got so far
this menu will show if the user is not the author of the post

and this will show to the author of the post

What do you think ?

Updated version for other users post menu

Looks great!! I will merge in a bit. Just added the re-mumble functionality so I will be testing that before adding user registration.


Bug 1833195: delete bundle objects after CSV gets deleted

Description of the change:
OLM now supports secrets and configmaps in the bundle. These should have ownerrefs associated with the CSV in the bundle. When the CSV is deleted, these objects should be deleted as well. Since they are namespaced scoped, the kube gc will delete them automatically.
Motivation for the change:
Reviewer Checklist

[ ] Implementation matches the proposed design, or proposal is updated to match implementation
[ ] Sufficient unit test coverage
[ ] Sufficient end-to-end test coverage
[ ] Docs updated or added to /docs
[ ] Commit messages sensible and descriptive

/retest

/retest

Unit tests are failing.

Unit tests are failing.

didn't check things again after rebasing off master 🤦

Added two additional e2e tests for the 2 upgrade scenarios @ecordell mentioned.
Also added the ownership code to service objects in the bundle as well for consistency across all bundle objects OLM resolves.

/retest

Tests should pass once Quay is back in a healthy state

/retest

/lgtm


[Windows] Add Microsoft Reporting Services Projects support

Description
[Windows] Add Microsoft Reporting Services Projects support
Related issue:
https://github.com/actions/virtual-environments/issues/5340
Check list

[x] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

/azp run windows2022

/azp run windows2022


[$500] Request money - Distance - Keyboard is showing for brief moment of time while coming from delete waypoint

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Click on FAB icon
Then select request money
Click on distance TAB
Click on add stop button
Click on stop which is added
Then select three dots (top most right)
And delete this waypoint

Expected Result:
Keyboard should not be show
Actual Result:
Keyboard is showing for brief moment of time
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[x] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: v1.3.66-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/115492554/ae9dac71-d047-417f-95b4-7970e65caad7
https://github.com/Expensify/App/assets/115492554/e37667df-bea6-491a-970a-62ac05fdd188
Expensify/Expensify Issue URL:
Issue reported by: @niravkakadiya25
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1693941099472519
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~018ef807527bb44c5b
Upwork Job ID:<PHONE_NUMBER>677569536
Last Price Increase: 2023-09-11

Proposal
Please re-state the problem that we are trying to solve in this issue.
What is the root cause of that problem?
We have the textInput element focus set on the load here:
https://github.com/Expensify/App/blob/89598d8793b5db05488a1358c1fd13d6da73fe1f/src/pages/iou/WaypointEditor.js#L163
What changes do you think we should make in order to solve the problem?
Call the textInput.current.blur() when we open the "Delete Stop" modal (setting isDeleteStopModalOpen) as well as when deleting the stop – at deleteStopAndHideModal.

What alternative solutions did you explore? (Optional)

@sobitneupane what do you think of the current proposals?

Thanks for the proposal everyone.
@hungvu193 I don't think your proposal will do the job as we see keyboard between two modals in the delete process (i.e modal with Delete waypoint menuitem and Delete waypoint modal.

According to https://github.com/Expensify/App/issues/20608#issuecomment-1616813839, only iOS refocuses after closing the modal, so we don't have to reopen the keyboard after the modal is dismissed, to be consistent with other platforms.

Oh yes. I think it makes sense to dismiss keyboard when we press threedotsmenu to open the first modal as it is the case in other platforms as well.
Proposal from @tienifr looks good to me.
🎀 👀 🎀 C+ reviewed

@sobitneupane @hayata-suenaga looks like we have a proposal we like, what's the next step?

@sobitneupane @hayata-suenaga bumping :)

We are waiting for response from @hayata-suenaga on https://github.com/Expensify/App/issues/27201#issuecomment-1722927165

Yup @tienifr Let's hold this issue for now.
cc: @sonialiap

On hold for https://github.com/Expensify/App/issues/27182
might get incorporated into https://github.com/Expensify/App/issues/24452

Still on hold

@tienifr can you check the issue against which we're holding this issue and check what they have decided on?

On hold for https://github.com/Expensify/App/issues/24452
FYI, I'm OOO until Oct 22

changing the priority until the hold is lifted

On hold for https://github.com/Expensify/App/issues/24452

on HOLD for https://github.com/Expensify/App/issues/24452

On hold for https://github.com/Expensify/App/issues/24452

Here is the latest update for https://github.com/Expensify/App/issues/24452

On hold for https://github.com/Expensify/App/issues/24452, the PR for that issue is still in review

on hold for #24452

on hold for https://github.com/Expensify/App/issues/24452

Still on hold for https://github.com/Expensify/App/issues/24452. Looks like the PR for that issue is still in Draft

A new contributor was assigned to https://github.com/Expensify/App/issues/24452 to review the PR.
We're still on hold for that PR.

The PR related to this issue was divided into segments, but progress is continuously being made.

Still on hold for the PR

PR is still being reviewed

the latest comment

Still on hold https://github.com/Expensify/App/issues/24452

latest update here

Still on hold for https://github.com/Expensify/App/issues/24452

https://github.com/Expensify/App/issues/24452 is still being worked on

Still on hold for https://github.com/Expensify/App/issues/24452

Still on hold

Still on hold

Still on hold for https://github.com/Expensify/App/issues/24452 but looks like we're getting closer!

PR for https://github.com/Expensify/App/issues/24452 was deployed on June 12 https://github.com/Expensify/App/pull/43041#issuecomment-2161775499. I'm asking QA to retest this to see if it's still an issue - slack

@sonialiap The steps are outdated; the "add stop" button is no longer present.
https://github.com/Expensify/App/assets/93399543/d8f3b04f-cd11-46e6-8865-86788494695c

Looking at this today! (completing another task and then this one's next on my list)

@tienifr chatted with the team, does 50% sound reasonable to you?

@tienifr I've adjusted the contract in upwork and issued payment ✔
Payment summary:

@tienifr $250 via upwork

HI
@sonialiap can you laso adjus amount in my upwork as well.


Remove arbitrary concurrency limiter for uplods and replace with a configurable limit that is disabled by default.

The existing limit doesn't really help to achieve the goals it states as it cannot help in conditions where multiple clients are using RE at the same time, which are the common exercise conditions. Since the flag can be used to limit quickly uploads in case of some issues with RBE I opten in for keeping it in some form, by making it a setting, though it likely isn't very useful, so it's disabled by default. Instead we should rely on other control mechanisms like number of connections and TCP or HTTP/2 flow control or, like in this case, explicit errors coming from the RE protocol itself (like RESOURCE_EXHAUSTED or others).

cc @benbrittain who originally added this

The existing limit doesn't really help to achieve the goals it states as it cannot help in conditions where multiple clients are using RE at the same time, which are the common exercise conditions

Yeah, but even if they're the common exercise conditions, it's still pretty easy to imagine that they might not be the failure conditions. Ie you could imagine some server that has one thread per client or something like that and falls over if a client shows up trying to write too many big files at once.
I think there's a nice in-between option to go for though, which both preserves this property and sounds closer to what you actually want: Can we only apply this concurrency to the non-batched uploads, and let batched uploads proceed at unlimited concurrency?

Yeah, but even if they're the common exercise conditions, it's still pretty easy to imagine that they might not be the failure conditions.

Which is why there is an optional configuration to limit concurrency. Any default value would be making likely incorrect assumptions about the implementation and exercise conditions of a RE server and this is just wrong as there are multiple implementations and all of them are wildly different and can run on extremely large distributed clusters with a wide array of different hardware and software specifications. This invariably results in lower performances by default with no real impact. On the other hand if such a limit is necessary for a RE server they can state this setting in their docs. I think this is already a middle ground and brings the potential benefit of limiing concurrency whilst keeping a performant setup by default, wdyt?

friendly ping @JakobDegen :)

Ping :)

I tested this on our RE setup, and without it the client seemed to be stuck for 40+ minutes on re_upload stages, but with this PR, it's no longer blocked on re_upload.

Agree that it would be great to see some forward progress on this.  @JakobDegen is there anything you're waiting for?

ping @JakobDegen

@TheGrizzlyDev yeah, sorry, I realized that in my previous message I managed to actually not right any of the things that I was actually most worried about.
The reason that I think this kind of a concurrency limiter makes sense isn't so much the server side of things, but rather the client side. When it comes to batched uploads, or other uploads that are coming from memory and not from disk, I think I'm willing to trust the system network stack to make sure that resources are effectively utilized. But for non-batched uploads, those are generally reading from disk as a part of the request, and limiting concurrency the concurrency of disk accesses in buck2 does seem like something we should do, if for no other reason than that we don't want to run out of file descriptors. Thoughts?

those are generally reading from disk as a part of the request, and limiting concurrency the concurrency of disk accesses in buck2 does seem like something we should do, if for no other reason than that we don't want to run out of file descriptors. Thoughts?

if that's the case then this limiter still wouldn't help and you'd need a completely different type of limiter that controls open file descriptors within a whole build. Also, I find it highly unlikely this could happen, whereas the current limiter is certain to slow down uploads by a huge factor.

I’ll also add that for large projects, you’ll already hit your fd limit just by the fs monitor, so in practice everyone with a large enough project is already going to uncap.
Even beyond that, the concurrency in inherently limited by the max number of concurrent actions, as well as by natural serialization points in the build, which is all to say that the present behavior is far worse than any potential impact caused by increasing the number of file descriptors

@JakobDegen ping :)

There may be better longer term solutions here, but this seems like a fine improvement over the current state. I think we should just change the key to something like max_concurrent_uploads_per_action (it's per upload_request, but that should basically be 1 per action and per_action i think is more understandable). I'd lean towards prefixing it with oss_ since it's only going to affect the oss client (or maybe with grpc_), but either way is fine.


Dynamic/Fragments

[ ] #127
[ ] support ordering by aggregates
[ ] support fragments when filtering?
[ ] support filtering by aggregates?

Totally could use this.  Considering that we sometimes use case statements in the order by whether it's a new case clause or referencing a complex case statement already established within the select clause and we want to simply refer to it as "order by 1".
For fragment filtering and aggregates I'd say this should be applied to those as well.  Fragment filtering in turn may offer the escape hatch I was looking for so that I don't I'm not forced to set up Ecto associations (assoc) in my schemas and migrations and I can refer instead to the virtual field within a well known schema, or possibly even a field within an standard map for specific custom queries spanning many models.

There are a few things to consider here.

Flop needs to be able to reference fragments and/or dynamics by name.
It must be possible to order by a fragment/dynamic.
Flop needs to set the order direction dynamically from the parameters.
It must be possible to filter by a fragment/dynamic.
Flop needs to be able to apply all existing filter operators to a fragment/dynamic.

Ordering
Flop only needs to add the order direction dynamically. Ecto allows to use a dynamic within the keyword list passed to Ecto.Query.order_by/2.
order_by = [
  desc: dynamic([p], fragment("?>>?", p.another_field, "json_key"))
]

To dynamically order by a dynamic like this, all we need is to allow configuring dynamics with Flop.Schema:
@derive {
  Flop.Schema,
  dynamic_fields: [
    city: dynamic([u], fragment("? ->> ?", u.location, "city")),
    comment_count: dynamic([comments: comments], count(comments.id))
  ]
}

We could then add order conditions on these fields to the parameters.
params = %{
  order_by: [:city, :comment_count],
  order_directions: [:desc, :asc]
}

Cursor-based pagination
The solution above works fine if all you want to do is to order by a value without retrieving it. However, it is likely that you want to return and display this value, e.g. in a table column. Also, if you are using cursor-based pagination, Flop needs to be able to retrieve the values of all fields in the order by clause in order to build the cursor.
First, let's have a look at how we can retrieve the values manually. Expanding on the example above, our query might look like this.
User
|> join(:left, [u], c in assoc(u, :comments), as: :comments)
|> group_by([u], u.id)
|> select([u, comments: comments], {u, %{comment_count: count(comments.id), city: fragment("? ->> ?", u.location, "city")}})
|> Flop.validate_and_run!(params, for: User)

This query selects into a tuple, with the first element being the user and the second element being a map with the derived fields. Alternatively, you might want to either select into a single map, or select the derived fields into virtual fields on the schema struct.
For cursor-based pagination, Flop needs to know how to find the field values in the return items. You could define a custom cursor value function like this.
def get_cursor({user, extra}, order_by_fields) do
  Enum.into(order_by_fields, %{}, fn
    :city -> {:city, extra.city}
    :comment_count -> {:comment_count, extra.comment_count}
    field -> {field, Flop.Schema.get_field(user, field)}
  end)
end

# pass function in options
|> Flop.validate_and_run!(params, for: User, cursor_value_func: &get_cursor/2)

Reusing derived field definitions
While Flop is not meant to replace the Ecto query DSL and thereby is not concerned with select expressions, it would still be nice if you could reuse the field definitions in the query instead of duplicating them.
The problem with the solution above is that Ecto doesn't allow you to use dynamic in select expressions. However, it is allowed to use fragments in select expressions. So instead of defining the complete dynamic expression in Flop.Schema, we might define only the fragments:
@derive {
  Flop.Schema,
  fragment_fields: [
    city: ~s(fragment("? ->> ?", u.location, "city")),
    comment_count: ~s(fragment("count(?)", comments.id))
  ]
}

Since fragment can only be used within the Ecto.Query functions, we need to define the fragments as strings here. Alternatively, we could add a macro to Flop and do something like this:
@derive {
  Flop.Schema,
  fragment_fields: [
    city: Flop.define_fragment("? ->> ?", u.location, "city"),
    comment_count: Flop.define_fragment("count(?)", comments.id)
  ]
}

The define_fragment macro would escape the arguments, and Flop.Schema would generate order and filter expressions for the fragment fields at compile time.
The query could now look like this:
User
|> join(:left, [u], c in assoc(u, :comments), as: :comments)
|> group_by([u], u.id)
|> select([u, comments: comments], {u, %{comment_count: ^Flop.Schema.fragment(User, :comment_count), city: ^Flop.Schema.fragment(User, :city)}})
|> Flop.validate_and_run!(params, for: User)

I don't think Flop could or should make assumptions on how to select these derived fields, but at least you can reuse the Flop.Schema configuration this way.
As for the cursor value function: If you were to select the derived field values into virtual fields on the schema, the default cursor value function would already handle this case. If you need the data in another format, like the tuple in the query example, you will still need to define your own cursor value function as in the example above. I don't think there is a good way to handle this through more configuration.
Generalizing
If we're going to add a macro to define reusable fragments anyway, we might as well generalize this and add a macro for defining any partial query expression allowed in order_by instead.
@derive {
  Flop.Schema,
  join_fields: [
    comment_id: {:comments, :id}
  ],
  derived_fields: [
    city: Flop.derived_field(fragment("? ->> ?", u.location, "city")),
    comment_count: Flop.derived_field(count(comments.id))
  ]
}

In this case, Flop would generate the necessary query functions at compile time, setting the bindings parameter automatically from the configured fields, allowing you to reference any configured join field and any named binding defined in the join field configuration. The implementation details for this are still a bit vague.
Filtering
This is a bit trickier.

Flop needs to be able to use the existing filter operators on the expression.
Not all operators are applicable to all dynamic expressions, e.g. a dynamic expression that returns a string cannot be used in combination with the :in operator. At the moment, Flop does not validate the operator depending on the field type even for normal fields, so we can neglect this for now and add stronger operator validation for all field types in a later release.
If the dynamic expression returns an aggregate, Flop needs to use a having expression instead of a where expression to this field.

Applying filter operators
If derived fields are configured with the derived_field macro as defined above, Flop.Schema would be able to define filter functions for each derived field by injecting the expression into a where or dynamic expression at compile time.
Where or Having
There is no way for Flop to automatically determine whether to filter on a derived field using where or having, so this would have to be configured. The default would be where, and to use having, you would have to pass an option.
@derive {
  Flop.Schema,
  derived_fields: [
    city: Flop.derived_field(fragment("? ->> ?", u.location, "city")),
    comment_count: {Flop.derived_field(count(comments.id)), clause: :having}
  ]
}

This would be so awesome 💚. Can I help implementing this somehow?

Hey @andreasknoepfle, PRs (also with partial functionality) are certainly welcome. It's a bit tricky, since it requires some macro magic.

This might have gotten a lot easier to implement with Ecto 3.9: https://hexdocs.pm/ecto/Ecto.Query.API.html#selected_as/2

I just merged #242, which introduces alias fields, allowing you to sort by aliases defined with Ecto.Query.selected_as/2. Find the documentation here: https://github.com/woylie/flop/blob/main/lib/flop/schema.ex#L102.
I also added a short example on how to filter by calculated values using a lateral join on a subquery: https://github.com/woylie/flop/blob/main/lib/flop/schema.ex#L295.
I think that should cover most use cases for this issue, without awkwardly adding a partial query DSL to Flop. If there is something you cannot do this way, let me know and I might reconsider.


Race condition in max_gauge.go

Bug Report
What did you do?
I'm running tests for remote write API, and I'm seeing race condition
go run -race -tags netgo  ./cmd/prometheus/main.go --config.file prometheus.yml

What did you see instead? Under which circumstances?
Environment

System information:
Linux 4.15.0-51-generic x86_64

Prometheus version:
prometheus, version 2.10.0 (branch: upstream-master, revision: 819205ed0315069dfa4236925b3f2ec4615a8fb9)
go version:       go1.12.4

Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s
  evaluation_interval: 30s
  # scrape_timeout is set to the global default (10s).

remote_write:
  - url: http://mydb/push

scrape_configs:
- job_name: test

  honor_labels: true
  # scrape_interval is defined by the configured global (15s).
  # scrape_timeout is defined by the global default (10s).

  # metrics_path defaults to '/metrics'
  # scheme defaults to 'http'.

  static_configs:
  - targets: ['localhost:12345']

Logs:

==================
WARNING: DATA RACE
Write at 0x000004cc9248 by goroutine 43:
  github.com/prometheus/prometheus/storage/remote.(*maxGauge).Set()
      /home/vagrant/gopath/src/github.com/prometheus/prometheus/storage/remote/max_gauge.go:32 +0xb8
  github.com/prometheus/prometheus/storage/remote.(*timestampTracker).Commit()
      /home/vagrant/gopath/src/github.com/prometheus/prometheus/storage/remote/write.go:73 +0x153
  github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit()
      /home/vagrant/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:167 +0x481
  github.com/prometheus/prometheus/scrape.(*timeLimitAppender).Commit()
      <autogenerated>:1 +0x62
  github.com/prometheus/prometheus/scrape.(*scrapeLoop).append()
      /home/vagrant/gopath/src/github.com/prometheus/prometheus/scrape/scrape.go:1218 +0x105a
  github.com/prometheus/prometheus/scrape.(*scrapeLoop).run()
      /home/vagrant/gopath/src/github.com/prometheus/prometheus/scrape/scrape.go:935 +0xac6

I will post a PR soon.

fixed in #5647


added precompilation support

This PR adds precompilation support for exqlite, and it still allows the user to set their advanced configurations, https://github.com/elixir-sqlite/exqlite#advanced-configuration.
I also have a working one on my fork, and here is the log of the precompilation workflow, https://github.com/cocoa-xu/exqlite/actions/runs/4210648732 and all the precompiled binaries, https://github.com/cocoa-xu/exqlite/releases/tag/v0.13.2.
It should address all the concerns in #230, but if not, please let me know, and I'll be happy to help!

Oh this is awesome. I was never able to find time to do this. 🙌🏻 thanks!

@cocoa-xu released under v0.13.3 seems like a simple enough workflow.

Oh this is awesome. I was never able to find time to do this. 🙌🏻 thanks!

No worries! Happy to help!


Allow a path to declare a constant

Allow Foo::Bar = 1 to declare a constant Bar inside a module Foo.
Previously it was broken: it declared a constant Foo, discarding Bar completely.
Fixes #5742

@ysbaddaden it should have the same semantics as declaring class Foo::Bar when Foo doesn't exist - which is to create Foo as a module. If we want to change that semantic is a different discussion but this should be consistent with what we already have.

I'm using the same method as in the class name resolution, so it should work the same. But I'm can add a test for it if you want!

Yes please.

@ysbaddaden I've never gotten that error, maybe you're misremembering? I'm pretty sure it's always worked.


'storage' keyword should be highlighted in vs-code

Technically speaking, I'm not certain that "storage" is actually a keyword, but it should be highlighted regardless.

https://github.com/FuelLabs/sway-vscode-plugin/blob/5f7e27c41d5ec6fe5f750bb079dad44fd59cab68/syntaxes/sway.tmLanguage.json#L254

I'm not sure if this is something that should already be working, or needs to be done?
I'm using the latest version (v0.2.2) and it doesn't work yet.
cc @mitchmindtree


fix: ignore new spectrum packages in renovate

:tada: This PR is included in version 1.21.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


does ionic team have plan to transfer typings to npm @types?

npm @types seems more simple, no more extra modules needed.

Yes.
This is coming in beta 12 later this week or early next.
Thanks,
Dan


Weird data showing up in recent definitions

As originally discovered by @AlexWebYourmind  there are several licenses that recently started to show up and should most likely be entirely filtered out such as No_licend, See_license, No_license_found and more... I would guess these come from the addition of Fossology to the definition? These are not SPDX licenses, so IMHO anything not SPDX should be filtered out of the definition.
See for instance search reactive UI and 8ed94f5d40248f409177e2cdd82e2029e9f7c601 e.g. at https://api.clearlydefined.io/harvest/git/github/reactiveui/reactiveui/8ed94f5d40248f409177e2cdd82e2029e9f7c601?form=raw
(I would have given a website link but #318 is missing for now)

Note there was also '.git directories showing in the UI... but I cannot find that anymore.

Lets keep the issues separate. One is random text showing up for the license. this is likely a Fossology thing. More generally we should be checking the values coming from the harvested data when we put it into the definitions.
There is a separate problem of "ignoring" files/folders we know to not to be interesting. The .git folder is one example. I'm assuming that scancode is skipping those. It may be the way we are running fossology that is not.
Both of these are service issues and should be addressed there. @AlexWebYourmind , can you open/guide these appropriately?

I can take this on next week

This is now fixed for all components.

@dabutvin Thanks!


User auth-5-ua landing page-01

Draft PR.. Still have to fix the css to have it replicate the demo Brandon sent. Forgot password not yet working. Sign-in and Sign-up page not yet registering user's info because db hasn't been created yet. Lots of stuff still incomplete.
Regardless of all this, any and all feedback is greatly appreciated, thanks.

Thank you for the update, Robel!

Feel free to convert this from a draft pull request to a regular pull request.
Also don't forget to rebase your branch to the most recent main branch so you can resolve conflicts.

Just need another review to get this merged!

From my perspective, everything seems to be functioning properly. I have tested the home navigation, forgot password, and sign-up functions working correctly, redirecting users to the appropriate pages. The input text color matches the input field color, which may be difficult to see but can be adjusted later on.