170

QUESTION 170
This question requires that you evaluate the underlined text to determine if it is correct.
All Azure services that are in public preview are provided without any documentation.
Instructions: Review the underlined text. If it makes the statement correct, select “No change is needed”. If the
statement is incorrect, select the answer choice that makes the statement correct.
A. No change is needed
B. only configurable from Azure CLI
C. excluded from the Service Level Agreements
D. only configurable from the Azure portal

C


Cranelift: "register allocation: SSA(VReg(vreg = 145, class = Int), Inst(0))" in upcoming 0.93 when compiling with optimizations

.clif Test Case
test compile
set opt_level=speed_and_size
target aarch64

function u0:33() system_v {
    ss0 = explicit_slot 32
    sig0 = (i64, i64, i64, i64, i64) -> i64, i64 system_v
    fn0 = colocated u0:0 sig0
    jt0 = jump_table [block36, block38]

block0:
    v80 = iconst.i32 0
    v91 = iconst.i64 0
    v92 = iconst.i64 0
    v96 = iconst.i64 0
    v235 = iconst.i64 0
    v236 = iconst.i64 0
    v237 = iconst.i64 0
    v238, v239 = call fn0(v236, v237, v91, v92, v235)  ; v236 = 0, v237 = 0, v91 = 0, v92 = 0, v235 = 0
    v97 = iadd v238, v96  ; v96 = 0
    br_table v80, block37, jt0  ; v80 = 0

block36:
    trap user0

block37:
    trap unreachable

block38:
    v98 = load.i8 notrap v97
    v99 = fcvt_from_uint.f64 v98
    stack_store v99, ss0
    trap user0
}

Steps to Reproduce

clif-util test repro.clif

Expected Results
Compiles
Actual Results
Regalloc checker error: register allocation: SSA(VReg(vreg = 145, class = Int), Inst(0))
Versions and Environment
Cranelift version or commit: Cranelift 0.93 (43022c862a1d497f4b8c7c2fce47148775503add on the release-6.0.0 branch) which is not yet released.
Operating system: Linux
Architecture: AArch64

I'm taking a look at this -- there's something weird going on with an iconst dedup'ing to one that does not dominate all uses.


[Request] Gray Out Inactive Controls

Small request/suggestion from one of my users to make it more noticeable/prominent. Gray out the PvP rank value elements just like the element labels are currently for the PvP rank filters when disabled/inactive.

Like this?


Regarding lstm's full integer quantisation, does it quantise its intermediate output? The lstm full integer quantisation I get has only one input and one output quantisation parameter (minus the arguments)

1. System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

TensorFlow installation (pip package or built from source):

TensorFlow library (version, if pip package or github SHA, if built from source):

2. Code
Provide code to help us reproduce your issues using one of the following options:
Option A: Reference colab notebooks

Reference TensorFlow Model Colab: Demonstrate how to build your TF model.
Reference TensorFlow Lite Model Colab: Demonstrate how to convert your TF model to a TF Lite model (with quantization, if used) and run TFLite Inference (if possible).

(You can paste links or attach files by dragging & dropping them below)
- Provide links to your updated versions of the above two colab notebooks.
- Provide links to your TensorFlow model and (optionally) TensorFlow Lite Model.

Option B: Paste your code here or provide a link to a custom end-to-end colab
(You can paste links or attach files by dragging & dropping them below)
- Include code to invoke the TFLite Converter Python API and the errors.
- Provide links to your TensorFlow model and (optionally) TensorFlow Lite Model.

3. Failure after conversion
If the conversion is successful, but the generated model is wrong, then state what is wrong:

Model produces wrong results and/or has lesser accuracy.
Model produces correct results, but it is slower than expected.

4. (optional) RNN conversion support
If converting TF RNN to TFLite fused RNN ops, please prefix [RNN] in the title.
5. (optional) Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Hi @1778813838, can you please provide a reproducible code example? So that we may verify that what you are seeing is expected or not expected. Though in general, we recommend that use Transformers instead of LSTMs.


Input image form web url

Feature request
What are you trying to achieve?
It would be great if sharp is able to input images from web urls.
Similar to Jimp library like in this working example:
var Jimp = require('jimp');

  Jimp.read(
    "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png"
  )
    .then((image) => {
      image.write("GoogleImage.jpg");
    })
    .catch((err) => {
      console.log(err);
    });

When you searched for similar feature requests, what did you find that might be related?
See example here. For now I am using node fetch to get the image from web url.
What would you expect the API to look like?
See example here-above
What alternatives have you considered?
Using node fetch or use Jimp.
Please provide sample image(s) that help explain this feature
N/A

You'll need to add your own code to handling networking and all the additional problems this can bring - please see #1901


Reported issued after importing Python libraries (at the beginning), running Python 3.6

Description
What steps will reproduce the problem?

At the beginning when importing libraries and attempting to read the files;, running Python 3.6
Spyder screens come up alerting it should be reported and the program does not run (it ran before)

What is the expected output? What do you see instead?
Please provide any additional information below
File "C:\Users\t\Anaconda3\lib\site-packages\spyder\utils\external\lockfile.py", line 150, in lock
symlink(str(os.getpid()), self.name)
File "C:\Users\t\Anaconda3\lib\site-packages\spyder\utils\external\lockfile.py", line 88, in symlink
rename(newlinkname, filename)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\Users\t\.spyder-py3\spyder.lock.1508512464094.newlink' -> 'C:\Users\t\.spyder-py3\spyder.lock'
Traceback (most recent call last):
File "C:\Users\t\Anaconda3\lib\site-packages\spyder\app\start.py", line 109, in main
lock_created = lock.lock()
File "C:\Users\t\Anaconda3\lib\site-packages\spyder\utils\external\lockfile.py", line 159, in lock
pid = readlink(self.name)
File "C:\Users\t\Anaconda3\lib\site-packages\spyder\utils\external\lockfile.py", line 96, in readlink
fObj = _open(os.path.join(filename, 'symlink'), 'rb')
PermissionError: [Errno 13] Permission denied: 'C:\Users\t\.spyder-py3\spyder.lock\symlink'
Traceback (most recent call last):
File "C:\Users\t\Anaconda3\lib\site-packages\spyder\app\mainwindow.py", line 2446, in execute_in_external_console
console.ex

Are you running this program as an user with administrator rights? If not could you try that and see what happens?

Please open a terminal (cmd.exe), run
spyder --reset

and try to start Spyder again.

This worked.   Thanks

Great! Thanks for letting us know about it!


ARROW-16715: [R] Bump default parquet version

Also removes deprecated args to write_parquet()

@github-actions crossbow submit test-r-arrow-backwards-compatibility

@jonkeane do you want to take a look at this? I seem to recall you set up the backwards compat testing.

This looks good to me. I'm a little bit surprised that the backwards compatibility all the way back still is fine, but we must not test any types that have differed since the 2.0 version.

It may also be that the library can read whatever version file it receives (at least as far back as we test)


SendGrid Access Issue

Expected Behavior
Sendgrid will work in production enviornment.
Actual Behavior
Sendgrid does not work in production. This appears to be related the same issue as the AWS compromise. A new account has been instantiated and will be implemented following a fix for this issue.
Steps to Reproduce
Try to use Sendgrid for emails in production on Heroku.

Fixed with #44


Allow explicitly specifying file 'language'regardless of file extension

this is especially helpful when creating a new file or opening a temporary edit window
you won't get any language feature like syntax highlighting until you save the file with the right extension which is very annoying

If you are running brackets from source, you can fix this by commenting the following statement in https://github.com/adobe/brackets/blob/master/src/editor/EditorStatusBar.js
languageSelect.$button.prop("disabled", doc.isUntitled());
Brackets uses tern to generate code hints for a file using other files in the same directory. Since this information is not available for an untitled file, code hints will not work for the untitled file, but syntax highlighting will work.

A number of Brackets features will break if you were to change the language of an Untitled document, including the JS code hints mentioned above.
Please upvote Switch language / type of Untitled documents on our feature backlog for this to be officially supported.  I'll close this bug to avoid duplication with what's already tracked on the backlog.


PromQL engine: Support automatic inclusion of info metric data labels.

Make the PromQL engine, when configured to do so, automatically include info metric data labels when expanding time series. This is intended as experimental functionality, although I haven't yet marked it as such (the PR is WIP after all).
See proposal Simplify joins with info metrics in PromQL for background. The automation in this PR should form the basis of the proposed info function, just with configurability to execute without explicit function calls in queries. I don't know for sure if this will be desirable UX in the end, but at least it's helpful during development.
This is especially useful for OpenTelemetry, since it enables automatically adding back (select) OTel resource attributes (encoded as target_info metric labels) to metrics (cheaper than promoting them to labels at ingestion time).
BenchmarkInfoFunction shows great performance gains vs. traditional join queries against info metrics (due to only fetching info metrics with matching join labels mostly):
$ benchstat join.txt info.txt
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/promql
cpu: AMD Ryzen 9 3950X 16-Core Processor
                                                                  │   join.txt   │              info.txt              │
                                                                  │    sec/op    │   sec/op     vs base               │
InfoFunction/Joining_info_metrics_with_other_metrics_example_1-32    7.013m ± 8%   1.858m ± 2%  -73.51% (p=0.002 n=6)
InfoFunction/Joining_info_metrics_with_other_metrics_example_2-32   12.814m ± 6%   1.907m ± 3%  -85.12% (p=0.002 n=6)
geomean                                                              9.480m        1.882m       -80.14%

                                                                  │   join.txt   │              info.txt               │
                                                                  │     B/op     │     B/op      vs base               │
InfoFunction/Joining_info_metrics_with_other_metrics_example_1-32   451.6Ki ± 0%   431.2Ki ± 0%   -4.50% (p=0.002 n=6)
InfoFunction/Joining_info_metrics_with_other_metrics_example_2-32   705.3Ki ± 1%   433.5Ki ± 0%  -38.54% (p=0.002 n=6)
geomean                                                             564.3Ki        432.4Ki       -23.39%

                                                                  │  join.txt   │              info.txt              │
                                                                  │  allocs/op  │  allocs/op   vs base               │
InfoFunction/Joining_info_metrics_with_other_metrics_example_1-32   3.474k ± 0%   3.272k ± 0%   -5.81% (p=0.002 n=6)
InfoFunction/Joining_info_metrics_with_other_metrics_example_2-32   5.463k ± 0%   3.299k ± 0%  -39.62% (p=0.002 n=6)
geomean                                                             4.357k        3.285k       -24.59%

TODO:

[x] Fix instant queries involving rate
[ ] Don't try to augment info metrics

I think as a PoC, this is interesting. It would be interested to feed this to a test audience to see how this flies.
However, I have concerns if this should ever be generally accessible, even behind a feature flag. In general, introducing a whole new configuration section on the PromQL level rubs me the wrong way.
Feature flags are already putting conditions to how a given PromQL query is executed, but at least those are fairly simple binary decisions. This config is quite complex and deep, and it will essentially change how PromQL works in a fairly invasive fashion. For exploring a feature, that's fine. But I don't want to live in a world where people look at a PromQL expression, and the next question they ask is if they could see the PromQL engine config used to evaluate the expression.
I'm ~fine with configuring scrapes so that info metrics can be described there. That's just changing the input data on ingestion. (But of course, for that to work, we need to implement storing this information in the TSDB, and that's a much bigger barrier.)

But wouldn't this have similar result of "give me PromQL expression and also your scrape target /metrics text,so I know what are the target labels configured there?"

I think it's a reasonable thing to say "the outcome of your PromQL expression depends mostly on what's in your TSDB". That's why I think if there is ingestion config that changes the date ultimately stored in the TSDB, that's nothing new and expected. Adding an additional config is a different thing. (Feature flags are already like that, but they are simple and inherently meant for experimental things.)

Mechanisms looks fine, but do you mean you want to change your design to attach your labels by default with info function? That might be easy to use, but likely introduce overhead and potential surprises. It feels an explicit function might be better?

Yeah, I am also in the "explicit is better than implicit" camp. I imagine the desire for "auto-adding" labels from info metrics comes from one problem with the "map all the labels on ingestion" approach: With the latter, you cannot change your mind later. If you realize you want to map more labels, you cannot easily apply that retroactively to historic data. But you can always change the query-time config and use it to query historic data.

I share Beorn's and Bartek's concern here, I don't think a query language should give different results depending on configuration. I believe the main use-case for the info() function is to unblock folks coming from OpenTelemetry background and it's already set in stone that their info metric is target_info. It's fine to have this hardcoded.
I understand that this would unblock with other info metrics, mostly coming from kube-state-metrics, but I'd love to see this being solved with an approach that doesn't require query language configuration. Bonus points if we don't even need the info() metric at all, e.g. storing info as metadata and make metadata part of query results 😬

I imagine the desire for "auto-adding" labels from info metrics comes from one problem with the "map all the labels on ingestion" approach: With the latter, you cannot change your mind later. If you realize you want to map more labels, you cannot easily apply that retroactively to historic data.

The second problem that arises is that when changing which OTel resource attributes to promote to labels, you also have a temporary doubling in series (i.e. old label set + new label set).

I understand that this would unblock with other info metrics, mostly coming from kube-state-metric.

@ArthurSens actually, the point of this approach is rather the different UX of not having to call info in queries. It's been a user feedback, that they don't want to have to change their queries. Potentially, this PR could also be changed to only support target_info with instance/job as the identifying labels.

the point of this approach is rather the different UX of not having to call info in queries. It's been a user feedback, that they don't want to have to change their queries.

I think that's the core of what we are discussing here. Some users want to "automatically" change the result of their queries without actually changing the queries. This rubs some of us the wrong way, because now we make the outcome of a query depend on some external setting.
I think it is fine to "preprocess" the query in a wrapper, separate from core Prometheus. I assume the kind of users requiring "change the result of my query without changing the query" will work through some kind of UI that could do that for them.
To solve this within PromQL, maybe it could be piggybacked into the vaguely planned with feature, which currently is meant to just add additional label matchers to all selectors in their scope. But now we are talking about combining two not-yet-implemented features (info function and with), so maybe let's first see how the info function flies.

To solve this within PromQL, maybe it could be piggybacked into the vaguely planned with feature

Thanks @beorn7! I had never heard of with.

It is very vaguely planned. Hidden in a dev summit note from ages ago…
https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit?pli=1#bookmark=id.m8eq3rv3yopu

Closing PR, as it's not what the team would like to proceed with :)


TabView with badges - When header text is long with badges, it overlaps and looks all messed up

Description
Create tabviewitems with longer texts and show badges also. The tab header does not consider the Badge space and kind of overlaps with one another.

Another issue with the last tab

On IOS, badge shows in the middle,

Expected Behavior
It should have proper spaces around itself
Actual Behavior
It overlaps and gets messed up
Version information
Xamarin Toolkit - Latest version - 1.0.2
Android version - API 30
Xamarin 5
Workaround
Tried setting padding etc., but not successful yet.
Any workarounds if any appreciated

Is there any fix or workaround for this issue? I'm using XCT 1.2.0 and the problem still persists. I'm trying to use badges in iOS, my tabs do not have icons but the badge still appearing in the center


Added new module mail_id_harrvest as per issue #143 also solved issue #200

Checklist

[x] I have followed the Contributor Guidelines.
[x] I have added the relevant documentation.
[x] My branch is up-to-date with the Upstream master branch.

Changes proposed in this pull request
I have added tool mail_id_harvest for better passive information gathering and it has no conflict with master.
-right now google is only making request but there is few exception is there which are been added soon as i am tackling that issue
Your development environment

OS: Parrot
OS Version: 4.19.0
Python Version: 3.x

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 8

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  7

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 6

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  7

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 5

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  7

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 1

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 1

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 1

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 2

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  5

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 1

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  5

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  19

Clones added
============
- lib/scan/mail_id_harvest/engine.py  5

See the complete overview on Codacy

@itsdivyanshjain, please look into the PEP8 standards for writing code.

@shaddygarg can you review and test this one also?

 Here is an overview of what got changed by this pull request:

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  20

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  22

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  22

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- lib/scan/mail_id_harvest/engine.py  22

Clones added
============
- lib/scan/mail_id_harvest/engine.py  6

See the complete overview on Codacy

I also solved issue #200 for py3dns by changing it to version 3.2.1 and also do review for mail_id_harvest module

closing due to high time complexity of a code

closing due to high time complexity of a code


Sort optimization suggestions

Description
We are replacing presto in the system with velox, but we have done a lot of self-research and optimization on performance based on presto. We found that there is a performance gap between directly using velox to replace presto and using our optimized presto. In the specific Sort scenario, we concluded after analysis that it may be caused by the misalignment of the processing methods in the following places:：

Null value check
When the 'null value' in velox is stored in RowContainer, it is stored in the form of null flags + T() default value. When comparing, nullFlags will be read first, and then the data will be read.
The improvement we made is that we use the min/max of T according to compareFlags when storing, and directly read the data when comparing, eliminating the overhead of reading the null flag in one step.

Prefix sort
When compare(), the data is read directly from the underlying data structure. Whether it is Presto or Velox, there is some additional memory access overhead. In Presto, a compare requires reading data in different columns, while in Velox, the data Stored in a row-base manner, in addition to the sort key, the RowContainer also contains null flags, normalize data, and payload. In this way, the data required for sorting (sort key) is stored in a larger memory block than actually required. There will be additional cache misses.
The optimization we did was to introduce a prefix column in PageIndex/RowContainer, which is a vector< long > or vector< byte[] >:

For Byte/Integer/Bigint, it is directly stored in the prefix
For Varchar, take the first 8 bytes of the string and store them in prefix
For the case of multiple columns, use a fixed-length byte[] to store a prefix. If Byte/Integer/Bigint is written into the prefix, when the first Varchar is encountered, the bytes of the remaining length of the prefix are written into the prefix. Also ignore remaining sort keys
When compare(), compare the prefix first. If the prefix is equal, then read the data from the underlying data structure. If the prefix has a higher selectivity, the performance of sort will be improved.

Range Sort
In our system, we previously implemented "range sort" based on the column-base data structure in PagesIndex (presto). The general logic is:
Suppose there is a sort statement order by a, b, c. First sort column a. If there are repeated data in column a, use the repeated data to construct several ranges, and then sort column b. When sorting, you only need to sort the items in the range, and then process column c in the same way.
However, this algorithm is based on the column-base data structure. Currently, RowContainer stores data in row-base. Can RowContainer support column-base storage?

To summarize the issues :

Are the above improvements to Sort reasonable?
Currently, velox’s sort implementation is simply std::sort. Does velox have plans to optimize sort performance in the future?
In practice, how should we quickly incorporate optimization suggestions into velox?

Hi @skadilover, welcome to the community. Would you introduce yourself? Where are you from?
The optimizations you described above sound interesting. Are you interested in trying them out, iterating and eventually incorporating them into Velox?

@mbasmanova
I am an engineer from Alibaba, we are trying to use Velox to speed up Presto jobs.
Very glad that the community is interested in these ideas. Add something that was not mentioned in the above description:

The core idea of 'prefix sort' is similar to the 'Binary String Comparison' described in duckDB article(https://duckdb.org/2021/08/27/external-sorting.html#binary-string-comparison), 'It encodes all columns in the ORDER BY clause into a single binary sequence', in our case it is called 'prefix'.
Based on "remove null check" and "prefix sort", I have completed a prototype. In a single column BIGINT scenario, E2E tested the sort operator with a 40% performance improvement. I am organizing the code in the hope of integrating it into the community.
Finally, I currently need some help and guidance on how to benchmark the sort operator. I have not found any examples of benchmarking an operator. Can anyone give me some advice?

I am an engineer from Alibaba, we are trying to use Velox to speed up Presto jobs.

Welcome. What's your name? I only see GItHub handle skadilover.
I'm curious whether the optimizations to Presto you mentioned earlier are available in the prestodb repo or if these are internal enhancements.

Based on "remove null check" and "prefix sort", I have completed a prototype. In a single column BIGINT scenario, E2E tested the sort operator with a 40% performance improvement. I am organizing the code in the hope of integrating it into the community.

Fantastic. Looking forward to the PR.

Finally, I currently need some help and guidance on how to benchmark the sort operator. I have not found any examples of benchmarking an operator. Can anyone give me some advice?

Check out velox/functions/prestosql/aggregates/benchmarks/ReduceAgg.cpp
This is a benchmark for reduce_agg aggregate function which benchmarks a mini query (table scan + aggregation). I suggest you write similar one to benchmark table scan + order-by. One may comment that this doesn't benchmark order by in isolation, but that's Ok. After all there is no point to speed a portion of the query that's not using significant amount of CPU time to begin with. E2e query speed up is what we are after.
Hope this helps.

The core idea of 'prefix sort' is similar to the 'Binary String Comparison' described in duckDB article(https://duckdb.org/2021/08/27/external-sorting.html#binary-string-comparison), 'It encodes all columns in the ORDER BY clause into a single binary sequence', in our case it is called 'prefix'.

This makes a lot of sense. Excited to see this optimization coming to Velox.
CC: @amitkdutta @aditi-pandit

I am an engineer from Alibaba, we are trying to use Velox to speed up Presto jobs.

Welcome. What's your name? I only see GItHub handle skadilover.

My name is Heng Jiang.

I'm curious whether the optimizations to Presto you mentioned earlier are available in the prestodb repo or if these are internal enhancements.

Not available in the prestodb repo, these are internal enhancements.

Fantastic. Looking forward to the PR.

Next week is our National Day holiday (10 days). After the holiday, I will continue to organize the code and submit the PR as soon as possible

@mbasmanova @skadilover
here's one related patch on s/std::sort/timsort.
https://github.com/facebookincubator/velox/pull/6745
We got ~2x performance on some cases.
thanks, -yuan

Hi guys, sorry about the delay to follow up on this thread. I've recently helped reviewed this great paper about sorting optimization (which eventually got published at TODS); linking it here for folks who are interested in the subject:
https://arxiv.org/pdf/2209.08420.pdf
This sounds all very interesting. It would be super helpful if as a first step we established a microbenchmark to be able to tweak and reproduce these optimizations.
A few specific questions:

Have you guys also looked at offset-value coding? By reading the paper is seems like a great idea, but I have never seen a real system implementing it - the google folks at the paper claim they use it internally though.

"The improvement we made is that we use the min/max of T according to compareFlags when storing, "
How does this work if you actually have the min/max values as part of the data?

"Prefix sort"  this sounds like the the same idea as what we have implemented for StringViews. Does that not help in this situation, or we only need a similar optimization for other variable-length types?

Hi guys, sorry about the delay to follow up on this thread. I've recently helped reviewed this great paper about sorting optimization (which eventually got published at TODS); linking it here for folks who are interested in the subject:
https://arxiv.org/pdf/2209.08420.pdf

@pedroerp  Glad to hear your response for this issue!~  The paper mentioned above is very useful.
PrefixSort is not just a normalizekey technology: it is different from velox’s existing storage structure (RowContainer
) It hopes to encode the sort key separately like duckdb. At the same time, in order to avoid the performance degradation of the swap method in scenarios where the sort keys are particularly long, we only encode the first few columns in the form of prefix.
In order to facilitate communication, I submitted a PR https://github.com/facebookincubator/velox/pull/7230 (it is not 100% ready yet, some code will be adjusted based on my E2E test results), and the design document is also being prepared (it will take a few days).If you have time, you can help me take a look at this PR first.

Have you guys also looked at offset-value coding? By reading the paper is seems like a great idea, but I have never seen a real system implementing it - the google folks at the paper claim they use it internally though.

@pedroerp  Yes, for some specific columns we may refer to the offset-value encoding format, but this work may be done later. I hope to optimize the single-column bigint/varchar scenario first, especially in the bigint scenario. radix-sort

"The improvement we made is that we use the min/max of T according to compareFlags when storing, "
How does this work if you actually have the min/max values as part of the data?

Directly store min/max instead of T() (what velox does now)

"Prefix sort"  this sounds like the the same idea as what we have implemented for StringViews. Does that not help in this situation, or we only need a similar optimization for other variable-length types?

Like question 1, I think this is a coding issue. I will discuss it scene by scene in the subsequent PRs.

This sounds all very interesting. It would be super helpful if as a first step we established a microbenchmark to be able to tweak and reproduce these optimizations.

@pedroerp
At present, I can only verify the optimization results through E2E testing (our framework is similar to presto-cpp). The sort optimization results are affected by the size of the data set. I am currently using a 1T tpch table. Regarding building similar scenarios microbenchmark, could you give me some reference examples? That would be very helpful in my current job

Regarding building similar scenarios microbenchmark, could you give me some reference examples? That would be very helpful in my current job

This is a good example. Maybe come up with something similar to this that generates and sorts a dataset, then we could experiment with your optimization and see how much they improve performance?
https://github.com/facebookincubator/velox/blob/main/velox/exec/benchmarks/MergeBenchmark.cpp

Regarding building similar scenarios microbenchmark, could you give me some reference examples? That would be very helpful in my current job

This is a good example. Maybe come up with something similar to this that generates and sorts a dataset, then we could experiment with your optimization and see how much they improve performance?
https://github.com/facebookincubator/velox/blob/main/velox/exec/benchmarks/MergeBenchmark.cpp

Thanks a lot,  This example looks like it could work. I will try it soon

@skadilover Wondering if you had a chance to review Arrow Row format: https://docs.rs/arrow-row/latest/arrow_row/index.html
"Rows are normalized for sorting, and can therefore be very efficiently compared, using memcmp under the hood, or used in non-comparison sorts such as radix sort. This makes the row format ideal for implementing efficient multi-column sorting, grouping, aggregation, windowing and more, as described in more detail in this blog post."

Some additional references: https://citeseerx.ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>.1080&rep=rep1&type=pdf

Some additional references: https://citeseerx.ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>.1080&rep=rep1&type=pdf

@mbasmanova  Thanks for the references aboeve.
The encoding tech (include  "normalized key", "memory compare" etc. ) can be very efficiently , and we have implemented them in our inernal presto before. In PrefixSort optimization, I hope to implement them step by step. This week I plan to prepare a design document (including some research E2E tests and benchmarks) to describe my plan in detail.

@skadilover Looking forward to hearing more from you on this topic. Thanks.

Overview
PrefixSort's optimization plan includes two optimization points: normalized keys optimization and cpu cache optimization.
Normalized Key For Multi Sort Keys
The core idea of normalized key is: assuming that sort keys can be encoded into a binary string and the sorting semantics remain unchanged, then all sort keys can be regarded as a whole for sorting (see [1]). At this time, memcmp can also use simd to speed up. In the submitted PR implementation, in the scenario of two columns of BIGINT, we can get a performance improvement of 200%-300%.

The benchmark test results are as follows:

PrefixSort currently supports BIGINT, and the current research results of other fixed-length types are that they can be implemented. The VARCHAR scenario will be a little complicated. We also studied the implementation of duckDB, and the following somewhat special method was used in its implementation. Range sort is used to process varchar. Further research and testing of the encoding format will be conducted to improve this part of the design:

Memory Data Structure
In order to implement the above-mentioned Normalized Key technology in velox, we consider redesigning a data structure to store the normalized key for the following two reasons:

The Sort algorithm requires an iterator with "Random Access Iterator" (TODO link) semantics see[3][4], and the RowContainer data currently uses non-contiguous allocation to store row data, which is implemented on the basis of non-contiguous memory. Random Access Iterator has performance problems and requires complex pointer mapping.
Currently, the normalized field in RowContainer does not serve for sort and does not contain compareFlags semantic information. In addition, the processing of null values will also affect the implementation of the compare method.

First, we review the current OrderBy operator implementation of velox, as shown in the figure below:

The general process of implementation is:

Get the std::vector<char*> returnRows of the row pointer through the listRow() method of RowContainer. The reason for this step is that the sort algorithm requires a contiguous memory block to facilitate the implementation of the iterator.
Give the begin/end iterator of returnRows to std::stort for sorting. The logic of the custom compare method provided for std::vector is: first get the row ptr from returnRows, and then use ptr to access the data in the RowContainer.
From the perspective of CPU cache, the problem with this implementation is:
Research on the sort algorithm basically has an assumption: the memory access operation is constant and low-time consuming. This assumption is wrong in engineering practice. In engineering practice, we need to consider the issue of memory access efficiency. Here is a classic example see [2]:

Going back to the implementation of the OrderBy operator, we can find that the compare method in the sort loop needs to access two memory blocks (address vector and row container) at the same time. As the memory grows, memory access problems will significantly affect the efficiency of sort execution. . The solution to this problem is to store the sort key and row Num separately. Take duckDB as an example:

In this solution, the performance of the compare method is improved, while the performance of the swap method will decrease (swapping all sort keys will be slower than swapping only row addresses). When the number of sort columns increases to a certain extent, the overhead of swap will exceed the benefits of the compare method. , so there needs to be a rollback mechanism here. The design of PrefixSort is as follows:

Use row ptr to replace row num in the implementation of PrefixSort. The purpose of this is:

There can be a fallback mechanism when the size of sort keys is greater than a certain value.
At the same time, using row ptr can also better adapt to the current velox operator implementation and facilitate subsequent output operations.
Data in RowConatiner can be reused without re-storing payload data.

In the E2E test of single column BIGINT, we can get 30%-40% gain. In fact, it will be faster if we use reintercept_cast to get the value, but for the unified implementation of normalize key, memcmp is used here for comparison. In long scenarios, memcmp is slower than reintercept_cast. In the implementation, template memcmp in duckDB is used to replace std::memcmp. There is some improvement in the effect, but it is still slower than the implementation of reintercept_cast.
In addition, directly using std::vecotr to store keys during prototype testing can achieve greater performance improvements. However, considering the maintenance cost of the overall code and the possibility of changing the subsequent sort algorithm to RadixSort, this is not currently possible to consider this optimization:

Conclusion and plan
By optimizing the memory data structure and using Normalize technology, PrefixSort will have performance improvements compared to the original implementation in both single-column and multi-column cases. It is planned to be divided into 4 stages to achieve complete functions.

[done] PR#1  a Implement PrefixSort framework code b Support BIGINT c Implement the BenchMark testing framework
[on going] PR#2 Supports all basic types including (INT/SORT/TIME type/DOUBLE...)
[on going] PR#3 Support VARCHAR type
[suspend] PR#4  Support Radix Sort

How To Reproduce Optimization
Just Run PrefixSortBenchmark.cpp && OrderByTest.cpp
reference：
[1] https://citeseerx.ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>.1080&rep=rep1&type=pdf
[2] https://courses.cs.washington.edu/courses/cse373/21sp/lectures/lecture24.pdf
[3] https://en.cppreference.com/w/cpp/iterator/random_access_iterator
[4] https://en.cppreference.com/w/cpp/algorithm/sort

@mbasmanova  updates for the topic.

@skadilover Thank you for detailed description of the proposed design. I have some questions to make sure my understanding is correct. Does this design involve making a copy of sorting keys and row pointers for all rows using continuous allocation? I'm thinking about the table with 2 columns (normalized key and row ptr) in the diagram.

@skadilover Thank you for detailed description of the proposed design. I have some questions to make sure my understanding is correct. Does this design involve making a copy of sorting keys and row pointers for all rows using continuous allocation? I'm thinking about the table with 2 columns (normalized key and row ptr) in the diagram.

yes ，there is a copy

@skadilover Got it. What are your thoughts on the limit of the normalized key size? There can be lots of columns and some columns can be variable-width, hence, I assume we need to put a hard limit on the "prefix" of the normalized key. Also, can you extend the design to support complex types as well?

@skadilover Got it. What are your thoughts on the limit of the normalized key size? There can be lots of columns and some columns can be variable-width, hence, I assume we need to put a hard limit on the "prefix" of the normalized key.

If the type of column that needs to be sorted can be normalized, I think there is no need to limit the size of the Prefix. The reason is that as seen in the benchmark results, the execution time in this scenario does not increase significantly as the number of columns increases.
If only one part can be normalized, such as String_view, the remaining parts need to be compared using pointers. From a design perspective, if the remaining columns can be normlized, we can continue memcmp after comparing using pointers.But this goes against our original intention of treating Prefix as a whole and using memcmp.In this scenario I think it would be better to only store the first part normalized column in prefix.When VARCHAR scenarios are supported in the future, judgments will be made based on benchmark results.Currently in code design, we can flexibly define behavior through PrefixSortLayout(holding relevant information)
In some special scenarios, our customers know the data distribution. For example, the first column that needs to be sorted has high selectivity, and there is no need to store the remaining sorted columns in Prefix (especially if there are many columns left). , we hope that our customers can control the size of Prefix through SQL Hint, leaving room for optimization for specific scenarios.
Therefore limit size is not a "hard code". Users can set it through queryConfig. There is no limit by default.

Also, can you extend the design to support complex types as well?

There is no essential difference between complex type and varchar. If necessary, we can store some necessary meta-information and potential sorting prefixes in Prefix. Like Varchar, if complex cannot obtain benefits through memcmp, I feel that it is possible to use only pointers for comparison will be better .

@mbasmanova  the pr close above is not used, stil use  https://github.com/facebookincubator/velox/pull/7230 to merge code.

@mbasmanova Is there more questions about the topic ? Could you help me to review the code?

@skadilover Thank you for working on this. Took a quick look and left some initial comments on the PR.

@skadilover

Therefore we`d better to provide a config option for users to set limit of prefix size, and in our practice there is no limit by default.

I think we need to have a sensible limit by default. I'm also interested in figuring out support for varchar and row() keys as these are common.

I think we need to have a sensible limit by default.
I will check it later.

I'm also interested in figuring out support for varchar and row() keys as these are common.
Our previous method of dealing with varchar was to 1. analyze the maximum length of the data before sorting and then process it according to a fixed size, 2. only retain the prefix for those that exceed the length limit. I hope to find a better way. Currently  I am still doing relevant research work.

@mbasmanova
As we discussed before, I have started to split the original PR into 3 sub PRs and merge them according to the dependencies. Could you help me  to review this PR first?

@skadilover How does the prefix sort compared with radix sort and Google's SIMD quicksort? On integer types of fixed length, the combined MSB/LSB radix sort can easily be 3x faster than std::sort on the scale of millions and up numbers. Do you have benchmarks to compare the prefix sort with these algorithms? My current impression is that the Prefix sort may be good for sorting strings, and in general, compare based sorting algorithms can't beat the radix type sorting on fixed width integers on large scale.

@skadilover How does the prefix sort compared with radix sort and Google's SIMD quicksort? On integer types of fixed length, the combined MSB/LSB radix sort can easily be 3x faster than std::sort on the scale of millions and up numbers. Do you have benchmarks to compare the prefix sort with these algorithms? My current impression is that the Prefix sort may be good for sorting strings, and in general, compare based sorting algorithms can't beat the radix type sorting on fixed width integers on large scale.

The core idea of this opt is 'use binary string to opt mulit columns sort', use quick-sort just for speeding up things.

We will also try to optimize sort, initial proposal is:
Currently, the common usage is to request space for each row.
Firstly, we can request all the rows memory at once, and newRow is only for fixed size column, maybe we can also optimize string column at the same way.
for (auto row = 0; row < input->size(); ++row) {
    char* newRow = inputData_->newRow();

    for (auto i = 0; i < inputs_.size(); ++i) {
      inputData_->store(decodedInputs_[i], row, newRow, i);
    }

    addNewRow(groups[row], newRow);
  }

Secondly, currently we will spill input for memory pressure, we can also request all the memory in the addInput function, and test if we can get the memory before request, if not try to sort and spill output as spark does, sort will not request any memory in that case.
What Spark does is:

Insert record and check if the memory is enough, if not, sort and spill https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L491
Spill and sort: https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L234

Will add a sort benchmark and sort spill benchmark to track the performance.
First PR is https://github.com/facebookincubator/velox/pull/10041
Please let me know if it is acceptable. Thanks!

@jinchengchenghh
I think this feature you proposed has no conflict with the optimization of prefix-sort.
I think you can file a separate issue and let the code maintainer know, which may make things progress faster.

Yes, because this title is Optimize sort, same topic with mine, so I add it here, And I would like to apply the second optimization after prefix sort merged. Otherwise it will make it hard to enable prefixsort because it requires sort does not require memory.


Add files field to reduce package size #6

Addresses #6

Coverage remained the same at 100.0% when pulling aab114a1d61debc07da26081d0f55b27f8aee884 on smuszel:master into bc898eeee723833fc9d604523e00014350bcc4e1 on epoberezkin:master.

@epoberezkin would be cool to have index.d.ts added and then this merged ❤


exec error: attribute "noe" not found on module object

I'm getting an error: exec error: attribute "noe" not found on module object when I run the Risor as a REPL and make a typo like so:
>>> time.noe()
exec error: attribute "noe" not found on module object

This error is expected, but from then on I get the exact same error, no matter if I fix it:
>>> time.now()
exec error: attribute "noe" not found on module object

Have to close the REPL and restart.

Thanks @avioli - I just found this too. I'll have a fix tonight or tomorrow!

No rush. All else I played with is looking great. Excited to play with Risor.

I stumbled upon another REPL issue - when it evaluates an expression as an error, that error persists until I restart the REPL:
~> risor
Risor

>>> uuid.v5()
type error: uuid.v5() takes exactly 2 arguments (0 given)
>>> uuid.v5("one", "two")
type error: uuid.v5() takes exactly 2 arguments (0 given)
<Ctrl-D>

~> risor
Risor

>>> uuid.v5("one", "two")
"e3908a09-438b-5436-92b8-8e17887a14d3"
>>>

Fixed in https://github.com/risor-io/risor/pull/104


Minified js Returns First Item Only

When I am using typeahead minified (boostrap3-typeahead.min.js), it does not allow me to choose any other items in my dropdown list except the first item. This means even if the dataset consists few items like
source: [     {id: "someId1", name: "Display name 1"},     {id: "someId2", name: "Display name 2"},     {id: "someId3", name: "Display name 3"}   ],
but no matter which display name I click, it returns the display name 1. Same issue does not happen to the full js, just the minified js.

It looks like this question was opened in the wrong repository.
For the file boostrap3-typeahead.min.js, the correct repository is: bassjobsen/Bootstrap-3-Typeahead,


Examples to convert tensorflow trained model to Keras model

It would be nice to include an example showing how to programmatically convert a tensorflow model to keras model such that model learns from Tensorflow can be imported into Keras for further development/experiment.

TF is much more flexible than Keras. Making a parser from TF to Keras would be a humongous task.
Has anyone started a project like that?
TF at its core has no concept of layers it's just a big graph.

How can I convert a. pb file to .h5.file? I have frozen_inference_graph.pb from tensorflow, and I need to have h5 format for loading weights in Keras

@dextroza @GustavZ I think you might benefit from this issue:
https://github.com/keras-team/keras/issues/8026

If you want to convert only weights (suppose you have code for the same model), you have to create model with random weights (you can find InceptionV3 in keras.applications) then read the TensorFlow .ckpt file with tf.train.NewCheckpointReader then call set_weights() method for each corresponding layer.

@bottydim thank you for the suggestion!

This repo has a .ipynb notebook for this.
https://github.com/nyoki-mtl/keras-facenet/blob/master/notebook/tf_to_keras.ipynb


Attack parameters are wrongly saved to model checkpoint

https://github.com/IntelLabs/MART/blob/2cb258dec4abd52e2468fc8445ab9464d02c9e9c/mart/attack/adversary.py#L94
We used to do self.perturber = [perturber] and reference it as self.perturber[0]. Do you know why it was changed? @dxoigmn

There was no comment as to why that particular line was necessary, so I removed it because none of the tests failed.

I wonder if PyTorch supports a more elegant way of excluding certain members from module.parameters().


New version: ManifoldSoftware.Manifold.9 version 9.0.180

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.4 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

I do not see what is wrong?

Duplicate of #100297

Duplicate of #100297

No, not a duplicate. #100297 is for Manifold Viewer and this is for full Manifold application.


js2xml and CDATA

Hi everyone.
I can't handle CDATA insertion on my js formated object using js2xml.
Here is a piece of code i tried :
import x2js from 'x2js';
[...]
const parser = new x2js();
const jsonData = {};
jsonData.items = {};
jsonData.items.item = [];

const someValue = 'hello world';

jsonData.items.item.push({
  someKey: `<![CDATA[${someValue}]]>`,
})

console.log(parser.js2xml(jsonData);)

(code can be executed here : https://runkit.com/5857a3ffdd1b580013a9d722/5857a3ffdd1b580013a9d723)
Here is the output : "<items><item><someKey><![CDATA[hello world]]></someKey></item></items>"
How can I get rid of <![CDATA[ and ]]> and place the legit <![CDATA[ ans ]]> ?
Regards,
Florent

As far as I know, the library does not have a feature that enables you to accomplish this. All emitted values are XML-escaped, which should eliminate any need for CDATA.
Is there some particular reason you desire to use CDATA instead of normal XML-escaping (given that they are just alternatives to each other)?

Hi.
Back from the dead :)
I don't remember well how I managed to resolve this, but to answer your question, the reasons I desired to use CDATA instead of normal XML-escaping, was purely a business request, impacting though the developpement process.
Regards,
Flo

Put your data in a __cdata field. Plain content, the surrounding cdata-tag is being added by x2js.
var myTitle = 'The brown fox';
var myDescription = 'The quick brown fox jumps over the lazy cat.";
let data = {
title: myTitle,
description:  [{__cdata : myDescription}],
};
will result in

The brown fox
The quick brown fox jumps over the lazy cat.


V3

Codecov Report

Merging #384 into master will decrease coverage by 0.02%.
The diff coverage is 96.29%.

@@             Coverage Diff              @@
##             master     #384      +/-   ##
============================================
- Coverage     91.48%   91.45%   -0.03%
- Complexity      192      194       +2
============================================
  Files            10       10
  Lines           493      503      +10
============================================
+ Hits            451      460       +9
- Misses           42       43       +1

Impacted Files
Coverage Δ
Complexity Δ

src/Commands/GenerateDocumentation.php
94.69% <ø> (ø)
35 <0> (ø)
:arrow_down:

...c/Tools/ResponseStrategies/ResponseTagStrategy.php
100% <100%> (ø)
6 <1> (ø)
:arrow_down:

.../Tools/ResponseStrategies/ResponseCallStrategy.php
94.69% <100%> (+0.09%)
38 <5> (ø)
:arrow_down:

src/Tools/ResponseResolver.php
100% <100%> (ø)
7 <4> (ø)
:arrow_down:

...ols/ResponseStrategies/TransformerTagsStrategy.php
86.04% <100%> (ø)
23 <1> (ø)
:arrow_down:

src/Tools/Generator.php
94.57% <91.66%> (ø)
53 <0> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92226ed...b7aedfd. Read the comment docs.


Patrickbrown/ch13686/fix appstatuslistener performance again

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.

100%ed thread:

retained jobs which should not be:


Support custom registries in commands

Same as #90 but rebased

Depends on #105

As a maintainer of a custom registry this PR definitely looks interesting !
Is there anything I can do to help move it forward ?

First https://github.com/microsoft/vcpkg/issues/18967 must be "resolved" and then it can be implemented

I understand, I'll follow the discussion there to keep an eye on this then.


fin-click event details are incorrect

The fin-click event handler receives an object with properties detail.row and detail.gridCell.y. Both appear to be incorrect, and refer to the row below the one that was clicked, without considering how the grid was scrolled. That means clicking the top visible row triggers an event for the second row of the entire grid.
This is my workaround:
grid.addEventListener('fin-click', (event) => {
  let row = grid.getRow(event.detail.gridCell.y + grid.vScrollValue - 1);
  // Handle the event.
});

By the way the renderer tutorial says:
jsonGrid.addFinEventListener('fin-cell-click', function(e){

Seems both the method and event name have since changed.

@jjrv We will have this in our backlog for a future sprint. Not enough bandwidth to address this bug right now as our queue is full

@jjrv @Dwaynekj These are known issues with the version currently on master (v.1.2.1).
The new version v1.3.0, scheduled to be merged to master this evening, provides e.detail.dataCell which contains (in .y) the zero-based scrolled row index, which is what you're looking for.
Furthermore, e.detail.row now returns the correct data row.
Note that e.detail.gridCell was correct as is; it refers to the currently viewable grid rows, i.e., the visible grid rows (not the data rows).
Caution that v1.3.0 has breaking changes!

@jjrv This https://github.com/openfin/fin-hypergrid/pull/498 should address your concerns.


Add language support for VHDL

Adds support for VHDL (#490). Sadly VHDL has no official logo, so I just used the name.

Thank you! Very nice 3D effect with the ASCII.

Sadly VHDL has no official logo

It's definitely hard to make an ASCII logo with no official logo for reference :cry:
How did you decide on green for the color? What do you think about making the letters different colors to make the ASCII more eye-catching?

What about using a micro chip as an illustration? similarly to what we did for SQL and Fortran:

       |  |  |  |
      ------------
   --|    ----    |--
   --|   |    |   |--
   --|   |    |   |--
   --|    ----    |--
      ------------
       |  |  |  |

__     ___   _ ____  _
\ \   / / | | |  _ \| |
 \ \ / /| |_| | | | | |
  \ V / |  _  | |_| | |___
   \_/  |_| |_|____/|_____|

Could you resolve the conflict? #630 introduced a new field when defining language support.

I don't really know. I think it looks nice

I think it looks nice

Agreed :slightly_smiling_face:
Although I wonder if green and yellow would better represent a microchip. @o2sh What do you think?


LICENSE for the repo

Is this a unique feature?

[X] I have checked "open" AND "closed" issues and this is not a duplicate

Is your feature request related to a problem/unavailable functionality? Please describe.
the repository does not have a license
Proposed Solution
adding a license
Screenshots
adding a license
Do you want to work on this issue?
Yes
If "yes" to above, please explain how you would technically implement this (issue will not be assigned if this is skipped)
adding a license

@akku-07  i assign you this issue

make sure you star ⭐ repo if you like and follow on github

thankn you sir!could you also add it under hacktoberfest please?

could you also add it under hacktoberfest please?


Port tests from CoreCryptoTests

Porting TCE tests from devdiv

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: Gary-Zh sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


✨ Improve user experience and accessibility

This PR is ready (not verified), maintainers can help contribute to it in forked repository
This PR is ready but has not been verified for readiness
Website
Switch from Quirks Mode to Standards Mode
https://developer.mozilla.org/en-US/docs/Web/HTML/Quirks_Mode_and_Standards_Mode
Optimise User Interaction on Mobile
https://developer.chrome.com/blog/300ms-tap-delay-gone-away/
Add more fallback font
Add arial font before going to san-serif.
Add Language Attribute
Add en-US or English American to every HTML file
GitHub
Add Issue template

Clean-up
⚠ This might result in potential conflict.
Improve the readability of the script.
Improve README.md
Add License section.
Add FUNDING
Add a way to fund you!

Done™
Other stuff not pertaining to this PR will not be included (but will be in other PR)

Done™
Rebase done
*Older commit (1/2/2022 UTC+7) will not be signed
*Newer commit (as of 1/3/2022 UTC+7) will be signed


npm fails to install redux-saga because it exits with error exit status 0

Npm fails to install redux-saga because it exits with multiple errors and an error exit status 0, any hints?

You would have to share some logs to get help

I am unable to get logs. All I know is that it exits with all errors.

There endless possibilities for an error, most of which have nothing to do with redux-saga.

The issue is with npm.

I will edit npm later.

This is actually becoming a problem.


Debugger refactor

The debugger is undoubtedly (by far!) in the poorest shape of any part of the SDK codebase and is almost entirely untested. It's not a critical piece of the SDK, so that's not a huge problem, but with stuff like screen capture being increasingly important, it'd be worthwhile trying to make it better. This PR does some significant refactoring and adds a bunch of new tests.
Changes

Break up DebugViewModel. This >500 line file has a massive amount of stuff in it.
I've moved models like StatusItem and LoggedEvent to their own files.
I've created standalone classes APIVerifier and DeepLinkVerifier that encapsulate that functionality.
DebugUI now has views for each of the Status items. This better separates concerns and will make testing these in the future easier (all the tests in this PR are just related to the logic—nothing about the presentation yet).
I've created ScreenCapturer which encapsulate the screen capture logic instead of it being bolted on to UIDebugger. It uses the new protocol ScreenCaptureUIto communicate back toUIDebugger`.
Moved LifecycleEvent.EventProperties to the debugger directories as StructuredLifecycleProperties since the only place it's used is in the debugger.

Benefits

Testability! There's now tests for all this stuff.
The DebugViewController and friends are tied the the lifecycle of the debug view and not the UIDebugger class. So when the debugger is dismissed, all that stuff is deinited instead of sticking around forever after it's shown once.
Simpler data flow. The events that come in via AnalyticsSubscribing are now piped through a single combine Publisher that gets used to show the fleeting log items, update the event list in the debugger, and calculate the active experiences section of the debugger.

How to review this PR
Looking at the final diff will make it quite challenging to follow where everything moved to. Reviewing each of the 3 "♻ Refactor..." commits might be simpler. Note that some of the things in ac6c4f0 were subsequently changed further in 118543b (like where the APIVerifier and DeepLinkVerifier are inited).
Some of the tests are complicated. For example, the ScreenCapturer still does the 4 changed network requests, so to properly mock/test the last step, we need to mock/test all 3 before it. Also testing the Combine pipeline stuff is a bit awkward since you need to set an expectation to fulfill once you see the values you want coming through the pipeline.

Actual tests passed. The codecov updater failed to download.


The model to consider.
I am interested in deploying the HuatuoGPT-Vision 7B model, as detailed in the repository HuatuoGPT-Vision. However, I noticed that the model architecture LlavaQwen2ForCausalLM used by HuatuoGPT-Vision-7B is not currently supported by vLLM.
As this model is a multimodal model, it's hard to add it to vllm framework by myself.
Could you please consider adding support for this model? If there are any specific challenges or requirements needed to enable this support, I would be happy to assist or provide more information.
Thank you for your attention to this request.
The closest model vllm already supports.
LlavaQwenForCausalLM
What's your difficulty of supporting the model you want?
No response
Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

If the only difference between this model and the vanilla LlavaForConditionalGeneration is the different language backbone, you should be able to load the model in vLLM by setting the text_config in config.json to load Qwen2 instead of Llama.

Thank you for your reply. Follow your instructions and modify the corresponding JSON file. I have deployed it on my GPU, but I don’t know how to match it with https://github.com/FreedomIntelligence/HuatuoGPT-Vision/blob/.
I tested https://docs.vllm.ai/en/latest/getting_started/quickstart.html#offline-batched-inference
The calling method of offline the printing result is
Prompt: 'Hello, my name is', Generated text: '!!!!!!!!!!!!!!!!'
Prompt: 'The president of the United States is', Generated text: '!!!!!!!!!!!!!!!!!'
Prompt: 'The capital of France is', Generated text: '!!!!!!!!!!!!!!!!!!!'
Prompt: 'The future of AI is', Generated text: '!!!!!!!!!!!!!!!!!'

Need your help!

This sounds a lot like the problem encountered by @fyabc when implementing Qwen2-VL in #7905. In this case, the problem may be inside Qwen2 backbone itself.

By the way, please provide a bit more details about how you modified the JSON file.

By the way, please provide a bit more details about how you modified the JSON file.

I add the text_config in the config.json file:
{
  "_name_or_path": "HuatuoGPT-Vision-7B",
  "architectures": [
    "LlavaForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "image_aspect_ratio": "pad",
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "mm_hidden_size": 1024,
  "mm_projector_type": "mlp2x_gelu",
  "mm_vision_select_feature": "patch",
  "mm_vision_select_layer": -2,
  "mm_vision_tower": "./vit/clip_vit_large_patch14_336",
  "model_type": "llava_qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_theta": 1000000.0,
  "sliding_window": 131072,
  "tie_word_embeddings": false,
  "tokenizer_model_max_length": 4096,
  "tokenizer_padding_side": "right",
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.0.dev0",
  "tune_mm_mlp_adapter": false,
  "use_cache": false,
  "use_mm_proj": true,
  "use_sliding_window": false,
  "vocab_size": 152064,
  "text_config": {
    "_name_or_path": "Qwen/Qwen2-7B-Instruct",
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "hidden_act": "silu",
    "hidden_size": 3584,
    "initializer_range": 0.02,
    "intermediate_size": 18944,
    "max_position_embeddings": 32768,
    "max_window_layers": 28,
    "model_type": "qwen2",
    "num_attention_heads": 28,
    "num_hidden_layers": 28,
    "num_key_value_heads": 4,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sliding_window": 131072,
    "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "transformers_version": "4.41.2",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 152064
  }
}

and add a preprocessor_config.json file referring https://huggingface.co/llava-hf/llava-1.5-7b-hf/blob/main/preprocessor_config.json
{
    "crop_size": {
      "height": 336,
      "width": 336
    },
    "do_center_crop": true,
    "do_convert_rgb": true,
    "do_normalize": true,
    "do_rescale": true,
    "do_resize": true,
    "image_mean": [
      0.48145466,
      0.4578275,
      0.40821073
    ],
    "image_processor_type": "CLIPImageProcessor",
    "image_std": [
      0.26862954,
      0.26130258,
      0.27577711
    ],
    "processor_class": "LlavaProcessor",
    "resample": 3,
    "rescale_factor": 0.00392156862745098,
    "size": {
      "shortest_edge": 336
    }
  }

Some of the fields in LlavaQwen2Config actually belong to to the language backbone. You may have to replace some of the values in the original Qwen2Config with those values.

Thanks for your response.
I'm trying to figure out how to use both images and text as input in this case. The code at https://github.com/FreedomIntelligence/HuatuoGPT-Vision/blob/main/cli.py seems to handle this, but it uses a different approach than the VLLM API.
Do you have any suggestions or examples of how to combine image paths and text for prompting  LLM, ideally compatible with the VLLM framework?

For now, this is best handled by the OpenAI-compatible server which supports multi-modal inputs. The offline LLM.chat method currently only supports text inputs.

I try the example in https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html:
 from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
  model="NousResearch/Meta-Llama-3-8B-Instruct",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)

However get nothing in io output.
The log shows:
INFO 09-03 11:22:28 logger.py:36] Received request chat-7b8a6a1dfefa41b89eb94b03d4bda69b: prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131052, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 151645, 198, 151644, 872, 198, 9707, 0, 151645, 198, 151644, 77091, 198], lora_request: None, prompt_adapter_request: None.
INFO 09-03 11:22:28 async_llm_engine.py:205] Added request chat-7b8a6a1dfefa41b89eb94b03d4bda69b.

Has the model finished downloading? The example is using a different model than the one you originally used.

Sorry for leading misunderstanding, I have changed the script for loading my local download model.

Sorry for leading misunderstanding, I have changed the script for loading my local download model.

For detailed, I used CUDA_VISIBLE_DEVICES=2 vllm serve /home/chuyun/hf_hub/HuatuoGPT-Vision-7B --dtype auto --api-key token-abc123 --port 7996 for load HuatuoGPT-Vision-7B
Then I run
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:7996/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
  model="/home/chuyun/hf_hub/HuatuoGPT-Vision-7B",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)

It keeps running without ending and doesn't print any information.

Can the model run in offline mode with just text? What does your JSON config look like now?

offline mode returns:
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 174.82it/s]

INFO 09-03 12:30:06 model_runner.py:917] Loading model weights took 14.8443 GB
WARNING 09-03 12:30:06 model_runner.py:1084] Computed max_num_seqs (min(256, 512 // 576)) to be less than 1. Setting it to the minimum value of 1.
INFO 09-03 12:30:09 gpu_executor.py:121] # GPU blocks: 23818, # CPU blocks: 4681
INFO 09-03 12:30:18 model_runner.py:1208] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 09-03 12:30:18 model_runner.py:1212] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 09-03 12:30:40 model_runner.py:1327] Graph capturing finished in 22 secs.
Processed prompts: 100%|█████████████| 4/4 [00:00<00:00, 14.00it/s, est. speed input: 77.03 toks/s, output: 224.08 toks/s]
Prompt: 'Hello, my name is', Generated text: '!!!!!!!!!!!!!!!!'
Prompt: 'The president of the United States is', Generated text: '!!!!!!!!!!!!!!!!'
Prompt: 'The capital of France is', Generated text: '!!!!!!!!!!!!!!!!'
Prompt: 'The future of AI is', Generated text: '!!!!!!!!!!!!!!!!'

The config.json is like this:
{
  "_name_or_path": "HuatuoGPT-Vision-7B",
  "architectures": [
    "LlavaForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "image_aspect_ratio": "pad",
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "mm_hidden_size": 1024,
  "mm_projector_type": "mlp2x_gelu",
  "mm_vision_select_feature": "patch",
  "mm_vision_select_layer": -2,
  "mm_vision_tower": "./vit/clip_vit_large_patch14_336",
  "model_type": "llava_qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_theta": 1000000.0,
  "sliding_window": 131072,
  "tie_word_embeddings": false,
  "tokenizer_model_max_length": 4096,
  "tokenizer_padding_side": "right",
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.0.dev0",
  "tune_mm_mlp_adapter": false,
  "use_cache": false,
  "use_mm_proj": true,
  "use_sliding_window": false,
  "vocab_size": 152064,
  "text_config": {
    "_name_or_path": "Qwen/Qwen2-7B-Instruct",
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151643,
    "hidden_act": "silu",
    "hidden_size": 3584,
    "initializer_range": 0.02,
    "intermediate_size": 18944,
    "max_position_embeddings": 131072,
    "max_window_layers": 28,
    "model_type": "qwen2",
    "num_attention_heads": 28,
    "num_hidden_layers": 28,
    "num_key_value_heads": 4,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sliding_window": 131072,
    "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "transformers_version": "4.40.0.dev0",
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 152064
  }
}

I tried client.completions.create.
The code is shown below:
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:7996/v1",
    api_key="token-abc123",
)

completion = client.completions.create(model="/home/chuyun/hf_hub/HuatuoGPT-Vision-7B", prompt="San Francisco is a")
print(completion.choices[0])

it returns
CompletionChoice(finish_reason='length', index=0, logprobs=None, text='!!!!!!!!!!!!!!!!', stop_reason=None, prompt_logprobs=None)

For the online inference using Chat Completions, you may be missing the chat template. Nevertheless, we still have the problem of nonsense output.

Is the model designed for text-only input or is it necessary to input both text and image?

Is the model designed for text-only input or is it necessary to input both text and image?

This model is designed for both image and text, and they provide an offline mode CLI code: https://github.com/FreedomIntelligence/HuatuoGPT-Vision/blob/main/cli.py
I need to input both text and image.

I would like to inquire whether know vllm can support LlavaQwen2ForCausalLM.

Is the model designed for text-only input or is it necessary to input both text and image?

This model is designed for both image and text, and they provide an offline mode CLI code: https://github.com/FreedomIntelligence/HuatuoGPT-Vision/blob/main/cli.py I need to input both text and image.

Sorry that I forgot about your reply. I don't have bandwidth to debug the model right now, see if anyone else is willing to pick this up.

@Jiwoo0822 @Chuyun-Shen You may need to convert the FreedomIntelligence/HuatuoGPT-Vision-7B to HF format llava model to use it in vllm, since we use HF format llava as reference.
You can use the convert_llava_weights_to_hf.py to convert the model:
python convert_llava_weights_to_hf.py --text_model_id Qwen/Qwen2-7B --vision_model_id openai/clip-vit-large-patch14-336 --output_hub_path HuatuoGPT-Vision-7B-hf --old_state_dict_id FreedomIntelligence/HuatuoGPT-Vision-7B


Auto update QR code

Auto update QR code

Source File:
Line(s):
Question:for French tousanticovid Android app, English language, Wallet,  should existing QR codes for vaccination be updated automatically to the eu digital QR codes? This is not happening for my group.

The French app is not part of this GitHub organization, however the code base here does not support any conversion / update of pre-existing qr codes, so I consider it unlikely the French app has this feature.


Unify assign_vector_potential and assign_balloon function input arguments in Maxwell 2D

Description of the feature
The 2 functions are used in the same way (and often they are interchangeable), but the input argument names are different in pyAEDT:
assign_balloon(self, edge_list, bound_name=None):
assign_vector_potential(self, input_edge, vectorvalue=0, bound_name=None)
I can implement the change, but I'm not sure how to maintain backward compatibility.
Steps for implementing the feature
No response
Useful links and references
No response

Closing as part of the keywords refactoring.


Module not found

Can not found it as shortcut nor in the list of installed modules to select apps :/
Using lsposed 1.8.4

@wolfxana  I am using 1.9.2(7024) LSposed and the API version is 100. This Xposed module is based on Modern Xposed provided by the LSposed team.
These are the minimum requirements :
minApiVersion=100
targetApiVersion=100
The same has been declared in module.prop file. Let me know if the above requirements are satisfied by your phone or not.

I can not go beyond magisk v23 as am using lsposed 1.8.5  (6649) now with riru . to update is to update magisk which I can not 🚫 Ms Authenticator works for me.
How to check API version ?
Would be possible to downgrade? I know selfish of me, as only option flagged form me now : SE linux flag is enabled.


Cpfp descriptor

This implements https://github.com/re-vault/practical-revault/pull/25.
It's based on #11 .

Ack 1abb577dba9631b0abd7a7c3c54f568a5e4a0ad1


Mention downtime when deploying to a cluster

@feddovanede and @cwesdorp
Can you review what I've written and confirm that it is OK.
I notice that it doesn't say that the After startup is only performed by the cluster leader. Can either of you confirm whether this should be added, or shall I go to Runtime to confirm?

Thanks for the review, Feddo.


checkout.pph error

What am I trying to do..
..What's happening
An uncaught Exception was encountered
Type: Error
Message: Call to undefined method SendMail::clearAddresses()
Filename: C:\xampp\htdocs\Primedawn_Hardware\application\controllers\Checkout.php
Line Number: 72
Backtrace:
File: C:\xampp\htdocs\Primedawn_Hardware\application\controllers\Checkout.php
Line: 41
Function: sendNotifications
File: C:\xampp\htdocs\Primedawn_Hardware\index.php
Line: 292
Function: require_once

@ValeraT17
This issue is fixed. Download/Refresh you code again


When creating an Item and setting MaximumUsages but not setting UsagesLeft, the UsagesLeft value should equal the MaximumUsages value

This somehow doesn't seem to work as it should. The UsagesLeft value is 0, or rather the item returns the NoUsagesLeftResponse although it shouldn't.
Code:
      .addItem()
        .setName('shovel')
        .setDescription('A sturdy shovel that just has the perfect size.')
        .setInSceneDescription('On the right side of the entrance is a shovel.')
        .setCanPickUp(true)
        .setCanUseFunction((item: InGameItem, currentScene: Scene) => {
          if(currentScene.getName() === 'Driveway'){
            return true;
          }

          return false;
        })
        .setCannotUseItemResponse('You don\'t think that using a shovel inside the hut is a good idea.')
        .setMaximumUsages(3)
        .setItemUsedResponse('You play around in the snow until it\'s not fun anymore.')
        .setNoUsagesLeftResponse('Again? You don\'t feel like it.')
        .finish()

The ItemBuilder now sets the UsagesLeft value to the MaximumUsages value, if it has not been set yet.


Added source-maps support

I added support for source-maps using the node-sass sourceMap option (#33)

Travis builds working fine on my forked repo :confused: https://travis-ci.org/iMoritz/node-sass-middleware/builds/63678281

Thanks for the PR @iMoritz! :+1:
LGTM modulo few nits.
@kevva, PTAL.

Looks good apart from my comments.

Travis build was probably timing out. I restarted the job and it's green now.

Thanks @iMoritz ! :tada: :star2:


Make this handle the empty string.

https://github.com/Tombert/Vertigo.CsvParse/blob/f3d9fc6c7f868c61066ba3f645e60202d6d45bc5/Library.fs#L34

For the record, I found out this issue :D

Fixed in the latest commit: https://github.com/Tombert/Vertigo.CsvParse/commit/6c342b5ff4dd5a40ca9cebf9457259393adbb73f


.net core 2.0 preview app w/ Docker problem in linux

.net core 2.0 preview app w/ Docker problem in linux
Details
I just upgraded our project from netcoreapp1.1 to netcoreapp2.0, in which run in Docker, built by VSTS private Linux build agent. Yet I got this error in build agent.
:~/myagent/_work/3/s$ dotnet restore References.sln
/home/xxx/myagent/_work/3/s/docker-compose.dcproj : error MSB4236: The SDK 'Microsoft.Docker.Sdk' specified could not be found.
/home/xxx/myagent/_work/3/s/src/References.Web.csproj : error NU1003: PackageTargetFallback and AssetTargetFallback cannot be used together. Remove PackageTargetFallback(deprecated) references from the project environment. [/home/xx/myagent/_work/3/s/References.sln]

Does this mean now .net core 2.0 app w/ docket is not supported when built in Linux?

Close this, as tried further and verified this is not an issue.


Big refactoring to node implementation

The application is deployed at http://de-mucdevondepl01:8091
To read the emails sended by the application you can go to http://de-mucdevondepl01:8092
IMPORTANT: The application is working but there is something peding:

Test: all test are missing
JSDoc: all classes/methods do not have a proper JSDoc.
Better exception handling: layer service shouldn't know anything about the HTTP response to the client. Thats why we throw a normal Error in the service layer and then we catch them at controller layer and send to the client the proper HTTP error. This mechanism is OK, but we can improve it by creating errors and then adding an error handling which is the responsible to send the correct HTTP error to the client. This feature will arrive soon.

Hi @dario-rodriguez, should we merge this now or wait until pending features?

Just checked the deployed version and the bookings are not shown in the waiter private area. Please check it. Thanks!

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: dario-rodriguez:x: rootroot seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Hello Santos,
I solved the error in the private area and redeployed the application.
If you want to merge now for me is ok. I will create a new PR when everything are done.
Kind regards

Hi @dario-rodriguez,
Thanks for the fix, but I have found another one 🙃... Sorting in the waiter area  is not working in bookings nor orders.
Please check it when you come back from holidays!
Kind regards

Hello @sjimenez77 ,
PR updated and application redeployed.

Hi @dario-rodriguez, everything seems to work fine now. Merging!
Thanks!


Why is the number of predicted bounding boxes so high?

Hi,
I'm using PVRCNN with a custom data loader for another subset of KITTI.
Things seem to work. However, when I check the number of predicted bounding boxes ('batch_box_preds'), I see that it's quite high -> 211200.
Any ideas why the number of boxes is so high at the very first epoch and why it is specifically 211200?

I guess 211200 is the number of anchors rather than predicted bounding boxes.

Thanks @jihanyang!


check_matching_dims with std::vector

I am trying to use the new offset & multiplier feature with nested vector in array data structures. The stanc3 compiler does indeed give me a C++ file, but then I get a compiler error.
Compiler error:
In file included from stan/src/stan/model/model_header.hpp:13:
/Users/sebi/work/cmdstan-2.25.0/stan/src/stan/model/model_base_crtp.hpp:191:50: error: no matching member function for call to 'log_prob'
    return static_cast<const M*>(this)->template log_prob<true, true>(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
../partial-warmup/poisson3.hpp:147:3: note: in instantiation of member function
      'stan::model::model_base_crtp<poisson3_model_namespace::poisson3_model>::log_prob_propto_jacobian' requested here
  ~poisson3_model() final { }
  ^
../partial-warmup/poisson3.hpp:491:14: note: candidate template ignored: substitution failure [with propto__ = true, jacobian__ = true, T__ = stan::math::var_value<double,
      void>]
  inline T__ log_prob(std::vector<T__>& params_r__,
             ^
../partial-warmup/poisson3.hpp:1021:15: note: candidate function template not viable: requires at most 2 arguments, but 3 were provided
    inline T_ log_prob(Eigen::Matrix<T_,Eigen::Dynamic,1>& params_r,
              ^
12 errors generated.
make: *** [../partial-warmup/poisson3] Error 1

poisson3_data_R.txt
poisson3_stan.txt
The problematic offset & multiplier is for z_1.
Thanks for looking into this!
(my ambition is to automate this stuff with brms... which could be a big deal possibly/maybe/hopefully)

It also compiles if I replace the require here
https://github.com/stan-dev/math/blob/d47c331a693a3d5ebc4453360743b9353a8e3ed1/stan/math/prim/err/check_matching_dims.hpp#L26
to
template <typename T1, typename T2, require_all_std_vector_t<T1, T2>* = nullptr>
inline void check_matching_dims(const char* function, const char* name1,
                                const T1& y1, const char* name2, const T2& y2) {

Though I think makes check_matching_dims not work with scalars. Not a problem for this model but it possibly is a problem in general? Maybe?


[BUG] Investigate single datapoint for London population

Describe the bug
http://staging.datacommons.org/place?dcid=wikidataId%2FQ84

There are stats from UNData and Wikidata sources. UNData is a single 2011 data point, while Wikidata is a time-series (http://staging.datacommons.org/browser/dc/p/qlyvv2jyqszzb).
I think the cache should build in both. I know the mixer does some ranking among sources [1]. For ones that aren't listed there, is the choice arbitrary? Should it be based on some heuristics on the data (has time-series or has the latest data, etc.)
[1] - https://github.com/datacommonsorg/mixer/blob/104d4db5b44f862beda42ee268d63a0678f7caae/server/stats.go#L33-L45

@shifucun
Should we add more to this list then? The preference for a time series is only for UI purposes, but if we weight it towards latest data, that will resolve to Wikidata which works better for both UI and analysis.

I will note that the date on Wikidata population stats will be all over the place across places. So won't be surprised if, for another place UNData was more recent.

Looks like we need a bit complicated ranking on both data freshness and time span. I propose to:

First compare data time span. If one dataset has twice or more data than another, then pick it.
If two datasets have similar time span, pick the one that has latest data.

Any other ideas?


Generate code for "x-ms-long-running-operation-options"

It's given by the modeler:
    extensions:
      x-ms-long-running-operation: true
      x-ms-long-running-operation-options:
        final-state-via: location

Just need to integrate it in the template

https://github.com/Azure/autorest.python/commit/5a23589fa864a7fd8a893bd82fb77e382586651f


Check weaveDNS subdomain on REST PUT

Fixes #1516.

@rade now with added contexts.

small enough fix to go into 1.2.1. please re-base.

GitHub YUNO modify PR target branch


Support for compiling Java modules

Implementation for #12258.

I did some more refinements on this while working on the execution story (#12580).
This PR now includes adding a Version and/or MainClass to module-info.class. The integration test coverage for these features is part of #12580.


Refactor form rendering configuration according to more appropriate way

Reference: https://discourse.laminas.dev/t/documentation-about-form-component-rendering-with-laminas-mvc/3027

I finally had time to read the thread in the forum.
Here is a couple things to consider.

Since laminas-mvc-form is only a meta package, you can NOT just add it to the modules.config.php etc. You run a composer require for it. During that install, it installs the 3 packages that are needed for usage in the MVC if you are going to integrate it fulling into the MVC. Two of which are absolute requirements if you are going to use any of the view helpers for form rendering. 99.9% of the time that will be the case.
The reason why the i8ln package is required, is because laminas\form\src\view\helper\abstracthelper extends Laminas\I18n\View\Helper\AbstractTranslatorHelper Which is your coupling.
Its usually much simpler to just tell new laminas users to not select the "minimal" build, have them select yes when it prompts for laminas-form, and any other package that is required for the project. Result, you end up with doctrine annotations that will not be used in the text. If by some chance they are advanced composer users and select to require dev it, then they will get the package anyway as a dev requirement for laminas-form, which would make the point of not loading doctrine annotations moot anyway.

In the "suggest" block of the laminas-form package composer.json it gives the following reason for suggesting laminas/laminas-i18n. The read me of the laminas-mvc-form repo also explains this, but not as well IMO.

"laminas/laminas-i18n": "^2.12, required when using laminas-form view helpers"

As a general rule of thumb, any package that exposes a Module.php file will need to be added to the modules array, ie /config/modules.config.php.
Which brings me to my next point, which applies to all laminas packages. If they contain a Module.php they need to be added to the modules.config.php file. I think what tripped you up (possibly) is that laminas-form was installed before the skeleton, which means that laminas-component-installer most likely was not in use. Which in turns means that it was not automatically injected into the modules.config.php array. You brought laminas-form over from the pre MVC context (possibly, its hard to recall, since I have bounced back and forth between the early chapters and not had a chance to read every page of them). Which means that its Module.php file was not called, and its ConfigProvider was not called, which means, the view helper services and aliases was never registered with the service manager. Of course they are still available via autoloading. When you ran the Eclipse tooling to update the dependencies that makes the classes available via autoloading. That, however, does not add it to the modules.config.php file and the execution of the config provider in this case is what makes them available as services via the service manager, whether it be by alias or by service name.
You may want to take a look at Laminas/View/Renderer/PhpRenderer since they are called through its context via __call which just proxies the call to the view helper plugin manager.


test: add faulty server tests

This change is

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.42%. Comparing base (4269f5b) to head (1692290).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #400      +/-   ##
==========================================
+ Coverage   83.24%   83.42%   +0.17%
==========================================
  Files          36       36
  Lines        1695     1695
  Branches     1695     1695
==========================================
+ Hits         1411     1414       +3
+ Misses        209      205       -4
- Partials       75       76       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Export Types

Hello @bhaveshpatel200 ,
Thanks for the time taken to create this package.
I was wondering if is possible to cerate & export types for the package? That way we can get TS help when creating a column array etc...

Hi,
Sure i will check and update in next version if not.

I would open a PR, but I am not sure how bundling works for vue packages. If I learn how, I will send a PR later today

Hello,
Reviving the conversation to add : In a properly configured IDE, it will mark import errors because of the type definitions not being resolved.

Could not find a declaration file for module '@bhplugin/vue3-datatable'.
...
The '@bhplugin/vue3-datatable' library may need to update its package.json or typings.

@BayBreezy You mentioned you were opening a PR for this, is it being worked on?

@alexis-massa Sadly, I have not had the time to look into this.

Hey @alexis-massa , I opened the PR for this just now.
@bhaveshpatel200 please take a look when you get the chance #33

Seems you have changed lots of code other than types related and also changed props name which will be break package for existing users.
as i have not deeply check fully code, so will check proper and test, once will get more time.

Yeah... Check when you get the chance and let me know.
It was not a prop that was changed( so no breaking). I created a local variable to store the value of the prop. I saw where you were directly changing the value of the prop(which is bad practice)


ParseException, cause: Built-in function not yet implemented (only aggregates like COUNT so far)

https://qlever.cs.uni-freiburg.de/wikidata/5KqxJw
I assume this is about:
 BIND (xsd:integer(SUBSTR(STR(?item),33)) AS ?itemId)

Which is kind of a workaround anyway.
What would be an alternative?
the issue for me here is not the non implemented feature but the way the error message is displayed. It should e.g. point out where the problem is (what line/what function ...)

The error messages will be improved when we switch to the new ANTLR-based parser (we are already quite far with this). The query has two problems:

Functions like xsd:integer, SUBSTR and STR are not yet implemented. That is relatively easy to do. Anyway, that's pure post-processing of the query result, so rarely criticial (since the results of a SPARQL query are usually processed further in one way or the other anyway).

If you GROUP BY a subset of the variables, the other variables have to be aggregated using COUNT, SAMPLE, MIN, etc. When you don't aggregate one of them, QLever currently returns the cryptic error message absl::container_internal::raw_hash_map<>::at. This will also be fixed soon, when the new parser is in place.

@hannahbast  thanks for the info. For this one there is an easy workaround. I don't need the SPARQL query to return the numeric id value  - the python code analysing the result can easily extract that.
# get the items that have the given label in the given language
# e.g. we'll find human=Q5 as the oldest type for the label "human" first
# and then the newer ones such as "race in Warcraft"
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX schema: <http://schema.org/>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX wikibase: <http://wikiba.se/ontology#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT
  #?itemId
  ?item
  ?itemLabel
  ?itemDescription
WHERE {
  VALUES ?itemLabel {
       "human"@en
  }
  #BIND (xsd:integer(SUBSTR(STR(?item),33)) AS ?itemId)
  ?item rdfs:label ?itemLabel.
  ?item schema:description ?itemDescription.
  FILTER(LANG(?itemDescription)="en")
}
#ORDER BY ?itemId

is now somewhat misleading in the comments but https://qlever.cs.uni-freiburg.de/wikidata/TxJdYE works except the sorting needs to be done externally now.

But why would you want to sort by the integer value of the QID? A sorting by "popularity" would be more meaningful, for example, by the number of sitelinks: https://qlever.cs.uni-freiburg.de/wikidata/XDUlft

@hannahbast  thank you for the idea of a different "relevancy" function. The sorting by number just shows the "older" entries first. The idea here is that the original encyclopaedic entries will be ranked higher than folksonomy style new entries or amendments being done later.

For over half a year now, SPARQL queries are parsed via ANTLR and there are proper error messages when GROUP BY is used wrongly.


Remove SqliteEngine.UseWinSqlite3

Resolves #262

😱 this is a big breaking change. Although "winsqlite3" has issues on UWP, it works just fine on Windows Server Nano and Win 10 desktop. Before we remove this, we should provide a way for users to still provide a way to use "winsqlite3" in those scenarios. Or, we could move the API to ".Internal" since it isn't fully-baked yet.

Making it internal doesn't sound bad to me. @bricelam thoughts?

Recalling. Let's move it to an Internal namespace.