Having fun with linalg.generic fusion heuristics

Fixes #14414.
Note: CI will fail due to iree-opt unit tests which I haven't updated yet, but e2e tests succeed locally. Wanted see if you think that the change makes sense before proceeding with updating tests.
Note: this PR is now playing with a simple cost model. Original PR description preserved below, referred to the first commit in this PR.
There is already code in isFusableWithConsumer that checks that the consumer is all parallel before fusing. This PR adds a similar check about the producer, and replicates the same logic in isFusableWithProducer -- maybe we could let these two functions share a helper.
I think I understand a simple rationale why we should not fuse reduction producers: as reductions produce an output that is of lower rank than the iteration space (since reduction iterators don't participate in the output indexing), not fusing  reductions is a caching optimization. As a toy example, if the producer is a complete sum reduction of a vector down to a scalar, and the consumer is a broadcast of that scalar onto a vector, clearly we don't want to compute the same sum reduction over and over again.
I'm less clear on the rationale for the existing check about not fusing reduction consumers. Is it due to some incidental limitation of tile size heuristics? Take for instance a function returning the sum of squares of elements in a vector. So there's a producer that has a parallel iterator and computes x -> x * x and there's a consumer with a reduction iterator, computing the sum-reduction. Why not fuse?
Another question, why the existing discrepancy between isFusableWithConsumer, which was checking that the producer is all parallel, and isFusableWithProducer, which was not checking that? I'm concerned that fusion decisons might hinge on pattern application order etc?

@MaheshRavishankar @hanhanW @dcaballe , thoughts?

Thinking more about it, there is no way around the fact that a cost model is needed here, even a simple one. Heuristics like my "don't fuse reduction producer" always stumble on special cases, for instance it does make sense to fuse consecutive partial reductions -- e.g. if the input is a 2D matrix, the producer is performing a partial sum reduction along rows, the consumer is computing the sum of the row sums, then we do want to fuse.  Any simple cost model would see that, e.g. just evaluate the cost of any linalg op as the number of loop iterations times (the number of ops in its basic-block, plus some constant K for each input/output operand to account for loads/stores, to reward fusions avoiding round-trips to memory.  The constant K could be tuned to adjust how much we value avoiding going back to memory, but even just the value 1 would probably reward enough fusions.  Another parameter in the cost model would be what value to give static dimensions. It probably shouldn't matter, as most fusion decisions are the same for all except perhaps the tiniest sizes, so just giving dynamic dims the value 16 in the cost model would be good enough. With higher values we would need to start worrying about integer overflow in many-iterator generics used in practice on smaller dynamic dim sizes).

+1 for simple cost models - often we've found people getting caught in the perfect-is-the-enemy-of-good trap and planning out massive infra (global cost models, auto tuning, etc) when basic orders of magnitude comparison are often enough (and what humans do when making the decisions at this level) and we should have what we need in linalg ops to how much data each is traversing and the rough order of magnitude on the number of ALU ops each is performing. Not perfect, and in some cases not good, but better than nothing :)

the template metaprogramming one exists, so I think it wins :)

I think we agree, we were just phrasing it above as saying we need a really simple cost model that doesn't even try to get into details (device or otherwise) because we are really just asking for really basic things there. Basic fusion decisions are more or less universal.
So yes, we just need to make the current heuristics a bit better, and some of that might or might not look like an embryonic cost model.
Re 1. what I was missing/unsure about is the notion of a root op that you explain in 2.
Re 2. thanks for explaining the algorithm, this is what I was guessing what happening, but then I still have a concern that it might be indeterminate in some cases. What if a producer-consumer pair of ops both qualify as root ops? E.g. two linalg.generic with reduction? Is the fusion decision going to be made by isFusableWithProducer or isFusableWithConsumer?

I think we agree, we were just phrasing it above as saying we need a really simple cost model that doesn't even try to get into details (device or otherwise) because we are really just asking for really basic things there. Basic fusion decisions are more or less universal. Talking about really basic things here, like "do fuse back-to-back elementwise" and "dont fuse reduction producer when that would multiply overall arithmetic cost by the reduction size".

These should be already accounted for. The back-to-back elementwise get fused with FusionOfTensorOps. Today the fusion heuristics enforce no redundant computation is added.

So yes, we just need to make the current heuristics a bit better, and some of that might or might not look like an embryonic cost model.

Always up for improvements, but I would first recommend basing it off of what is already done. This is something that has been looked into in quite amount of detail. So a lot of basic issues (like fuse back-to-back elementwise operations) are already accounted for. There might be bugs, but I am just saying we arent starting at 0. I would recommend looking at
https://github.com/openxla/iree/discussions/9348 and https://discourse.llvm.org/t/rfc-next-iteration-of-fusion-of-elementwise-operations/59955 and the code in FusionOfTensorOps pass and FormDispatchRegions pass to get an idea of what is implemented currently.

Re 1. what I was missing/unsure about is the notion of a root op that you explain in 2.

Basically this is the "seed" of the dispatch and drives the compilation of the dispatch. For example a GEMM or a convolution.

Re 2. thanks for explaining the algorithm, this is what I was guessing what happening, but then I still have a concern that it might be indeterminate in some cases. What if a producer-consumer pair of ops both qualify as root ops? E.g. two linalg.generic with reduction? Is the fusion decision going to be made by isFusableWithProducer or isFusableWithConsumer?

Actually this is something I want to fix. The initial approach was to not fuse two linalg.generic with reductions. But there was a lot of discussion on "fusing softmax" that probably pushed this beyond what it should have been. Currently the fusion decision is made in isFusableWithProducer but again making sure that the reduction operations have "common outer parallel loops". I do want to dial that back though.

I haven't looked at IREE fusion but in Affine we even fused ops without a producer-consumer relationship when they shared the same input, for example. Regarding reductions, I would expect fusing a generic_op [parallel, parallel, reduction] into a generic_op[parallel,parallel] to be beneficial because we save the intermediate allocation + improved temporal locality. Same for the opposite scenario. It could be, though, that we are not ready to handle some of these fused cases down the road and some patterns/optimization strategies are not that effective, making fusion not beneficial for now.

Thanks for the reply.  I've pushed a naive implementation of the cost model idea, which still fixes https://github.com/openxla/iree/issues/14414. WDYT?

I haven't looked at IREE fusion but in Affine we even fused ops without a producer-consumer relationship when they shared the same input, for example.

Yes, this would be worth adding.

Regarding reductions, I would expect fusing a generic_op [parallel, parallel, reduction] into a generic_op[parallel,parallel] to be beneficial because we save the intermediate allocation + improved temporal locality. Same for the opposite scenario.

These are also done by default today (unless they are tripped up by some implementation detail. They are intended to be handled.

It could be, though, that we are not ready to handle some of these fused cases down the road and some patterns/optimization strategies are not that effective, making fusion not beneficial for now.

Thats exactly right.

I dont think this is the root cause of the issue you are trying to fix.


Small comma error on PPP

Add a comma

Codecov Report

Merging #325 (48e36a5) into master (3839af9) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #325   +/-   ##
=========================================
  Coverage     70.03%   70.03%
  Complexity      593      593
=========================================
  Files            85       85
  Lines          1972     1972
  Branches        227      227
=========================================
  Hits           1381     1381
  Misses          526      526
  Partials         65       65

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


[QA-677] Updated AddressCRI load profiles

QA-677
What?
Added new profiles for March 2025 Volumes to the existing Orange(Address) script
Changes:

Added loadMar2025, soakMar2025, spikeNFR & spikeSudden profiles

Why?
For the March 2025 Volumes

This PR contains the commits for both IPV core and Address CRI. This is likely because you checked out in the new branch (for Address) from the old feature branch (IPV core). Ideally you would check out to main, pull latest changes and then check in to the feature/QA branch.
Can you remove the IPV core commit from this PR as there is a separate PR for this?


fix depth subpath bug

Due to a bug discovered during the provisioning of module velero with @simonemex  and @lnovara   that has. a subdir of multiple path, here it is the patch for this bug. Now the function will handle whatever hierarchy level of subdir will exist in the repo

hi @ralgozino , you are right, is my vim configuration that is set in that way. Is pretty the usual configuration, but if is something that have to adapted to a standard, just let me know and i will change it  👍

Thanks @lzecca78 ! I'm seeing a lot of format changes in the diff (changed from 2 to 4 spaces identation), I think w

I was working on it but there is not ready yet. So feel free to continue the PR

Anyway I made a search and the standard in yaml Is 2 spaces so I changed my vim setup to match this standard.

LGTM! Waiting for one more feedback, @ralgozino  or @phisco


Loosing network on guest when host goes into standby mode.

Describe the issue
When host goes into standby and later is powered on the guest cannot get ip from the bridged interface.
Same happens when the guest runs in nat mode.
Only solution is to restart UTM.
To simulate this we need to either remove network card from usb or standby host and wait some time before powering on.
Configuration

UTM Version: Version 3.2.4 (58)
OS Version: W11 or MacOS
Intel or Apple Silicon? ARM APPLE

There is no log for mac os guest.

config.plist.zip

Also believe I've seen this. It seems the guest requires a restart (did not have to restart UTM). This was on a macOS Virtualization.framework macOS 12.5 guest virtual machine within UTM wherewith I experienced this.


Truncated window

Hello,
I have a problem display on mono Linux, the window is truncated and I can not change its size. :

An idea ?
Thanks

Hi,
As a workaround, you can edit the following file: https://github.com/NikolayIT/RatioMaster.NET/blob/master/Source/RatioMaster/MainForm.Designer.cs and change values of:
this.ClientSize = new System.Drawing.Size
this.MaximumSize = new System.Drawing.Size
this.MinimumSize = new System.Drawing.Size

@rptx create a pull request for that if this workaround doesn't break anything for other users.


Cannot store datetime.timedelta objects in config

Cannot store datetime.timedelta objects in config
Although python manage.py constance list shows the value correctly, navigating to constance in Django admin throws Server Error (500):
django.core.exceptions.ImproperlyConfigured: Constance doesn't support config values of the type <class 'datetime.timedelta'>. Please fix the value of 'VARIABLE'.

Steps to reproduce

In config, define variable with type timedelta:

# (settings.py)
from datetime import timedelta
CONSTANCE_CONFIG = {
    'VARIABLE': (
        timedelta(days=5), 'Sample timedelta variable', timedelta
    ),
}

Navigate to http://localhost:8000/admin/constance/config/

System configuration

Django version: 1.10.4
Python version: 3.6.1

I'd be happy to use this feature from pip, could you publish a new release please?


Update imports on file move not working in VSCode

Env info

volar: v0.40.13
vscode: 1.71.2

Steps to reproduce

codebase used below HenryC-3/reproduce-volar_update_import_dont_work

disable all extensions except volar
move src/components/HelloWorld.vue to src/HelloWorld.vue
checkout App.vue, the import remain the same.

Fixed in v1.0


add "additional_tags" and "status" metatags in the metadata ipynb file

having the metadata inside the ipynb file restricts the user to only a handful of metatags:
"title", "date", "category", "tags", "slug", "author"

This pull request adds an additional meta-tag "additional_tags" list inside the metadata object in order to expand the above list. additional keys can be accessed directly inside the pelican themes.
additionally, to fix #59 (the status: 'draft' issue when using ipynb metadata dict) i added "status" to the keys to check against in the metadata aswell.
Example
# inside the .ipynb file
{
   "metadata": {
      "additional_tags": ["Modified"],
      "Modified": "2017-04-30 08:58",
      "status": "draft"
}

usage inside a pelican theme:
<!-- articles.html -->
{% if article.modified %}
Last modified: {{ article.modified }}
{% endif %}

Thanks for the PR!
Now that there is more options to pass the metadata i think i am going to deprecate the option to modify the metadata in the ipynb file.


Activate SDL2 compilation and usage by default

This PR includes:

usage of OSX Frameworks by default instead of the default method with pkgconfig. This implies that the requirements (SDL2, SDL2_ttf, SDL2_mixer, SDL2_image, GStreamer) must be installed into /Libraries/Frameworks. This can be disabled with USE_OSX_FRAMEWORKS=0.
generate a kivy/setupconfig.py, the python equivalent of the kivy/graphics/config.h / config.pxi
use the setupconfig.USE_SDL2 to determine if pygame is allowed in the core provider (audio/text/window/image).

@akshayaurora about the #2818, did you have tested on mobile, or is it just a guess? Cause unless it's tested, i don't think it will be sufficient, and so, the PR could wait until the mobile is working too


Allow SourceMapDevToolPlugin.append option to be a function

https://github.com/webpack/webpack/issues/783
Allow SourceMapDevToolPlugin's append option to be a function so that source-map could be better customized. https://github.com/webpack/webpack/issues/783#issuecomment-671410045
(Though it seems that SourceMapDevToolPlugin might be able to solve the URL issue with the publicPath and append options).

What kind of change does this PR introduce?

feature
Did you add tests for your changes?

Yes
Does this PR introduce a breaking change?

No
What needs to be documented once your changes are merged?

Allow SourceMapDevToolPlugin's append option to be a function
https://webpack.js.org/plugins/source-map-dev-tool-plugin/#options
I didn't change the behavior, but maybe EvalSourceMapDevToolPlugin also.  Because setting append option to false doesn't disable the appending but just set the default value ("//# sourceURL=[module]\n//# sourceMappingURL=[url]").
https://webpack.js.org/plugins/eval-source-map-dev-tool-plugin/

@snitin315 Just a small job - resend it :smile:


Metrics sometimes sent every 1 minute even though reporting period manually set higher

What version of OpenCensus are you using?
From go.mod:
go.opencensus.io v0.22.0
contrib.go.opencensus.io/exporter/stackdriver v0.12.2
What version of Go are you using?
> go version
go version go1.12.5 linux/amd64

What did you do?
If possible, provide a recipe for reproducing the error.
In a Go web app that was already instrumented with OpenCensus stats with a Stackdriver Monitoring exporter, changed the code:
view.RegisterExporter(exporter)
view.SetReportingPeriod(60 * time.Second)

to:
view.RegisterExporter(exporter)
view.SetReportingPeriod(10 * time.Minute)

What did you expect to see?
In Stackdriver Monitoring, less data points. One sent every 10 minutes per process.
What did you see instead?
No change in frequency of data points. One every 1 minute.
Additional context
When I noticed this issue, I created as small as possible a Go web app to try to find out if it was an issue with OpenCensus or the Stackdriver Monitoring exporter, or with how we used it in our web app. I created this app, using the official documentation for the Go Stackdriver Monitoring exporter (https://opencensus.io/exporters/supported-exporters/go/stackdriver/) as a guide: https://github.com/groupby/beacons-go-opencensus-stackdriver-test
In my minimal example, it works properly. When I change the code view.SetReportingPeriod(60 * time.Second) from the example to view.SetReportingPeriod(10 * time.Minute), I get just one data point in Stackdriver Monitoring every 10 minutes.
In my work's web app that was already instrumented, I noticed the original coder had:
if err := exporter.StartMetricsExporter(); err != nil {
	log.Errorf("failed to start metric exporter: %v", err)
}

Since the official example does not have that call to exporter.StartMetricsExporter, and the comments for that method in the library indicate that it seems to be an alternate way to start things than view.RegisterExporter, and that you shouldn't use both ways at once, I removed it from our web app to see if it would make a difference. At this point, it stops sending metrics altogether. Note that my minimal working example, which sends metrics every 10 minutes, does not have a call to exporter.StartMetricsExporter. It only call view.RegisterExporter.
Unfortunately I can't show the code from this web app, since it's proprietary, but it was a standard looking Go web app. Some clients like Redis and GCP PubSub created first, then business logic service structs using them, then some HTTP routes set up on a Chi router, and then that router started on a standard library HTTP server. The OpenCensus tag keys, measures, and views were all created and registered properly (confirmed because we know the metrics were being sent every 1 minute this entire time).
The next thing I did was dig further in the code for the OpenCensus core library and the Stackdriver Monitoring library. At that point, I discovered there was another way to sset the reporting period. You would provide it as a constructor option as you create the exporter. So I changed our web app's code to this:
exporter, err := stackdriver.NewExporter(stackdriver.Options{
	ProjectID: config.GCloud.ProjectID,
	ReportingInterval: 10 * time.Minute,
})

I also removed our code that used the method view.SetReportingPeriod to set the reporting period. Now, it was just being set on the exporter.
At this point, our web app worked. It sent data points every 10 minutes.
So at this point, I'm confused about why the view.SetReportingPeriod method worked in my minimal example but not in my original, larger web app. I don't know where to start to try to reproduce that because there are so many things about that web app to try, like how our code is arranged, where we declare the tags, measures, and views (we use a microservice skeleton in a shared library), etc. I'm also confused about what is the recommended way to set reporting periods in OpenCensus. The official documentation implies it is at the view layer, not the exporter layer. But for me, it only worked when done at the exporter layer.
I considered opening an issue in the repo for the exporter (https://github.com/census-ecosystem/opencensus-go-exporter-stackdriver) but since the official documentation for OpenCensus makes this look like a view layer thing, not an exporter layer thing, I decided to open the issue here.

If you are still wondering, you should NOT call view.RegisterExporter if you also use exporter.StartMetricsExporter.
https://github.com/google/exposure-notifications-verification-server/issues/474#issuecomment-700351001 contains some additional details.

Thanks for leaving a comment. It's been a while since we touched this code. We also had an issue with many errors in our logs like "could send... too many time series", so maybe we ended up having the double metric problem you mentioned in that link. We also just put up with that. I'll review how we implemented it.


Login error

Started  a few days ago.   1.18.0.  Working beautifully for months and all of a sudden
SystemERRORloginSession Error: GET https://www.alarm.com/login failed: GET https://www.alarm.com/login failed: request to https://www.alarm.com/login failed, reason: getaddrinfo EAI_AGAIN www.alarm.com
alarm.com app works fine.   Please close.  Somehow both my joins boxes lost their dns settings.  Bad name server

Just to be clear for anyone in the future, the error plugin site unavailable is related to you having some sort of device on your network which wasn't able to communicate out to the internet properly?


add desc key for keymaps, change error to notification

as https://github.com/nanotee/nvim-lua-guide#vimkeymap at section vim.keymap  suggested,
we should add keymap descriptions for keymaps defined by vim.keymap.set, since lua function doesn't have good string representation, the keymap defined by lua function is not informative unless we give it a description

Thanks a lot for the PR!

Is there any way to manually provide these? i have about 100 hotkeys under leader with nice names and then all the iron ones are ugly. Im not sure how to define them as I dont think im able to reachint o the plugin to call those commands manually, can i?
For example how would i call visual_send directly? Then I could just make my own keymap


Bindings to other language : java

I would like to use this embedded library in some of our production service. Are there any jni bindings ?
What would be the recommended way to create some jni bindings , any codepointer appreciated

Hi @patelprateek Thanks for your support!
We don't have any JNI bindings for now, sorry about this.


[Bug] inconsistent blocks number

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug
Many tasks tasks of spark jobs will throw the exceptions that the inconsistent blocks number. The stacktrace is as follows:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 9.0 failed 6 times, most recent failure: Lost task 2.5 in stage 9.0 (TID 7653, BJLFRZ-10k-152-228.hadoop.jd.local, executor 159): org.apache.uniffle.common.exception.RssException: Blocks read inconsistent: expected 7 blocks, actual 0 blocks
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.checkProcessedBlockIds(ShuffleReadClientImpl.java:215)
at org.apache.spark.shuffle.reader.RssShuffleDataIterator.hasNext(RssShuffleDataIterator.java:135)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
at org.apache.spark.shuffle.reader.RssShuffleReader$MultiPartitionIterator.hasNext(RssShuffleReader.java:227)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:768)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:134)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:129)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:467)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1478)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:470)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2083)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2032)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2031)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2031)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:979)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:979)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:979)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2263)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2212)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2201)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Caused by: org.apache.uniffle.common.exception.RssException: Blocks read inconsistent: expected 7 blocks, actual 0 blocks
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.checkProcessedBlockIds(ShuffleReadClientImpl.java:215)
at org.apache.spark.shuffle.reader.RssShuffleDataIterator.hasNext(RssShuffleDataIterator.java:135)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
at org.apache.spark.shuffle.reader.RssShuffleReader$MultiPartitionIterator.hasNext(RssShuffleReader.java:227)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:768)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:134)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
Affects Version(s)
0.6.0
Uniffle Server Log Output
No response
Uniffle Engine Log Output
No response
Uniffle Server Configurations
rss.coordinator.quorum=xxx:19999,xxx:19999,xxx:19999
rss.jetty.http.port=19998
rss.prometheus.push.enabled=true
rss.prometheus.uniffle.cluster.name=test100
rss.rpc.executor.size=2000
rss.rpc.message.max.size=1073741824
rss.rpc.server.port=19999
rss.server.app.expired.withoutHeartbeat=120000
rss.server.buffer.capacity=30g
rss.server.commit.timeout=600000
rss.server.flush.cold.storage.threshold.size=64m
rss.server.flush.thread.alive=5
rss.server.flush.threadPool.size=10
rss.server.heartbeat.interval=10000
rss.server.heartbeat.timeout=60000
rss.server.localstorage.initialize.max.fail.number=6
rss.server.preAllocation.expired=120000
rss.server.read.buffer.capacity=15g
rss.storage.type=MEMORY_HDFS

Uniffle Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Maybe #276 is help for you. Coming soon.

Could you try to use the master code?

Maybe #276 is help for you. Coming soon.

Maybe #276 is help for you. Coming soon.

The patch has been updated. The problem still exists.

Could you try to use the master code?

Let me try

Could you try to use the master code?

The same problem exists with the master version.

Could you give me the shuffle server's log? What's your shuffle data size and job's concurrency?

Maybe you should give me the executor's logs, too.

Problems with the company's HADOOP version,
Now it is resolved.

Problems with the company's HADOOP version, Now it is resolved.

@lixy529  hi, I meet the same error , can  I add you wechat to  help me solve this ?

Problems with the company's HADOOP version, Now it is resolved.

@lixy529 @jerqi
I meet same error,  build uniffle   must match HADOOP version ?
company's HADOOP version is 3.3.4,  the build command should be  ./build_distribution.sh --hadoop-profile hadoop3.3 ?

We don't verify the Hadoop version 3.3. We only verify the Hadoop version 3.2.

We don't verify the Hadoop version 3.3. We only verify the Hadoop version 3.2.

@jerqi  thanks !!!   have another 2 questions need your help :
1、if hadoop version is 3.3,  no matter whether it's correct or not ，the build command should be ./build_distribution.sh --hadoop-profile hadoop3.3 ?
      2、want to  write shuffle data into hdfs ， my conf is correct ？
            client.conf :
                             spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager
                             spark.rss.coordinator.quorum=<IP_ADDRESS>:19999,<IP_ADDRESS>:19999,<IP_ADDRESS>:19999
                             spark.rss.storage.type=MEMORY_LOCALFILE_HDFS
                             spark.rss.remote.storage.path=hdfs://ns1/rss/data

            coordinator.conf:
                                       rss.coordinator.quorum <IP_ADDRESS>:19999,<IP_ADDRESS>:19999,<IP_ADDRESS>:19999
                                       rss.rpc.server.port 19999
                                       rss.jetty.http.port 19998
                                       rss.coordinator.server.heartbeat.timeout 30000
                                       rss.coordinator.app.expired 60000
                                       rss.coordinator.shuffle.nodes.max 3
                                       rss.coordinator.exclude.nodes.file.path file:///app/rss-0.7.1/conf/exclude_nodes

                 server.conf：
                                      rss.rpc.server.port 20000
                                      rss.jetty.http.port 20001
                                      rss.storage.basePath /app/rss-0.7.1/data
                                      rss.storage.type MEMORY_LOCALFILE_HDFS
                                      rss.coordinator.quorum <IP_ADDRESS>:19999,<IP_ADDRESS>:19999,<IP_ADDRESS>:19999

We don't verify the Hadoop version 3.3. We only verify the Hadoop version 3.2.

@jerqi thanks !!! have another 2 questions need your help : 1、if hadoop version is 3.3, no matter whether it's correct or not ，the build command should be ./build_distribution.sh --hadoop-profile hadoop3.3 ?
      2、want to  write shuffle data into hdfs ， my conf is correct ？
            client.conf :
                             spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager
                             spark.rss.coordinator.quorum=<IP_ADDRESS>:19999,<IP_ADDRESS>:19999,<IP_ADDRESS>:19999
                             spark.rss.storage.type=MEMORY_LOCALFILE_HDFS
                             spark.rss.remote.storage.path=hdfs://ns1/rss/data

            coordinator.conf:
                                       rss.coordinator.quorum <IP_ADDRESS>:19999,<IP_ADDRESS>:19999,<IP_ADDRESS>:19999
                                       rss.rpc.server.port 19999
                                       rss.jetty.http.port 19998
                                       rss.coordinator.server.heartbeat.timeout 30000
                                       rss.coordinator.app.expired 60000
                                       rss.coordinator.shuffle.nodes.max 3
                                       rss.coordinator.exclude.nodes.file.path file:///app/rss-0.7.1/conf/exclude_nodes

                 server.conf：
                                      rss.rpc.server.port 20000
                                      rss.jetty.http.port 20001
                                      rss.storage.basePath /app/rss-0.7.1/data
                                      rss.storage.type MEMORY_LOCALFILE_HDFS
                                      rss.coordinator.quorum <IP_ADDRESS>:19999,<IP_ADDRESS>:19999,<IP_ADDRESS>:19999

Yes
Ok for me.  rss.coordinator.shuffle.nodes.max is a little small.

@jerqi  thanks !!!

rss.coordinator.shuffle.nodes.max is a little small.
only deployed them in 3 machines,   every machine  had deployed one coordinator server and shuffle server ， if don`t increase machine  and shuffle server instances，only increase  coordinator server instances  is useful ?

@jerqi thanks !!!

rss.coordinator.shuffle.nodes.max is a little small.
only deployed them in 3 machines,   every machine  had deployed one coordinator server and shuffle server ， if don`t increase machine  and shuffle server instances，only increase  coordinator server instances  is useful ?

@jerqi Could you help me explain this ?

We should increase server instances.

@jerqi thanks !!!

rss.coordinator.shuffle.nodes.max is a little small.
only deployed them in 3 machines,   every machine  had deployed one coordinator server and shuffle server ， if don`t increase machine  and shuffle server instances，only increase  coordinator server instances  is useful ?

@jerqi Could you help me explain this ?

We should increase server instances.

@jerqi  got it , thanks !!!


unify config strategy / config cleanup

Currently there are 3 ways of setting config values in Sqlpad - the UI, cli flags, and environment variables.
I think it'd be great if all 3 methods could accept the same config key/value setting where appropriate.
As discussed in #71, env variables would have the format SQLPAD_CONFIGKEYNAME. SqlPad command line flag --CONFIGKEYNAME should be honored as well.
Existing config key/values that don't fit the convention should be honored.

This is covered in wiki roadmap


Dynamically determine storage and 2nd drive mount points

Background
Depending on if the script is run on the following setups, what gets reported as the storage and 2nd drive may or may not be useful.

OS on MicroSD, Data on attached SSD - storage will report space free on microsd, and 2nd drive report space free on the SSD
OS + Data on attached SSD bootable - storage and 2nd drive will report space free on SSD
OS + Data on attached SSD bootable, MicroSD used as a backup - storage and 2nd drive will report space free on SSD

This is because in the script we make assumptions about what the storage and 2nd drive are.  Specifically, storage is the $/ or root of the file system, and the 2nd drive is either that which is mounted to /mnt/ext (Raspibolt < v3) or /data (Raspibolt >= v3)
Considerations
The real estate in the rapsibolt-pulse / system overview script is constrained because in part a desire to be able to show all the data within one screen, and stick to no wider then terminal width (80 characters).  Two lines are dedicated to resources, giving us the space for reporting on 2 drives.
Suggestions
1. Get drive mounts
We could make enhancements to the script to try to dynamically get drive and mount information.
For example, leverage lsblk to list out mountpoints, and sort to push the root to the top (excluding the boot partition)
$ lsblk --output MOUNTPOINT | grep / | grep -v /boot | sort

Example output on a Raspibolt 2 system using microsd as OS and attached SSD for data:
/
/mnt/ext

A Raspibolt 3 system would appear as follows with no additional drive
/

A Raspibolt 3 system where microsd is used as OS and attached drive for the data
/
/data

And for those using SSD drive for the root, and microsd for backups
/
/backups

2. Determine free space based on drive mounts
From the above, once we've determined what to check free space on, that can be done via the existing checks by updating the variables referenced.  The free space check on primary drive doesn't change, as still gets disk free on root.  If the result of the lsblk command above returns more then 1 line (pipe it to | wc -l), then the second line can be assigned to the variable ext_storage2nd.
Sample code
drivecount=$(lsblk --output MOUNTPOINT | grep / | grep -v /boot | sort | wc -l)
if [ $drivecount -gt 1 ]; then
  ext_storage2nd=$(lsblk --output MOUNTPOINT | grep / | grep -v /boot | sort | sed -n 2p)
else
  ext_storage2nd=""
fi

Addressed in #17


[Feature] Allow ConfigurationGenerator to register custom annotation handlers

Currently the ConfigurationGenerator is working with a fixed set of annotations which is ok for now but in the future I`d like to see a way to add custom annotation handlers which should be able to generate methods (and maybe more?) in the generated proxy class so that frameworks could add own annotations (e.g. an annotation to configure the routing or security settings).

Instead of generating magic methods that can be called from the bean configuration method we should simply extend the "Parameter" mechanism and allow to inject those "values". That way the generated code will be syntactically correct - which is one of the key features of Disco.


[SPARK-21150][SQL] Persistent view stored in Hive metastore should be case preserving

What changes were proposed in this pull request?
This is a regression in Spark 2.2. In Spark 2.2, we introduced a new way to resolve persisted view: https://issues.apache.org/jira/browse/SPARK-18209 , but this makes the persisted view non case-preserving because we store the schema in hive metastore directly. We should follow data source table and store schema in table properties.
How was this patch tested?
new regression test

Test build #78277 has started for PR 18360 at commit 439c950.

cc @gatorsmile @viirya

LGTM

Oh. I just thought there might be an issue when doing CREATE OR REPLACE VIEW viewName AS SELECT ... in CreateViewCommand.
The CatalogTable used to replace existing view is also coming from prepareTable. It doesn't contain the properties. And in alterTable we don't explicitly add the properties for VIEW too.
So once we replace an existing view, the properties will not be there anymore.

Test build #78277 has finished for PR 18360 at commit 439c950.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78277/
Test PASSed.

@viirya good catch! fixed

Test build #78289 has started for PR 18360 at commit ff3b1aa.

Test build #78289 has finished for PR 18360 at commit ff3b1aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78289/
Test PASSed.

Thanks! Merging to master/2.2


1388: Add the Others category to the filter bar in the app

Short description
Add the "Others" category button to the filter bar.
Proposed changes

return the Others category in the filter bar

Side effects
Actually this category was removed intentionally in this PR https://github.com/digitalfabrik/entitlementcard/pull/709/files
But I didn't find why?
Maybe it shouldn't be displayed in Nürnberg, but then we could remove it from the config probably?
Resolved issues
Fixes: #1388

The category was not removed by this pr it was only adjusted to be removed for entitlementcard and sozialpass. @seluianova
I think its fine for nuernberg but probably should be discussed with @dkehne

@f1sh1918 right, I just meant it was "removed" from the widget, by calling removeWhere() function :)

I think the decision (back at the time) was that we do not show Others as this is not a real category. But it's also fine to me if we show it :D
We should not encourage the Administrators to put too many stores into the  others category. instead they should improve their categorization scheme in this case.

We can add the Others-category to all entitlement-cards without any flag etc. It's suitable for Nuremberg and Bavaria.


feat: do not use finalizer to stop metrics cache loop

Signed-off-by: Derek Wang<EMAIL_ADDRESS>

How does that interplay with peeking?

How does that interplay with peeking?

I don't want to clean up the key from the dead keys queue when successfully hitting the endpoint during peeking - it's unnecessary to do so which will cost a little bit to check the map, just leave it to the GoRoutine to check Step existence.

Sure. When we get an informer re-sync (every 20m) am I right it thinking it is likely to put any key incorrectly removed back into the map?


I need rowspan & colspan，but have not

Hey @BarretL i know it. So far I dont have time on maintain this project. I will try to implement it when I have enough time, thanks. Any PR is welcome and close it due to duplicate


Stacked area on left Y axis gets re-checked after every simulation

Describe the bug
The new PV-battery default graphs attempt to re-assert themselves after every simulation run, even after initial graphics have been set up.
To Reproduce
Steps to reproduce the behavior:

Create a PV-Battery behind-the-meter case
Click simulate
Change the graphs and uncheck "stacked area on left Y axis"
Click simulate and go back to graphs
Stacked area on left Y axis is checked again

Expected behavior
Time series graphs settings shouldn't change when the user hits simulate
Operating System and Version

Version: 2024.12.12

In my testing, this was only the case when nothing is selected on Time Series graphs, similar to how gen populates when nothing is selected. I'm not opposed to changing the behavior, but that is how it's supposed to work now.

That hasn't been my experience today - maybe something change late in the process? Here are screenshots of an example. Before pressing simulate:

After:

If you're having trouble reproducing it I can send over a SAM file that's giving me issues. If it makes a difference it was upgraded from the previous version of SAM.

Need to move max window sizing within the check for variables selected.


embeds_one requires id field on cast_embed

Environment

Elixir version (elixir -v): 1.8.1
Database and version: PostgreSQL 11.3
Ecto version (mix deps): 3.1.4
Database adapter and version (mix deps): postgrex 0.14.3
Operating system: OSX 10.14.5

Current behavior
I have this schema:
schema "suppliers" do
  field :name, string
  embeds_one :info, Info, on_replace: :delete do
    field :account, :string
  end
end

def changeset(supplier, attrs) do
  supplier
  |> cast(attrs, [:name])
  |> cast_embed(:info, with: &info_changeset/2)
end

def info_changeset(schema, attrs) do
  schema
  |> cast(attrs, [:account])
end
attrs = %{ "name" => "Herman", "info" => %{ "account" => "1234" }}

%Supplier
|> Suppliers.Supplier.changeset(attrs)
|> Repo.insert()

fails with
** (Ecto.NoPrimaryKeyValueError) struct `%Suppliers.Supplier.Info{account: nil, id: nil}` is missing primary key value

Expected behavior
It should insert the record, not complain about a missing id.
if I  add an id field to the embedded info, it works, uses a generated id though.
Or if I force the primary_key to false in the embeds_one's options then it works.
I see 2 issues with this:

Why should I need to add an id field to my info to make it work out of the box. I do not need to do this with embeds_many
Why is there an id in embeds_one in the first place, there can be only one.

Can you provide the complete stacktrace? Is it raising in the insert line?
José Valim
www.plataformatec.com.br
Skype: jv.ptec
Founder and Director of R&D

[error] #PID<0.1069.0> running OvlInvWeb.Endpoint (connection #PID<0.1068.0>, stream id 1) terminated
Server: localhost:4000 (http)
Request: PUT /api/suppliers/12
** (exit) an exception was raised:
    ** (Ecto.NoPrimaryKeyValueError) struct `%OvlInv.Suppliers.Supplier.Info{account: nil, client_ref: nil, id: nil}` is missing primary key value
        (ecto) lib/ecto/embedded.ex:156: anonymous fn/3 in Ecto.Embedded.autogenerate_id/5
        (elixir) lib/enum.ex:1940: Enum."-reduce/3-lists^foldl/2-0-"/3
        (ecto) lib/ecto/embedded.ex:155: Ecto.Embedded.autogenerate_id/5
        (ecto) lib/ecto/embedded.ex:111: Ecto.Embedded.to_struct/4
        (ecto) lib/ecto/embedded.ex:62: anonymous fn/6 in Ecto.Embedded.prepare/5
        (stdlib) maps.erl:257: :maps.fold_1/3
        (ecto) lib/ecto/repo/schema.ex:332: anonymous fn/15 in Ecto.Repo.Schema.do_update/4
        (ecto) lib/ecto/repo/schema.ex:914: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
        (ecto_sql) lib/ecto/adapters/sql.ex:887: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
        (db_connection) lib/db_connection.ex:1415: DBConnection.run_transaction/4
        (ovl_inv) lib/ovl_inv_web/controllers/supplier_controller.ex:31: OvlInvWeb.SupplierController.update/2
        (ovl_inv) lib/ovl_inv_web/controllers/supplier_controller.ex:1: OvlInvWeb.SupplierController.action/2
        (ovl_inv) lib/ovl_inv_web/controllers/supplier_controller.ex:1: OvlInvWeb.SupplierController.phoenix_controller_pipeline/2
        (ovl_inv) lib/ovl_inv_web/endpoint.ex:1: OvlInvWeb.Endpoint.instrument/4
        (phoenix) lib/phoenix/router.ex:275: Phoenix.Router.__call__/1
        (ovl_inv) lib/ovl_inv_web/endpoint.ex:1: OvlInvWeb.Endpoint.plug_builder_call/2
        (ovl_inv) lib/plug/debugger.ex:122: OvlInvWeb.Endpoint."call (overridable 3)"/2
        (ovl_inv) lib/ovl_inv_web/endpoint.ex:1: OvlInvWeb.Endpoint.call/2
        (phoenix) lib/phoenix/endpoint/cowboy2_handler.ex:33: Phoenix.Endpoint.Cowboy2Handler.init/2
        (cowboy) /Users/herman/Projects/ovl_inv/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2

I am sorry, upon reflection, the error was on update not create. But even then the issue still stands, I believe, with regards to the necessity of the primary for an embeds_one.

Can you please post the whole schema module? Do you have @primary_key set
anywhere?
The reason why we require the ID is to know when you are replacing or
keeping the same embed. In any case, it is not something we can change, as
it would be a breaking change. But the ID should be generated by default. I
am surprised that it isn’t. Or you should explicitly disable it as you
mentioned.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

defmodule OvlInv.Suppliers.Supplier do
  use Ecto.Schema
  import Ecto.Changeset

  schema "suppliers" do
    field :city, :string
    field :email, :string
    field :name, :string
    field :street, :string
    field :tel, :string
    field :vat, :string
    field :supplier, :boolean, default: true, load_in_query: false
    field :supplier, :boolean, default: false, load_in_query: false
    field :volunteer, :boolean, default: false, load_in_query: false

    embeds_one :info, Info, on_replace: :delete do
      field :account, :string
      field :client_ref, :string
    end

    timestamps()
  end

  @doc false
  def changeset(supplier, attrs) do
    IO.inspect(attrs)

    supplier
    |> IO.inspect()
    |> cast(attrs, [:name, :street, :city, :vat, :email, :tel])
    |> cast_embed(:info, with: &info_changeset/2)
    |> validate_required([:name, :street, :city])
  end

  def info_changeset(schema, attrs) do
    schema
    |> cast(attrs, [:account, :client_ref])
  end
end

hah, I noticed a bug with posting it here, I have the field suppliers double, one should be customer.

Yeah, I double checked the docs and it does generate a primary key by
default, with autogenerated values, so you have to opt out. Can you tell me
if it is complaining that the new embed doesn’t have a primary key or is it
the one being replaced?
--
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

Looking at the stack trace, I would think the one being replaced, as it shows all fields as nil and has id: nil in it.

Definitely the one being replaced, I tested it. So if your embedded field has no id, you MUST set primary_key to false or it will blow up, or you need to supply a dummy key field in your changes.

So it seems there is a dummy value in there where everything is nil? How is
this value getting in there in the first place? :)
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

I think it comes from here, https://github.com/elixir-ecto/ecto/blob/32677929ccbb02d619d3205ab7bc0a445107125b/lib/ecto/embedded.ex#L110

I see, can you isolate this in a minimal app or a failing test case to this repo so we can take a further look? Thanks.

ok, I'll try

I have create a test in a fork, do I make a PR for it?

@Hermanverschooten yes please!

done #3020

Thanks @Hermanverschooten. I understand how you can trigger the failure by directly calling into the Ecto.Embedded private API but I still cannot reproduce it via the Repo API. If I call Ecto.TestRepo.insert(...) with the changeset in the test you  provided, it will fail because the parent changeset is an insert and the embedded is updating, which is not logically possible. But it won't fail due to the reason you mentioned above.
I can reproduce the error on update, but only if I directly change the embedded field to contain an invalid or non-persisted embedded before calling update.
So can you please describe why your code is trying to update an embedded that was not persisted? Are you directly setting the field? Or are you calling some public API that is causing it?
If you can update the example to contain the full workflow, without calling private APIs, it would be very useful. Thank you.

@josevalim I am going to try to rollback my app to the version I had then, see If it still behaves the same and give you access.

Thanks, that’s very appreciated but I would prefer to not get access to the
app unless really necessary. I believe this can be isolated, we just need a
bit more context around it. :)
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

I have got the error again.  Ok, so what happens is this, the database contains a record in which the info jsonb-field has no id. When I read this back from the id is filled in as nil, when I then try to update it, it throws the error. I updated the test without using the internal api. But it would be better if I could simulate the reading first.

The put_meta make no difference, I thought they might be the trigger, but without the error also appears.

To show you the sequence
iex(29)> OvlInv.Suppliers.create_supplier(%{name: "CJ", street: "Lam", city: "Deurne"})
[debug] QUERY OK db=8.4ms queue=0.7ms
INSERT INTO "suppliers" ("city","customer","name","street","supplier","volunteer","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5,$6,$7,$8) RETURNING "id" ["Deurne", false, "CJ", "Lam", true, false, ~N[2019-06-11 15:16:20], ~N[2019-06-11 15:16:20]]
{:ok,
 %OvlInv.Suppliers.Supplier{
   __meta__: #Ecto.Schema.Metadata<:loaded, "suppliers">,
   city: "Deurne",
   customer: false,
   email: nil,
   id: 5,
   info: nil,
   inserted_at: ~N[2019-06-11 15:16:20],
   name: "CJ",
   street: "Lam",
   supplier: true,
   tel: nil,
   updated_at: ~N[2019-06-11 15:16:20],
   vat: nil,
   volunteer: false
 }}
iex(30)> s = OvlInv.Suppliers.get_supplier! 5
[debug] QUERY OK source="suppliers" db=0.7ms
SELECT s0."id", s0."city", s0."email", s0."name", s0."street", s0."tel", s0."vat", s0."info", s0."inserted_at", s0."updated_at" FROM "suppliers" AS s0 WHERE (s0."id" = $1) [5]
%OvlInv.Suppliers.Supplier{
  __meta__: #Ecto.Schema.Metadata<:loaded, "suppliers">,
  city: "Deurne",
  customer: false,
  email: nil,
  id: 5,
  info: %OvlInv.Suppliers.Supplier.Info{account: nil, client_ref: nil, id: nil},
  inserted_at: ~N[2019-06-11 15:16:20],
  name: "CJ",
  street: "Lam",
  supplier: true,
  tel: nil,
  updated_at: ~N[2019-06-11 15:16:20],
  vat: nil,
  volunteer: false
}
iex(31)> OvlInv.Suppliers.update_supplier(s, %{ id: 2, name: "CJ", street: "Lam", city: "Deurne", info: %{ account: "1234"}})
[debug] QUERY OK db=0.2ms
begin []
[debug] QUERY OK db=0.2ms
rollback []
** (Ecto.NoPrimaryKeyValueError) struct `%OvlInv.Suppliers.Supplier.Info{account: nil, client_ref: nil, id: nil}` is missing primary key value
    (ecto) lib/ecto/embedded.ex:156: anonymous fn/3 in Ecto.Embedded.autogenerate_id/5
    (elixir) lib/enum.ex:1940: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto) lib/ecto/embedded.ex:155: Ecto.Embedded.autogenerate_id/5
    (ecto) lib/ecto/embedded.ex:111: Ecto.Embedded.to_struct/4
    (ecto) lib/ecto/embedded.ex:62: anonymous fn/6 in Ecto.Embedded.prepare/5
    (stdlib) maps.erl:257: :maps.fold_1/3
    (ecto) lib/ecto/repo/schema.ex:332: anonymous fn/15 in Ecto.Repo.Schema.do_update/4
    (ecto) lib/ecto/repo/schema.ex:914: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
    (ecto_sql) lib/ecto/adapters/sql.ex:887: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
    (db_connection) lib/db_connection.ex:1415: DBConnection.run_transaction/4

Thanks! This puts us on a tough spot. We could remove the exception here,
but from ecto perspective you have invalid data and now we will stop
detecting such cases. It would be important to know how the bad data got
there in the first place but I would update the database to add a random ID
if none is there.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

that's what I initially detected, just adding a random id to your new data, solves the issue.
But wouldn't it be better if the id was set on read?  We know the id is necessary, and in combination with on_update: :delete is going to be replaced, so instead of nil and auto-generated id. It would solve this case. Just a thought.

But the bad data is created by ecto... if you have no information in the embedded, it will be written as nil to the database, and up read id is nil, and ....

But the bad data is created by ecto...

Can you show how this is happening? Ecto will generate the ID on insertion
and Ecto should not allow an embed with nil ID to be inserted. If this is
happening, then that’s the bug.
José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

if you look at the code I posted 3 hours ago (5 posts up), you see exactly this.
And this is the schema that was used:
defmodule OvlInv.Suppliers.Supplier do
  use Ecto.Schema
  import Ecto.Changeset

  schema "suppliers" do
    field :city, :string
    field :email, :string
    field :name, :string
    field :street, :string
    field :tel, :string
    field :vat, :string
    field :supplier, :boolean, default: true, load_in_query: false
    field :customer, :boolean, default: false, load_in_query: false
    field :volunteer, :boolean, default: false, load_in_query: false

    embeds_one :info, Info, on_replace: :delete do
      field :account, :string
      field :client_ref, :string
    end

    timestamps()
  end

  @doc false
  def changeset(customer, attrs) do
    customer
    |> cast(attrs, [:name, :street, :city, :vat, :email, :tel])
    |> cast_embed(:info, with: &info_changeset/2)
    |> validate_required([:name, :street, :city])
  end

  def info_changeset(schema, attrs) do
    schema
    |> cast(attrs, [:account, :client_ref])
  end
end

and this the create/update functions:
def create_supplier(attrs \\ %{}) do
    %Supplier{}
    |> Supplier.changeset(attrs)
    |> Repo.insert()
  end

def update_supplier(%Supplier{} = supplier, attrs) do
    supplier
    |> Supplier.changeset(attrs)
    |> Repo.update()
  end

It is not allowing an insert with a nil id, it is inserting the whole embedded as nil

Thanks for updating #3020. You are still explicitly  marking the data as loaded and adding a schema without an ID. So Ecto is correct (so far) in complaining. In other words, if you are directly changing the struct state, you need to do so consistently.
What I would like to see is: a way to insert a schema with an embedded in the database in a way that the IDs of the embedded are not properly set (i.e. they remain nil). Otherwise, based on all of the examples given so far, there is something directly bypassing Ecto's behaviour, which means it is not an Ecto bug. Thank you.

Reading back from your replies, I see this:
iex(30)> s = OvlInv.Suppliers.get_supplier! 5
[debug] QUERY OK source="suppliers" db=0.7ms
SELECT s0."id", s0."city", s0."email", s0."name", s0."street", s0."tel", s0."vat", s0."info", s0."inserted_at", s0."updated_at" FROM "suppliers" AS s0 WHERE (s0."id" = $1) [5]
%OvlInv.Suppliers.Supplier{
  __meta__: #Ecto.Schema.Metadata<:loaded, "suppliers">,
  city: "Deurne",
  customer: false,
  email: nil,
  id: 5,
  info: %OvlInv.Suppliers.Supplier.Info{account: nil, client_ref: nil, id: nil},
  inserted_at: ~N[2019-06-11 15:16:20],
  name: "CJ",
  street: "Lam",
  supplier: true,
  tel: nil,
  updated_at: ~N[2019-06-11 15:16:20],
  vat: nil,
  volunteer: false
}

get_supplier! is returning an empty "info" embed instead of simply returning nil for info. Are you doing something in get_supplier that is potentially setting a default value?

And here is a test that shows that embeds written as nil are loaded as nil: b0baf80cb2afe90a22615c451582f7bbd16f01fd

Ping. :)

If you have more information, please let us know, thanks!

For reference, I have run into this issue when changing a field definition in an Ecto schema, from one that does not expect a primary key, to one that does expect a primary key. While the schema field was not expecting a primary key, it allowed data with no primary key to be inserted. After I changed the schema definition to expect a primary key for that field, a Ecto.NoPrimaryKeyValueError was raised when trying to update old data that did not have a primary key.
I think this is ok and to be expected, we shouldn't expect to necessarily be able to change an Ecto schema field's definition without migrating existing data to comply with the new definition. I have tripped myself up on this more than once though, so I hope this note may be useful.
There are two common cases that can cause this:

Changing a field's definition from :map to embeds_one without updating existing data to ensure it contains primary keys.
Removing @primary_key false from an embeds_one without updating existing data to ensure it contains primary keys.

If you end up in this situation, a simple approach is to find records with no primary key and set them to null (will cause data loss of course). UPDATE my_table SET my_col = NULL WHERE my_col -> 'id' IS NULL;. Alternatively you could update the json values, adding ids where they were missing.

I just ran into the same issue as described by @brucepom because of a migration add :params, :map, default: %{}, having lots of empty json values that are not considered as non-existent by ecto.
@josevalim

What would be the best place to document this behaviour? I would like to draft some text and send a suggestion
Would it maybe be possible to treat an empty json value the same as a non-existent embedded value (null) ?


Local Storage Deprecation

Hello.
Not really an issue.  More of a request.
Please do not deprecate the local storage option.  It is very useful for those of us that never send or store keepass files over the net.  There are strong security reasons to never put a keepass file online.
thank you

I am not sure why the authors feel that the users expect databases to stay current automatically.  Most uses of keepass are probably read only anyway.  Keeping a reference set of keepass files on a USB drive is fairly normal and considerably safer than storing them on Dropbox, Google Drive, or other online storage solutions.
As the only reasonable option so far on keepass usage in chrome (and chromebooks) it seems reasonable to expect that the authors have a particularly keen understanding of security and privacy.  Online storage of anything is neither secure nor private.
=-=-=-=
CKP no longer supports this feature because it cannot work the way that people expect.
=-=-=-
Loading files this way is ONLY suitable for old files that are never updated. Please use Google Drive or Dropbox or OneDrive for the best CKP experience!
=-=-=-=
I realize that files stored in this way won't update themselves
=-=-=-=

I have no plans to actually remove the feature.  I use it myself too, in some scenarios.
It just falls short of my and other users' expectations, because ideally the feature would be able to read the file from a USB drive directly, but that just cannot be done by a Chrome Extension.

Since the core feature that I would use is being deprecated, I can not justify adopting this extension.  I'll look else where for my solutions.

Indeed, really unfortunate. CKP seemed promising. I have my own personal cloud with my own synchronization mechanism between my devices so i use the local file storage so my password are where i want them to be.


not an issue, but...

just reading the source code now, how did you manage to pull someone's license data (mainly the license name)?
just thought I'd ask because i'm thinking of forking (with your approval) this repo + adding my own little touches to it.

It comes from the event FLP_Text_Licensee, which is a regular string except it's obfuscated. The code to deobfuscate it is in that event handler code as well.
Curious what you're gonna do with a fork! 👀

I was thinking about adding parts of this to something bigger a few friends and I had an idea for. I could talk about it more but talking through GitHub issues isn't the best thing haha

Sounds interesting, but yes indeed, the issue tracker isn't the best place for this 👍


Patch cfg factory

The generic variable S was misplaced on the function instead of the class; this makes it hard to change what S exactly is.
The function also requires a filePath since the CFG factory will make use of the babel visitor in the javascript implementation which also requires a filePath to be specified.

@dstallenberg could you add in the description why the change is necessary

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.05%) to 53.879%

Totals

Change from base Build<PHONE_NUMBER>:
0.05%

Covered Lines:
832

Relevant Lines:
1453

💛  - Coveralls


Change the way Survol differentiates URLs, not only by domain, but also subdomain

For example, this and this are completely different website, although has same domain name. But, Survol treat those as same.

This changes will make Survol treat those websites different. So if you disable Survol on one page, the other page have no effect.

It makes sense, then maybe we should add some kind of regex in the settings when disabling inner links also, so people can type in something like *.github.com or just gist.github.com for example.

like this https://developer.chrome.com/extensions/match_patterns?

hum no, this is manifest related. Currently the extension has 2 options to disable it :

The page setting : which disables Survol completely on the page, when disabled you won't be able to preview any link, this option is toggled with the switch in the popup.
The inner-link setting : which disables Survol for some links on certain pages, e.g: github.com links are disabled on github. The list of websites where inner-link preview is disabled is situated in the global settings.

Things that should be done :

Display a list of websites disabled completely, and let the user input directly into the list with regex inputs *.github.com, github.com, gist.github.com
Modify the current inner-link setting to work with regex inputs

I don't actually understand what you mean by "manifest related", but the first point, basically you want user to be able to see a list of the pages they disabled right? So it must be something like this:

The link you sent, explains how to enable / disable the extension via the manifest.json file, which is not what we want, we want to enable / disable the extension programatically through the extension itself.
But yea basically we should implement it like you did in the image you sent, the user should be able to input things like "*.github.com" (== "github.com") "gist.github.com", ...

Ohh, you misunderstood me 😆
I sent that link just to provide a pattern examples that maybe we might use, but not to implement into manifest.json

I don't actually understand what you mean by "manifest related", but the first point, basically you want user to be able to see a list of the pages they disabled right? So it must be something like this:

So, if I want to try to implement this, should I add a commit here or create a new PR?

I think all the changes should be implemented in this PR as otherwise everything would break

I'm not sure if you're done with this, but we need to adapt the way we disable the extension in core.js - master - 230 in order to make sure if someone blocks the extension on github.com it blocks it on every subdomain, but if someone blocks only 1 subdomain the extension will be disabled only on this one.
Should also apply the same way of blocking to inner link preview core.js - master - 218

Wouldn't it be better to be left like this? If user blocks the extension on github.com and it blocks on every subdomain, then why do we need regex you've mentioned earlier. So my suggestion, if user wants to disable extension on every subdomain, they can do something like `.github.com.

Yes but right now without modifying anything if you block "github.com" it will only block the extension on github.com and not on every subdomains

That's what I mean mate.
But there's a problem, what if user wants to block all subdomains except for one.
For example, someone wants to block all netlify-hosted sites, so he adds *.netlify.app in the settings page. Then, he goes to sample.netlify.app for example, the switch should be turned off right, but what will happen if he turns the switch on, will it remove the whole *.netlify.app or how? Should we make a whitelist mechanism?

I guess we could modify the regex to to add an exception for a list of selected subdomains

Maybe we could use exclamation mark ! or something. If so, the above case can be resolved with *.netlify.app,!sample.netlify,app.

Or could be like UserScript, but that would be very complicated.

I'm not a regex pro but something like this I guess  (assuming gist is blocked)
^((gist|cdn).*).github.com|github.com$
In this example gist.github.com, github.com and cdn.github.com are whitelisted, everything else is blocked,
You could remove the last github.com to remove the main domain


[Indexer] Add bcs SuiSystemState to epochs table

Description
The existing system_state column is a BCS serialization of a JSON-RPC type, which is not evolvable.
This PR adds a new column that is the JSON serialization of SuiSystemStateSummary without BCS.
It also fixes two bugs along the way:

We were storing the wrong version of the system state, offsetting by 1 epoch. This PR fixes it by storing the right version of the json column at the beginning of the epoch instead of at the end.
We were ignoring a deserialization error and swallow it in some place. Made that throw instead.

Test plan
Added a debug assert to see that this column is populated with the correct epoch.

Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

[ ] Protocol:
[ ] Nodes (Validators and Full nodes):
[ ] Indexer:
[ ] JSON-RPC:
[ ] GraphQL:
[ ] CLI:
[ ] Rust SDK:
[ ] REST API:

Why don't we instead store the serialized JSON form instead of a BCS form? JSON is a much more evolvable format that bcs and the SystemState type is not a core data structure that has a "canonical" bcs form.

Why don't we instead store the serialized JSON form instead of a BCS form? JSON is a much more evolvable format that bcs and the SystemState type is not a core data structure that has a "canonical" bcs form.

We certainly could. SuiSystemState type uses enum at the top so it should be "canonical"?

Why don't we instead store the serialized JSON form instead of a BCS form? JSON is a much more evolvable format that bcs and the SystemState type is not a core data structure that has a "canonical" bcs form.

We certainly could. SuiSystemState type uses enum at the top so it should be "canonical"?

Its just this will always be transmitted in json form, so we should just store and use the json form

All feedback addressed. Updated the PR description.


[GLUTEN-1695][VL] Bug: Show the right writen row number of DataWritingCommandExec in spark UI

What changes were proposed in this pull request?
(Fixes: #ISSUE-ID)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Run Gluten Clickhouse CI


Editing project settings breaks every Control node's (and its derivatives) size_flags_changed signal.

Tested versions
Godot Engine v4.1.4.stable.official.fe0e8e557
System information
Windows 10, Godot Engine v4.1.4.stable.official.fe0e8e557. Compatibility
Issue description
In any project where there are any Control nodes, and said project is exported as a Release build on Windows, the console will throw errors such as these:
Godot Engine v4.1.4.stable.official.fe0e8e557 - https://godotengine.org
OpenGL API 3.3.0 - Build <IP_ADDRESS>49 - Compatibility - Using Device: Intel - Intel(R) HD Graphics 4600
USER ERROR: Attempt to disconnect a nonexistent connection from 'ColorRect:<ColorRect#22884123746>'. Signal: 'size_flags_changed', callable: ''.
at: _disconnect (core/object/object.cpp:1355)
USER ERROR: Attempt to disconnect a nonexistent connection from '_v_scroll:<VScrollBar#22850569312>'. Signal: 'size_flags_changed', callable: ''.
at: _disconnect (core/object/object.cpp:1355)
USER ERROR: Attempt to disconnect a nonexistent connection from '_h_scroll:<HScrollBar#22817014878>'. Signal: 'size_flags_changed', callable: ''.
at: _disconnect (core/object/object.cpp:1355)
This happens either when I changed the Stretch Mode to 'canvas_items', or toggled Advanced Settings on, or toggled 'convert text resources to binary' on. It specifically targets the 'size_flags_changed' signal.
Steps to reproduce

Create a project
Set any node as the root node
Add a bunch of Control nodes, or its derivatives (just one is enough, as well)
Add some way to quee_free() the Control nodes
Try exporting the project on Windows, in Release mode, include console
Run console, everything should be fine if the Control nodes are released
Now change any of the following project settings:
-change the Stretch Mode to 'canvas_items'
-toggle Advanced Settings on
-toggle 'convert text resources to binary' on
Export as in step 5
Releasing any Control node now should give an error similar to this: USER ERROR: Attempt to disconnect a nonexistent connection from '_v_scroll:<VScrollBar#22850569312>'. Signal: 'size_flags_changed', callable: ''.

Minimal reproduction project (MRP)
(File size exceeds maximum size, so I cannot upload a sample here.)

Sounds like:

https://github.com/godotengine/godot/issues/89657

Can you check if this occurs in 4.1.3? If not then it's likely the same issue as it was caused by a PR cherry picked into 4.1.4

I apologize, but I cannot test it at the moment, as I don't have 4.1.3, or
the export templates yet. I'll see if I find some time later.
ned, 28. jul 2024. 12:41 A Thousand Ships @.***> је
написао/ла:

Sounds like:

#89657 https://github.com/godotengine/godot/issues/89657

Can you check if this occurs in 4.1.3? If not then it's likely the same
issue as it was caused by a PR cherry picked into 4.1.4
—
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/94868#issuecomment-2254469168,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A774YDVVSPK3YYIKPZRD6FTZOTDFDAVCNFSM6AAAAABLSUVS5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJUGQ3DSMJWHA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Remove legacy stacktrace parsing from Opera 9, 10

This was a lot of special-case code to dedicate to really old browsers that are in the 0.01% range of worldwide usage (according to caniuse).
I've verified that Opera 12.16 (last version built on Presto) and Opera 15 (first version built on Webkit) both provide stack traces after this change.
cc @dcramer @macqueen @mattrobenolt @wheresrhys

This change is

Filesize changes:

Type
Before
After
% Change

unminified
88,284
82,131
-6.9%

minified
23,140
21,695
-6.2%

minified + gzip
8,946
8,529
-4.7%


[FLINK-2567]

Allow quoted strings in CSV fields to contain quotation character inside of the field, as long as its escaped Ex: 'Hi my name is 'Flink' '. If a quotation character is found, the previous character is checked to see if it is the escape character. If yes, the string continues to be searched for the closing quotation. Test classes were updated to include strings with escaped quotation characters.

This PR should also update documentation of the CSV parser (see "Data Sources" section in programming_guide.md").
Thanks, Fabian

Agree with Fabian. Other then the inline comments and the documentation, this looks good.

Looks good, merging this...

Thanks for your contribution @tammymendt. You pull request has been merged but the auto-close didn't work :) Could you close the pull request?

Is this merged? I cannot find this commit in ASF Git repository.

Pending merge... it is in my travis branch, tests ran over night...

Oh sorry, that was a reference from Stephan's repository.

@tammymendt Sorry for the confusion. Could you please re-open the pull request? I thought it had been merged but it was only merged into Stephan's repository...

Now it's merged. This time for real :)


New Generation HopeField & Transform Networks Support

Hi , there is a new type network named Transform.
It beats LSTM at all costs.
Now many people migrating to Transform on Keras and Python
Currently , seems like DL4J doesnt have Transform Network support
Also with new Transform Network , we can use new generation HopeField network that seems  usable.
Here is :
Hopfield Networks is All You Need (Paper Explained)
https://www.youtube.com/watch?v=nv6oFDp6rNQ
Illustrated Guide to Transformers Neural Network: A step by step explanation:
https://www.youtube.com/watch?v=4Bdc55j80l8
Hope to see these libraries as , easy to use on DL4J

I have found one of them also java wrapper
https://github.com/zhihu/cuBERT
https://github.com/zhihu/cuBERT/blob/master/java/src/test/java/com/zhihu/cubert/ModelTest.java

Also there is TurboTransformers Library on C++
https://github.com/Tencent/TurboTransformers
maybe implemented or add to DL4J can be good choice!

@eix128 you should be able to run transformer models using our tensorflow import and samediff.
https://deeplearning4j.konduit.ai/samediff/importing-tensorflow
more here: https://github.com/eclipse/deeplearning4j-examples/blob/master/tensorflow-keras-import-examples/README.md


Jank/flicker in the visual editor

Ze Ye we need to guard view while the placement of the nodes and edges are calculated and re-drawn. This isn't a ship-blocker for R9, but a strong nice to have.

fixed in #2780


DataStore not working with @connection

Describe the bug
I am trying to implement Has Many. The following model does not work i.e. I do not get results that were created using the App Sync console:
type User
@model
{
  id: ID!
  actions: [Action!]! @connection
}

Using the App Sync console, everything looks ok, I can do get and list and results come back, but with Amplify.DataStore.query(User.self) I always get case .success(let result) but result is empty and there are no errors.

@amuresia Have you found a solution for this? I think I experience the same problem

@amuresia @Nicolaidam can you confirm if the record created in the AppSync console have the conflict detection fields _deleted, lastChangedAt and _version like described in the docs? https://docs.amplify.aws/lib/datastore/how-it-works/q/platform/ios#writing-data-from-the-appsync-console
DataStore uses an augmented AppSync schema to reconcile distributed offline/online. When interacting with a DataStore backend via the console, you have to provide _version explicitly as well as adding all three in the mutation result selection set.

Closing this due to inactivity. Feel free to re-open if you're still experiencing the same issue.

I have the same issue with a very simple model:
type Course @model {
  id: ID!
  name: String!
  owner: String
  lectures: [Lecture!]! @connection(keyName: "byCourse", fields: ["id"])
}

type Lecture @model @key(name: "byCourse", fields: ["courseId", "order"]) {
  id: ID!
  name: String!
  owner: String
  courseId: ID
  order: String
}

Amplify build by model gen is also good:
export const schema = {
    "models": {
        "Course": {
            "name": "Course",
            "fields": {
                "id": {
                    "name": "id",
                    "isArray": false,
                    "type": "ID",
                    "isRequired": true,
                    "attributes": []
                },
                "name": {
                    "name": "name",
                    "isArray": false,
                    "type": "String",
                    "isRequired": true,
                    "attributes": []
                },
                "owner": {
                    "name": "owner",
                    "isArray": false,
                    "type": "String",
                    "isRequired": false,
                    "attributes": []
                },
                "lectures": {
                    "name": "lectures",
                    "isArray": true,
                    "type": {
                        "model": "Lecture"
                    },
                    "isRequired": true,
                    "attributes": [],
                    "association": {
                        "connectionType": "HAS_MANY",
                        "associatedWith": "courseId"
                    }
                }
            },
            "syncable": true,
            "pluralName": "Courses",
            "attributes": [
                {
                    "type": "model",
                    "properties": {}
                }
            ]
        },
[...]

By when I query the model:
await DataStore.query(Course, id);

there is no field lectures in the response.
{ "id": "3960b325-e924-4b11-b59c-ad9cd6a19cc2", "name": "Main test", "owner": null, "_version": 1, "_lastChangedAt":<PHONE_NUMBER>907, "_deleted": null }
If I look at the request send to the backend I see that indeed lectures was not part of the query:
{"query":"query operation($limit: Int, $nextToken: String, $lastSync: AWSTimestamp) {\n  syncCourses(limit: $limit, nextToken: $nextToken, lastSync: $lastSync) {\n    items {\n      id\n      name\n      owner\n      _version\n      _lastChangedAt\n      _deleted\n    }\n    nextToken\n    startedAt\n  }\n}\n","variables":{"limit":1000,"nextToken":null,"lastSync":1596705659144}}

Note that the queries.js built by amplify codegen is good, but doesn't seems to be the one used by DataStore
export const syncCourses = /* GraphQL */ `
  query SyncCourses(
    $filter: ModelCourseFilterInput
    $limit: Int
    $nextToken: String
    $lastSync: AWSTimestamp
  ) {
    syncCourses(
      filter: $filter
      limit: $limit
      nextToken: $nextToken
      lastSync: $lastSync
    ) {
      items {
        id
        name
        owner
        lectures {
          items {
            id
            name
            owner
            courseId
            order
            _version
            _deleted
            _lastChangedAt
            createdAt
            updatedAt
          }
          nextToken
          startedAt
        }
        _version
        _deleted
        _lastChangedAt
        createdAt
        updatedAt
      }
      nextToken
      startedAt
    }
  }
`;

any further updates on this issue ?

Hi @shivaping @djou-13 @drochetti @amuresia @Nicolaidam - this issue was closed, but I can't see anywhere where the issues were addressed.
I have a model called "asset" and on that model I have a connection to multiple "photo" records.
photo: [Photo] @connection (keyName: "byPhotoAssetId", fields: ["id"])

Amplify codegen generates the following entry on the asset model in schema.js
                "photo": {
                    "name": "photo",
                    "isArray": true,
                    "type": {
                        "model": "Photo"
                    },
                    "isRequired": false,
                    "attributes": [],
                    "isArrayNullable": true,
                    "association": {
                        "connectionType": "HAS_MANY",
                        "associatedWith": "asset"
                    }
                },

In my GraphQL back end I can execute the query
query getManyPhotos{
  getAsset(id:"xxx")
  {
      photo{
       items{
          photoType
       }
     }
   }
 }

Which generates the following result
{
  "data": {
    "getAsset": {
      "photo": {
        "items": [
          {
            "photoType": "EQUIPMENT"
          },
          {
            "photoType": "NAMEPLATE"
          }
        ]
      }
    }
  }
}

But in DataStore when I query the Asset the photo field is not even visible for me to query against - which is what is being said in @djou-13 comment (https://github.com/aws-amplify/amplify-ios/issues/547#issuecomment-669824476) above.
There is a HAS_MANY associated field generated by amplify codegen - how do we use it?


QoL

what i've done:

make the installer automaticticalt target stable
kill the inatsller instead of letting it continue on a fatal error
improve error messages

I'm not sure I would want it to install to stable automatically, that feels wrong. Also generally in this PR you seem to have messed with some style stuff including spacing (4 -> 2), can you please revert?


Maya: os.getcwd()

Describe the bug
The cwd of Maya application should be set properly.
To Reproduce
Launch Maya from Launcher for a new task, open Python script Window, use
import os
print os.getcwd()

to inspect the cwd, it would print the D:\Development\OpenPype-repo\igniter which is the igniter folder path.
Expected behavior
This path should be a path for current asset.
Desktop (please complete the following information):

OS: Windows
Host: Maya

Additional context
3.9.0-nightly.9

This path should be a path for current asset.

Can you describe the use case? Because I'm afraid that is irrelevant request to set right cwd to be able use os.getcwd to get directory of opened scene.
CWD (Current work directory) is set at the start of Maya process and does not change during it's lifetime. If you would open scene from different folder using File > Open Scene the cwd would not change.
Also you can open .ma/.mb file using cmd (or Poweshell):
> cd C:/Windows
> D:/path/to/your/file.ma

The cwd would be C:/Windows or anything you pass after cd ....


Broken when using nested theme properties

🐛 Bug Report
Broken when using properties nested in the theme object.
At @smooth-ui version 10.0.7
Error message

Couldn't parse the color string. Please provide the color as a string in hex, rgb, rgba, hsl or hsla notation.

To Reproduce

Define colors object for theme.

const colors = {
  skyBlue: '#fff',
  paleVio: 'palevioletred'
}

Create nested properties theme object

example of bug
// Cause an error
const theme = {
  colors
}

Create non nested properties theme object

example of work fine.
// Work fine
const theme = {
  ...colors
}

Expected behavior
Should work even if the theme object is deep.
Link to repl or repo (highly encouraged)
Codesandbox: https://codesandbox.io/embed/n0vxz7j0om

Look both <BugThemeExample /> and <WorkThemeExample />

In fact, Smooth UI already uses theme.colors, you have to specify the default one from Smooth UI. See this updated example: https://codesandbox.io/s/k14xmwow95.
We should definitively explain it in the documentation.

Ah I see. It conflicts with the default theme of Smooth-UI.
Thank you for telling the solution of sandbox example.


Add local ENR change listener

PR Description
Address the case when the client code needs to be notified on the local ENR updates.
Fixed Issue(s)
the part of

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Disable SuperSafe warning...

Firstly, apologies for using the ScalaTest issue tracker for this, but there doesn't appear to be a separate tracker for SuperSafe...
I'm using SuperSafe Community Edition 1.1.9 with Scala 2.13.1, and it produces a warning about one test file when running sbt test:

[warn] [Artima SuperSafe] 8 errors found in /home/someuser/someproject/src/test/scala/somepackage/SomeTest.scala that are not reported by the Community Edition.  To get full information about the errors, please install your license file in the .supersafe subdirectory of your home directory on this machine. If you don't have an activated license, you can get a 30 day free trial at: http://www.artima.com/shop/supersafe.

I understand the issues, and I'm not concerned about them: they're all valid tests.
However, the warning prevents me from compiling my project with the -Werror Scala compiler option, which I regard as essential.
Is there any way to disable this warning, or inhibiting SuperSafe from checking the code that leads to the warning? If not, can I raise a solution to this problem as an enhancement request? I'm currently faced with either disabling -Werror or not using SuperSafe Community Edition at all, neither of which are appealing. (The project concerned is an open-source project, and I can't justify purchasing a commercial license for it, nor could I then use Travis to perform builds.)
BTW, as a general point, there appears to be no documentation that lists and differentiates the features in the Community Edition and the full commercial product. One thing that I have noticed is that the Community Edition does not appear to recognize the specification of a SuperSafe configuration file as a Scala compiler option. That is, I get the following when I try to do that:

[error] bad option: -P:artima-supersafe:config-file:supersafe.cfg

Could you add a comparison of the two versions, or indicate in the documentation those features that are not part of the Community Edition?
Thanks for your assistance with this!

I have this same problem. Seems I will have to just remove the community edition of this plugin unless this WARN can be suppressed somehow.

I have this same problem. Seems I will have to just remove the community edition of this plugin unless this WARN can be suppressed somehow.

I wonder if this can be filtered somehow in Scala 2.13.x now that -Wconf exists...

@cheeseng @bvenners we are considering removing this plugin entirely because we end up having to disable it on most projects due to the warning. Are there any plans for a workaround? Maybe a setting to disable the message? I would prefer having some support rather than none at all, as I have seen the plugin catch issues during compilation that otherwise would have failed during test execution, but we have no plans on upgrading away from the community edition at this time.

@solarmosaic-kflorence Can you describe the warning you'd like to suppress?

@bvenners it's the one mentioned in the first comment:

[warn] [Artima SuperSafe] 8 errors found in /home/someuser/someproject/src/test/scala/somepackage/SomeTest.scala that are not reported by the Community Edition. To get full information about the errors, please install your license file in the .supersafe subdirectory of your home directory on this machine. If you don't have an activated license, you can get a 30 day free trial at: http://www.artima.com/shop/supersafe.

The comment is actually very thorough about why this is a problem.

@solarmosaic-kflorence Yes, I reread that. Will think about how to enable -Werror.


Koushikphy/FORTRAN-Topics is no longer available

README.md:1686: References github.com/Koushikphy/FORTRAN-Topics which is no longer available.
[FORTRAN-Topics](https://github.com/Koushikphy/FORTRAN-Topics): codes to demonstrate Modern Fortran features, and codes for numerics and I/O, by Koushikphy

removed


Building fails.

When building I get the following error.
build/server/_serverless.js:2:21: error: Could not resolve "../index.js"

The same here

Hi @ivands @DanielRios549, Thanks for taking an interest in my project.
I think the issue may be the version of my adapter, and the version of svelte kit you are using are incompatible, could you try instead using v0.4.2 and let me know if that fixes your issue.
To give you an idea of how fast the adapter API is changing, a last released a build 19 days ago, to fix issues with the then current API, those changes are already out of date and the Svelte team seem to have changed the way that want to do things yet again.

You may also want to rebase your project onto the latest svelte kit build 1.0.0-next.301 and use the latest adapter version

Upgrading to 1.0.0-next.301 doesn't help.
The build folder doesn't have an index.js file inside.

Upgrading the adapter to 0.6.0 fixed the issue tho.

Hi @ivands That is expected behaviour as the file is removed at the end of the build process.
Im glad updating fixed your issue however :)


feat: basic manual instrumentation

we have diff on the check-traces results
manual:

auto:

@pnaik1 PTAL :+1:


[INLONG-6747][Sort] StarRocks connector supports transferring all tables for all schemas in one database

Prepare a Pull Request

[INLONG-6747][Sort] StarRocks connector supports transferring all tables for all schemas in one database

Fixes #6747

Motivation
Inspired by doris-inlong, starrocks-inlong can also support transferring all tables in one database on time.
StarRocks FE api has changed, is different from Doris FE api. If you use doris-cdc-inlong to handle StarRocks tables writing, it will give a error: http://x.x.x.x:xxxx/api/backends?is_alive=true failed. java.io.IOException: Failed to get response from Doris. That is why we need a starrocks-inlong connector.
Based on flink-starrocks from community, starrocks-inlong supports all tables writing in one database one time.
Modifications

StarRocksDynamicSinkFunction's invoke method supports handling debezium canal-json upstream parsing. It picks data from upstream, assembles data with json format.

StarRocksSinkManager's init method will only validate table structure in single table transferring, and skip validating in all database migration.

StarRocksStreamLoadVisitor's checkLabelState method add String database parameter.

StarRocksSinkManager removes checkFlushException method invoking in waitAsyncFlushingDone, close and flushInternal methods. If there is a error when writing to StarRocks, flink cdc will fail to make a checkpoint because of this error.

Verifying this change
(Please pick either of the following options)

[ ] This change is a trivial rework/code cleanup without any test coverage.

[ ] This change is already covered by existing tests, such as:
(please describe tests)

[ ] This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

@liaorui Please add packaging config for this connector in inlong-distribution/src/main/assemblies/sort-connectors.xml

@liaorui Please add base shade in starrocks.pom, ref to iceberg pom. Because It can avoid modifying base lead classloader problem.
     <relocations>
                                <relocation>
                                    <pattern>org.apache.inlong.sort.base</pattern>
                                    <shadedPattern>
                                        org.apache.inlong.sort.starraocks.shaded.org.apache.inlong.sort.base
                                    </shadedPattern>
                                </relocation>
                            </relocations>

@liaorui Please add packaging config for this connector in inlong-distribution/src/main/assemblies/sort-connectors.xml

done

@liaorui Please add base shade in starrocks.pom, ref to iceberg pom. Because It can avoid modifying base lead classloader problem.
     <relocations>
                                <relocation>
                                    <pattern>org.apache.inlong.sort.base</pattern>
                                    <shadedPattern>
                                        org.apache.inlong.sort.starraocks.shaded.org.apache.inlong.sort.base
                                    </shadedPattern>
                                </relocation>
                            </relocations>

done