[SPARK-6028][Core]A new RPC implemetation based on the network module

Design doc: https://docs.google.com/document/d/1CF5G6rGVQMKSyV_QKo4D2M-x6rxz5x1Ew7aK3Uq6u8c/edit?usp=sharing

Merged build triggered.

Merged build started.

Test build #33660 has started for   PR 6457 at commit 2011e94.

Test build #33660 has finished for   PR 6457 at commit 2011e94.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class MessageLoop extends Runnable
class NettyRpcEnvFactory extends RpcEnvFactory with Logging
class NettyRpcEndpointRef(@transient conf: SparkConf)
class NettyRpcHandler(
class KryoSerializationStream(
class KryoDeserializationStream(

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #33748 has started for   PR 6457 at commit e41f2e8.

Test build #33748 has finished for   PR 6457 at commit e41f2e8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class MessageLoop extends Runnable
class NettyRpcEnvFactory extends RpcEnvFactory with Logging
class NettyRpcEndpointRef(@transient conf: SparkConf)
class NettyRpcHandler(

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #33783 has started for   PR 6457 at commit 91fc944.

Test build #33783 has finished for   PR 6457 at commit 91fc944.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class MessageLoop extends Runnable
class NettyRpcEnvFactory extends RpcEnvFactory with Logging
class NettyRpcEndpointRef(@transient conf: SparkConf)
class NettyRpcHandler(

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #33850 has started for   PR 6457 at commit 91fc944.

Test build #33850 has finished for   PR 6457 at commit 91fc944.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class MessageLoop extends Runnable
class NettyRpcEnvFactory extends RpcEnvFactory with Logging
class NettyRpcEndpointRef(@transient conf: SparkConf)
class NettyRpcHandler(

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #33852 has started for   PR 6457 at commit 64f1f04.

Test build #33852 has finished for   PR 6457 at commit 64f1f04.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class MessageLoop extends Runnable
class NettyRpcEnvFactory extends RpcEnvFactory with Logging
class NettyRpcEndpointRef(@transient conf: SparkConf)
class NettyRpcHandler(
case class SortOrder(child: Expression, direction: SortDirection) extends Expression
abstract class BinaryMathExpression(f: (Double, Double) => Double, name: String)

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #33854 has started for   PR 6457 at commit 879ecd5.

Test build #33854 has finished for   PR 6457 at commit 879ecd5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class MessageLoop extends Runnable
class NettyRpcEnvFactory extends RpcEnvFactory with Logging
class NettyRpcEndpointRef(@transient conf: SparkConf)
class NettyRpcHandler(

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34367 has started for   PR 6457 at commit 1735f2d.

Test build #34367 timed out     for PR 6457 at commit 1735f2d     after a configured wait of 175m.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34450 has started for   PR 6457 at commit 019334a.

Test build #34450 has finished for   PR 6457 at commit 019334a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class MessageLoop extends Runnable
class NettyRpcEnvFactory extends RpcEnvFactory with Logging
class NettyRpcEndpointRef(@transient conf: SparkConf)
class NettyRpcHandler(

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #35687 has started for   PR 6457 at commit 24aea28.

Test build #35687 has finished for   PR 6457 at commit 24aea28.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Build triggered.

@zsxwing can you bring this up to date? I'd like to review this this week.

Merged build triggered.

Merged build started.

Test build #38094 has started for   PR 6457 at commit f905c72.

Test build #38094 has finished for   PR 6457 at commit f905c72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #38146 has started for   PR 6457 at commit f905c72.

Merged build triggered.

Merged build started.

Test build #68 has started for   PR 6457 at commit f905c72.

Test build #38146 has finished for   PR 6457 at commit f905c72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #68 has finished for   PR 6457 at commit f905c72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38247 has started for   PR 6457 at commit a4fd9d2.

Test build #38247 has finished for   PR 6457 at commit a4fd9d2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

I definitely need a second read of the code, but I'm a little concerned about the confusing way thread-safe endpoints are handled, and the way quite some code is duplicated in the two Inbox implementations. I think there's a lot of room for improvement there.

@vanzin thanks for reviewing this one. I will try to combine two types of Inboxes into a single one.

Merged build triggered.

Merged build started.

Test build #38356 has started for   PR 6457 at commit 0de34c8.

Test build #38356 has finished for   PR 6457 at commit 0de34c8.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38359 has started for   PR 6457 at commit ac69cb5.

Test build #38359 has finished for   PR 6457 at commit ac69cb5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

@zsxwing will you have time to update this sometime soon? it would be good to get this in early in the 1.6 cycle, so that we can test it thoroughly and iron out any kinks.

@zsxwing will you have time to update this sometime soon? it would be good to get this in early in the 1.6 cycle, so that we can test it thoroughly and iron out any kinks.

I will update it ASAP

Merged build triggered.

Merged build started.

Test build #42430 has started for   PR 6457 at commit 56b5beb.

Test build #42430 timed out     for PR 6457 at commit 56b5beb     after a configured wait of 250m.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42430/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #42481 has started for   PR 6457 at commit 9ae0873.

Test build #42481 has finished for   PR 6457 at commit 9ae0873.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42481/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #42489 has started for   PR 6457 at commit 9ae0873.

Test build #42489 has finished for   PR 6457 at commit 9ae0873.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42489/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #42517 has started for   PR 6457 at commit e8ecab8.

Test build #42517 has finished for   PR 6457 at commit e8ecab8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42517/
Test PASSed.

@vanzin I have updated this PR and implemented a single Inbox. Could you take another look? Thanks.

I'll pick this up tomorrow (stopped at NettyRpcEnv.scala), but I have a question: it seems like whenever NettyRpcEnv is initialized, it will start listening on a port; that's ok for now since akka behaves that way, but isn't it the case that it may not be necessary? If you're using an external shuffle service, there should be no need for executors to listen for incoming connections, right?

If you're using an external shuffle service, there should be no need for executors to listen for incoming connections, right?

Do you mean using RpcEnv in the external shuffle service? The shuffle service cannot use it since it doesn't depend on core now.
However this is a good point. Some module may not need to listen for connections during the initialization. We can add a public start method to RpcEnv after removing Akka.

Merged build triggered.

Merged build started.

Test build #42605 has started for   PR 6457 at commit e130a4c.

Merged build triggered.

Merged build started.

Test build #42606 has started for   PR 6457 at commit c904a9d.

Test build #42605 has finished for   PR 6457 at commit e130a4c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42605/
Test PASSed.

Test build #42606 has finished for   PR 6457 at commit c904a9d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42606/
Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #42620 has started for   PR 6457 at commit c904a9d.

Do you mean using RpcEnv in the external shuffle service?

No, I mean executors. With akka out of the picture, the executor's RpcEnv does not need to listen for incoming connections anymore.

Test build #42620 has finished for   PR 6457 at commit c904a9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42620/
Test PASSed.

No, I mean executors. With akka out of the picture, the executor's RpcEnv does not need to listen for incoming connections anymore. The block manager listens on its own port, so even without an external shuffle service, you shouldn't need this (to correct my previous comment).

Ah, so it means master/driver should run a RpcEnv server, and worker/executor should run a RpcEnv client? Never thought about it. I will investigate it later when removing Akka.

Hi @zsxwing,
There were a couple of things that were bothering me in the code but I couldn't really put my finger on them. Instead of trying to point them out in comments, I played a little with your code and made changes to it.

thread-safety: https://github.com/vanzin/spark/commit/ea62714d32c764957c8bb6fa13fbd46b312f9538

This follows up on my "using AtomicInteger" comments. The synchronization-based code had at least a couple of races in it; my changes avoid using synchronization in Inbox and fix those races too.

too many maps: https://github.com/vanzin/spark/commit/e6674673518ef95cc20c5f0d0f113ce6711b7917

I think this looks a little bit cleaner and is easier to follow than the previous code. There are some changes here, particular to the tests, that should actually be part of the previous commit.
Could you take a look at those and  see what you think? There's some synchronization code in Dispatcher also that I didn't try to remove; it didn't seem necessary to do that at this point, since that would be mostly for performance reasons.

thread-safety: vanzin@ea62714
This follows up on my "using AtomicInteger" comments. The synchronization-based code had at least a couple of races in it; my changes avoid using synchronization in Inbox and fix those races too.

Could you clarify the races? I post a comment about a race condition in your commit.

too many maps: vanzin@e667467
I think this looks a little bit cleaner and is easier to follow than the previous code. There are some changes here, particular to the tests, that should actually be part of the previous commit.

This commit is much simpler. I stole your idea to my latest commit and made some changes.

Merged build triggered.

Merged build started.

Test build #42717 has started for   PR 6457 at commit d540a80.

Test build #42717 has finished for   PR 6457 at commit d540a80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42717/
Test PASSed.

After reading the code again, I'm not sure the thread-safe inbox race that I saw before really exists. But the OnStop race does exist.

Nevermind my last comment; I think I finally got my head around all the checks in the code.

@zsxwing this is looking pretty good. I left some mostly minor comments.

Merged build triggered.

Merged build started.

Test build #42726 has started for   PR 6457 at commit 3262dd1.

@vanzin addressed your comment

Test build #42726 has finished for   PR 6457 at commit 3262dd1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42726/
Test PASSed.

Left some minor feedback. After those are taken care of, this LGTM. @rxin did you mean to also take a look at this?

Will take a look later. But given it is early in the release cycle, I'm also ok with merging it first and do post-hoc reviews & feedback addressing later.

Merged build triggered.

Merged build started.

Test build #42806 has started for   PR 6457 at commit 1c1ec99.

Test build #42806 has finished for   PR 6457 at commit 1c1ec99.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42806/
Test PASSed.

LGTM; the duplicate event issue is very minor and doesn't happen with the default config (where connections per peer is 1), so not super critical to fix now.

Merged build triggered.

Merged build started.

Test build #42872 has started for   PR 6457 at commit 90de095.

Test build #42872 has finished for   PR 6457 at commit 90de095.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42872/
Test PASSed.

@zsxwing did you intend to leave "netty" as the default rpc implementation? Just double checking. Let me know and I'll merge this.

@zsxwing did you intend to leave "netty" as the default rpc implementation? Just double checking. Let me know and I'll merge this.

Yes. Actually it's better to use netty now so as to test it and see if it's stable. We can change it to akka if we find any critical issue before releasing it.

I'm going to merge this and review it post-hoc.

@zsxwing @rxin I saw this test failure on Jenkins master. If this happens frequently, we may need to revert this patch. I created https://issues.apache.org/jira/browse/SPARK-10799 for the issue.
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3596/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.2,label=spark-test/testReport/junit/org.apache.spark.rpc.netty/InboxSuite/post__multiple_threads/
org.apache.spark.rpc.netty.InboxSuite.post: multiple threads

Failing for the past 1 build (Since Failed#3596 )
Took 9 ms.
add description
Error Message

org.apache.spark.rpc.netty.InboxSuite$$anonfun$3$$anon$1@73986812 was not empty
Stacktrace

sbt.ForkMain$ForkError: org.apache.spark.rpc.netty.InboxSuite$$anonfun$3$$anon$1@73986812 was not empty
	at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
	at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)
	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:466)
	at org.apache.spark.rpc.netty.InboxSuite$$anonfun$3.apply$mcV$sp(InboxSuite.scala:94)
	at org.apache.spark.rpc.netty.InboxSuite$$anonfun$3.apply(InboxSuite.scala:66)
	at org.apache.spark.rpc.netty.InboxSuite$$anonfun$3.apply(InboxSuite.scala:66)
	at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
	at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42)
	at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
	at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
	at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
	at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
	at org.scalatest.Suite$class.run(Suite.scala:1424)
	at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
	at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
	at org.scalatest.FunSuite.run(FunSuite.scala:1555)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)
	at sbt.ForkMain$Run$2.call(ForkMain.java:294)
	at sbt.ForkMain$Run$2.call(ForkMain.java:284)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Also this might be related: org.apache.spark.deploy.StandaloneDynamicAllocationSuite
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/3622/HADOOP_PROFILE=hadoop-2.4,label=spark-test/testReport/junit/org.apache.spark.deploy/
org.apache.spark.deploy.StandaloneDynamicAllocationSuite.dynamic allocation default behavior

Failing for the past 1 build (Since Failed#3622 )
Took 0.12 sec.
add description
Error Message

1 did not equal 2
Stacktrace

      org.scalatest.exceptions.TestFailedException: 1 did not equal 2
      at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
      at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)
      at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:466)
      at org.apache.spark.deploy.StandaloneDynamicAllocationSuite$$anonfun$1.apply$mcV$sp(StandaloneDynamicAllocationSuite.scala:78)
      at org.apache.spark.deploy.StandaloneDynamicAllocationSuite$$anonfun$1.apply(StandaloneDynamicAllocationSuite.scala:73)
      at org.apache.spark.deploy.StandaloneDynamicAllocationSuite$$anonfun$1.apply(StandaloneDynamicAllocationSuite.scala:73)
      at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
      at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
      at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42)
      at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
      at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
      at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
      at org.apache.spark.deploy.StandaloneDynamicAllocationSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(StandaloneDynamicAllocationSuite.scala:33)
      at org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:255)
      at org.apache.spark.deploy.StandaloneDynamicAllocationSuite.runTest(StandaloneDynamicAllocationSuite.scala:33)
      at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
      at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
      at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
      at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
      at scala.collection.immutable.List.foreach(List.scala:318)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
      at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
      at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
      at org.scalatest.Suite$class.run(Suite.scala:1424)
      at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
      at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
      at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
      at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
      at org.apache.spark.deploy.StandaloneDynamicAllocationSuite.org$scalatest$BeforeAndAfterAll$$super$run(StandaloneDynamicAllocationSuite.scala:33)
      at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
      at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
      at org.apache.spark.deploy.StandaloneDynamicAllocationSuite.run(StandaloneDynamicAllocationSuite.scala:33)
      at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
      at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
      at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
      at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
      at org.scalatest.Suite$class.runNestedSuites(Suite.scala:1526)
      at org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:29)
      at org.scalatest.Suite$class.run(Suite.scala:1421)
      at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:29)
      at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
      at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
      at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
      at scala.collection.immutable.List.foreach(List.scala:318)
      at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
      at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
      at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
      at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
      at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
      at org.scalatest.tools.Runner$.main(Runner.scala:860)
      at org.scalatest.tools.Runner.main(Runner.scala)

@zsxwing @rxin I saw this test failure on Jenkins master. If this happens frequently, we may need to revert this patch. I created https://issues.apache.org/jira/browse/SPARK-10799 for the issue.

I have already sent #8905 to fix this test

Actually, the second is worse, which broke most recent master builds.

@zsxwing I reverted this patch due to master build failures. Please fix the tests and re-submit a PR. Thanks!
https://amplab.cs.berkeley.edu/jenkins/view/Spark QA Test (Dashboard)/job/Spark-Master-Maven-pre-YARN/4232/HADOOP_VERSION=2.0.0-mr1-cdh4.1.2,label=spark-test/testReport/junit/org.apache.spark.deploy/StandaloneDynamicAllocationSuite/dynamic_allocation_default_behavior/history/

@mengxr I saw some race conditions in StandaloneDynamicAllocationSuite after reviewing it. This patch just exposed them. I'm thinking about how to fix them. It's better to remerge this patch after fixing StandaloneDynamicAllocationSuite to avoid breaking the master branch.

Just realized we don't need to revert it. The new RPC can be disabled...


very curious about how good about the accuracy without fine-tuning pruned network

I'm very curious about how good about the accuracy without fine-tuning the pruned network. It  may be not as good as directly trained network. If not the best prediction from Pruning network. Maybe you can not make sure the searched results are the best

Hi, I think there is no guarantee in 'best' even with fine-tuning. With meta-learning, we find even without fine-tuning, it is almost very satisfied for predicting good-preforming pruned networks. Fine-tuning is too time costly.


Port #6503 and #6504 to release

Codecov Report

:exclamation: No coverage uploaded for pull request base (release-2021.07@b60e3a1). Click here to learn what that means.
The diff coverage is n/a.

@@                Coverage Diff                @@
##             release-2021.07   #6507   +/-   ##
=================================================
  Coverage                   ?     69%
=================================================
  Files                      ?     410
  Lines                      ?   28261
  Branches                   ?    4191
=================================================
  Hits                       ?   19762
  Misses                     ?    6824
  Partials                   ?    1675


Integrating master direct pacakge [BREAKING CHANGES]

Direct package internal types only (Only one single public type)
All REST contacts *Settigns Types are now open-sourced
o	Contracts code is simple and clean
o	No JsonSerializable or Resource/CosmosResource
o	You will see mix of older and newer contracts through-out (DocumentClient doesn’t use the OSS contracts)
o	Contract is limited to public surface area only
Json navigator code is open-sourced
o	No more references in master
o	All Json tests moved to unit-tests project
Some code moved back into direct package
o	Region and proximity util’s
o	Direct package only tests
o	StoreClientFactory

All gates passed. One optional build-def is still running. But it never succeeded before.
So for now merging it.


DockerComposeFixture: Support custom container names

The YugaByteFixture needs to give their containers custom names which means that these won't be prefixed by the compose application name.  The underlying DockerComposeFixture currently assumes that all containers created by the compose file will be named with the application prefix.  This means that DockerComposeFixture won't be able to remove these containers when necessary.
The fix is to add optional parameters to the DockerComposeFixture start methods where the custom container names csn be specified.

DONE for neonLIBRARY-v2.4.0


fix(rear-panel): incorrect limiting of msg size

Missed a test failure that had a real bug! The raw iterator manipulation does not correctly handle the case where we're out of space in our write buffer.

Also I'm just gonna merge this when it passes


DNSoverQUIC

Implementation of DoQ for Unbound, DNS over QUIC transport. This implements doq for downstream, clients that query unbound server, RFC9250.
Compile this with the ngtcp2 library. And with openssl+quic. Like this:
git clone --depth 1 -b OpenSSL_1_1_1o+quic https://github.com/quictls/openssl openssl+quic
cd openssl+quic
./config enable-tls1_3 no-shared threads --prefix=/path/to/openssl+quic_install
make
make install
cd ..
git clone https://github.com/ngtcp2/ngtcp2 ngtcp2
cd ngtcp2
autoreconf -i
./configure PKG_CONFIG_PATH=$PWD/../openssl+quic_install/lib/pkgconfig LDFLAGS="-Wl,-rpath,$PWD/../openssl+quic_install/lib" --prefix=/path/to/ngtcp2_install
make
make install
cd ..
git clone -b dnsoverquic https://github.com/NLnetLabs/unbound unbound_dnsoverquic
cd unbound_dnsoverquic
./configure <other flags> --with-ssl=/path/to/openssl+quic_install --with-libngtcp2=/path/to/ngtcp2_install LDFLAGS="-Wl,-rpath -Wl,/path/to/ngtcp2_install/lib" --prefix=/path/to/unbound_install
make

With the compile, it can be turned on. This is governed by the config option in unbound.conf, quic-port: 853. When an interface is on that port number, the UDP socket receives DoQ queries.
With this unbound.conf:
interface: <IP_ADDRESS>@2853
quic-port: 2853

Then unbound serves quic queries to localhost on the 2853 port number. Also other interfaces work, like ::1@2853. Unbound can be started attached to the console for debug, with ./unbound -d -c theconfig.conf. With -dd it prints logs to the terminal as well. Ctrl-C can exit, or send a term signal.
With make doqclient the test tool can be created to send a query. Send a query with ./doqclient -s <IP_ADDRESS> -p 2853 www.example.com A IN. With -v it prints more diagnostics, also unbound logs more diagnostics, also from the internals of libngtcp2, when verbosity is 4 or more. An example of output from doqclient is:
./doqclient -s <IP_ADDRESS> -p 2853 www.example.com A IN
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
www.example.com.	IN	A

;; ANSWER SECTION:
www.example.com.	86400	IN	A	<IP_ADDRESS>
www.example.com.	86400	IN	RRSIG	A 8 3 86400<PHONE_NUMBER>4414<PHONE_NUMBER>1500 17695 example.com. DcGGeVQlXf2W91/d5SReEjVlwgJ1W67axWGLBQ6hNgsn5s0gT1pQdOE96YfDb3VP3UKUMyFwR9O7SWs7Cfue3RAs0j3S1b8rIS1CNUt4SGMAjKut3krBS/8nfpQsb8WpCII3Qv1VsCyGuIQmGCQ8wxaFIQ931uyQyloR+oVLK3M= ;{id = 17695}

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:
; EDNS: version: 0; flags: do ; udp: 1232
;; MSG SIZE  rcvd: 231
;; SERVER: <IP_ADDRESS> 2853

It is possible to have the TCP port on the same interface as DoQ server DoT or DoH, dnsovertls or dnsoverhttp, or also serve over TCP.
The resource consumption can be configured with quic-size: 8m. More queries are turned away. The number of quic queries is output in num.query.quic in the statistics. The mem.quic statistic outputs memory used.

I followed your tutorial and reported an error in ./configure of unbound_dnsoverquic. The error message is as follows. How can I solve this problem?
configure: error: No QUIC support detected in OpenSSL. Need OpenSSL version with QUIC support to enable DNS over QUIC with libngtcp2

The error message is that the openssl library found by configure does not have quic support, the one in https://github.com/quictls/openssl has it.

The error message is that the openssl library found by configure does not have quic support, the one in https://github.com/quictls/openssl has it.

I use this openssl. openssl-3.0.10+quic

Which version of ngtcp2 did you use during the implementation? I'm currently compiling unbound with the latest version of ngtcp2 and reporting some errors like this:
In file included from services/mesh.c:65:
./services/listen_dnsport.h:587:39: error: field 'last_error' has incomplete type
587 |  struct ngtcp2_connection_close_error last_error;
|                                       ^~~~~~~~~~

Which version of ngtcp2 did you use during the implementation? I'm currently compiling unbound with the latest version of ngtcp2 and reporting some errors like this: In file included from services/mesh.c:65: ./services/listen_dnsport.h:587:39: error: field 'last_error' has incomplete type 587 | struct ngtcp2_connection_close_error last_error; | ^~~~~~~~~~

Now configure will run without an error, but make will run without an error.
In file included from services/mesh.c:65:
./services/listen_dnsport.h:587:39: error: field 'last_error' has incomplete type
587 | struct ngtcp2_connection_close_error last_error;
| ^~~~~~~~~~

There are still some problems compiling with your modified version. The specific error message is as follows:
checking for ngtcp2/ngtcp2_crypto_openssl.h... no
checking for ngtcp2/ngtcp2_crypto_quictls.h... yes
checking whether ngtcp2_conn_server_new is declared... yes
checking whether ngtcp2_crypto_encrypt_cb is declared... yes
checking for ngtcp2_crypto_encrypt_cb in -lngtcp2_crypto_openssl... no
checking for ngtcp2_crypto_encrypt_cb in -lngtcp2_crypto_quictls... yes
checking for ngtcp2_crypto_encrypt_cb... yes
checking for ngtcp2_ccerr_default... yes
checking for SSL_is_quic... no
configure: error: No QUIC support detected in OpenSSL. Need OpenSSL version with QUIC support to enable DNS over QUIC with libngtcp2.

The fixes updates the code branch from a libngtcp2 version of slightly after 0.5 from its code repository, to a version of slightly after 0.18 from its code repository. The unit test works with the new version. That should fix the initial configure stage error and the incomplete type errors.
Also this compiles with openssl-3.0.10+quic and this works here. The SSL_is_quic test does not fail here, something must be wrong; perhaps try again with the extra code fixes, or look in the output of config.log - the part that is pertinent for the SSL_is_quic test, search in there for it, and the compile command line of the test and the error output of the compiler could give more information.

How do I get a certificate that can be used for testing? I try to generate a certificate with openssl in OpenSSL-3.0.10 +quic, but I always get the following error:
[root@localhost unbound_dnsoverquic_my]# ./unbound -dd
[1692538673] unbound[36447:0] info: is quic called, ::1 2853 2853
[1692538673] unbound[36447:0] info: is quic called, <IP_ADDRESS> 2853 2853
[1692538673] unbound[36447:0] notice: init module 0: validator
[1692538673] unbound[36447:0] notice: init module 1: iterator
[1692538673] unbound[36447:0] error: doq: error for cert file: /root/key/quic_certificate.pem
[1692538673] unbound[36447:0] error: doq: error in SSL_CTX_use_certificate_chain_file crypto error:80000002:system library::No such file or directory
[1692538673] unbound[36447:0] error: and additionally crypto error:10080002:BIO routines::system lib
[1692538673] unbound[36447:0] error: and additionally crypto error:0A080002:SSL routines::system lib
[1692538673] unbound[36447:0] error: doq: error for cert file: /root/key/quic_certificate.pem
[1692538673] unbound[36447:0] error: doq: error in SSL_CTX_use_certificate_chain_file crypto error:80000002:system library::No such file or directory
[1692538673] unbound[36447:0] error: and additionally crypto error:10080002:BIO routines::system lib
[1692538673] unbound[36447:0] error: and additionally crypto error:0A080002:SSL routines::system lib
[1692538673] unbound[36447:0] info: start of service (unbound 1.17.2).
^C[1692538679] unbound[36447:0] info: service stopped (unbound 1.17.2).
[1692538679] unbound[36447:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
[1692538679] unbound[36447:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0

The log output contains a debug message, the commit removes it. It also explains that the file does not exist.
Perhaps chroot: somedir is enabled, it is enabled by default, and this modifies the paths unbound uses. Put the file inside the chroot, perhaps.

The log output contains a debug message, the commit removes it. It also explains that the file does not exist.
Perhaps chroot: somedir is enabled, it is enabled by default, and this modifies the paths unbound uses. Put the file inside the chroot, perhaps.

Your inference is correct. I disabled chroot by modifying the conf file and now it works. Thks :)

ssl.h: No such file or directory
95 | #include <ngtcp2/ngtcp2_crypto_openssl.h>
|          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:323: listen_dnsport.lo] Error 1
This is because the latest version removes openssl specific libs, I think git checkout 1d0cad6697992cf71661e69a6cccb347f63e4aaa should work for ngtcp2.

With ngtcp2-0.19.1, available from the releases and tags, together with openssl+quic, the code works with this API. That uses nghttp3-0.15.0 for the examples in ngtcp2, by the way. Since the configure script detects a number of changes, it could also work for some other, intermediate, versions.

@gthess  any update on this?

This is pending review from my part to then go into the next feature release. Due to other developments, I will refocus on this on January.

Hello,
I'm in a similar situation to venus1234, where I get an error that QUIC isn't supported.
configure: error: No QUIC support detected in OpenSSL. Need OpenSSL version with QUIC support to enable DNS over QUIC with libngtcp2.
Below is the log related to SSL_is_quic.
configure:21910: checking for SSL_is_quic configure:21910: gcc -o conftest  -g -O2 -D_GNU_SOURCE -flto   -I/usr/local/include -I/path/to/ngtcp2_install/include -Wl,-rpath -Wl,/path/to/ngtcp2_install/lib -L/usr/local/lib -L/path/to/ngtcp2_install/lib conftest.c  -lcrypto -lngtcp2 >&5 /usr/bin/ld: /tmp/cc6EkE5f.ltrans0.ltrans.o: in functionmain':
/home/nekoy/unbound_dnsoverquic/conftest.c:193: undefined reference to SSL_is_quic' collect2: error: ld returned 1 exit status configure:21910: $? = 1 configure: failed program was: | /* confdefs.h */ | #define PACKAGE_NAME "unbound" | #define PACKAGE_TARNAME "unbound" | #define PACKAGE_VERSION "1.19.4" | #define PACKAGE_STRING "unbound 1.19.4" | #define PACKAGE_BUGREPORT<EMAIL_ADDRESS>or https://github.com/NLnetLabs/unbound/issues"     | #define PACKAGE_URL "" | #define CONFCMDLINE "--with-libngtcp2=/path/to/ngtcp2_install LDFLAGS=-Wl,-rpath -Wl,/path/to/ngtcp2_install/lib --prefix=/path/to/unbound_install" | #define HAVE_STDIO_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 ... | #if defined __stub_SSL_is_quic || defined __stub___SSL_is_quic | choke me | #endif | | int | main (void) | { | return SSL_is_quic (); |   ; |   return 0; | } configure:21910: result: no configure:21917: error: No QUIC support detected in OpenSSL. Need OpenSSL version with QUIC support to enable DNS over QUIC with libngtcp2. ... ac_cv_func_SSL_is_quic=no

The check indicates that the openssl+quic version is not detected. That has the function that is looked for. If the openssl+quic version is in use, the error makes it seem like '--disable-flto' could fix the issue, if the lto optimization is causing it.
So, using the system default openssl version is not likely to work, as that does not have the quic functionality.

I tried the --disable-flto option immediately, but got the same result.
Also, I found out that the default OpenSSL doesn't support QUIC, so I installed the QUIC-compatible version yesterday, but it didn't improve anything.
$ openssl version
OpenSSL 3.2.0 23 Nov 2023 (Library: OpenSSL 3.2.0 23 Nov 2023)
below is the log again.
configure:21910: checking for SSL_is_quic
configure:21910: gcc -o conftest  -g -O2 -D_GNU_SOURCE   -I/path/to/openssl+quic_install/include -I/path/to/ngtcp2_install/include -Wl,-rpath -Wl,/path/to/ngtcp2_install/lib -L/path/to/openssl+quic_install/lib -L/path/to/ngtcp2_install/lib conftest.c  -lcrypto -lngtcp2 >&5
/usr/bin/ld: /tmp/cclsRvi8.o: in function main': /home/nekoy/unbound_dnsoverquic/conftest.c:188: undefined reference to SSL_is_quic'
collect2: error: ld returned 1 exit status

Is that the openssl that is just a version increase, where openssl has more quic support. But what the code needs is the openssl version from the branch of code, linked at the top post, that has the quic functions that are used by libngtcp2. That prints a version line like OpenSSL <version>+quic. If this is an install of that line of versions, then there is a failure in the configure detection logic possibly, but I think, since the version line is wrong, it is likely the openssl version.

I have OpenSSL version 3.3.2 which natively supports QUIC.
Can I use this version? I have not yet swapped over to the forked version hoping I can use the native OpenSSL.
But the quic-port command isn't recognized

I believe that only implements the client part, and the branch implements server code, with the other library.

No, I do not know. The upstream part is present on plans.

i have a few questions. Does this installation before or after i setup an unbound resolver? because i 've done everything above but i cannot find quic -port in unbound.conf. ( I have installed unbound resolver before starting this)

The instructions above are for compilation, you would need to compile before running the unbound server. That means using the version of the software that is compiled, and not a version that is installed from a package system. Perhaps ports are firewalled on the computer, and this is dropping all traffic, and thus timeouts, instead of a compilation problem.

The instructions above are for compilation, you would need to compile before running the unbound server. That means using the version of the software that is compiled, and not a version that is installed from a package system. Perhaps ports are firewalled on the computer, and this is dropping all traffic, and thus timeouts, instead of a compilation problem.

I dont think that is the case. My firewall is down and my unbound is built from source. I did restart my unbound server but it did not work (it still can perform normal query with dig but cannot perform DoQ one with /doqclient . Maybe because i built the server first then the instructions. They just dont work together.
unbound-checkconf even gives me error for putting "quic-port" in unbound.conf. So i think it is a compilation problem.

Yeah the build of the server needs the result from those instructions. So the instructions are earlier, but I cannot tell what the compilation problem is, apart from that in the unbound that is attempted the doq does not seem to be present.

I build the server again. This time when i try to start the server, it fails and log looks like this:
[1727840988] unbound[66391:0] error: doq server socket create: no tls-service-key
[1727840988] unbound[66391:0] error: could not create doq comm_point
[1727840988] unbound[66391:0] error: can't create commpoint
[1727840988] unbound[66391:0] error: could not create listening sockets
[1727840988] unbound[66391:0] fatal error: Could not initialize main thread

The dnsoverquic needs tls-server-key and tls-server-pem, just like for TLS service, with the certificate for the server. This is the server public and private key files. But it looks like it is the correct code that starts from the failure.


NuGet Package Not Working on Linux-x64

TLDR: Seems to be some kind of issue with the NuGet package, at least on Arch Linux x64. Building from repo works fine.
I wanted to add physics to a new 3d project I started and stumbled across this project and cloned the repository to look at the Demos. They seemed to work well, even on my dated laptop, so I decided to go ahead and add the Jitter2 NuGet package to my project. I copy-pasted some of the code from the Hello World example in the docs into my project and ran it, and not a single frame was rendered, program completely locked up. Pausing the debugger did not work.
After some investigation, the hangup seemed to be somewhere in world.Step(1f/100f,false);, but I figured it's probably something I did wrong, so I took a step back and followed the Hello World tutorial step-by-step. Same thing.
Checked my .NET SDK version and it seems the arch repo package is out of date, so I uninstalled it and installed the latest version of the binaries from Microsoft, and tried again. Still no luck.
Went back and tried the Demos from the cloned repo, and they still worked. Then I decided to just try copying the Jitter2 project into my solution and ditched the NuGet package in favor of a project reference, and lo and behold: everything just started working.
My system:
tomatoes@tomato-can[~]$ dotnet --version
8.0.401
tomatoes@tomato-can[~]$ uname -a
Linux tomato-can 6.10.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 04 Sep 2024 15:16:37 +0000 x86_64 GNU/Linux

I'm not too sure how to debug a NuGet package when the repo code works just fine, so apologies for not being able to offer up more detail, but I felt I ought to report it ASAP. I should have time to try replicating this issue on a few other machines come the weekend.

Thank you for this qualified report. I'm also using Linux, but can not reproduce your issue.

After some investigation, the hangup seemed to be somewhere in world.Step(1f/100f,false);

We already noticed some strange hangups (https://github.com/notgiven688/jitterphysics2/issues/115) for which the root cause has not been found yet. I assume it has something to do with threading. However in your example (world.Step(dt, false) ) you have multi-threading disabled.

I should have time to try replicating this issue on a few other machines come the weekend.

Looking forward to your analysis.

Weekend started a bit early so I decided to do a bit more testing and a bit of poking around before trying it out elsewhere... I do think it is related to the issue you mentioned -
With the NuGet package:
I tried each build configuration [of the test project]: hangs regardless if Debug or Release
Debugger/No Debugger: worked perfectly with no debugger, hangs with debugger
(continued using debugger after this)
true / false in World.Step parameters: hangs either way
So it certainly seems like there's something fishy going on with the debugger.
I figured out how to configure VS Code to load symbols and sources for the package and was able to step through the call to world.Step(1f/100f, false);  I deliberately left multi-threading off for debugging this one.
It seems on World.Step.cs:729 (compiler must have optimized this because debugger insists it is on L728), ThreadPool.Instance is accessed for the first time (with MT off) and therein calls the constructor, which then calls ChangeThreadCount.
Inside ChangeThreadCount is a for loop that starts new threads, and I determined that the program was consistently hanging for me at ThreadPool.cs:143 / threads[i].Start() when i held the value of 2. Strangely enough the program would also sometimes hang at other points stepping through parts of the ThreadPool class, but this was far rarer, whereas I could count on the hang occurring at L143 when i == 2.
Stepping in further, the closest I could get to a final location for the problem was in the dotnet code itself, inside Thread.CoreCLR.cs:90 at a call to StartInternal.
Here's what's going on at this point:

Going any further than this resulted in the infinite hangup.
I also tried turning off my CPU's hyper-threading (from 8 logical processors to 4), and the program started working again, but in this case, i never got up to 2 inside the aforementioned loop.
Either way, it seems like the thread pool is spinning up threads regardless of whether they're desired/needed or not, so perhaps addressing that would reduce the inconvenience of the underlying bug.
But ultimately, your comments in the other thread that it could be a problem with the debugger itself seem to check out so far. Wish I knew more about how StartInternal worked across different platforms.
I'm still gonna try a couple other computers with different configurations of Linux to see if there's any other reasons why it seems so hard to reproduce. Still unable to repro this with the ProjectReferenced version of my test project still.

Thank you very much @tin-can-tomatoes
Here is my minimal reproducer:
Console.WriteLine("Hello, World!");

SpinUpThreads();

Console.WriteLine("Bye, World!");

void SpinUpThreads()
{
    void ThreadProc()
    {
        while (true)
        {
            // Thread.Yield();
        }
    }

    var threads = new Thread[4];

    Console.WriteLine("Initializing.");

    var initWaitHandle = new AutoResetEvent(false);

    for (int i = 0; i < 4; i++)
    {
        threads[i] = new Thread(() =>
            {
                initWaitHandle.Set();
                ThreadProc();
            });

        threads[i].IsBackground = true;
        threads[i].Start();
        initWaitHandle.WaitOne();
    }
}

For me (under Linux) I get with the debugger:
Hello, World!
Initializing.
{Hangs}
The code works if I comment in "Thread.Yield()". However this does only seem to "hide" the bug, as doing so in Jitter itself makes the engine hang at a later point.

Hmm I played with your repro code and see the same thing with Thread.Yield.
Back in the Jitter2 repo I figured out that
#if DEBUG
    public const float ThreadsPerProcessor = 0.0f;
#else
    public const float ThreadsPerProcessor = 0.9f;
#endif

was why I wasn't having the issue when compiling the library myself, so I changed it to always be 0.9 and now I can reproduce the hangup in the Jitter2 sln itself.
I played around with adding Thread.Yield()s and Thread.Sleep(...)s into ThreadProc and it seemed to alleviate the problem for me - I didn't experience any other hangups at later points.
Here's what I ended up with:
    private void ThreadProc()
    {
        while (running)
        {
            if (taskQueue.TryDequeue(out ITask? result))
            {
                result.Perform();
                Interlocked.Decrement(ref tasksLeft);
            }
            else
            {
                Thread.Yield();
                mainResetEvent.Wait();
            }
        }
    }

Having Thread.Yield() follow the mainResetEvent.Wait() statement also seems to work. Having Thread.Yield outside of the if statement also works, but appears to be unnecessary when the if statement's condition is true.
I'm figuring the overall issue is the debugger struggling to get/do something with the blocked or infinite-looping thread, and having a Yield or Sleep in there allows the debugger a chance to do what it needs to. Perhaps this is the fix if we can make sure it's not hanging elsewhere like you experienced.
Or if that fix still proves lacking, perhaps a good workaround for the time being would be to allow the consumer of the library to set the initial thread count of the ThreadPool before the instance is created - either by changing ThreadsPerProcessor to a modifiable static property instead of a const, or by having a way to create the instance with an explicit number of threads.

That took a while...
but https://github.com/dotnet/runtime/issues/33561 came to the rescue!
It turns out there's an actual bug in the .NET runtime. From what I can see, the .NET 9 preview includes the commit, which seems to fix our issues.

That is good news!  Looks like .NET 9 will land in mid-November, so I'm good with using workarounds until then.

@tin-can-tomatoes: We should be safe with: https://github.com/notgiven688/jitterphysics2/commit/1c2aca9cfe91157b8ac97d2911ee3c01de189cd3 (?)

I'm at work so I won't be able to test properly (with the nuget artifact) until tonight, but a quick clone of the jitterphysics2 repo seems to debug just fine on my work laptop (also linux x64).

If you're working on putting out a new NuGet release, I think it may also be worth looking into changing World.Step to avoid calling ThreadPool.Instance altogether when multithreading is set to false.
My attempt to do this is the following changes:
Refactored DetectCollisions (World.Step.cs) to:

    private void DetectCollisions(bool multiThread)
    {
        const int taskThreshold = 1024;
        if(multiThread){
            int numTasks = DynamicTree.PotentialPairs.Slots.Length / taskThreshold + 1;
            numTasks = Math.Min(numTasks, ThreadPool.Instance.ThreadCount);

            if (numTasks > 1)
            {
                Parallel.ForBatch(0, DynamicTree.PotentialPairs.Slots.Length,
                    ThreadPool.Instance.ThreadCount, detectCollisions);
                return;
            }
        }

        DetectCollisionsCallback(new Parallel.Batch(0, DynamicTree.PotentialPairs.Slots.Length));

    }

Added a multiThread parameter to CheckBagCount in DynamicTree.cs:
    private void CheckBagCount(bool multiThread)
    {
        int numThreads = multiThread? ThreadPool.Instance.ThreadCount : 1;
        if (lists.Length != numThreads)
        {
            lists = new SlimBag<IDynamicTreeProxy>[numThreads];
            for (int i = 0; i < numThreads; i++)
            {
                lists[i] = new SlimBag<IDynamicTreeProxy>();
            }
        }
    }

And lastly, changed the last part of World.Step() function to:
        // Signal the thread pool that threads can go into a wait state. If threadModel is set to
        // aggressive this will not happen. Also make sure that a switch from (aggressive, multiThreaded)
        // to (aggressive, sequential) triggers a signalReset here.
        if (ThreadModel == ThreadModelType.Regular && multiThread /* || !multiThread */)
        {
            ThreadPool.Instance.SignalReset();
        }

I'm pretty sure the change I made to DetectCollisions doesn't cause any side effects - but for the other two changes, I'm wary that they may have implications I haven't thought of yet.

notgiven688 I tested the prerelease nuget package on the BoxDrop demo and my own project, and it works in the debugger without issue.

@tin-can-tomatoes https://github.com/notgiven688/jitterphysics2/pull/182
This includes your suggestions to not initialize the ThreadPool ever.

Fixed in Jitter 2.4.4.


Unique users

This is more a question than an issue.... We've been using this plugin in our app and have been running an external beta test. Google analytics is reporting 170 users, but we know that we only have 30 beta testers. How does the plugin / google identify unique users? Is there something in the use that we should be doing to avoid app sessions counting as new users?

Hi @tomelsner, what are you looking at on the GA site? Sessions or users?
This module doesn't really do anything special wrt unique users, so that is up to the native GA library. But what you could do is use setUser and provide a unique ID if you want to make sure users are unique: https://developers.google.com/analytics/devguides/collection/ios/v3/user-id


Upload to static file in flask project

def create():
    if request.method == 'POST':
            file = request.files["video"]
            stream = ffmpeg.input(file)
            stream = ffmpeg.hflip(stream)
            stream = ffmpeg.output(stream, app.root_path + '/' + app.config['UPLOAD_FOLDER'] + '/videos/dd.mp4')
            ffmpeg.run(stream)

-filter_complex: No such file or directory (error from ffmpeg)

Any solution for my problem? I try change directory and check directory before pass a path but error still same.

I don't know exactly how Flask deals with POSTed files but I think it gives you a file-like object you can read from.
If so, you should call ffmpeg manually (i.e. p = subprocess.Popen(stream.compile(), stdin=subprocess.PIPE)), use stdin as input in ffmpeg (use pipe:1 as the file name, see this), then write to the pipe with a loop, i.e. (code not tested)
CHUNK_SIZE = 1024
data = file.read(CHUNK_SIZE)
while data:
    p.stdin.write(data)
    data = file.read(CHUNK_SIZE)

Check Flask's documentation to see how to stream POST data like that.

Full demo:
import json
import logging
import subprocess

import ffmpeg
from flask import Flask, request

app = Flask(__name__)

def save_audio(file):
    f_path = 'upload_file.wav'
    stream = ffmpeg.input('pipe:0')
    stream = ffmpeg.output(stream, f_path, acodec='pcm_s16le', ac=1, ar='16k')
    # TODO, prompt and log when overwrite
    stream = ffmpeg.overwrite_output(stream)
    p = subprocess.Popen(stream.compile(), stdin=subprocess.PIPE)
    CHUNK_SIZE = 1024
    data = file.read(CHUNK_SIZE)
    while data:
        p.stdin.write(data)
        data = file.read(CHUNK_SIZE)

@app.route("/", methods=['POST', 'GET'])
def save_and_verification_audio():
    if request.method == 'GET':
        return '''
        <!doctype html>
        <title>Upload new File</title>

Upload new File

        <form method=post enctype=multipart/form-data>
          <input type=file name=file>
          <input type=text name=id value="bala">
          <input type=submit value=Upload>
        </form>
        '''

    task_id = request.form['id']
    # get file
    file = request.files['file']
    filename = file.filename
    save_audio(file)
    logging.info('save file: {}'.format(json.dumps({'filename': filename, 'task_id': task_id}, ensure_ascii=False)))
    return 'ok'

if __name__ == '__main__':
    app.run(debug=True)

@Honghe I tried this and i'm getting the error that 'upload_file.wav' doesn't exist. Any help/advice?

@krrishdholakia Do you have the write permission of the target directory?

where do i set that ? i'm running this on a heroku instance

@Honghe here's the error i got from the subprocess call:
ValueError: write to closed file
i think it's referencing the 'upload_file.wav', but i'm curious how it can be closed if it's never been written to ?

@Honghe Did some further digging,
here's what i'm getting:
pipe:0: Invalid data found when processing input


feat(batch): reduce boilerplate with process_partial_response

Issue number: #2083
Summary
New process_partial_response and async_process_partial_response higher level functions.
Changes

Please provide a summary of what's being changed

Add two new higher-order functions to reduce boilerplate and make it less error-prone for new customers using Batch, where the additional return often cause confusion.
This addition has the added benefit of reducing decorators therefore call stacks too (more memory/CPU efficient).
User experience

Please share what the user experience looks like before and after this change

Before

After

Async

Checklist
If your change doesn't seem to apply, please leave them unchecked.

[x] Meet tenets criteria
[x] I have performed a self-review of this change
[x] Changes have been tested
[ ] Changes are documented
[x] PR title follows conventional commit semantics
[x] Update examples to use new higher-level function first
[ ] Add FAQ section on when to choose this vs batch_processor and async_batch_processor, and uncertain decision about future deprecation for the more verbose DX
[x] Review Custom Batch Processor and other areas to favour new higher-level function

[x] Replace all code snippets using batch_processor with process_partial_response
[x] Replace all code snippets using async_batch_processor with async_process_partial_response
[x] Fix Pydantic examples aren't working
[ ] Review each processor intro for the no longer necessary .response()
[ ] Review any call out to batch_processor and async_batch_processor
[ ] Add callout banner for Streams on checkpoint behaviour
[ ] Review this

You will not have access to the processed messages within the Lambda Handler; use context manager for that.

Is this a breaking change?
RFC issue number:
Checklist:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

Converting to Draft as I still need to work on a doc revamp ;)
PS: check this out @BakasuraRCE, I'm sure you'll enjoy the new slim DX.

Codecov Report
Patch coverage: 100.00% and project coverage change: +0.01 :tada:

Comparison is base (ec1b346) 97.45% compared to head (93fa497) 97.46%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2090      +/-   ##
===========================================
+ Coverage    97.45%   97.46%   +0.01%
===========================================
  Files          146      147       +1
  Lines         6748     6781      +33
  Branches       477      478       +1
===========================================
+ Hits          6576     6609      +33
  Misses         136      136
  Partials        36       36

Impacted Files
Coverage Δ

aws_lambda_powertools/utilities/batch/__init__.py
100.00% <100.00%> (ø)

aws_lambda_powertools/utilities/batch/base.py
94.70% <100.00%> (-0.42%)
:arrow_down:

...ws_lambda_powertools/utilities/batch/decorators.py
100.00% <100.00%> (ø)

aws_lambda_powertools/utilities/batch/types.py
81.25% <100.00%> (+8.52%)
:arrow_up:

...wertools/utilities/idempotency/persistence/base.py
99.39% <100.00%> (ø)

...ools/utilities/idempotency/persistence/dynamodb.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Notes for maintainers:
Additional tech debt paid

[x] Fixed syntax highlighting for all code snippets
[x] Fixed incorrect code snippets for Pydantic integration
[x] Moved code snippets using decorators as-is under examples (still needs to be refactored as our docs theme)
[x] Renamed As a decorator to As a decorator (legacy) to keep customers informed along with a FAQ -- whichever way they find batch_processor they'll know the reason.

spoke to @leandrodamascena who's on public holiday today - approved to go.


[full node] Fix memory leak at full node follower

We forgot to remove entries for digests we have processed. There are still possibilities for clean ups to be missed, but this 1 line fixes the common case.

Folks, are we good to land this? This is a very approved PR :)

@mystenmark since you did the clean up I will let you the honours of landing too.


Timesheet - 01/09/2021 - Chrisha Canete

Task 1
Marketing Analytics
Proof of Work

https://www.canva.com/design/DAEorFfl0TQ/share/preview?token=3kvINyKk1C8pMqjzTytY-w&role=EDITOR&utm_content=DAEorFfl0TQ&utm_campaign=designshare&utm_medium=link&utm_source=sharebutton

Time Spent
3.5 hours

Total Time Spent
3.5 hours
Account Number
7053f27870d388b0bca4149fe72765172c3e6d1b4ffc7cd0149a296cbefe8f57

@Chrisha1212, is this ready?


[C-2315] Filter by channel in dashboard

Add a channel filter to the dashboard page

Only show filter if any platform is selected or there is only one platform connected (in this scenario we don't show the platform filter)
Only show channel filter if there are channels for the selected platform

Channel filter should follow the same UI as in other pages. The only difference is that channels don't need to be grouped by platform, since they will only belong to a platform

C-2315

@joanagmaia does this affect only activities?

Hi @joanagmaia please assign this issue to me and @Gracelyn-Mystica

Hi @joanagmaia please assign this issue to me and @Gracelyn-Mystica

@YaswanthGotham Sure! This will require some backend as well because for members and organizations, you will need to support channel filter in the API


Streets in a OSM-Relation are not displayed

OSM-Street's that are member of a Relation wont get rendered in the Map.
Example:
Wyssgasse

https://www.openstreetmap.org/way/5944788
https://zurich.equalstreetnames.eu/de/index.html#18.41/47.373802/8.525517

Stauffacherstrasse

https://www.openstreetmap.org/way/11917566
https://zurich.equalstreetnames.eu/de/index.html#18.1/47.378372/8.520375
-> The northwestern Part of this street is not in a Relation and gets rendered on the map

Strange, for Brussels most of the streets are also relations with type=associatedStreet and it works just fine.
I'll need to investigate.
Can you check that your Overpass queries extract correctly those streets ?

The Overpass query works. But somehow most of the geometries in relations.geojson do not have there wikidata tag anymore. There is also Hallwylstrasse in Bern with the same issue.
I tried to figure out how to run your processing toolchain, but I didn't succeed.

Thanks for your feedback @CaptainInler !
I'll have a look when I have some free time.

I'm not sure if this helps: but I checked the relation of Wyssgasse and added the name:etymology:wikidata attribute directly on the relation (rather than on the way in the relation) and this seems to do the trick. At least looking at the map now, Wyssgasse is rendered.

Nice, thanks for the update @metaodi 👍
@CaptainInler Could you check again and let me know if this is still an issue ? Thanks.

@jbelien Yes this did the trick.


chore: Update bug issue template to use valid Markdown

When previewing the generated text, it rendered as ** Brief Description **, etc.
Also tidied up filename and updated the description for the steps (was leftover from copying from a previous issue template)

🚀 Thank you for your contribution! 🚀


SCRD-6376  Updated to use playbook constants

Yep I see that these are used with and without the .yml, missed that the first time.
Sounds good, just wanted to clarify that that was the decision.

Looks good to me :+1:


Legit Visuals feature (x88's Dead ESP) + Help

demo: https://streamable.com/k838x
ESP / Occluded / XQZ only shows when your dead (forgot to demo it) but this one has an override key in case you really need walls.
ps: Im trying to get a name changer working but im going nowhere i tried obtaining the name cvar, doing a call back then setting its value but i know im doing it wrong.

+++1 code for it ?

There's no specific source code for it but its simply making a Boolean flag for all the visuals that lets you see through walls then you can just change that flag by checking for a key press or if local player is dead.


Talk to Alibaba about what they had to hack in their app and why

Talk to Alibaba about what they had to hack in their app and why

After discussion, it looks like the obvious VC and FlutterView hacks are no longer a problem after some engine changes that have already landed.  Therefore, nothing more to do here.


Improve windows tooling

Allow cd tools && go generate -tags=tools -x create binaries with .exe extension.
Add main.windows.go to build windows version and eliminate linter errors.

CI fails, branch is out-of-date. Please create draft PRs if it is not ready for review yet


[MRG] Keep prior when normalizing Integer spaces.

Fixes #1018

Sorry I messed things up when merging. Can you resubmit the patch but for master?

@glouppe The code in master was rewritten and the bug is no longer there, but it's in the (only) released pip.

I think your merge was correct (into the 0.8.x bugfix branch) and revert that you did not needed.


Interpersonal relations plugin

Please when adding PR's about plugins only include the plugin content + the new images, I think you have also included on many PR's some stuff that is not related, closing for now.

Please, when closing a PR be more specific. I find your closing PR is just
as aggressive as you find My approach to things
Here’s a really really nice blog post that I think really captures what I
have experienced throughout my career
https://github.blog/2016-03-15-kindly-closing-pull-requests/
Bruce Bookman
On Sat, Jun 29, 2024 at 9:41 PM Joan Cabezas @.***>
wrote:

Please when adding PR's about plugins only include the plugin content +
the new images, I think you have also included on many PR's some stuff that
is not related, closing for now.
—
Reply to this email directly, view it on GitHub
https://github.com/BasedHardware/Friend/pull/372#issuecomment-2198398284,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD5ZUQ6CCCNUMPALKRTINSTZJ5O57AVCNFSM6AAAAABJ66OZHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJYGM4TQMRYGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>


[Blazor] OnValidSubmit not invoked when submit button clicked

Describe the bug
When the ValidationSummary is define at the top of the EditForm, and you have existing errors in the form, if you fix the error and then immediately click the submit button, the errors are fixed however the submit button does not invoke OnValidSubmit.
If you define the ValidationSummary at the end of the EditForm and perform the same task, it will work as expected.
This doesn't work
<EditForm Model="@person" OnInvalidSubmit="@HandleInvalidSubmit" OnValidSubmit="@HandleValidSubmit">

    <DataAnnotationsValidator />
    <ValidationSummary />

    <InputText @bind-Value="person.Name"></InputText>
    <InputNumber @bind-Value="person.Age"></InputNumber>
    <button type="submit">Submit</button>
</EditForm>

This works:
<EditForm Model="@person" OnInvalidSubmit="@HandleInvalidSubmit" OnValidSubmit="@HandleValidSubmit">

    <DataAnnotationsValidator />

    <InputText @bind-Value="person.Name"></InputText>
    <InputNumber @bind-Value="person.Age"></InputNumber>
    <button type="submit">Submit</button>

    <ValidationSummary />
</EditForm>

I believe this is occurring because when you click the submit button, focus is lost from the inputs which causes the errors to be fixed, however the submit button is not actually clicked. Therefore the OnValidSubmit event is not fired.
To Reproduce
Steps to reproduce the behavior:

Using this version of ASP.NET Core 3.0.100-preview7-012821
Run this code (see above snippets)

Expected behavior
When the ValidationSummary is defined at the top of the EditForm, and an error is fixed. I should should be able to click the submit button and have the OnValidSubmit event invoked
Screenshots
Video of bug: https://clips.twitch.tv/AuspiciousDepressedSproutPeteZarollTie
Video of workaround: https://www.twitch.tv/videos/463432083

Added videos clips of the twitch stream which demonstrates the issue.

From the video it looks like the button is not clicked the first time because the content jumps up because the validation message is removed. I would suggest adding some CSS so the content does not jump or try use the keyboard navigation e.g. tab -> enter then it should work the first time.

@campersau That was exactly it! Thanks!


EFCore 2.1 lazy-loaded proxies do not load for new entity instances.

When I create a new instance of an entity that has properties that link to another entity, those proxies do not get loaded after the new entity starts getting tracked when I try to access them.  Accessing the properties of existing entities works as expected, so I know the lazy-loading is working at least in that regard.
The SQL scripts and application code are below.
SQL scripts
CREATE DATABASE EFCoreLazyLoadingTest
GO

USE EFCoreLazyLoadingTest

CREATE TABLE [TestTable1]
(
    [Id]        INTEGER IDENTITY    NOT NULL    PRIMARY KEY,
    [Value]     VARCHAR(MAX)        NOT NULL,
)
GO

CREATE TABLE [TestTable2]
(
    [Id]            INTEGER IDENTITY    NOT NULL    PRIMARY KEY,
    [TestTable1Id]  INTEGER             NOT NULL    FOREIGN KEY REFERENCES [TestTable1]([Id]),
    [Value]         VARCHAR(MAX)        NOT NULL,
)
GO

INSERT INTO TestTable1([Value])
VALUES
('TestTable1Value1'),
('TestTable1Value2')

Application code
static void Main(string[] args)
{
    var context = new EFCoreLazyLoadingTestContext();

    var tr = new TestTable2() { TestTable1Id=1, Value = "Record 1" };

    context.TestTable2.Add(tr);
    context.SaveChanges();

    Console.WriteLine(tr.TestTable1.Value); //NullReferenceException.

    Console.Write("Press any key to continue . . . ");
    Console.ReadKey();
}

In the model for TestTable1 I have public virtual ICollection<TestTable2> TestTable2 { get; set; }.  In the model for TestTable2 I have public virtual TestTable1 TestTable1 { get; set; }.  In EFCoreLazyLoadingTestContext I have the call to  .UseLazyLoadingProxies() above the call to UseSqlServer().  All the rest is as it was generated by the following scaffold command:
Scaffold-DbContext "Server=DbServer;Database=EFCoreLazyLoadingTest;User ID=<REMOVED>;Password=<REMOVED>;" Microsoft.EntityFrameworkCore.SqlServer -Force -OutputDir Models
Is there something I'm missing here that I should be doing?  Or is this a bug or a feature of the lazy-loading functionality that has not yet been implemented?
Further technical details
EF Core version: netcoreapp2.1
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Windows 10
IDE: Visual Studio 2017 15.7.1

@Metritutus Lazy-loading proxies work by creating a proxy type that inherits from your entity type. This type then overrides navigation properties to perform lazy-loading. In the code above, the entity instance is created by calling new, which means that it is not an instance of the proxy type and hence does not have lazy-loading behaviors. This situation can be handled in two ways:

Build lazy-loading behavior into the entity type itself, as described in the docs here: https://docs.microsoft.com/en-us/ef/core/querying/related-data
Use context.CreateProxy<TestTable2>() to create the entity instance to force it to be a proxy.

This makes a lot of sense and makes me feel a bit dumb for not thinking properly through how the lazy-loaded proxies were actually working (using derived classes).  It would be useful if that CreateProxy() method was mentioned on the related-data page you linked to though (unless we're not normally supposed to use it?).
Also this does lead to my next question:  Is there any convenient way to create a proxy instance from an existing instance of the base model?  ie an equivalent of doing something like context.CreateProxy<TestTable2>(testTable2Instance);, where there's a constructor in the model that takes an instance of itself.

@Metritutus I created a docs issue for CreateProxy: https://github.com/aspnet/EntityFramework.Docs/issues/780
There isn't a method for creating a proxy instance from a non-proxy instance, partly because it can cause issues if not uses very carefully. Specifically, if the original instance is already tracked, then it would need to be detached before the proxy instance can be used, Further, if the original instance is part of a graph with references to other instances, then these would need to be moved to point to the new instance. And then the original instance should stop being used, which may not be obvious if it is being held onto by another part of the application. However, assuming all that is under control, then it can be done this way:
var blog = new Blog {  Id = 7, Title = "Cats and their muses" };
var blogProxy = context.CreateProxy<Blog>();
context.Entry(blogProxy).CurrentValues.SetValues(blog);

This creates a new proxy instance, then uses methods designed to work mainly with DTOs to set property values from the original instance onto the new one.

The model instance is in fact being created from a DTO (whose property names differ slightly from the model's).  I was also previously unaware of this SetValues() functionality!  It looks like it is briefly covered in the Disconnected Entities article.
Thanks very much for your help and in highlighting these pieces of functionality!


Fakenet Issue

Devs,
Was testing out Fakenet today and was hitting some errors. I noticed that the it attempts to run it using Python 2.7. However when I checked the git site for the project, they just ported the project to Python3. Looking at the salt package, it appears to install Fakenet and it's dependencies using python2. I think this might be causing some issues with the program. Thought it might be something worth looking into.
Also, Mandiant https://github.com/mandiant/flare-fakenet-ng appears to have take over the git for this project from Fireeye. The link for the salt package still goes to the old git  but I think it auto redirects to the Mandiant anyways so it may be moot.  But just in case you feel like changing the link.
Cheers!

Hi @geeksailor , I went over the current Python 3 fakenet version, and it's still in Alpha right now. I ran a few tests with installation and practice, and there are a few issues which are present and still unfixed by the repo owner.
I'm going to take another approach over the next few days to see if Mandiant will fix the issues, and I'll see if I can come up with some form of a workaround until they do.
Cheers

@digitalsleuth Yeah the whole alpha thing confused me considering what is installed through Remnux says is Fakenet 3.0 according to pip.Not sure if that is meant to be v3.0a.
All good though. Do what you do and I'm sure you will figure it out one way or another. Just wanted to make sure the issue was caught and brought up to ensure Remnux is running it's best self.
Cheers!

A recent update in fakenet (py3 version) may resolve this issue. Will look into it further.


Update trial template

This PR updates the trialforce template ID.

Critical Changes
Changes
Issues Closed
New Metadata
Deleted Metadata
Testing Notes

Any specific date we want this reviewed and merged by, @jstvz?

Anytime, @ceiroa.
(Looks like I thought the review was requested automatically, but scratch org defs aren't covered by CODEOWNERS.)


save figures instead of showing them in browser.

Is your feature request related to a problem? Please describe.
most times, I don't want to just view the figures.
i like to save them instead, or at least the code have to have an option to do so.
Describe the solution you'd like
maybe add a save_graph attribute? to the basic model class?
Describe alternatives you've considered
IMO, models class should be purely models, and plotting should have been another completely separate module.
Additional context
maybe we can use mplfinance to do part of the plottings.

Do model.save().
P.S: ALSO, there is an option to test the model but not graph. Just do model.test(show_graph=False)
P.P.S: I had the same idea about splitting the plotting into a function, but I have not gotten around to that


.min() doesn't work on np.datetime64 with a chunked Dataset

Hi all,
if a xr.Dataset is chunked, i cannot do ds.time.min(), i get an error : ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]'). I don't know if it is expected ?
Moreover, ds2.time.mean() works
Thanks
What happened:
raised an UFuncTypeError: ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')
What you expected to happen:
compute the min & max on a chunked datetime64 xarray.DataArray
Minimal Complete Verifiable Example:
import xarray as xr
import numpy as np

obs=200
t0 = np.datetime64("2010-01-01T00:00:00")
tn = t0 + np.timedelta64(123*4, "D")

ds2 = xr.Dataset(
    {
        "time": (["obs"], np.arange(t0, tn, (tn-t0)/obs)),
    },
    coords={
        "obs": (["obs"], np.arange(obs)),
    },
).chunk({"obs": 100})

ds2.time.min()

Anything else we need to know?:
ds2.time.mean() works, max & min raise Exception
Environment:
Output of xr.show_versions()

INSTALLED VERSIONS
commit: None
python: 3.7.9 (default, Aug 31 2020, 12:42:55)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.15.0-133-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
LOCALE: fr_FR.UTF-8
libhdf5: 1.12.0
libnetcdf: 4.7.4
xarray: 0.16.2
pandas: 1.2.1
numpy: 1.19.5
scipy: 1.6.0
netCDF4: <IP_ADDRESS>
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.6.1
cftime: 1.3.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2021.01.1
distributed: 2021.01.1
matplotlib: 3.3.4
cartopy: None
seaborn: None
numbagg: None
pint: 0.16.1
setuptools: 52.0.0.post20210125
pip: 20.3.3
conda: None
pytest: 6.2.2
IPython: 7.20.0
sphinx: 3.5.0

core.duck_array_ops.mean seems to have a custom wrapper for datetime arrays.
It should not be a problem to generalize this to min and max as well.
Maybe there a more generic wrapper would be the best solution?

Yeah that's a good idea. We should check whether dask & numpy supports this now.


Side load jsx and tsx

Needs some careful thought when used with Proscenium::Phlex::ReactComponent or Proscenium::ViewComponent::ReactComponent, as these currently expect the lazy loading via component manager.

Component manager might not even be needed at all if component managers is already side loaded.


Build system

summary: would we like a build system (that makes an index.html) for yaml2ics?  Preview:   https://coderefinery.github.io/calendar/
I've been using yaml2ics as part of something I called "git calendar", which takes a repo of yaml files and complies them to the ics feeds.  This is all pretty standard, and basically what yaml2ics can do already:
https://github.com/coderefinery/git-calendar
The difference is that I made a slightly nice interface, that makes an index.html file for it (still doesn't look too nice, but can be improved).  This is an example of this in use for CodeRefinery:
https://coderefinery.github.io/calendar/
I previously did this with a Makefile (it was a huge hack), but now has been improved and written in python with a jinja2 template.  It's still not perfect, there are remaining hacks, formatting is bad, I would want to get rid of the legacy Makefile, but it's getting better now.
It's still a bit chaotic, but the essence is one python file:
https://github.com/coderefinery/git-calendar/blob/main/build.py

a simple Github Actions file to build it and the template.  The hardest part is that this embeds yaml2ics as a submodule, and git-calendar itself is embedded as a submodule into the projects that use it.  This has required a lot of manual work when I want to add features that require coordination of yaml2ics and git-calendar.

So, I am wondering, could this functionality be added directly to yaml2ics, then I could deprecate git-calendar.  After yaml2ics becoming install able, it would become a lot easier for me to maintain this and for sub-projects to depend on this and use it (no submodules, instead use venv+pip install).
In practice, I think the #19 (packaging) PR could be altered to make this a package, instead of a module.  And then I would add another module into the package in a separate PR after improving some more.
What do you all think?

So, I am wondering, could this functionality be added directly to yaml2ics, then I could deprecate git-calendar. After yaml2ics becoming install able, it would become a lot easier for me to maintain this and for sub-projects to depend on this and use it (no submodules, instead use venv+pip install).

I am not sure we want to specialise into anything at this stage.

In practice, I think the #19 (packaging) PR could be altered to make this a package, instead of a module. And then I would add another module into the package in a separate PR after improving some more.

The current PR would already allow to import yaml2ics. So you could use all features already available.

As another example, here's our HTML rendering of calendars: https://scientific-python.org/calendars/
I feel this is outside the scope of this project, though.  There's nothing that requires any specialized calendar knowledge in order to build HTML pages.  Jinja, Hugo, or any similar template rendering engine will do the job.


CosmosDB logic app connection only seems to support SQL API

The Cosmos DB connector only seems to support the Cosmos accounts created using the SQL API. If you use an account using the Table API and attempt to create a new document the logic app reports no errors but it just doesn't work (doesn't create or update any documents).
The requirement that the input payload have an id property might be considered a cryptic clue to this, since Table API entries don't have an id, instead they have a PartitionKey and RowKey. Indeed when you dig deeper into the documentation for the Cosmos DB Connector it does make mention of this in an out of the way location. However it might take you a long time to stumble across this.
If there are limitations on which Cosmos API's the connector will work with, this really needs to be clearly stated up front and center.
I wasted a whole day finding this out.

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4921f5f1-3cee-8400-2d45-f7d25d8bf148
Version Independent ID: 99e5e49e-a7f1-8d7b-14df-6c9ae720dd5e
Content: Connect to Azure Cosmos DB - Azure Logic Apps
Content Source: articles/connectors/connectors-create-api-cosmos-db.md
Service: logic-apps
GitHub Login: @jcocchi
Microsoft Alias: jucocchi

Thank you for your feedback! We have assigned this issue to the author to review further and take the next course of action.

@jcocchi, could you please address this feedback from @Neutrino-Sunset?

@Neutrino-Sunset thank you for your feedback! You are correct that the connector only supports the SQL API. I will call this limitation out more clearly in the beginning of the document.

@Neutrino-Sunset: A doc update will follow later today. Thanks for finding this issue!


Missing blank line when calling fixusings and codeactions

I am copying the issue over from the vim plugin issue https://github.com/OmniSharp/omnisharp-vim/issues/509. If you need more information please let me know. The issue for both of these scenarios is a missing blank line after the new line is inserted into the document when using code actions and fix usings. See the images and scenarios below:

No blank line is inserted between the constructor and last newly added field when calling code actions to create, for example, a private readonly field. I believe it is using /v2/codeactions from the vim script code.

No blank line is inserted between the class name line and the last using statement when calling fix usings. This only seems to happen when the usings are nested in the namespace and the new using statement is added to the bottom. If the using is outside the namespace, it gets added with the appropriate spacing.  I believe it is using /fixusings from the vim script code.

Here is an example server response for fixing usings if it helps. I was trying to debug it to determine if it was truly a server error or if it was vim, but with limited knowledge and time I have not found anything yet...
Nested using  (does not insert the blank line)
namespace IdentityService.Web.IntegrationTest
{
    using System;
    using System.Data.Common;
    using System.Data.SqlClient;
    using FakeItEasy;
    using IdentityService.Data;
    using IdentityService.Domain;

    public class DatabaseFixture : IDisposable
    {
        ...<snip>

server response:
{ 'StartColumn': 34, 'EndColumn': 2, 'StartLine': 8, 'NewText': '^@    using Microsoft.EntityFrameworkCore;^@', 'EndLine: 9}

Normal using (inserts the blank line as expected)
using System;
using System.Data.Common;
using System.Data.SqlClient;
using FakeItEasy;
using IdentityService.Data;
using IdentityService.Domain;

namespace IdentityService.Web.IntegrationTest
{
    public class DatabaseFixture : IDisposable
    {
        ...<snip>

server response:
{ 'StartColumn': 1, 'EndColumn': 2, 'StartLine': 7, 'NewText': 'using Microsoft.EntityFrameworkCore;^@', 'EndLine: 7}

thanks, this looks like a bug indeed

This patch should fix this issue in vim. @filipw Maybe you could shed some light on why omnisharp-roslyn returns EndColumn = 2 for a empty line in the example provided in the pull request?


Fixed Usage.md links

Sorry for the confusion @acroca , the links are for when you view the documentation on Hex docs. Instead, I'm going to remove them as they don't work when you view the guide on GitHub.

Removed by fd279c252ca3c1d2325c4ab3e0ef43830abfec7e.


Add Covalent API to Cryptocurrency

add covalent api's in cryptocurrency

[x] Your submissions are formatted according to the guidelines in the contributing guide
[x] Your additions are ordered alphabetically
[x] Your submission has a useful description
[x] The description does not end with punctuation
[x] Each table column should be padded with one space on either side
[x] You have searched the repository for any relevant issues or pull requests
[x] Any category you are creating has the minimum requirement of 3 items
[x] All changes have been squashed into a single commit

hey , I already submitted covalent to this directory https://github.com/public-api-lists/public-api-lists/pull/18


Major outcomes from L1 feedback meeting 2023-11-29

[ ] Significant preference for units as part of research_name (e.g. soil_vwc_10cm_degC). Roy and Ben both groan :) but many find this a lot easier!
[ ] Have L1 data include basic visualizations for all variables
[ ] Include OOB limit values in metadata
[ ] How to handle out-of-service events recorded by techs?
[ ] BBL and SP to work towards design_link tables for all sites
[ ] Want a mechanism to handle the issue of e.g. sapflux sensor being shifted to a new tree after original dies -- basically need a valid_until column in design link table and code to handle that
[ ] RR gave good example of hierarchical flag: fluctuating voltage flag should cascade to inform all sensors on that datalogger

Hierarchy examples

Fluctuating voltage flag should cascade to inform all sensors on that datalogger
Does a bad TEROS temperature reading informs VWC and EC on same probe?

Hierarchy. With the addition of a single parent column in the design_link table, we can do something like this:

This visualizes how a flag on the solar battery voltage would flow (propagate) to the sapflow, TEROS, and AquaTroll sensors.


Move Twig cache into the storage directory

This is set in the initiation in the container; use the dirname(FILE, 3) structure for consistency.
As part of this change, we need to make sure the current cache location is not committed. The new location will also need .gitignore configuration.

I'll look into this. I'm sorry, I thought this was already done.

@MisterBrownRSA No problem. I figured it was another easy PR for Hacktoberfest. The change appears to be relatively straightforward. Let me know if you run into any problems.
Currently it saves the cache to lib/Framework/storage/sessions

#87

There are some tests in the repo; they do use phpunit.
I really was just hacking at it though. tests are a weakness for me. There is also an issue for testing routes, but @andradedev said they would work on it. Feel free to open issues on other tests, or anything you see. I would like to add badges for test coverage and try to get good coverage. But, like I said, I still have much to learn with testing.

@RogerCreasy yes, actually i working fixing the issues, and adding to routes, after that i will add to other stuff too

@andradedev Cool! Thanks!

Resolved with PR #87


⚠ Slack has degraded performance

In 82b603e, Slack (https://status.slack.com/api/v2.0.0/current) experienced degraded performance:

HTTP code: 200
Response time: 47 ms

Resolved: Slack performance has improved in e6b729c.


Add license and copyright

I propose using the same license & copyright as OpenShift Origin.  I'll create a PR if everyone agrees.

Sounds good to me


Change breakpoints

Is it possible to customize breakpoints value?

sorry i've seen now on the readme
ResponsiveGridBreakpoints.value = ResponsiveGridBreakpoints(
  xs: 600,
  sm: 905,
  md: 1240,
  lg: 1440,
);


Any plans for a new release?

The latest release is 2.4.2 made on September 5 2016, any plans for a new release?

Done


UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'

I can't resolve this error.
UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
Also, I finally get an error like this too.
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Also, when I run run_gui.ps1, I get an error like this.
ModuleNotFoundError: No module named 'gradio'
I checked with "pip list" and found that "gradio" is installed correctly.
Therefore, I have no choice but to run "python.exe demo.py" from the command prompt.

After installing gradio in venv, the error ModuleNotFoundError: No module named 'gradio' disappeared.
So, when I executed run_gui.ps1 again, I got the following new error.
\layerdivider\venv\Lib\site-packages\gradio_client\documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix'   warnings.warn(f"Could not get documentation group for {cls}: {exc}") \layerdivider\venv\Lib\site-packages\gradio_client\documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix'   warnings.warn(f"Could not get documentation group for {cls}: {exc}")


Benchmarking image classification on RPi failing with this error:

Please close if a duplicate of #760.


Use default object when options is falsy

The SSHAgentClient constructor did not set this.timeout when options was falsy which resulted in a TypeError when #_request called Socket#setTimeout with undefined as the first argument. This pull request adds an object containing the default value of timeout to remove the conditional, ensuring that this.timeout is set—unless options is truthy but not an object (e.g., a Number).
Fixes #3 (and the tests).

ah, i didn't see this when i created https://github.com/mcavage/node-ssh-agent/pull/5, as at first i thought this was a node-manta problem and not a problem with this module.

Closing because #5 has been merged.