[SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

What changes were proposed in this pull request?
Spark SQL supports some window function like NTH_VALUE.
If we specify window frame like UNBOUNDED PRECEDING AND CURRENT ROW or UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING, we can elimate some calculations.
For example: if we execute the SQL show below:
SELECT NTH_VALUE(col,
         2) OVER(ORDER BY rank UNBOUNDED PRECEDING
        AND CURRENT ROW)
FROM tab;

The output for row number greater than 1, return the fixed value. otherwise, return null. So we just calculate the value once and notice whether the row number less than 2.
UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING is simpler.
Why are the changes needed?
Improve the performance for NTH_VALUE, FIRST_VALUE and LAST_VALUE.
Does this PR introduce any user-facing change?
'No'.
How was this patch tested?
Jenkins test.

Test build #128867 has started for PR 29800 at commit 9610c86.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33489/
Test PASSed.

Test build #128867 has finished for PR 29800 at commit 9610c86.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128867/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33548/
Test PASSed.

Test build #128926 has started for PR 29800 at commit 1dd2d36.

Test build #128926 has finished for PR 29800 at commit 1dd2d36.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait OffsetWindowSpec extends Expression
case class Lead(input: Expression, offsetExpr: Expression, default: Expression)
case class Lag(input: Expression, offsetExpr: Expression, default: Expression)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128926/
Test FAILed.

Test build #128927 has started for PR 29800 at commit d4e1cb2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33549/
Test PASSed.

Test build #128927 has finished for PR 29800 at commit d4e1cb2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128927/
Test FAILed.

Test build #128929 has started for PR 29800 at commit fb7778e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33551/
Test PASSed.

Test build #128932 has started for PR 29800 at commit f1f80d3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33554/
Test PASSed.

Test build #128929 has finished for PR 29800 at commit fb7778e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128929/
Test PASSed.

Test build #128932 has finished for PR 29800 at commit f1f80d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128932/
Test PASSed.

cc @cloud-fan

Test build #129058 has started for PR 29800 at commit b5a48b8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33679/
Test PASSed.

Test build #129058 has finished for PR 29800 at commit b5a48b8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class WorkerDecommission(
  case class DecommissionExecutor(executorId: String, decommissionInfo: ExecutorDecommissionInfo)
class InstallCommand(install):

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129058/
Test FAILed.

retest this please

Test build #129070 has started for PR 29800 at commit b5a48b8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33691/
Test PASSed.

Test build #129070 has finished for PR 29800 at commit b5a48b8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class WorkerDecommission(
  case class DecommissionExecutor(executorId: String, decommissionInfo: ExecutorDecommissionInfo)
class InstallCommand(install):

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129070/
Test FAILed.

Test build #129073 has started for PR 29800 at commit ae71344.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33694/
Test PASSed.

Test build #129073 has finished for PR 29800 at commit ae71344.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129073/
Test PASSed.

Test build #129101 has started for PR 29800 at commit 875b92b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33722/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33722/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33722/
Test PASSed.

Test build #129101 has finished for PR 29800 at commit 875b92b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129101/
Test FAILed.

retest this please

Test build #129107 has started for PR 29800 at commit 875b92b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33728/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33728/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33728/
Test FAILed.

Test build #129107 has finished for PR 29800 at commit 875b92b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129107/
Test PASSed.

Test build #129209 has started for PR 29800 at commit dab9e30.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33824/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33824/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33824/
Test FAILed.

Test build #129209 has finished for PR 29800 at commit dab9e30.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129209/
Test FAILed.

retest this please

Test build #129220 has started for PR 29800 at commit dab9e30.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33835/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33835/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33835/
Test PASSed.

Test build #129220 has finished for PR 29800 at commit dab9e30.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129220/
Test PASSed.

Test build #129682 has started for PR 29800 at commit fcd3f44.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34286/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34288/

Test build #129685 has started for PR 29800 at commit 4d21236.

Test build #129685 has finished for PR 29800 at commit 4d21236.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129685/
Test FAILed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34286/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34286/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34288/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34288/
Test PASSed.

Test build #129686 has started for PR 29800 at commit 0f2ab8c.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34291/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34292/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34291/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34291/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34292/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34292/
Test PASSed.

Test build #129682 has finished for PR 29800 at commit fcd3f44.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129682/
Test PASSed.

Test build #129686 has finished for PR 29800 at commit 0f2ab8c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129686/
Test PASSed.

Test build #129978 has started for PR 29800 at commit 28dc1a2.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34586/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34586/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34586/
Test PASSed.

Test build #129978 has finished for PR 29800 at commit 28dc1a2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129978/
Test FAILed.

retest this please

Test build #129995 has started for PR 29800 at commit 28dc1a2.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34601/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34601/
Test FAILed.

Test build #129995 has finished for PR 29800 at commit 28dc1a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129995/
Test PASSed.

Test build #130146 has started for PR 29800 at commit d7b9862.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34753/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34753/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34753/
Test PASSed.

Test build #130146 has finished for PR 29800 at commit d7b9862.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130146/
Test FAILed.

retest this please

Test build #130158 has started for PR 29800 at commit d7b9862.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34765/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34765/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34765/
Test PASSed.

Test build #130158 has finished for PR 29800 at commit d7b9862.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130158/
Test FAILed.

retest this please

Test build #130186 has started for PR 29800 at commit d7b9862.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34788/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34788/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34788/
Test PASSed.

Test build #130186 has finished for PR 29800 at commit d7b9862.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130186/
Test FAILed.

retest this please

Test build #130192 has started for PR 29800 at commit d7b9862.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34793/

Test build #130192 has finished for PR 29800 at commit d7b9862.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130192/
Test FAILed.

retest this please

Test build #130195 has started for PR 29800 at commit d7b9862.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34793/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34793/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34796/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34796/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34796/
Test PASSed.

Test build #130201 has started for PR 29800 at commit 22adc21.

Test build #130202 has started for PR 29800 at commit 9e84f54.

Test build #130195 has finished for PR 29800 at commit d7b9862.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130195/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34802/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34803/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34802/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34802/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34803/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34803/
Test PASSed.

Test build #130201 has finished for PR 29800 at commit 22adc21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130201/
Test PASSed.

Test build #130202 has finished for PR 29800 at commit 9e84f54.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130202/
Test PASSed.

Test build #130318 has started for PR 29800 at commit cde6170.

Test build #130318 has finished for PR 29800 at commit cde6170.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FrameLessOffsetWindowFunctionFrame(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130318/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34920/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34920/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34920/
Test FAILed.

Test build #130322 has started for PR 29800 at commit ca574d9.

Test build #130324 has started for PR 29800 at commit fd59f6e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34924/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34926/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34924/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34924/
Test FAILed.

Test build #130325 has started for PR 29800 at commit 72e7805.

Test build #130326 has started for PR 29800 at commit 9c35ddd.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34926/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34926/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34927/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34929/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34929/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34929/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34927/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34927/
Test FAILed.

Test build #130322 has finished for PR 29800 at commit ca574d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130322/
Test PASSed.

Test build #130324 has finished for PR 29800 at commit fd59f6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130324/
Test PASSed.

Test build #130326 has finished for PR 29800 at commit 9c35ddd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #130325 has finished for PR 29800 at commit 72e7805.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130326/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130325/
Test PASSed.

Test build #130348 has started for PR 29800 at commit 1c0e82b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34950/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34950/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34950/
Test FAILed.

Test build #130348 has finished for PR 29800 at commit 1c0e82b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130348/
Test FAILed.

thanks, merging to master!

@cloud-fan Thanks for your help!


Uncaught TypeError: marked.setOptions is not a function

I think it might be related to #825

You're right, @ajboni - it's related. That's the exact same TypeError that's caused by the marked package in version 4.x. When the @sparksuite will implement my PR into their codebase, the TypeError you mentioned will be gone.

Thanks for confirming, @staszek998 . For now, a workaround I'm using, was to move to yarn and use resolutions on package.json. Not really ideal, but it works...
  "resolutions": {
    "marked": "3.0.8"
  }

I'm using the same workaround and it works for me, too 🙂


Support of sdp(Session Description Protocol) file?

Hello, I would like to ask if your vlc player has a way to provide sdp file?
may be like this
rtp.sdp
v=0
m=video 5002 RTP/AVP 96
a=rtpmap:96 H264/90000
c=IN IP4 <IP_ADDRESS>

Because flutter_vlc_player can use controller.setMediaFromFile(flie) to set rtp.sdp file, so I don't know if you can do this?


feat(storage): prefetch data for compaction

Signed-off-by: Little-Wallace<EMAIL_ADDRESS>What's changed and what's your intention?
Our default configuration for block-size is 64 KB. But according to some tests, a larger read request can improve scan request.

A simple idea is reading multiple block for compactor.
Optimization:

prefetch multiple blocks for reading in compaction job.
upload the results of compaction to s3 in parallel
prefetch meta data before compaction task starts, because there may be a lot of files in L0 and each of them is just generated by compute-node, which can not exist in meta-cache of compactor node.

Checklist

[x] I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

cc @skyzh  Maybe it can help your load data in test.

Good catch.... But the problem only cause some of useless request. Because all data will be placed to block-cache and it would never exceed the capacity limit

The next step I will make the prefetch size configurable and user can pass key-range to ReadOptions so that we would not read data blocks which keys has exceed the target key-range.

I will make the prefetch size configurable

+1, but set a reasonable default (and try to benchmark the different prefetch size if possible?)
I think basically the pattern is something like an async queue that will try to fetch N blocks in advance and it can buffer N blocks?

I will make the prefetch size configurable

+1, but set a reasonable default (and try to benchmark the different prefetch size if possible?)
I think basically the pattern is something like an async queue that will try to fetch N blocks in advance and it can buffer N blocks? (so it also has built-in back pressure)
For remote compaction like s3, read  1MB-size data only cost a little more time than 48KB. So we do not need to fetch them in another thread.


fix travis + cache

Tasks

[x] fix build
[x] cache

travis cache is only available for private repos

addressed in #13


Scroll Top Feature added for Todo Apllication - New Pull request

Avinash,
I had closed that pull request, once again raising a new pull request for scrollTop feature in ToDo application.
Added code editor screenshot for conflicts resolved in css files. Added UI recent screenshot for reference.
Adding screenshot for your references, kindly approve it.

sorry Prakash , you have to resolve once again !
good work , keep doing !


There's now no way to pass the path to build.gradle

We just started integrating CodePush into a hybrid application (an originally native application that React Native is injected in), so the path to build.gradle is not android/app/build.gradle (the default one used for most React Native applications). In our case the folder is named differently, for instance android/our-app and the whole path to build.gradle is android/out-app/build.gradle.
So, with
"appcenter-cli": "2.6.3"

every our attempt to make a code-push release for Android fails with:
{ Error: ENOENT: no such file or directory, lstat 'android/app'
    at Object.lstatSync (fs.js:845:3)
    at Object.getHermesEnabled (......./node_modules/appcenter-cli/dist/commands/codepush/lib/react-native-utils.js:327:12)
    at CodePushReleaseReactCommand.<anonymous> (......./node_modules/appcenter-cli/dist/commands/codepush/release-react.js:112:72)
    at Generator.next (<anonymous>)
    at fulfilled (......./node_modules/appcenter-cli/dist/commands/codepush/release-react.js:11:58)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  errno: -2,
  syscall: 'lstat',
  code: 'ENOENT',
  path: 'android/app' }

in function getHermesEnabled at https://github.com/microsoft/appcenter-cli/blob/a63017dcc66f6eca4874757cead23eafba0ff2b6/src/commands/codepush/lib/react-native-utils.ts#L409
because the path is hardcoded:
https://github.com/microsoft/appcenter-cli/blob/a63017dcc66f6eca4874757cead23eafba0ff2b6/src/commands/codepush/lib/react-native-utils.ts#L405
Here's a slightly bigger piece of code:
...
export function getHermesEnabled(gradleFile: string): boolean {
  let buildGradlePath: string = path.join("android", "app");
  if (gradleFile) {
    buildGradlePath = gradleFile;
  }
  if (fs.lstatSync(buildGradlePath).isDirectory()) {
   ..

For the time being, we just patch this file on postinstall hook (I know this is a bad piece of duct-tape), otherwise we cannot use code-push at all.
What would be a better approach here? Is this already planned to be implemented, or it's intentional and should never be changed? I can raise a PR to introduce a CLI argument for that.

Have you tried giving the gradleFile value to the method getHermesEnabled ?That can be done by setting args -g or --gradle-file. Run $ appcenter codepush release-react --help will show the help text for the args.

Oh... there's already an arg for that.... thanks for pointing at it. I actually missed it. Sorry about that. Such a shame :) And, yes it does work.


Allow Geo-Dash created imagery to be added to a project

Currently Geo-Dash created imagery by default is just added to the widget, but has the option to add to the institution.  This issue will extend the functionality to a radio button collection:
(X) Add to widget (Default selected)
(  ) Add to project collection basemaps
(  ) Add to institution basemaps
These options will cascade up.

I think this is complete now that we redirect users to create the imagery at the institution level.


Communication between frontend and backend

Currently, both the frontend (javascript UI / code) and the backend (Python code) are separate pieces of code that do not interact with each other.
The currently envisioned process to create a QEC experiment is the following:

The user starts the frontend.
The user builds the experiment he/she wants to perform, still 100% in the frontend.
Once the experiment is fully caracterised in the frontend, the user asks for the Stim file generation (or multiple Stim files).
[internally] the frontend represents the QEC experiment in a JSON format.
[internally] the frontend sends the generated JSON to the backend (Python code).
[internally] the backend generates a Stim file from the input JSON.
[internally] the backend sends back the generated Stim file to the frontend.
The frontend provides a way to download / export the generated Stim file.

The above process is still in discussion. It has shortcomings, but also has the good property of being really simple to implement on both sides (frontend and backend).
The current communication scheme using JSON, both the frontend and backend need to agree on the JSON format.
A tentative format can be found in the specification/schemas folder. We would like to improve this format and actually implement the serialisation / deserialisation procedures in both the frontend and the backend.

Another aspect that should be discussed, in my opinion, is exactly where
the simulations generating graphs of logical error rate versus physical
error rate will be run. On the user's machine? In some cloud service? As
our examples get bigger, a scalable cloud service would definitely be
desirable, but on this point I'm not sure what options are available, in
particular for free, and just how much compute power is available.
On Wed, Jan 10, 2024 at 2:36 AM Adrien Suau @.***>
wrote:

Currently, both the frontend (javascript UI / code) and the backend
(Python code) are separate pieces of code that do not interact with each
other.
The currently envisioned process to create a QEC experiment is the
following:

The user starts the frontend.
The user builds the experiment he/she wants to perform, still 100%
in the frontend.
Once the experiment is fully caracterised in the frontend, the user
asks for the Stim file generation (or multiple Stim files).
[internally] the frontend represents the QEC experiment in a JSON
format.
[internally] the frontend sends the generated JSON to the backend
(Python code).
[internally] the backend generates a Stim file from the input JSON.
[internally] the backend sends back the generated Stim file to the
frontend.
The frontend provides a way to download / export the generated Stim
file.

The above process is still in discussion. It has shortcomings, but also
has the good property of being really simple to implement on both sides
(frontend and backend).
The current communication scheme using JSON, both the frontend and backend
need to agree on the JSON format.
A tentative format can be found in the specification/schemas
https://github.com/QCHackers/tqec/tree/main/specification/schemas
folder. We would like to improve this format and actually implement the
serialisation / deserialisation procedures in both the frontend and the
backend.
—
Reply to this email directly, view it on GitHub
https://github.com/QCHackers/tqec/issues/38, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKAXTE5LEMEGHT7FR76VULYNZVKVAVCNFSM6AAAAABBURVZYGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA3TIMBXGU3DEMY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

@afowler The package can be installed and run anywhere. The user can decide if they want to run the simulations on their machine or a cloud service. We can put the Python code on a Flask server and have the UI communicate with the server over the wire.

a scalable cloud service would definitely be desirable, but on this point I'm not sure what options are available, in particular for free

This package should work on any cloud service as long as the user can get their hands on some free cloud credits or fork some cash.

Few options when it comes to credits:
https://cloud.google.com/free
https://aws.amazon.com/free/offers/
https://azure.microsoft.com/en-us/free

I'd also like to pitch that qBraid may be interested in being more involved in hosting this as well. We have most of infrastructure and deploying such applications is part of my technical capabilities.

+1 for @rryoung98's suggestion :fireworks:

As this matures, I will make sure that qbraid will be a nice home. In the mean time, the frontend and backend connection, assuming we're okay with the JSON will need to be REST API requests from the api and the frontend. This is a method of communication that is very common and shouldn't be hard to implement.
Out of the JSONs @nelimee I think QASM could be a good way of moving between the frontend and backend.

It works well with json/application and sending to the python api.
It is a very common way to get circuit information to and from a frontend and server for most major hardware providers.
Compatible with your cirq implementation.

QIR, could also be an optional as well but im not sure if we need all the auxiliary overhead that comes with the LLVM architecture... thoughts?
6,7,8 are trivial. We can just make a folder that's called stims or something and drop the files there as well as the JSON information associated and created from the frontend then use standard methods of accessing filesystems from the frontend.
6. [internally] the backend generates a Stim file from the input JSON.
7. [internally] the backend sends back the generated Stim file to the frontend.
8. The frontend provides a way to download / export the generated Stim file.

[internally] the frontend represents the QEC experiment in a JSON format.
[internally] the frontend sends the generated JSON to the backend (Python code).

How far are the above 4 and 5 right now? From our end 3. is virtually unimplemented, but we should be able to work in parallel with UI functionality improvements if we want to push ourselves to get this done end of Jan (haha).

Once the experiment is fully caracterised in the frontend, the user asks for the Stim file generation (or multiple Stim files).

@afowler about the server, I was thinking about a fully local implementation at the beginning. In later stages, having the possibility to install the backend (or run the generated stim files) on a different server might be a necessity, but I think this is something that should be primarily the user responsibility (as seems to say @vtomole). Basically, if you (as a user) want to run large circuits, you have to provide your own compute power.
@rryoung98 OpenQASM (2.0 I guess?) is fine on my side. I have no strong feeling about that part, anything that fits the needs (easy to generate on the frontend, easy to parse on the backend and able to represent everything that we need) is fine for me.
My thoughts about QIR is that this is way too much overkill for our needs, both in terms of representation capabilities, abstraction implemented and tooling required.
@rryoung98 4 is on the frontend side, but 5 is not implemented at all. Basically, I wanted to be sure about the JSON format and what we needed in it before implementing anything. The backend is quite flexible, and I can adapt myself or afford to perform some reasonable computations if the JSON input does not match exactly with what the backend needs.
If you are not really limited on your side either, I can finish the description in terms of JSON schema I started in the repository, wait for your validation, and implement that in the backend.
Implementing this step should be relatively easy, so I am not too worried about that part.

Self-assigning based on discussion in the Google Group.

Merged #40 to establish communication and live servers/endpoints
We still need to implement templates for the frontend; right now, only qubits and plaquettes are represented.

I think this should be spun off into more specific issues on the frontend/backend sides, as well as defining the serialization templates

Do you know how to split that issue? In my opinion, we can keep that one open as a placeholder until we start implementing the communication part and actual issues arise.

Do you know how to split that issue? In my opinion, we can keep that one open as a placeholder until we start implementing the communication part and actual issues arise.

@nelimee  I haven't given it a ton of thought lately. The frontend/backend developments have diverged somewhat over the past couple weeks, and I think once we can support building quantum circuits on the frontend (eg, #46), we can finalize this issue, since we'll actually have circuit data to send to the backend.

Marking this as task 7.5 in the task flow, and adding to the next milestone

This issue is not relevant anymore. Closing it. See #323 for more details.


Allow S3 or Oracle Cloud Object Storage

Feature description
Thank you for the great Micronaut integration with MicroStream. Really makes the integration easy and fast to get started.
I was wondering if there are plans to add configuration options for other Storage engines, such as:

Oracle Cloud Object Storage or AWS S3; or
SQL database storage

Or this even already supported?
Thanks

https://docs.microstream.one/manual/storage/storage-targets/index.html


changing the font of a statement to Roman

while trying to write (if) in the Roman font, I came across this
\textnormal{} % to use the LaTeX font of the main body text (usually serif)
\textrm{}     % to force Roman font (serif)
\textsf{}     % to force sans serif font
but these commands do not work, does this library support other way for forcing the roman font on parts of the text ?

WPS-Math uses several fonts for its glyphs (including a user-provided font for \text, but I don't know which of them may be a Roman one.


Fix OS display name

📝 Description
Fixing OS name in the VM details section in VM create with Instance type flow
🎥 Demo
Before:

After:

/lgtm


Bug Fix : ERR_UNESCAPED_CHARACTERS

Description :
After attempting to run this project on my system, I encountered the error ERR_UNESCAPED_CHARACTERS. I further tested it on different systems, but unfortunately, the same error persists. However, I have found a solution to resolve this issue. This PR will fix the issue.
Change :
This is the bug fix(non-breaking change which fixes an issue)
Checklist:

I have performed a self-review of my own code
I have checked my code and corrected any misspellings
My changes generate no new warnings

lgtm, thanks for the contribution


Fix crossbrowser date format

cherry pick from https://github.com/openshift/logging-view-plugin/pull/69

/lgtm

/cherry-pick release-5.6


Convert grpc malloc slice to use grpc_core::RefCount.

This shows up in profiles. Using grpc_core::RefCount lowers the time
spent in grpc_slice_malloc_large() by 8% in 1MB callback unary benchmarks.

Thank you, Vijay!


Configure the TLD, change from ddev.local to something else

Feature request:
Would it be possible to completely remove the .dev.local and replace it with another domain name? For example, dev.example.edu.
We have SSO that only works with one domain naming convention.
What you expected to happen:
Alter the YAML file to specific a completely different hostname.

Changed the title, hope it reflects your request.

I would love it if you could explain more why it's important to use a specific domain name. Most local dev environments don't matter what the FQDN is. Could you please explain more about that? Why would SSO matter for a local dev environment? Wouldn't putting a bogus site (like mysite.yoursso.domain) be a bad thing? And since it wasn't in DNS, wouldn't that break things?

It's really important for SAML authentication and similar authentication mechanisms. When you authenticate against a SAML identity provider (IDP), you are redirected to the IDP, you login, and then you get redirected back to a specific hostname set up in the IDP (with auth information in the URL in query params). Many SAML IDPs will not support redirecting back to an arbitrary URL, so your local environment has to live at a specific hostname. Since the IDP doesn't talk directly to the site you're logging into, not having the site in DNS doesn't affect the success.

Adding tags. Noting a somewhat related issue that could open the door for other FQDNs #416.

To answer the question of why you would want SSO for local development, it largely boils down to being able to test application(s) in development. In these apps, the user logs in, are registered as a user, and the username is used to fetch additional information via another server. These data are  added to user fields--and used for other applications.
As to your second point, I am not sure exactly what you mean by 'bad thing.' We have gotten SSO to work on local environments before (via Vagrant and Lando), and the applications worked in those environments.

In https://github.com/drud/ddev/blob/06267550c4af2a96d595ec89507c9e59e67a4032/pkg/version/version.go#L50 I see
// DDevTLD defines the tld to use for DDev site URLs.
const DDevTLD = "ddev.local"

The other items are var. Does that mean they can be overridden by environment variables? Or are these only considered when building the tooling itself, not on configuration

The others can be overridden by make a build time. Doesn't mean that this shouldn't be configurable at runtime.

Closing this one in favor of https://github.com/drud/ddev/issues/868, which solves this problem as well as others. Ever project can have a unique full URL.

The TLD does become configurable in https://github.com/drud/ddev/pull/1611, in ddev v1.9


Add support for printing nodes with multiple attributes on separate lines.

I would like to add support for printing attributes for nodes with multiple attributes on separate lines.
I will submit a PR for this shortly. Thanks for your consideration!
For example:
digraph multi {
  a[shape=square]
  aa[
    color=blue,
    shape=Mrecord
  ]
  subgraph v {
    aaa[shape=square]
    aaaa[
      color=red,
      shape=Mrecord
    ]
    aaa -> aaaa [label=FALSE]
  }
  a -> aa [label=TRUE,color=green]
}

Hi. Sure 😊


[RFC] Collapsible traces with messages

This PR contains a number of improvements to trace messages, most notably collapsible trace nodes can now have a message.   Nested traces are represented as .trace _ _ #[.trace .., .trace ..].  And there's a new RPC call lazyTraceChildrenToInteractive.  I avoided mutually inductive types because that's painful with the derive handlers.
The new API to produce traces is withTraceNode `trace.class msg do ... where msg : Except ε α → m MessageData is a function that can produce different messages depending on the return value.
traceCtx and withNestedTraces is removed, including the feature where traceCtx would drop all trace messages in its body if its trace option was not enabled.
Trace class inheritance is now opt-in.  set_option trace.foo true no longer automatically implies set_option trace.foo.bar true.  Inheritance can be enabled---after careful consideration---on a case-by-case basis by calling registerTraceClass `foo.bar (inherited := true).
Multiple trace messages for the same range are grouped into a single diagnostic.
Due to the structured transmission of the trace messages, the editor plugins now omit common prefixes of trace classes (with the full name as popup), gray out the trace class, etc.
I've edited a few trace messages.

To try out this PR, you need a compatible editor plugin:

neovim: https://github.com/Julian/lean.nvim/pull/267
vscode: https://github.com/leanprover/vscode-lean4/pull/237

When editing the trace messages, I've tried to follow some key principles:

Trace messages explain to the user what problem Lean solved and what steps it took.  This is not the place for debugging output or todos.  Think of trace messages like a figure in a paper.
Expansion progressively increases detail. Each level of expansion (of the trace node in the editor) should reveal more details.  For example, the unexpanded node should show the top-level goal.  Expanding it should show a list of steps.  Expanding one of the steps then shows what happens during that step.
One trace message per step.  The [trace.class] marker functions as a visual bullet point, making it easy to identify the different steps at a glance.
Outcome first.  The top-level trace message already shows whether the action failed or succeeded, as opposed to a "success" trace message that comes pages later.
Be concise.  Keep messages short.  Avoid repetitive text.  (This is also why the editor plugins abbreviate the common prefixes.)
Emoji are concisest.  Several helper functions in Lean.Util.Trace help with a consistent emoji language.
Good defaults. Setting set_option trace.Meta.synthInstance true (etc.) should produce great trace messages out-of-the-box, without needing extra options to tweak it.  (That's the motivation for making inheritance opt-in.)

Addresses #591.

@gebner This is great. Thanks for working on this. I didn't have to review it yet. BTW, the PR is currently marked as a draft, is something missing?
Could you please copy some of the comments above as docstrigs? For example, the inherited text could be copied as a docstring for registerTraceClass.

!bench

This is awesome! Could you comment on whether there is a clear path to implementing Daniel's [proposal] later as well?

This is awesome! Could you comment on whether there is a clear path to implementing Daniel's proposal later as well?

Is it possible to implement something like Daniel suggested (if I understand it correctly, a set_option trace.inside.Meta.isDefEq true that enables all trace messages inside a Meta.isDefEq node)?  Certainly.  But that would also enable a lot of junk trace messages as well, so I'm not sure how useful such a blanket setting would be.

Is it possible to implement something like Daniel suggested (if I understand it correctly, a set_option trace.inside.Meta.isDefEq true that enables all trace messages inside a Meta.isDefEq node)? Certainly. But that would also enable a lot of junk trace messages as well, so I'm not sure how useful such a blanket setting would be

Probably not much, but the suggestion of composable filters would be extremely useful:

One might want the Meta.synthInstance.newSubgoal calls that occur inside isDefEq calls.

With the trace explorer, the ideal UI might be something like a fuzzy search input box where we can type in isDefEq Meta.synthInstance.newSubgoal (think CSS selectors) and have the trace tree auto-expand to show those. It would either need server support so that we don't have to eagerly toInteractive the entire trace, or we could send just a "trace outline" containing only the classes but not the messages and have the client filter that.

It would either need server support so that we don't have to eagerly toInteractive the entire trace, or we could send just a "trace outline" containing only the classes but not the messages and have the client filter that.

Any thoughts on what the ideal RPC API for that would be?  With this PR there's only a LazyTraceChildren type, so even if we add an RPC call like searchTraceChildren{ children: LazyTraceChildren, query: String }: SearchResult we'd still need to do half of the search on the client (since we only have an RPC reference for the kids, but not the root node).  On the other hand, I didn't want to hide the root node behind an RPC ref, because that would require additional round-trips.
Specifying the filters on the Lean side is much easier of course, and requires less API work.  Something like enable_trace isDefEq>synthInstance in def ... would be fairly straightforward to implement.

BTW, the PR is currently marked as a draft, is something missing?

I wanted some discussion on the RPC API first (see exchange with Wojciech).  There's also plenty of trace messages to change so that they fit better into the new concept.  The only module I've really touched is the TC synthesizer, the rest is just the minimum to get rid of traceCtx and withNestedTraces.  But I can also do that in follow-up PRs.

It looks like this could also serve as the basis for a metaprogram that essentially tests a trace against a spec, right? The idea here is that I'd like to step through some instance searches in Functional Programming in Lean to help readers get an idea of how outParam and defaultInstance affect the process, but this is only tenable to keep up to date if I can test the resulting text against what Lean is actually doing. It seems that enabling the right traces, capturing them, and then validating them could be doable with this new .trace constructor, right?

Any thoughts on what the ideal RPC API for that would be?

Sadly no, you probably have a better idea of it by now.

Specifying the filters on the Lean side is much easier of course, and requires less API work. Something like enable_trace isDefEq>synthInstance in def ... would be fairly straightforward to implement.

The tradeoff seems to be:

If we send a trace outline, we could have a nice quickly-responsive UI with a fuzzy search box. But implementation effort is duplicated across infoview/NVim.
If we use an enable_trace command, the server does everything and no changes to clients may be required thanks to the collapseByDefault flag which we can just toggle appropriately. If done the obvious way though, there will be a pretty bad interaction with long-running commands which produce huge traces in that changing the enable_trace will re-elaborate the entire snapshot of the command below. So quick searching of the trace would not really work.

Thinking about the search API issue a bit more, I believe that all searching should happen server-side.  This is great because:

It also works on the command-line.
Filters can be shared in snippets on Zulip.
It reduces code duplication in editors.
The content of the traces can be searched as well.  For example, we can show all type-class traces with metavariables in them, etc.

We can implement this without breaking backwards compatibility by adding an additional field to the MsgEmbed for traces at the end, namely WithRpcRef MessageData for the .trace object itself (maybe with indentation and naming contexts etc.).  Then we only need to add an RPC call searchTrace{trace: MessageData, query: String}: MessageData and hook it up in the editors.

It looks like this could also serve as the basis for a metaprogram that essentially tests a trace against a spec, right? The idea here is that I'd like to step through some instance searches in Functional Programming in Lean to help readers get an idea of how outParam and defaultInstance affect the process, but this is only tenable to keep up to date if I can test the resulting text against what Lean is actually doing. It seems that enabling the right traces, capturing them, and then validating them could be doable with this new .trace constructor, right?

@david-christiansen Indeed, what you describe is a bit easier now since there is a structured trace representation so you don't need to search for the regex \[([^]+)\] (.*) in the diagnostic to find traces.  The trace messages are however completely free-form, you still need string processing to check if ✅ apply I2 to R A C is the expected output or not.
The trace messages are also not stable, and will be improved over time.

I've changed the serialization of trace to an object to make easier to extend in the future.  See comment above about server-side search.


fix inViewport

Hey @svapreddy could you merge this pull request so that I can send you another one for an optimization of the isVisible() function?


]UnInstallPackage does not reduce the dependency file

In case ]UInInstallPackage removes a principal package this package needs to be removed from the dependency file.

Not reproducible.


craftworker, ebauchoir (logical-conflict)

This pull request was generated by the 'mq' tool

/trunk merge


Flask error occuring when starting/refreshing development server.

I have been using Flask for over an year now and from time to time I get odd import errors whenever I start my server and make a request to it, or the server restarts due to changes I have made to my source code.
The easiest way to reproduce this issue is to start up a development server with a mature code base (something that is bigger, not just a hello world app) and try to make a bunch of requests as it starts or as it restarts.
It should just put those requests in a queue or something, and not try to serve them until the code is done compiling.
Here is the output:
File "/usr/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/lib/python3.9/site-packages/engineio/middleware.py", line 74, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/lib/python3.9/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.9/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1948, in full_dispatch_request
    rv = self.preprocess_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 2242, in preprocess_request
    rv = func()
  File "/usr/lib/python3.9/site-packages/flask_request_params/params.py", line 111, in bind_request_params
    setattr(request, attr_name, get_request_params())
  File "/usr/lib/python3.9/site-packages/flask_request_params/params.py", line 106, in get_request_params
    setattr(request, '_request_params', Params(base_params))
  File "/usr/lib/python3.9/site-packages/flask_request_params/params.py", line 19, in __init__
    if request.files:
  File "/usr/lib/python3.9/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/python3.9/site-packages/werkzeug/utils.py", line 90, in __get__
    value = self.func(obj)
  File "/usr/lib/python3.9/site-packages/werkzeug/wrappers/base_request.py", line 514, in files
    self._load_form_data()
  File "/usr/lib/python3.9/site-packages/flask/wrappers.py", line 98, in _load_form_data
    from .debughelpers import attach_enctype_error_multidict
ImportError: cannot import name 'attach_enctype_error_multidict' from partially initialized module 'flask.debughelpers' (most likely due to a circular import) (/usr/lib/python3.9/site-packages/flask/debughelpers.py)

It is nothing that would cause a problem in production but it would be a good idea to look into it and check if there is an easy fix in case there is someone out there who thinks its something wrong with their code.
Environment:

Python version: 3.9
Flask version: 1.1.0
OS: Arch Linux.

In years of using the dev server with large code bases that restart while requests are active, I've never seen this issue, and there's not enough information to reproduce it. The dev server (which is part of Werkzeug, not Flask) uses Python's built-in HTTP server, there's nothing for us to change.

I also see this exact error occasionally on the first request after restarting the server. It doesn't seem like it will actually matter in production and I can't reliably reproduce it, but does anyone have any idea why this happens?

The dev server (which is part of Werkzeug, not Flask) uses Python's built-in HTTP server, there's nothing for us to change.

So this is a Werkzeug issue?


Organiser profile option missing

The organiser profile should be available from the organiser menu at "Billing Settings" at slug /profile, e.g. https://app-test.eventyay.com/tickets/control/event/conferenceorganiser/profile/
This URL currently shows an internal server error. If the feature is not available please implement it. Same applies to billing details. It shows internal error, e.g. https://app-test.eventyay.com/tickets/control/event/conferenceorganiser/billing/
Instead of implementing the feature in the tickets component, implement it in eventyay-common.

Hi @mariobehling,
this feature is not implemented yet in eventyay-tickets, please add more detail what information will be showed in Settings - Billing  settings, Billing - Billing, Billing - One-off fess? please give as much info about what will be showed as possible, then we can implement it.

Ok, understand. Then the best is if this feature gets implemented in eventyay-common.
Please follow the same approach as on eventyay-tickets (but dont implement the dropdown expansion). Keep it simple.

Make "Settings" expandable and show "Billing" below the Settings menu item.
Use the following text for the form boxes

Primary Contact Name
Please provide your name or the name of the person responsible for this account in your organization.
Primary Contact Email
We will use this email address for all communication related to your contract and billing, as well as for important updates about your account and our services.
Company or Organization Name
Enter your organization’s legal name.
Address Line 1
Street address or P.O. box.
Address Line 2
Apartment, suite, unit, etc. (optional).
ZIP Code
Enter your postal code.
City
Enter your city.
Country
Select your country.
Tax ID (e.g., VAT, GST)
If you are located in the EU, please provide your VAT ID. Without this, we will need to charge VAT on our services and will not be able to issue reverse charge invoices.
Preferred Language for Correspondence
Select your preferred language for all communication.
Payment Information
Please enter your credit card details for monthly fee payments.

For credit card information we can use Stripe (https://docs.stripe.com/terminal/features/saving-cards/overview). Best would be to enable the organiser to save several credit cards.


TikTok.com - New URL

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2019.11.28
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

for profiles, playlists and songs:
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.tiktok.com/@dannero', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2019.11.28
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg git-2019-11-09-bb190de, ffprobe git-2019-11-09-bb190de
[debug] Proxy map: {}
[generic] @dannero: Requesting header
WARNING: Falling back on generic information extractor.
[generic] @dannero: Downloading webpage
[generic] @dannero: Extracting information
ERROR: Unsupported URL: https://www.tiktok.com/@dannero
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpwy0zjfmc\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpwy0zjfmc\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpwy0zjfmc\build\youtube_dl\extractor\generic.py", line 3347, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.tiktok.com/@dannero

for videos:
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.tiktok.com/@markoterzo/video/6762207861918420230', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2019.11.28
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg git-2019-11-09-bb190de, ffprobe git-2019-11-09-bb190de
[debug] Proxy map: {}
[generic]<PHONE_NUMBER>918420230: Requesting header
WARNING: Falling back on generic information extractor.
[generic]<PHONE_NUMBER>918420230: Downloading webpage
[generic]<PHONE_NUMBER>918420230: Extracting information
[download] Downloading playlist: M A R K O  on TikTok
[generic] playlist M A R K O  on TikTok: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://v19.muscdn.com/99bdb82ff18a6720f09602b8de495194/5de1edf9/video/tos/maliva/tos-maliva-v-0068/af15acd0acb44d8bbf38744b1857631d/?a=1233&br=1547&cr=0&cs=0&dr=0&ds=3&er=&l=2019112922200201011511513317E33486&lr=tiktok_m&qs=0&rc=ank1NDVyPGY2cTMzZDczM0ApNmlkO2hoOTtnNzg4ZTU6aWdoaWwzZG0tLWBfLS1eMTZzczYwLzMvNGNhLV4tMTFfMWE6Yw=='
[download] Destination: M A R K O  on TikTok-6762207861918420230.unknown_video
[download] 100% of 1.36MiB in 00:02
[download] Finished downloading playlist: M A R K O  on TikTok

Description

There is a new format of TikTok URLs that was recently introduced. None of them work in youtube-dl.
The "Discover" page on TikTok.com (https://www.tiktok.com/discover) features the following - and none of them work:

a profile - https://www.tiktok.com/@dannero
a video - https://www.tiktok.com/@markoterzo/video/6762207861918420230
a hashtag - https://www.tiktok.com/tag/eggchallenge
a song - https://www.tiktok.com/music/Juicy-Feat-Tyga-6746479732637908998

There is already a PR #22838 that fixes this issue.

There is already a PR #22838 that fixes this issue.

No it does not.

@yan12125 this one is closed...

this one is closed...

Hmm not sure what you mean

@yan12125 I meant this one: #26094 sorry for the misunderstanding.

For anybody wondering: pipx install https://github.com/runraid/youtube-dl/archive/tiktokwatermarkless.zip --force to install a version by @runraid (#25895) which can get download tiktoks without a watermark!
(Inspect changes here. If you don't use pipx yet, give it a try! Plain pip should work fine too.)

@bpenven #22838 actually does fix this, I've built 2020.07.28 with the changes to extractors.py and tiktok.py patched in.
@skyme5 I'm not sure if it's just my setup but in TikTokBaseIE/tiktok.py this doesn't work, the url variable ends up empty:
   formats = []
    formats.append({
        'url': try_get(video_info, lambda x: x['video']['urls'][0], str),
        'ext': 'mp4',
        'height': height,
        'width': width
    })

I've solved that by doing this:
    formats = []
    video_urls = try_get(video_info, lambda x: x['video']['urls'], list)
    video_url = video_urls[0]
    formats.append({
        'url': video_url,
        'ext': 'mp4',
        'height': height,
        'width': width
    })

I don't really code in python so I'm not really sure why your code wouldn't work to begin with, it seems perfectly valid to me.
If referencing the 0th element of a list like that doesn't work you've done the same thing here as well:
'thumbnail': try_get(video_info, lambda x: x['covers'][0], str),
but you probably just end up with no thumbnail whereas url being empty causes the below:
WARNING: "url" field is missing or empty - skipping format, there is an error in extractor
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/bin/youtube-dl/main.py", line 19, in
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 474, in main
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 464, in _real_main
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2019, in download
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 863, in process_ie_result
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1589, in process_video_result

@dtaust, Extractor in #22838 doesn't work on Python 2 (because of lambda expressions). Try Python 3.

Doesn't that mean that pull won't be merged because the youtube-dl project supports python 2.6/2.7?
From the project page:

It requires the Python interpreter (2.6, 2.7, or 3.2+), and it is not platform specific.

[debug] Python version 2.7.16 (CPython)
This is the versioning on my test machine and with the small changes I made the lambda expressions work fine on 2.7.

I think this is due to comparison of expected_type in try_get being different for Python 2 and 3, in Python 2 it has the type unicode while Python 3 has str.
Removing the expected_type comparison resolves the problem.
formats.append({
    'url': try_get(video_info, lambda x: x['video']['urls'][0]),
...

Looks like Tiktok updated their strategies again.
By inspecting the requests, we can see that a sort of verification/captcha is going on.

Looks like Tiktok updated their strategies again.

That has nothing to do with download the video.

Still have issues with this (using ver 2020.07.28). Any other info I can add to help?
youtube-dl 'https://www.tiktok.com/@rogeliobucktronar/video/6820475866087197958'
[generic]<PHONE_NUMBER>087197958: Requesting header
WARNING: Falling back on generic information extractor.
[generic]<PHONE_NUMBER>087197958: Downloading webpage
[generic]<PHONE_NUMBER>087197958: Extracting information
ERROR: Unsupported URL: https://www.tiktok.com/@rogeliobucktronar/video/6820475866087197958

@lihuelworks, PR #22838, fixes this issue which is still not merged yet, you can install this PR described here or you can replace the youtube_dl/extractor/tiktok.py in installation folder  with this.

Looks like Tiktok updated their strategies again.

That has nothing to do with downloading the video.
I 'm not sure I'm getting you @llacb47 . The download fails because of the captcha but that has nothing to do with downloading the video?
What am I getting wrong?

@skyme5 have tried @lihuelworks url before posting? The very reason of my first post is that I was under the impression that it doesn't work either.
I just tried and still doesn't work. Am I doing something wrong?
Best regards

Looks like Tiktok updated their strategies again.

That has nothing to do with downloading the video.

I 'm not sure I'm getting you @llacb47 . The download fails because of the captcha but that has nothing to do with downloading the video?
What am I getting wrong?

Look at the source code, the video URL is literally there. There is no captcha bypass needed to get the video.

My bad (or not) I had to checkout the tiktok branch to make it work and didn't know it existed.
Snippet should be :
git clone https://github.com/skyme5/youtube-dl.git
cd youtube-dl/youtube_dl
git checkout origin/tiktok
python __main__.py url

My bad (or not) I had to checkout the tiktok branch to make it work and didn't know it existed.
Snippet should be :
git clone https://github.com/skyme5/youtube-dl.git
cd youtube-dl/youtube_dl
git checkout origin/tiktok
python __main__.py url

can it be used to download all user videos ?

@MohamedElashri yes it worked afterwards

It works! Hope it gets implemented soon, I'm really missing that functionality on mpv.

pipx install https://github.com/skyme5/youtube-dl/archive/tiktok.zip --force to install the version from #22838. (Inspect changes here. If you don't use pipx yet, give it a try! Plain pip should work fine too.)

pipx install https://github.com/skyme5/youtube-dl/archive/tiktok.zip --force to install the version from #22838. (Inspect changes here. If you don't use pipx yet, give it a try! Plain pip should work fine too.)

Tried this version using py -3 -m pip install and pip3, and no dice, still got unsupported url. Anyone on Windows tried this one?

The only problem with installing these versions is that you wouldn't get updates from main youtube-dl branch (right now YouTube support is broken in both I think) until @skyme5 @runraid rebase their branches. If you only need TikTok support, you'll be fine though.

The only problem with installing these versions is that you wouldn't get updates from main youtube-dl branch (right now YouTube support is broken in both I think) until @skyme5 @runraid rebase their branches. If you only need TikTok support, you'll be fine though.

But watermarkless fork of @runraid do not works now (

@Al-Muhandis It did work for older videos (before July 2020 I think) last time I checked :thinking:

@Al-Muhandis It did work for older videos (before July 2020 I think) last time I checked 🤔

Yes, I noticed. It doesn't work for new users (

@notpushkin the solution for that is to clone the current ytdl-org/youtube-dl repo and replace youtube_dl/extractor/tiktok.py with skyme5's one and then make/install.

@dtaust Makes sense, but I'm too lazy personally :') (which is the reason I came up with the one-liners above)

I merged the PR from @skyme5 into my fork youtube-dlc. You can use pip to install it or download binaries. There are various other fixes which aren't merged into main project so far.

@blackjack4494 why can't it be merged here?
The recent youtube-dl version was released just recently and the tiktok support is not fixed in it.

@VasiliPupkin256 that PR has been opened for nearly 324 days and still not merged by @dstftw.

Really confused about why tiktok remains broken for so long. Can Sergey @dstftw @remitamine or any of the other maintainers give us some visibility into what's happening and why? Let's make it easy, check one of the following:

[ ] The PR is not sufficient because of reason [x] so we won't merge until it is fixed
[ ] We favor PR [x] instead, and plan to merge that one when blocker [y] is resolved
[ ] We have our own solution in the works that will probably be finished in [x] weeks, please be patient until then
[ ] We don't really care about tiktok, and will be removing it from the list of youtube-dl's supported sites

Hi! I downloaded link https://v16-web-newkey.tiktokcdn.com/blabla in browser and I got error 403. After I added header "Referer: https://www.tiktok.com/blabla" in my curl downloader example and I can downloaded tiktok video. We need to add header Referer when you download media link

@sjmueller:

[ ] TikTok blackmail admins threatening to kill their uyghur families if they don't remove it from youtube-dl

For others as a workaround, here are curl and wget examples:
curl --output "video.mp4" --referer "https://www.tiktok.com/@user/video/123456789" -O "https://v16-web-newkey.tiktokcdn.com/......"

wget --output-document="video.mp4" --referer="https://www.tiktok.com/@user/video/123456789" "https://v16-web-newkey.tiktokcdn.com/....."

For others as a workaround, here are curl and wget examples:
curl --output "video.mp4" --referer "https://www.tiktok.com/@user/video/123456789" -O "https://v16-web-newkey.tiktokcdn.com/......"

wget --output-document="video.mp4" --referer="https://www.tiktok.com/@user/video/123456789" "https://v16-web-newkey.tiktokcdn.com/....."

Where does "https://v16-web-newkey.tiktokcdn.com/....." come from ?

Where does "https://v16-web-newkey.tiktokcdn.com/....." come from ?

With the TikTok video open in the browser, right click it and inspect the element. Look for an element with video in it and a link similar to the above. That link is the actual video being requested, which is served by TikTok's CDN. However, the CDN requires the referral of the original link as it won't give you access to the video without it, hence the above workaround.

If installing via sudo pip3 install youtube-dl, is there any way to patch the files afterwards? Seems like most of the source code goes to /usr/local/lib/python3.8/dist-packages/youtube_dl while the executable resides in /usr/local/bin/youtube-dl.
But if extractors.py and tiktok.py is updated, does the original /usr/local/bin/youtube-dl reference those changes? Wouldn't the executable need to be rebuilt?

@sjmueller It is just a script file pointing to main installation of youtube-dl, Yes, you can directly make changes to extractors.py, and overwriting tiktok.py but these changes are lost whenever you update your installation of youtube-dl.

Issue fixed as well in haruhi-dl, a libre fork of youtube-dl.

@selfisekai @sjmueller I think tiktok.com is using CAPTCHA to prevent scraping.

@skyme5 they do use some 'verify' tokens for API requests, and user profiles are unavailable on hdl for this reason, but the JSON video metadata are contained inside the HTML document, probably this thing: https://nextjs.org/docs/api-reference/data-fetching/getInitialProps
Also, for some reason, using the new URL scheme (https://www.tiktok.com/@puczirajot/video/6878766755280440578) didn't work, but https://www.tiktok.com/share/video/6878766755280440578 sets 3 cookies and does a redirect to the same page (+ ?source=h5_t, but adding it to the URL doesn't help). I guess these cookies are the reason for this.
My logs for downloading a video:
 laura@iino  ~/haruhi-dl   master ●  http_proxy=http://<IP_ADDRESS>:8080 python3 -m haruhi_dl --no-check-certificate "https://www.tiktok.com/@markoterzo/video/6762207861918420230" -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--no-check-certificate', 'https://www.tiktok.com/@markoterzo/video/6762207861918420230', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] haruhi-dl version 2020.11.16
[debug] Git HEAD: c55393ce4
[debug] Python version 3.9.0 (CPython) - Linux-5.8.17_1-x86_64-with-glibc2.30
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1
[debug] Proxy map: {'http': 'http://<IP_ADDRESS>:8080', 'https': 'http://<IP_ADDRESS>:8080'}
[tiktok]<PHONE_NUMBER>918420230: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068/449076372907418aa65c8235093bd867/?a=1988&br=3094&bt=1547&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1605642934&l=2020111713552701019021820147016369&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=ank1NDVyPGY2cTMzZDczM0ApNmlkO2hoOTtnNzg4ZTU6aWdoaWwzZG0tLWBfLS1eMTZzczYwLzMvNGNhLV4tMTFfMWE6Yw%3D%3D&signature=132b3ae5b648b1017bf3786a539199a2&tk=tt_webid_v2&vl=&vr='
[download] Destination: M A R K O -6762207861918420230.mp4
[download] 100% of 1.36MiB in 00:00

haruhi-dl tiktok extractor source code: https://git.sakamoto.pl/laudom/haruhi-dl/-/blob/master/haruhi_dl/extractor/tiktok.py

This new URL scheme is supported as of version 2020.11.29, see fb626c05867deab04425bad0c0b16b55473841a2. I was able to successfully download a video using a URL like https://www.tiktok.com/@user/video/123456789.
I think this issue should be closed in favour of opening specific issues like about code 403, or supporting other URLs like profiles. This new TikTok URL scheme is supported in general.

@VasiliPupkin256 that PR has been opened for nearly 328 days and still not merged by @dstftw.
youtube-dl with tiktok download support is available at blackjack4494/youtube-dlc

youtube-dlc doesn't work for me now

https://github.com/ytdl-org/youtube-dl/issues/23264
https://github.com/ytdl-org/youtube-dl/issues/28741
https://github.com/ytdl-org/youtube-dl/issues/29591
https://github.com/ytdl-org/youtube-dl/issues/29849

https://github.com/ytdl-org/youtube-dl/issues/30251


[BUG] Tooltip placement is wrong with CSS zoom

Describe the bug
If the CSS zoom value of target element is different than 1, the Tooltip shows in a wrong position. This happens even if positionStrategy is set to fixed.
Version of Package
v5.7.5
To Reproduce
Add style={{zoom: 1.5}} to the target element.
Expected behavior
It should show on the top. but offset is way wrong
Screenshots

Desktop (please complete the following information if possible or delete this section):

OS: Windows 11
Browser Chrome
Version 109
Frameworks Remix.run / React 18

Additional context
None

According to the MDN docs, zoom is non-standard and you should avoid using it.
Try doing transform: scale(1.5) instead. If you run into any problems using it then, feel free to open another issue.


Published page contributorship and authorship... things to consider

Take this published page as an example:-
https://msdn.microsoft.com/visualfsharpdocs/conceptual/symbol-and-operator-reference-[fsharp]
Things to consider re. contributorship and authorship:-

Clarifying the relationship of "Den Delimarsky" to the page.  E.g. identify Den as the publisher, maybe (?)
Include a reference to the primary and original author of the bulk of the text, where known (which I take to be Don Syme in this and most other cases) for the purposes of attributability.

As things stand, the page - to my eyes - seems to suggest to people who wouldn't know otherwise that Den is the primary author of the text, which I don't think is the case.

The reason why all pages have me as an author is because I was the one that originally published them on GitHub - it doesn't necessarily reflect the first author.
Ultimately this makes little difference in terms of attribution, as both myself and Don are part of the Microsoft org - as you can see for new docs (e.g. when @ploeh added his work), we properly display the author.


Make the drop zone the full container

I'm opening this issue because:
supporting information:

I use sortable version:
I tested in the following browser (incl. versions): Firefox Quantum (69.0 (64-bit)), Chrome (Version 76.0.3809.132)
Windows, OS X/macOS, or Linux?: MacOs

How can the issue be reproduce the problem?
Have two lists:

On with all items
One with one item and with min-height

drag one item to the other list it needs to be precisely set to just under the other item.
What's the feature?
Add the possibility to drop an item any where in the container and be captured.
What problem is the feature intended to solve?
The problem described above
Is this feature similar to an existing feature in another project?
Not that I know of

It should be like this. Where exactly can you not drop it? Are you talking about an empty or a sortable with items in it already?

@lukasoppermann I have added a video of the issue.

Ahh, I see. You should be easily able to fix this, by adjusting the min-height of the ".sortable" you can adjust the dropzone for the first item.
However afterwards you will need to drop an item onto an item.
If you feel like sending a PR to allow for dropping items onto the sortable with items inside and always adding them at the end, that would be much appreciated.

I wouldn't mind doing a pr but could just poo t which file is the location
where you decide an item has to be dropped on another item and not on the
div
On Thu, 19 Sep 2019, 23:12 Lukas Oppermann<EMAIL_ADDRESS>wrote:

Ahh, I see. You should be easily able to fix this, by adjusting the
min-height of the ".sortable" you can adjust the dropzone for the first
item.
However afterwards you will need to drop an item onto an item.
If you feel like sending a PR to allow for dropping items onto the
sortable with items inside and always adding them at the end, that would be
much appreciated.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/lukasoppermann/html5sortable/issues/509?email_source=notifications&email_token=AGX234TUKEVBJKJGTZM3BL3QKOJGNA5CNFSM4IXA2722YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7D2KAQ#issuecomment-533177602,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGX234UPUPUSBEZ2F2OGXRLQKOJGNANCNFSM4IXA272Q
.

Hey, you should find what you need in the main html5sortable.ts file.

@crimson-med let me know if you need any more support.

@lukasoppermann Sorry th past few weeks and this weeks at work are pretty busy i'll try to have a look at it over the weekend.

Hey @crimson-med, no worries, looking forward to your contribution when you have the time. 👍

Is this a dupe of #172?

Thanks @erickoledadevrel, closing this as it is a duplicate of #172
If either @erickoledadevrel or @crimson-med could send a PR, I would really appreciate it and merge it asap. Thanks.


Issues with gpu usage

Hi,
Thank you for providing such a great tool. I am pretty new in python so my problems might be so dumb.
I want to use cell2location with my data but I could not figure out how to install it. I installed Ubuntu 20.04.4 LTS, I installed miniconda, and installed cell2location. The problem with Ubuntu is it does not see my Nvidia card, I tried different solutions but I could not fix it.
I am using Windows10, where my graphic card is seen, I created an isolated environment at anaconda. At mod = RegressionModel(adata_ref) step, I am getting AssertionError: Torch not compiled with CUDA enabled error. I again tried different solutions for this and I could not fix it.
I tried google colab, but I am getting installing and importing errors. I checked your issues section, found #136 and I could not make it fixed with your last solution.
I really need to use the package urgently. I am sure that I am missing some basics, but if you can lead me to use, I would be appreciated.
Thank you very much in advance.

Hi
I can't give an exhaustive reply now but to use the GPU on Ubuntu you need
to install the Nvidia GPU drivers, CUDA and a few other things - but I
never did it myself so can only give limited recommendations. This is not a
python issue.
If you have sudo access it should be relatively straightforward. You can
find which apt-get packages you need by looking at cell2location docker
recipe and at Nvidia container that we start with. In principle you can
build cell2location docker container on your system (that way it will have
latest cell2location).
On Mon, 4 Apr 2022, 20:59 MertD, @.***> wrote:

Hi,
Thank you for providing such a great tool. I am pretty new in python so my
problems might be so dumb.
I want to use cell2location with my data but I could not figure out how to
install it. I installed Ubuntu 20.04.4 LTS, I installed miniconda, and
installed cell2location. The problem with Ubuntu is it does not see my
Nvidia card, I again tried different solutions but I could not fix it.
I am using Windows10, where my graphic card is seen, I created an isolated
environment at anaconda. At mod = RegressionModel(adata_ref) step, I am
getting AssertionError: Torch not compiled with CUDA enabled error. I tried
different solutions for this and I could not fix it.
I tried google colab, but I am getting similar errors with #136
https://github.com/BayraktarLab/cell2location/issues/136 and I could
not make it fixed with your last solution.
I really need to use the package urgently. I am sure that I am missing
some basics, but if you can lead me to use, I would be appreciated.
Thank you very much in advance.
—
Reply to this email directly, view it on GitHub
https://github.com/BayraktarLab/cell2location/issues/140, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFMFTV2YB6EZFU4S53JEPQDVDNCZZANCNFSM5SQS23RQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Maybe @yozhikoff has more detailed recommendations because he used cell2location on his local PC.

@MertDemirdizen

If you want to use cell2locafion (or pytorch in general) on Ubuntu, you need to install NVIDIA graphics card drivers. You can use this tutorial for example. You don't need to install CUDA manually, it is included in the pytorch distribution.

In the Windows case it seems you have CPU-only pytorch version. You can install the GPU-version from here.

Both options of course work only if you have an NVIDIA graphics card.

Thank you for your quick and helpful replies!
I have NVIDIA graphic card, I tried to install drivers in Ubuntu but they did not work. I will both try this tutorial for Ubuntu and install pytorch version for Windows.
I will let you know about the outcomes.

I could not handle for Ubuntu but I installed pytorch version for Windows and it is working!
Thank you so much for your help.


Add test corpus from gists

@devoncarew
As discussed with @danrubel it would be useful to have a corpus of public dart files that we can run the fuzz tester on ahead of each release. This adds a corpus based on the sample gists.

LGTM

Re: the failures (https://travis-ci.org/dart-lang/dart-services/builds/242044827#L389), I'd suggest adding an excludes line in the analysis options to exlcude the corpus directory from analysis.
+1 to the corpus - sounds great. These files are going to get out of sync with the samples we ship however. I'd recommend adding a grind task to git clone the (10?) samples we ship with and move their dart source into the corpus. We can run this script manually - it'll be much easier to keep the corpus up to date.
lgtm w/ the analysis fix

(I did update those two samples to declare return types)

Thanks @devoncarew, I'll follow the grind task approach


Version Packages

This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.
Releases
<EMAIL_ADDRESS>Patch Changes

#499 6cc9437 Thanks @threepointone! - partysocket: add subpath support
This patch lets you add path: string to new PartySocket({...}) and point to a subpath in a room. This is a client side analog to the recent subpath support we added to multiparty .fetch()/.socket()

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Performance optimize directory structure in ProgramData

Issue description
If you request a certificate there are a lot of files generated inside the programdata folder. I have seen all files lay in the same folder for all certificates. I strongly suggest to create subdirectories with hostnames. Certbot also has such a structure.
The reason for doing this may not seen in small installations, but if you have a large installation with 5.000-10.000 certificates you will run into several performance issues.

Explorer will hang
Backups will hang on directory listing
This may end in inability to open the directory.

Solution/Workaround:

it is recommended to create no more than 60.000 files per folder.
Create one folder per domain and place the files in the subfolder
This will speed up backup, explorer and so on.

Steps to reproduce
Create 60.000-80.000 files in one folder. Try to open the folder, make a dir listing.
Client version: <IP_ADDRESS>
Windows version: Windows 2016
Relevant part of log file: none

This would be breaking some peoples workflows who don't use scripts or installers to move certs around, but rely on them directly appearing into the configured CertificatePath, so it'd have to be a optional feature.

Or make the hostname structure the new default and optionally allow the previous flat directory. Explain in the wiki why the new structure is better.

That would be the other option, but I prefer to keep the whole 1.x line as backwards compatible as possible so that people can easily upgrade and get the latest bug fixes without having to worry about having to change their scripts or processes.
For a future 2.x we can of course choose other (more sensible) defaults.

In version 2.0 the number of files generated in the certificate path per renewal has been reduced from 8 to 2, meaning that file system related bottlenecks should be reduced by 4x.

How about adding the hostname subfolder now?

It's possible, but are you really facing this problem or asking for puristic reasons? With the 2.0 changes you could do 10k renewals with 20k files in a folder, leaving plenty of headroom until you hit that 60-80k limit that you ran into before. How many renewals do you actually have?

As I know you keep all files forever. You never delete any older/expired/no longer used certificate files. That means we have several new files every 2 months and this is growing. That means 10 or 100 certificates can also easily grow over 60.000 over time. We have a lot more.

@alexhass a simple option is to schedule a task to cleanup files older than 180 days (for instance) e.g.: https://stackoverflow.com/questions/51054/batch-file-to-delete-files-older-than-n-days that way you are in control of the cleanup you need.

As I know you keep all files forever. You never delete any older/expired/no longer used certificate files. That means we have several new files every 2 months and this is growing. That means 10 or 100 certificates can also easily grow over 60.000 over time. We have a lot more.

This is not entirely true. When certificates are renewed the old cache files are overwritten. The only way the cache folder can get out of hand is when you have a high churn rate, e.g. new renewals being set up and old renewals being cancelled.  Even if you have 10.000 certificates you'd have 3 full cycles of all of them being removed and replaced.
If this is really the problem that you want to address, then dividing the cache files among sub folders is not the solution, because it would just delay the inevitable. The solution would be to simply delete the old files rather than merely warning about them.

The next build will have a new non-default setting that makes the program delete files older than 120 days from the certificate cache.

The new setting to clean up older files was released in 2.0.4. I'm going to close this issue for now because I'm not aware of anyone actually handling more than a couple of thousand of renewals, which is still very comfortably below these file system limits.
If someone actually wants to manage tens of thousands of certificates on a server then it's probably wise to have some separate discussions anyway.


Portmaster is not installing ports

Device: RG351V
Latest ArkOS 2.0 release installed -> Portmaster is  installing ports without any problems
(Same with ArkOS Update 07/30/2022)
ArkOS Update to 08/22/2022 -> Portmaster showed message when trying to install a port:
Files could not be installed please check Internet connection

Are you sure your device is connected to the internet when you try to install ports?  If so, try reinstalling portmaster.

Yes, the device is connected to the internet, even checking again the ArkOS Update function is working and some Ports are downloading, but right before installing it stops with this message.
Reinstalling Portmaster (Replacing folder and .sh file) showed the same error message.
This process was repeated six times with different SD cards and two different RG351V devices as well with 3x fresh downloaded img file (ArkOS 2.0 for RG351V) and two different internet network and the error could be reproduced.

This issue has now been resolved with the latest update to PortMaster.  Just update through the PortMaster menu or download the latest PortMaster.zip file from the PortMaster location.  Thanks for reporting the issue.


Add support for 2.0 health API to 1.x

Proposal:
Make 1.x support the 2.0 health API.
Together with #16898 it will allow users build robust system that will be prepared to smoothly upgrade to 2.
1.x Ping API:
https://docs.influxdata.com/influxdb/v1.7/tools/api/#ping-http-endpoint
2.0 Health API:
https://v2.docs.influxdata.com/v2.0/api/#tag/Health
Proposed Changes:
Add a health endpoint to handle the v2.0 server health check protocol.
InfluxDB v2 implementation: https://github.com/influxdata/influxdb/blob/master/http/health.go

Closed by #17252


New Design for Vue 3 JSX

Implementation Detail and new Design to discuss

vModel
slot
directives
modifiers
...

ref https://github.com/vuejs/jsx/issues/141

About '.sync' modifier
In vue-next, we should use v-model instead of .sync modifier.
<MyComp v-model={[value, "foo"]} />

But I think this is not suitable for TSX in some points.

TypeScript can't check type of value is assignable to foo or not.

When foo is defined as required, TS2322 (Property 'foo' is missing in ...) will occur

I guess it is not impossible to solve them by changing type definitions, but it will makes type definitions too complex.
Simple solution is giving up v-model.
<MyComp foo={value} on={{ "update:foo": v => value = v }} />

Is there any other (more smart) solutions ?

<MyComp foo={value} on={{ "update:foo": v => value = v }} />

Will you change type definitions when giving up v-model like above.
interface Props {
  foo?: string;
  on?: {
    'update:foo': (val: string) => void;
  }
}

Or
interface Props {
  vModel?: [string, 'foo']
}