verify that we can deploy multiple versions of chaise

Looks like the only "hardcoded" relative path for ermrestJS is in our Makefile for chaise.

The following variables, ERMRESTJS_RT_DIR and ERMRESTJS_BLD_DIR, need to be made conditional in the Makefile. i.e. ERMRESTJS_RT_DIR?=../../ermrestjs.
Better documentation for the 2 environment variables would make it more obvious how this can be done in chaise. These 2 environment variables can be defined for each deployment so the location of ermrestJS relative to chaise doesn't matter at all.
One change needs to be made in ermrestJS as well so that the JS variable we use for attaching the vendor scripts in the proper order is no longer purely relative to the server root.

I am closing this issue..


Add placeholder to the form

Add placeholder text for form input , to provide context on what is to be entered as input

Hello, I would like to work on this issue. Could you please assign it to me?

Here you go @rutvikk05 . 😄

@rutvikk05 are working on this issue?

Hey @Anurag30112003, yes I'm currently working on it. Thanks for assigning it to me :D

update : sent you the PR


[Feature] support array_agg(a order by b, c...)

What type of PR is this：

[ ] BugFix
[x] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #https://github.com/StarRocks/starrocks/issues/17638
Problem Summary(Required) ：

let aggregation framework support order by
let array_agg support more than one columns
can't support nested types now.
main idea: input columns result in intermediate result: struct{array[col0], array[col1], array[col2]... array[coln]}, return ordered array[col0']

mysql> select array_agg(a order by name), array_sortby(array_agg(a), array_agg(name)) from t;
+--------------------+---------------------------------------------+
| array_agg(a, name) | array_sortby(array_agg(a), array_agg(name)) |
+--------------------+---------------------------------------------+
| [2,11,1,1,1]       | [2,11,1,1,1]                                |
+--------------------+---------------------------------------------+
1 row in set (0.08 sec)

mysql> select array_agg(a order by name desc), array_sortby(array_agg(a), array_agg(name)) from t;
+--------------------+---------------------------------------------+
| array_agg(a, name) | array_sortby(array_agg(a), array_agg(name)) |
+--------------------+---------------------------------------------+
| [1,1,1,11,2]       | [2,11,1,1,1]                                |
+--------------------+---------------------------------------------+
1 row in set (0.07 sec)
mysql> select array_agg(a order by name, pv desc nulls last) from t group by pv;
+-----------------------------------------+
| array_agg(a ORDER BY name ASC, pv DESC) |
+-----------------------------------------+
| [1]                                     |
| [11]                                    |
| [2]                                     |
| [1]                                     |
| [1]                                     |
+-----------------------------------------+
5 rows in set (0.16 sec)

mysql> select array_agg(null) from t;
+----------------------------+
| array_agg(NULL)            |
+----------------------------+
| [null,null,null,null,null] |
+----------------------------+
1 row in set (0.06 sec)

mysql> select array_agg(null order by null) from t;
+-----------------------------------+
| array_agg(NULL ORDER BY NULL ASC) |
+-----------------------------------+
| [null,null,null,null,null]        |
+-----------------------------------+
1 row in set (0.05 sec)

mysql> select array_agg(a order by null) from t;
+--------------------------------+
| array_agg(a ORDER BY NULL ASC) |
+--------------------------------+
| [2,11,1,1,1]                   |
+--------------------------------+
1 row in set (0.06 sec)

mysql> select array_agg(a order by 1) from t;
+-----------------------------+
| array_agg(a ORDER BY 1 ASC) |
+-----------------------------+
| [1,1,1,11,2]                |
+-----------------------------+

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto backported to target branch

[x] 3.0
[ ] 2.5
[ ] 2.4
[ ] 2.3

[FE PR Coverage Check]
:disappointed: fail : 43 / 54 (79.63%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/catalog/AggregateFunction.java
6
10
60.00%
[92, 102, 324, 327]

:large_blue_circle:
com/starrocks/analysis/FunctionParams.java
11
15
73.33%
[150, 151, 167, 168]

:large_blue_circle:
com/starrocks/sql/analyzer/ExpressionAnalyzer.java
16
19
84.21%
[855, 856, 857]

:large_blue_circle:
com/starrocks/sql/parser/AstBuilder.java
4
4
100.00%
[]

:large_blue_circle:
com/starrocks/catalog/FunctionSet.java
3
3
100.00%
[]

:large_blue_circle:
com/starrocks/analysis/FunctionCallExpr.java
3
3
100.00%
[]

run starrocks_fe_unittest

run starrocks_fe_unittest

run starrocks_be_unittest

[FE PR Coverage Check]
:disappointed: fail : 71 / 124 (57.26%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/analysis/FunctionParams.java
14
47
29.79%
[102, 103, 104, 105, 106, 108, 113, 114, 115, 116, 117, 119, 151, 152, 153, 154, 155, 156, 157, 158, 160, 179, 180, 181, 182, 183, 186, 191, 192, 207, 208, 224, 225]

:large_blue_circle:
com/starrocks/analysis/FunctionCallExpr.java
11
27
40.74%
[247, 248, 249, 250, 271, 272, 273, 274, 304, 305, 306, 307, 315, 316, 317, 417]

:large_blue_circle:
com/starrocks/catalog/AggregateFunction.java
6
10
60.00%
[92, 102, 327, 330]

:large_blue_circle:
com/starrocks/sql/parser/AstBuilder.java
6
6
100.00%
[]

:large_blue_circle:
com/starrocks/catalog/FunctionSet.java
3
3
100.00%
[]

:large_blue_circle:
com/starrocks/sql/analyzer/AstToStringBuilder.java
7
7
100.00%
[]

:large_blue_circle:
com/starrocks/sql/analyzer/ExpressionAnalyzer.java
24
24
100.00%
[]

maybe implement array_agg order by as array_order_by(array_agg(), order by xxxx) is a better idea ?

maybe implement array_agg order by as array_order_by(array_agg(), order by xxxx) is a better idea ?

has analyzed at the first time https://starrocks.feishu.cn/docx/PtmtdbsuIog9nAxCTvccLZRqnXc.
which can't benefit from sorted agg in the future, as well as need to change array_orderby().

@mergify backport branch-3.0


Azure Resource Group Deployment Task Fails on AzureChina Cloud

Problem Statement
Azure Resource Group Deployment Task Fails When Targeting a Subscription in AzureChina Cloud
Environment
VSTS Environment

agrifirm.visualstudio.com
eCommerce - Website Agrifirm.com NL
CI-master-release-definition
Release-CI-master-build-definition_1.1.18095.01-agrifirmchina-1

Hosted Agent Queue Name
Hosted
Issue Description
Azure Resource Group Deployment Task Fails When Targeting Azure China Subscription

Can you verify your SPN via below mentioned steps

Using Azure RM endpoint dialouge

Using PowerShell script.

$servicePrincipalId = "<ServicePrincipalID>"
$servicePrincipalKey = "<servicePrincipalKey>"
$tenantId = "<tenantId>"
$azureSubscriptionId = "<azureSubscriptionId>"

$securePassword = ConvertTo-SecureString $servicePrincipalKey -AsPlainText -Force
$psCredential = New-Object System.Management.Automation.PSCredential ($servicePrincipalId, $securePassword)

Write-Verbose "Add-AzureRMAccount Started" -Verbose
$azureAccount = Add-AzureRMAccount -ServicePrincipal -Tenant $tenantId -Credential $psCredential -EnvironmentName AzureChinaCloud
$azureAccount
Write-Verbose "Add-AzureRMAccount Completed" -Verbose

Write-Verbose "Select-AzureRMSubscription Started" -Verbose
$newSubscription = Select-AzureRMSubscription -SubscriptionId $azureSubscriptionId -TenantId $tenantId
$newSubscription
Write-Verbose "Select-AzureRMSubscription Completed" -Verbose

I already did and it appears fine. I see the problem only in ARM Task. The App Service Deployment Task just runs fine

<EMAIL_ADDRESS>+31610405650
We can talk now 👍

@irajbalakrish
Posting observation during the call for future reference.

SPN was scoped to a resource group. When Azure RG task is executed with difference RG it's failing with 403 unauthorized.
In case of 403 error, error object doesn't contain a message and it's getting printed as an object in logs. ( We have to fix this code for logging error from our end )
If a resource type mentioned in the template doesn't exist in the cloud. In VSTS, we got an error and fail on it but visual studio got the same error but it doesn't fail on error.  ( Will discuss on this with the team)

@irajbalakrish
Apologize for the delay in response to the thread
As per discussion with the team, If you select "Continue on error" then Azure RG task will create all other resources except one that doesn't exist in the Azure Cloud. Can you please verify same behavior.
Added @ds-ms  any queries regarding Azure RG task.

@irajbalakrish are you still facing this issue?

The situation is fixed The outcome of the investigation is mentioned by Ajay in the prev thread

Closing the issue then.


Transaction delimiters not working

This is my global configuration

And this is the configuration for the data source

If I exec this script using the script execution (Alt + X) works:
CREATE TABLE AA (id NUMBER PRIMARY KEY )
/
INSERT INTO AA VALUES (1)
/
INSERT INTO AA VALUES (2)
/

Executing the same on the sql server data source for which I changed the configuration as above leads to an error

Incorrect syntax near 'GO'

CREATE TABLE AA (id NUMERIC PRIMARY KEY )
GO
INSERT INTO AA VALUES (1)
GO
INSERT INTO AA VALUES (2)
GO

Can't reproduce..
BTW default preconfigured SQL Server driver already has GO as delimiter.

There isn't any activity about this ticket very long time. It will be closed. However, if you think that it is actual just let us know.


chore(mempool_node): remove unused errors

Stack:

#1111
#1110
#1109
#1103
#1102
#1101
#1100
#1099
#1098
#1097
#1096
#1090
#1089
#1088 ⬅
#1058

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #1096


Tripott/bugs reported by chris 22

contact URL changed to hyper.io
getting started button bg hover color defaulted to purple and set up as Button prop export let hoverBgColor ='purple'
FAQ customer support team link changed to https://github.com/hyper63/hyper63/discussions.

ROADHOUSE!


Adding chart-boxplot to project gives an error with chartjs-chart-geo

I have an angular app that uses several chartjs plugins, including chart-geo.
When adding this plugin via
npm install --save chart.js @sgratzl/chartjs-chart-boxplot
My existing code that starts to produce errors that suggest the chart-geo plugin is not installed.
Errors:
TS2820: Type '"bubbleMap"' is not assignable to type 'keyof ChartTypeRegistry'. Did you mean '"bubble"'
Context

Version: 4.4.0 of boxplot, 4.3.1 or chart-geo and 4.4.3 of chart.js
Browser: chrome/mac

Can these two plugins co-exist in the same project?
Any help would be appreciated.

Relevant pieces of code
I define chartType in a 'parent' class as follows:
public chartType: ChartType;
then in a child class that extends from the parent I set it as:
this.chartType = 'bubbleMap'; /// doesn't work after installing the boxplot.
this.chartType = 'violin'; // works

closed in error

cannot replicate: https://codesandbox.io/p/sandbox/multi-chartjs-plugins-8n5y6j?file=%2Fsrc%2Findex.ts%3A37%2C13


simplify actors

Instead of actor client methods passing data to the server via the actor's channel, and the server in turn invoking server methods with that data, we get the client methods to pass a closure of the server method via the actor's channel. This reduces boilerplate a fair bit.
Closes #475.

@awh review, please.


"Skipping the delay kernel" warning emitted thousands of times.

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No - I get other errors when I install tf-nightly.
Source
source
TensorFlow version
2.17.0
Custom code
Yes
OS platform and distribution
Rocky Linux 9 and unknown
Mobile device
No response
Python version
3.12.5 and unknown
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
Cuda version 12.0 and unknown
GPU model and memory
No response
Current behavior?
I get an alarming number of warnings about skipping of a delay kernel.
This occurs on a pretty standard install of tensorflow that I use for a project. (I have been unable to reproduce it with tf-nightly because I get crashes due to incompatible shapes in some of my code.) But I have good news for reproducibility! The same log message is also created in the examples in the TF documentation, so however TF is built to generate the guides on the website (see below for links) also triggers this behavior. (This is why some of the versions are unknown - I haven't tracked down exactly how the docs are built.)
This occurs in training and in predicting, and it fills the log files with warnings that I cannot disable. I've pasted one in the section below. A typical run of my program produced just shy of 150,000 repeated warnings about skipping the delay kernel.
This warning does not seem to listen to TF_CPP_MIN_LOG_LEVEL, and it also still gets emitted after I add
tf.get_logger().setLevel("ERROR") and warnings.simplefilter("ignore"). The log file attached below includes all three of those attempts to suppress the messages.
I don't know what the solution is to these warnings, and I honestly don't even know if I'm doing anything wrong.
Is it something that I need to fix? If so, where do I specify that I would like to include delay kernels when building and training a model?
Is it something that should be ignored in every case? If so, why is it printed at all? Given that the documentation, which I would assume is using tensorflow correctly, produces these warnings, I'm leaning toward this being the case. In this instance, how do I disable the message?
Where does the warning even come from? Commit 10df33f84983f1f789f1215fc766e12ad046ff7b moved the warning message to a new file (so it's now in third_party/xla/xla/stream_executor/cuda/cuda_executor.cc instead of gpu_timer.cc), but without a traceback I can't tell which call to the function is triggering the bad behavior.
I would build my own copy of TensorFlow to debug further, but I don't have root access to a machine at work and building without that has been unsuccessful.
Standalone code to reproduce the issue
The message is present in many of the examples on the TF documentation, and often fills output boxes that are intended to contain other information, making reading the documentation rather frustrating.

https://www.tensorflow.org/guide/function

https://www.tensorflow.org/tutorials/quickstart/advanced

https://www.tensorflow.org/tutorials/generative/adversarial_fgsm

https://www.tensorflow.org/tutorials/images/transfer_learning

https://www.tensorflow.org/tutorials/generative/autoencoder

https://www.tensorflow.org/tutorials/audio/simple_audio

https://www.tensorflow.org/tutorials/generative/cyclegan

https://www.tensorflow.org/tutorials/generative/style_transfer

Relevant log output
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723232760.037824 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723232760.038491 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723232760.039293 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.060998 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.060998 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.060998 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.061204 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.061207 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.061210 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.308930 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.309064 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.309108 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.762708 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.762862 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232760.762911 1704933 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723232761.182515 1705056 service.cc:146] XLA service 0x7f67c8003f20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1723232761.182536 1705056 service.cc:154]   StreamExecutor device (0): NVIDIA GeForce RTX 3090 Ti, Compute Capability 8.6
I0000 00:00:1723232761.265672 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232761.265826 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232761.265879 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232761.742874 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232761.743078 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232761.743152 1704934 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723232762.175289 1705161 service.cc:146] XLA service 0x7f67d0003290 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1723232762.175319 1705161 service.cc:154]   StreamExecutor device (0): NVIDIA GeForce RTX 3090 Ti, Compute Capability 8.6
I0000 00:00:1723232762.269671 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232762.269891 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232762.269936 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232762.811310 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232762.811427 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1723232762.811474 1704937 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723232763.316489 1705273 service.cc:146] XLA service 0x7f67f80043a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1723232763.316524 1705273 service.cc:154]   StreamExecutor device (0): NVIDIA GeForce RTX 3090 Ti, Compute Capability 8.6
I0000 00:00:1723232763.328407 1705056 device_compiler.h:188] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
I0000 00:00:1723232764.801232 1705161 device_compiler.h:188] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
W0000 00:00:1723232764.833870 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.852954 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.853403 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.853845 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.854270 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.855588 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.856585 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.857226 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.857815 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.858271 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.859216 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.859872 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.864199 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.865096 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.866303 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.866899 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.867695 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.868588 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.869332 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.870090 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.870974 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.872071 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.872808 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.874219 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.874952 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.876521 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.877960 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.879262 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.879764 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.880946 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.882610 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.883286 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.884262 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.885159 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.938520 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.939871 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.941237 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.942563 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.944529 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.946130 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.947458 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.949099 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.950547 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.951902 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.953549 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.955035 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.956444 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.959359 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.961647 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.963095 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.964935 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.968955 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.970589 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.972500 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.974051 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.975503 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.977670 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.979426 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.981071 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.982999 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.984944 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.986547 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.988105 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.989497 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.991030 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.992826 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232764.994553 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.006351 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.008144 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.010465 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.012574 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.014353 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.015983 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.017785 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.019209 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.021683 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.023968 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.026816 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.030096 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.033443 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.036090 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.038314 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.040368 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.043280 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.045456 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.048798 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.051494 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.053658 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.055930 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.058870 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.063870 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.066426 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.069411 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.071951 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.074379 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.076916 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.079077 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.081423 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.083426 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.086207 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.095480 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.097701 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.099533 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.102417 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.104760 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.106645 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.109518 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.112559 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.114620 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.117068 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.119439 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.121426 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.125844 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.128501 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.129907 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.132639 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.135037 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.136905 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.140405 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.142656 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.144536 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.147934 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.150320 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.152360 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.158254 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.160801 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.162895 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.166604 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.168923 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.170685 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.174693 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.177255 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.179223 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.189563 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.192249 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.194193 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.196283 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.199712 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.201671 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.204049 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.207716 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.209591 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.212083 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.215755 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.220159 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.222598 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.225472 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.227479 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.229706 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.231615 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.232971 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.235788 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.237162 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.239782 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.242132 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.244283 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.248891 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.252746 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.254671 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.258255 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.260796 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.262522 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.265756 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.268230 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.270198 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.273803 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.283381 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.286579 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.288364 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.290891 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.293585 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.295439 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.297559 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.300255 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.302112 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.304314 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.307786 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.310909 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.315045 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.318333 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.320218 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.322896 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.326314 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.328383 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.331175 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.333914 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.335733 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.338057 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.339842 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.344540 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.346982 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.348444 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.351905 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.353990 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.356203 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.359343 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.361575 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.363981 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.367310 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.376307 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.378576 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.380323 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.383666 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.385889 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.387699 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.391181 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.393651 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.395483 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.398535 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.400612 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.402488 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.407495 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.409849 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.411846 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.415162 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.417432 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.419402 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.423073 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.425380 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.427480 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.431200 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.433669 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.436550 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.441645 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.444023 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.446073 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.448074 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.449769 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.451185 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.453771 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.455250 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.465666 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.469818 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.472128 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.475028 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.476893 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.479211 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.482098 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.483886 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.486365 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.489674 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.491510 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.494149 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.498504 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.501810 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.504348 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.506943 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.508713 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.510955 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.513580 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.515384 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.517688 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.521208 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.523155 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.525427 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.531938 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.533916 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.536442 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.539925 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.541828 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.544377 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.547393 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.549334 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.558568 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.563588 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.565909 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.568067 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.570669 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.572723 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.574810 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.577710 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.579802 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.581933 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.584333 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.586309 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.588492 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.592051 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.595625 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.597937 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.600781 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.602961 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.605194 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.607698 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.609791 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.612048 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.614991 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.617124 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.619411 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.623481 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.627261 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.629473 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.632518 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.634773 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.637166 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.639076 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.647533 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.649634 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.651332 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.654803 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.658555 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.660239 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.663157 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.665295 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.667099 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.670007 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.672266 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.674068 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.676654 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.679067 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.680874 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.683038 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.687995 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.689779 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.692386 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.694990 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.696832 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.698930 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.701470 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.703326 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.706019 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.708725 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.710649 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.713118 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.718185 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.720031 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.726136 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.727756 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.729836 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.730946 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.733076 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.735545 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.737739 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.739550 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.741139 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.742954 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.745182 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.747946 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.751953 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.753967 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.755742 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.758301 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.760613 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.762080 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.764203 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.767168 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.769898 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.773173 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232765.776661 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.099818 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.103232 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.110863 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.117101 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.118906 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.120300 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.125715 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.134527 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.140363 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.146259 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.147661 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.149023 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.156721 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.166398 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.173983 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.181315 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.182520 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.183984 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.184996 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.187285 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.196973 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.200989 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.219062 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.237505 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.248860 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.250557 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.252136 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.253462 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.257982 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.259694 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.261177 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.263185 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.264906 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.266256 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.268344 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.270020 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.271417 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.273704 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.275516 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.276927 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.279074 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.280939 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.282233 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.284802 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.291359 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.293467 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.302223 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.311843 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.327655 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.330126 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.331492 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.332706 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.336856 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.338229 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.339504 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.343685 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.345186 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1723232766.346497 1705064 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
I have not included the remaining 1,337 warnings.

Thank you for reporting the issue. This is a known issue where other issues are still open and developers are working on the same.
I request you to take a look at this https://github.com/tensorflow/tensorflow/issues/62075 and where a similar issue has been proposed and it is still open. Also I request to follow the similar issue which has been proposed to have the updates on the similar issue.
https://github.com/tensorflow/tensorflow/issues/70947
Thank you!

This occurs in training and in predicting, and it fills the log files with warnings that I cannot disable. I've pasted one in the section below. A typical run of my program produced just shy of 150,000 repeated warnings about skipping the delay kernel.

Got the same problem, partially resolved by this:
#!/usr/bin/env python
# -*- coding: utf-8 -*-

# Suppress tensorflow spam, see https://github.com/tensorflow/tensorflow/issues/26348
# Not the complete fix, cuda_executor.cc still ignores all these settings
# and spams something successful NUMA node read
# but probably it's best that can be done without redirecting stderr

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
import logging
logger = tf.get_logger()
logger.setLevel(logging.ERROR) # or logging.INFO, logging.WARNING, etc.

# Actual app starts here:

import argparse
import pathlib
import pprint
from enum import Enum
from tqdm import tqdm
...

So now instead of 1343 lines of Skipping the delay there's just 10 spam lines from the one line in cuda_executor.cc.
Still very spammy but better than 1500 repeats of the warning.
similarity ~/tmp/images/2024-08-18/
  0%|                                                                                | 0/280 [00:00<?, ?it/s]WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1724013025.313366 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.339050 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.339224 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.340675 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.340776 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.340832 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.391595 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.391736 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1724013025.391807 3889312 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 280/280 [01:24<00:00,  3.32it/s]

Actually, the above suggestion is not a fix. Setting TF_CPP_MIN_LOG_LEVEL = 3 logs only error and fatal messages. Similar for setLevel(logging.ERROR) only error is logged. So, no change.
In my case the 'Skipping the delay kernel' warning occurs numerous times during both training and subsequent build.
We await a response.

I am also affected by this issue and hope to get a way to clean my log from hundreds if not thousands of these messages 🫤
I would like to keep other warnings alive in case I am doing something wrong or inefficient somewhere else.

This is fixed by https://github.com/openxla/xla/pull/16566

Thanks, @klucke
This should be in Tensorflow 2.18.0, and likely 2.17.1, and of course, nightlies
Closing

For those coming here for Google like me and you don't want to bother compiling tensorflow yourself just to fix this. Just doing a simple
import warnings warnings.filterwarnings("ignore")
worked for me to suppress these warnings.

For future TensorFlownauts trying to solve weird logs, I can confirm that the issue is solved as of version 2.18.0 for GPU published on the 25th of October 2024.


Propagate callerName in relay-ad & disable circuits

This change does two things:

update callerName to be propagated.
Disable circuit breaker for ad or relay-ad.

After speaking with @kriskowal and @rf we realized
that the PeerReaper() will empty the entire cluster
if relay-ad is circuit broken.
We need to think harder about this failure mode and
what to do with it.
r: @jcorbin @rf @kriskowal

cool :+1:


Grants Eligibility

Hi All
Just wanted to ask if only projects building on polkadot stack are eligible for a grant. Do you also support Ethereum Projects?
Thanks and best regards
Timo

Hi Timo, sorry I didn't reply sooner. The W3F Grants Program is looking for projects that benefit the Substrate, Polkadot and Kusama ecosystems. Several teams and parachains here are EVM-compatible, so there is a bit of an overlap, but if you are planning to operate on Ethereum exclusively, wo won't support it.
Let me know if you have any further questions!


dataclass field doesn't change required schema

Bug
Please complete:

OS: Mac OS
Python version import sys; print(sys.version): 3.6.7 with dataclasses backport
Pydantic version import pydantic; print(pydantic.VERSION): 1.0

It is known that @pydantic.dataclasses.dataclass doesn't fully mimic the BaseModel. However, we'd like use .schema() representation faithfully. I'm wondering what's the best workaround.
Adding a Config for b with required=False doesn't carry over to .schema() either.
from typing import Dict

from dataclasses import field
from pydantic import Field, ValidationError
from pydantic.dataclasses import dataclass

@dataclass
class D1:
    a: int
    # b: Dict[str, int] = {"x":10} # mutable default, not allowed
@dataclass
class D2:
    a: int
    b: Dict[str, int] = Field({"x":10}) # pydantic.Field, unsafe and invalid default
@dataclass
class D3:
    a: int
    b: Dict[str, int] = field(default_factory=lambda: {"x":10}) # std lib, safe and invalid default

class C:
    fields = {"b": { "required": False}} # doesn't carry over!!

@dataclass(config=C)
class D4:
    a: int
    b: Dict[str, int] = field(default_factory=lambda: {"x":10})

print(D1(10))
try:
    print(D2(10))
except ValidationError as e:
    print(e) # pydantic/dataclasses.py:77

print(D3(10))
req = D3.__pydantic_model__.schema()["required"]
if req != ["a"]:
    print('only ["a"] should be required, is:', req)

print()
print(D4.__pydantic_model__.schema()["required"]) # same as D3

Output
D1(a=10)
1 validation error for D2
b
  value is not a valid dict (type=type_error.dict)
D3(a=10, b={'x': 10})
only ["a"] should be required, is: ['a', 'b']

['a', 'b']

This isn't a bug, but a feature request.
class Config:
    fields = {"b": { "required": False}}

Wouldn't work on a normal model.
I think what you want is better support for the field() function from dataclasses. If so, please describe exactly what parts of field() usage you'd like to support in pydantic. Is it just the default in schema?

Thanks for clarifying!
Only handling default* in schema is what we need to support. However, it would be nice to have a stable interface for .__pydantic_model__. I'm not sure if this might change.

__pydantic_model__ won't change, we should document it.
It would be wonderful if you could create a PR for that. :smile:

i'm going for it


⚠ Spryr has degraded performance

In 9c58fa5, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 4354 ms

Resolved: Spryr performance has improved in 95606f9 after 13 minutes.


Set up runner to refresh mdsinabox.com every night at midnight

closed with https://github.com/matsonj/nba-monte-carlo/commit/9c3c01a32d874d33951bfdf662a30a9edb9793dd


add p2p config

License: MIT
Signed-off-by: Overbool<EMAIL_ADDRESS>Feat: https://github.com/ipfs/go-ipfs/issues/5460

FYSA go-ipfs-config got merged into go-ipfs repo: https://github.com/ipfs/go-ipfs/tree/master/config
If you feel this PR is still relevant, please reopen as a PR against go-ipfs


Failed to run the cita-web3 example

Env

Ubuntu 18.04

$ cargo version
cargo 1.38.0-nightly (677a180f4 2019-07-08)

$ rustc --version
rustc 1.38.0-nightly (69656fa4c 2019-07-13)

$ rustup --version
rustup 1.19.0 (2af131cf9 2019-09-08)

command
cargo run --example query_height http://MY_ADDR:PORT
Error info:
Compiling futures-cpupool v0.1.8 (registry `https://mirrors.ustc.edu.cn/crates.io-index/`)
error: failed to run custom build command for `openssl v0.9.24 (registry `https://mirrors.ustc.edu.cn/crates.io-index/`)`

Caused by:
  process didn't exit successfully: `/home/flyq/workspaces/cita/cita-common/target/debug/build/openssl-aa083da41679e6c3/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Unable to detect OpenSSL version', /home/flyq/.cargo/registry/src/mirrors.ustc.edu.cn-b63e9dae659fc205/openssl-0.9.24/build.rs:16:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

relate
https://github.com/paritytech/substrate/issues/800
https://github.com/paritytech/substrate/issues/886

fixed by update web3's version
 Compiling cita-web3 v0.1.0 (/home/flyq/workspaces/flyq/cita-common/cita-web3)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 17s
     Running `/home/flyq/workspaces/flyq/cita-common/target/debug/examples/query_height 'http://x.x.x.x:1337'`
Response = Quantity(1228753)


map doesn't finish saving when running shutdown -r now

So I've only recently gotten into this linux thing so there might be something small I am doing wrong, but when I call "sudo shutdown -r now" systemd does not wait for factorio to close gracefully before restarting the system. When I do a "sudo service factorio stop" it runs for around 7 seconds and then stops gracefully (I can see it stop properly by checking the server.out file).
Should I just call "sudo service factorio stop" before each restart or is there something I could have messed up? Because what is happening now is that when I restart it creates a "/opt/factorio/saves/server-save.tmp.zip" file and then because it couldn't finish saving the server tries to load that file (The "latest" save) and complains that it is corrupt. So I have to delete that file and then restart the service.

I would append your ~/.bashrc to something like this:
alias clean-restart='sudo service factorio stop && sudo shutdown -r now'

That way all you need to do from now on is type clean-restart from your terminal.
Once you update your bashrc you will need to reload your shell for it to take effect, so logout, then back in.

From some research I just did, I gather that systemd might not wait for this init script to finish because:

This is a legacy init script, not a systemd unit, so systemd's SysV compatibility generator builds a minimal unit file for this using ExecStart and ExecStop directives. (Unix.SE)
systemd assumes that whatever command is run by ExecStop will wait synchronously for the operation to finish. (systemd issue comment) Glancing at the code, it looks like this init-script should synchronously block for some time, but my Bash isn't great, so it might not be waiting?

At any rate, you might have better luck with a real systemd unit, examples of which I was able to find with a quick Google search.

@dgw what exactly is wrong with https://github.com/Bisa/factorio-init/blob/master/factorio.service.example? A slightly changed version (different executable paths and group name) is used in my AUR package, and as far as I can tell it waits just fine.

Doing some cleaning here, please re-open if anyone still has the same issue.


2.7.0 doesn't work under Windows

This commit added a dependency on fcntl which isn't present in Windows.
Resulting in ModuleNotFoundError: No module named 'fcntl'
I can create a PR to exclude it, but would this issue be present in Windows too?

Already reported (https://github.com/eht16/python-logstash-async/issues/82) and fixed in master. Will be released later today.
I don't know if Windows is also affected by #81. If so, PRs are welcome but I won't work on this myself.


DataManagementApi: add PolicyService and PolicyTransformer

For the implementation of the policy endpoint we need an PolicyService (and an PolicyServiceImpl) with roughly the following signature:
public interface PolicyService {
  Policy findbyId(String policyId);
  Collection<Policy> query(QuerySpec query);
  Policy delete(String policyId);
  void create(Policy policy);
}

In addition, there needs to be an PolicyTransformer, that converts an Policy into a PolicyDefinitionDto and back. This depends on #779
Important aspects:

service must provide transactional consistency
service must perform defensive checks before performing an operation:

delete: a Policy can only be removed if not yet referenced in a ContractDefinition

invoking the transformer is done by the controller, not the service.
transformer does not perform any validation, it simply converts
depends on #803

Hi, since I'm not listed as contributor. I confirm that I take this issue.


Topic saturation

We have a VerneMQ cluster with 3 nodes inside a kubernetes cluster. In our use case, we have a single topic that can have a maximum throughput of around 500k / s in peak hours. Sometimes, due to the traffic, the latency might change and, when that happens, sometimes the VerneMQ nodes stops "operating normally".
By "operating normally" I simply mean "stops being fast as it normally is".
We found out that a way to fix this issue is to downscale VerneMQ to 1, scale it again to 3, and restart the first instance. This, I would say, is equivalent to restart the entire cluster.
I am very aware that this is a lot of assumptions over VerneMQ. Therefore, we have been thinking on how we can optimise its usage. That's why we were wondering if this particular topic is oversaturated and if that would be a bottleneck to, not only VerneMQ, but any other MQTT broker.
(As a side note, all the messages on that particular topic are also being published to another topic, so generating more traffic).
We would like to know if there is a way to optimise VerneMQ. Thus I ask:

What's the maximum throughput of a topic that VerneMQ cluster can deal with?
Performance wise, does it change if we split the traffic into sub-topics (meaning adding a "/" to it)? In this case, we would subscribe several consumers on a partition basis.
If adding partitions to the topic doesn't change the performance of it, what VerneMQ configuration details would you recommend for our case?

Environment

VerneMQ Version: erlio/docker-vernemq:1.6.1
OS: ubuntu 16.04
Kubernetes version: 1.12
VerneMQ configuration (vernemq.conf) or the changes from the default:

  DOCKER_VERNEMQ_ALLOW_PUBLISH_DURING_NETSPLIT: "on"
  DOCKER_VERNEMQ_ALLOW_REGISTER_DURING_NETSPLIT: "on"
  DOCKER_VERNEMQ_ALLOW_SUBSCRIBE_DURING_NETSPLIT: "on"
  DOCKER_VERNEMQ_ALLOW_UNSUBSCRIBE_DURING_NETSPLIT: "on"
  DOCKER_VERNEMQ_DISCOVERY_KUBERNETES: "1"
  DOCKER_VERNEMQ_ERLANG.async_threads: "1024"
  DOCKER_VERNEMQ_ERLANG__DISTRIBUTION__PORT_RANGE__MAXIMUM: "9109"
  DOCKER_VERNEMQ_ERLANG__DISTRIBUTION__PORT_RANGE__MINIMUM: "9100"
  DOCKER_VERNEMQ_KUBERNETES_APP_LABEL: vernemq
  DOCKER_VERNEMQ_KUBERNETES_INSECURE: "1"
  DOCKER_VERNEMQ_LISTENER__VMQ__CLUSTERING: <IP_ADDRESS>:44053
  DOCKER_VERNEMQ_MAX_ONLINE_MESSAGES: "-1"
  DOCKER_VERNEMQ_LISTENER__TCP__LOCALHOST: "<IP_ADDRESS>:1883"
  DOCKER_VERNEMQ_VMQ_PASSWD__PASSWORD_FILE: /etc/vernemq-passwd/vmq.passwd

Expected behavior

VerneMQ to handle a lot of traffic (few peaks up to ~500k / s).

Actual behaviour

VerneMQ gets slow when operating under a lot of traffic (few peaks up to ~500k / s).

Thanks

Hi, that's a really interesting case - I don't think we've every tried to push that many messages/s over a single topic before - most use cases have messages flowing over several topics at that levels. So this is of course really interesting. From a vernemq handling a message is done roughly like the following:

A message is published the erlang process handling the session will do a lookup into the routing table to see the subscribers the message should be forwarded to and forwards the message to the queue process for the subscribing client
For each of the subscribing clients the message is forwarded from the queue process (if the client is online) to the session process for the subscribing client and sent out over the socket.

point 1 happens in the context of the session process of the publisher, so this is in a way completely scalable as each process can run on any core. It may be that what you're experiencing is contention when the lookup to find the subscribers is done in the routing table as this is a shared datastructure (ETS tables). Point 2 happens in the context of each subscriber and all subscribers are independent of each other and thus are theoretically completely scalable over the number of cores.
It could certainly be the routing table lookup which is causing problems here, but before drawing any conclusions I think it would be good with some more information about the number of publishers/subscribers/qos levels/message sizes etc as well as load metrics (cpu/cores/memory/io) when the peak occurs.

Hi @larshesel , thanks for the quick answer.
I have just updated my message by adding the environment information and configuration. We have several grafana charts with that information. I will share them with you soon.

Hi @larshesel , here are some charts.
To give you some background about the service itself. The messages it receives are compressed. Their size, on average, is around 100 KB. The service will expand every incoming message into an array of messages and it will send each of them back to VerneMQ (back to the topic in question). During the peak hours, kubernetes scales up the service in question and the average ratio published/ received is around 500 /s per service during those hours.
In the image below, you can see two charts, one for the number of received and another for the number of published messages over time. I made some notes to highlight the events I described previously.

The next image shows the CPU, memory and Net I/O charts of the VerneMQ nodes (kubernetes generated charts). I also highlighted the same events in them.

Other things that might be of interest:

QoS for both subscription and publish mechanisms is 0 in the service
Produced messages have, on average, size equals to 0,5 KB (JSON messages only)
Number of services subscribed in the incoming topic is at least 2 (can go up to 30, managed by kubernetes)
Number of services subscribed to the outgoing topic is 2 (we are not scaling this one up, yet), sometimes 3 (due to scheduled jobs)
Slow means the drop (as you can see in the chart) of 200k produced messages to less than 3k on average, on all the services.

Please, let me know if you need more information.
Thanks.

Hi, thanks for the data! I don't really have any obvious ideas yet, but it may be helpful to know if anything is written to the logs when (and leading up to) the drop.

Yes, the logs from stdout. Metrics would also be helpful (maybe something looks weird?)

Hi @larshesel, thanks for the answer.
We will try to catch the problem in the logs. It happened before a few times, so I would expect it to happen again. Meanwhile, if we instead use a strategy to split the traffic in a  topic-partition fashion (meaning <topic-name>/<partition>), do you have an idea of how it would impact the performance?

If indeed the routing is the bottleneck as contemplated above then that might have an impact. But from your description and the graphs it looks like the system stalls for some reason, so I don't really think splitting the topic in that way would do much.

Closing for now - let us know if more data becomes available and we can reopen. Note that there's an interesting improvement which may/may not be of interest to you on the way via https://github.com/vernemq/vernemq/issues/988#issuecomment-446597814


IDataReader implementations need to be thread safe

Need to make CoreDumpReader and LinuxLiveDataReader thread safe.

Also need to validate that PR https://github.com/microsoft/clrmd/commit/54b64b91554873fff671156cf7cd0b5b34bed1c2 is thread safe.

Also need to validate that PR 54b64b9 is thread safe.

There is an internal lock in S.R.PE.
https://github.com/dotnet/runtime/blob/33bedaf3bcc95d91dde5f09251a5972fbac5f05e/src/libraries/System.Reflection.Metadata/src/System/Reflection/Internal/MemoryBlocks/StreamMemoryBlockProvider.cs#L44
https://github.com/dotnet/runtime/blob/33bedaf3bcc95d91dde5f09251a5972fbac5f05e/src/libraries/System.Reflection.Metadata/src/System/Reflection/Internal/MemoryBlocks/StreamMemoryBlockProvider.cs#L121

I will likely not tackle thread-safety of Linux data readers before RTM.  Moving this to "future".


Using built in filtering uses AND instead of OR

Describe the bug
I'm having an issue with the built in filtering in HotChocolate. I'm running a query to search over Contacts, checking if the search term is either in the FirstName or the LastName, but the SQL getting run uses an AND, requiring the search term to exist in both the FirstName and the LastName.
To Reproduce
Steps to reproduce the behavior:
On QueryType.cs
descriptor.Field(t => t.ContactGetAll(default))
    .UseFiltering()

The query I'm running
query GetAll($searchTerm: String) {
   contactGetAll (where: { firstName_contains: $searchTerm, OR: { lastName_contains: $searchTerm } }) {
      ...BasicContact
   }
}

The SQL generated
...
WHERE ([c].[FirstName] IS NOT NULL AND (CHARINDEX(N'name', [c].[FirstName]) > 0)) AND ([c].[LastName] IS NOT NULL AND (CHARINDEX(N'name', [c].[LastName]) > 0))

Expected behavior
I'm expecting the SQL generated to use OR functionality, allowing the search term to be in either of the strings, not requiring both.
Desktop (please complete the following information):

OS: Linux
Version: Docker Image - mcr.microsoft.com/dotnet/core/sdk:3.0)
HotChocolate: 10.3.2

Please let me know if any other information is needed.
Thanks!

We just discussed this on slack.
Adrian Frielinghaus

This might be the same as something that puzzled me... the OR conditions need to be in an array like so:
OR: [
{
lastName_contains: $searchTerm
},
{
lastName_contains: $searchTerm
}
]

Nick Beukema
contactGetAll (where: { OR: [{ lastName_contains: $searchTerm }, { firstName_contains: $searchTerm }] }) {
}

That is erroring now
{"0":{"message":"Unexpected Execution Error","locations":[{"line":10,"column":3}],"path":["contactGetAll"]}}
But it is working with the previously stated query (which now makes sense why it's doing an AND). It also works with each individual filter as well.

Actually this is not a bug but fully correct.
Let me explain.
The assumption that we have done to make it very easy to use is that the default is AND.
Meaning:
{ a: $a b: $b }

Will become a simple and query. So for simple queries there is no need to express an AND operator.
of you want an or query this is the way to write it:
{ OR: [ { a: $a } { b: $b } ] }

And you also can combine this since:
{ a: $a OR: [ { b: $b } { c: $c } ] }

would become
a = $a AND (b = $b OR c = $c)

the above can also be expressed more explicitly:
{ AND: [ { a: $a } ] OR: [ { b: $b } { c: $c } ] }

and also you can do more complex queries like:
{ AND: [ { a: $a } ] OR: [ { b: $b } { AND: [ { c: $c } { d: $d } ] } ] }

which would become:
a = $a AND (b = $b OR  (c = $c AND d = $d))

Does that clarify how the queries work... If you worked with mongo this should be quite familiar.
Hope this helps :)

Can we close this issue?

@NickBeukema  I added a issue especially for the bug you observed :+1:
I will close this issue

I think we also should write Moro docu on the filters. I think I will add a section on how the filters translate to sql.