Process get killed when running tensorflow.python.ops.stateful_random_ops._make_1d_state

Click to expand!
Issue Type
Bug
Have you reproduced the bug with TF nightly?
No
Source
source
Tensorflow Version
2.10.0
Custom Code
Yes
OS Platform and Distribution
Ubuntu 22.04
Mobile device
No response
Python version
3.9
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
Probably due to very large input argument.

Standalone code to reproduce the issue
import tensorflow as tf
import os
import numpy as np
from tensorflow.python.ops import stateful_random_ops
try:
  arg_0 =<PHONE_NUMBER>51
  arg_1 = 12345
  out = stateful_random_ops._make_1d_state(arg_0,arg_1,)
except Exception as e:
  print("Error:"+str(e))

### Relevant log output

```shell
2023-01-21 10:45:54.421270: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-21 10:45:54.535268: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-01-21 10:45:55.097219: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/lib/
2023-01-21 10:45:55.097403: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/lib/
2023-01-21 10:45:55.097412: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Killed

</details>

Hi, @nimashiri
Apologize for the delay, It seems like you're giving large value for state_size  argument to the function _make_1d_state()so due to Integer overflow to buffer overflow or due to insufficient memory (RAM), code is crashing or process getting killed and I was able to replicate issue on Google colab with GPU which is working as expected for other values with 8 or 9 digit ( I tried '9' digit value= 125091515, it's working as expected), for your reference I have added gist file here and you can refer python-file
If issue still persists please let us know ? or Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ? Thank you!

@nimashiri
You're welcome and yes, it's working fine with smaller integers as expected.
Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ? Thank you!

Typical OOM, not a vuln


r2.2-rc1 cherry-pick request: [Intel MKL] Fix performance regression in DNNL 1

Fix performance regression in DNNL 1 due to lack of primitive cache on reorder. This only affects TF-MKL and not stock TF.

@guizili0 Could you please post @googlebot I consent.?

@googlebot I consent.


Panic and unexpected content with hugo new foo.unknownExtension

Tested with https://github.com/gohugoio/hugo/commit/471ed91c60cd36645794925cb4892cc820eae626. Not reproducible with v0.88.1.
I found this accidentally while creating pandoc content. I meant to type hugo new foo.pdc but instead typed hugo new foo.pdoc.

panic: [BUG] no Page found for ...
panic: [BUG] no Page found for "/home/jmooring/code/hugo-testing/content/foo.pdoc"

goroutine 1 [running]:
github.com/gohugoio/hugo/create.(*contentBuilder).applyArcheType(0xc000268780, {0xc0007cf840, 0x31}, {0x0, 0x0})
        /home/jmooring/code/hugo/create/content.go:249 +0x354
github.com/gohugoio/hugo/create.(*contentBuilder).buildFile(0xc000268780)
        /home/jmooring/code/hugo/create/content.go:218 +0x1ad
github.com/gohugoio/hugo/create.NewContent(0xc0000e4fd0, {0x0, 0x0}, {0x7ffe6b78c03c, 0x684985})
        /home/jmooring/code/hugo/create/content.go:94 +0x6ab
github.com/gohugoio/hugo/commands.(*newCmd).newContent(0xc00015d1d0, 0xc000414a00, {0xc00079fcc0, 0x1, 0x0})
        /home/jmooring/code/hugo/commands/new.go:83 +0x150
github.com/spf13/cobra.(*Command).execute(0xc000414a00, {0xc00079fc90, 0x1, 0x1})
        /home/jmooring/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc00086a780)
        /home/jmooring/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/gohugoio/hugo/commands.Execute({0xc00003c0a0, 0x2, 0x2})
        /home/jmooring/code/hugo/commands/hugo.go:92 +0xb4
main.main()
        /home/jmooring/code/hugo/main.go:23 +0x58

The file (content/foo.pdoc) was created, but the content was unexpected.
With v0.881. the content is:
---
title: "Foo"
date: 2021-10-22T10:38:20-07:00
draft: true
---

With https://github.com/gohugoio/hugo/commit/471ed91c60cd36645794925cb4892cc820eae626 the content is:
---
title: "Content Placeholder"
_build:
  render: never
  list: never
  publishResources: false
---

I do not have an archetype for this file extension.
archetypes/
└── default.md

I'm having this issue on 0.91 extended....

@GGTruss This issue is closed. Please use the forum (https://discourse.gohugo.io) for questions and troubleshooting.


GM_registerMenuCommand not works

What is the problem?
GM_registerMenuCommand reg menu , mouse right click popup menu not found it in menu list
How to reproduce it?
// ==UserScript==
// @name popup menu test
// @namespace    test
// @match *://*/*
// @grant       GM_registerMenuCommand
// @run-at      document-start
// ==/UserScript==

(function() {
    'use strict';
    console.log("popup test run")
    GM_registerMenuCommand("popup menu test", function(){
      console.log("popup test click")
    })
})();

What is the expected result?
right menu has a menu item "popup menu test" and can click it
What is the actual result?
menu not exists
Environment

Browser:chrome
Browser version:93.0.4577.63（正式版本） (x86_64)
Violentmonkey version:v1.1.17
OS:Mac and Window

Sorry, my English is very poor. The following content comes from Baidu translation
found it in Violentmonkey extension icon click popop menu item "script common"
but it's a new Window,  It is expected that can right-click an element in HTML to further control it. However, when a new window is opened, it is difficult to directly associate and control the clicked target item


added tardis to CODE_NAMES

Tardis isn't included in CODE_NAMES and thus the models aren't shown in the plots

The individual notebooks are generated by the azure pipeline. You will have to update plotting_snia_toy06_test_template.ipynb, which serves as the jinja template.

Implemented by merge of #6.


Feature/dx2.0 stable frontend integration v1

Updated Contracts from feature/DutchExchange2.0_Alex_Code
--> Compiles successfully

@Velenir uhhhh can we close this, lmaooo

Closed as outdated


Explore offline generation of sequences

I generated a bunch of simulated sequences and dumped them to videos. This takes ~<1s per video and the mean video size is 200KB. This is a lot less than storing the images (raw bytes, JPEG compression might reduce it but OOT dataset takes 1,4G in images and 110M in videos for example)
60 frames x 320 width x 240 height x 3 channels x 1 byte = 13824000 bytes
but the video generation dumps the images to a temporal folder and then calls a process to encode them, but it seems to be done in parallel, as CPU usage was 300% and measuring time with %time returns
CPU times: user 814 ms, sys: 1.34 s, total: 2.15 s
Wall time: 837 ms
TODO: measure video reading time
Reading the video creates a dependency on OpenCV.
I tried generating an HDF5 file of a batch of activations also. This takes more space than the videos but less than the images (442368000 bytes for 32 sequences)
32 batch size x 60 frames x 7 width x 7 height x 1024 channels/filters x 4 bytes = 385351680 bytes
but saving the file took 2s, did not tried with a pickle file. Reading time is negligible, 4ms.
The GPU is the bootleneck for feature extraction, we could had a lot of machines generating videos and sending them to a central GPU node to at least keep it almost always busy. It would be better if there were also many GPU nodes sending features to a training node. But i dont know if the overhead is worth, specially when saving the activations. It would also be possible to avoid saving to disk by sending arrays over the network, but the transport overhead is still unknown.
I changed the parallel simulation and now the simulation and feature extraction seem to be more simultaneous. With a batch size of 128 and a GPU batch of 4 this saved ~10s, passing from ~40 to ~30s. The 40s are ~23s for simulation, ~15s forwarding and ~2s training, and the 30s are ~6s of exclusive simulation, ~24s of simultaneous extraction and simulation and ~2s training. I used 128 for batch size as this took ~9-12GB RAM when collecting frames and features (it could be more if a lot or RAM is available). I am not sure why feature extraction passed from ~15s alone to ~24s with simulation, as they are executed in CPU and GPU independently. Maybe because the process pool uses all cores by default so the main process has to also compete for CPU time?

A loop reading the 256 sample videos with OpenCV and storing each frame in a list uses 2.5 GB RAM and takes
Mean: 0.1 s
Sum: 26.27s
Min: 0.079s
Max: 0.24s

This is temporally closed as execution time has already been improved, and there is not yet an immediate use case for offline sequences. But it is definitely cheaper to just read pregenerated sequences. Possible use case: replay memory


add equalValueAt api for DecodedVector

Details in https://github.com/facebookincubator/velox/pull/6390#discussion_r1317580319

@bikramSingh91 I add the equalValueAt API in this PR according to our discussion in https://github.com/facebookincubator/velox/pull/6390#discussion_r1317580319. This API would combine with the isNullAt API since a DecodedVector may have extra nulls (eg, Dictionary encoding). Could you help to take a look? cc @mbasmanova


Initial

@mlofstrom,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/test/index
:white_check_mark:Succeeded

docs/test/markus
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

docs/test/index
:white_check_mark:Succeeded

docs/test/markus
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


az --version: CLI version cannot be parsed programmatically -> have -json switch or switch to output only version number

Is your feature request related to a problem? Please describe.
az --version outputs a lot of text that cannot be parsed easily and in a reliable way in order to determine the CLI version programmatically (e.g. from PowerShell)
Describe the solution you'd like
Either an option to output a machine parse-able output such as e.g. JSON or a switch to output only the version of the CLI as one string
Describe alternatives you've considered
No alternative, hacks like awk/grep/sed are common in the Unix world but are too much effort, too brittle and some of those are not available on Windows where the main automation tool is PowerShell (sometimes the az still has to be used though due to feature gaps of the azure powershell module)
Additional context
I had to check the az version to be greater than v2.0.69 as this is a minim requirement for the azure devops CLI extension.

Yes, good suggestion!
For your question, there should be a way to parse version string.
For too many outputs, we will think about it.

add to S163.

A potential solution is to add --short/-s option for az --version to output just the version of CLI. This involves changing code in knack. az --version is handled specially by checking args[0] == '--version' or args[0] == '-v', then invoke show_version() without passing in the args.
To avoid breaking changes in knack, we may want to add a show_short_version() method or more generic show_version_with_args().

After some discussion, we do not want overcomplicate the special case of az --version. @jiasli will author a new command to output short version.

This is the way I have done it:
(-join (az version) | convertFrom-Json).'azure-cli'


"counts" section of the /crawl endpoint has very inconsistent key names

Here's a current example of the data that is exposed:
"counts":
{
"AL_hit_rate":13.04347801208496,
"HashRouterEntry":3615,
"InboundLedger":82,
"Ledger":93,
"NodeObject":4009193,
"RCLCxPeerPos::Data":260,
"SLE_hit_rate":0.7297560975609756,
"STArray":3024,
"STLedgerEntry":918,
"STObject":32623,
"STTx":2986,
"STValidation":401,
"Transaction":330,
"dbKBLedger":2876,
"dbKBTotal":53629,
"dbKBTransaction":48894,
"fullbelow_size":1183851,
"historical_perminute":84,
"ledger_hit_rate":9.79262638092041,
"node_hit_rate":0.01132712606340647,
"node_read_bytes":1092703150,
"node_reads_hit":3954570,
"node_reads_total":3980180,
"node_writes":57087,
"node_written_bytes":30015185,
"treenode_cache_size":4009112,
"treenode_track_size":4009112,
"uptime":"20 minutes, 22 seconds",
"write_load":0
}

There is a mixture of snake_case, CamelCase, C++::Namespace::Names, Uppercase and lowercase in there.
The data that is exposed is also not really documented (https://developers.ripple.com/get_counts.html - "The list of fields contained in the result is subject to change without notice, but it may contain any of the following (among others) [...]"). The icing on the cake imho is the uptime that's reported as a custom string instead of as an integer of seconds. ;-)
Please decide on a format (you seem to use lowercase_snake_case everywhere else) and if possible also document what these numbers mean.

The names are "inconsistent" because they come from different sources. For example, STTx is the actual name of a class, and that name is used as the name of the counter associated with such classes. Honestly, I don't think that the particular choice of naming matters in this instance; this particular object is just a property bucket - a collection of name/value pairs.
I hear you about improved documentation: it is important and I'm going to open an issue to get that started. Also, fully agree that the uptime field should certainly be in seconds and not a human-readable string.

Closing in favor of https://github.com/XRPLF/xrpl-dev-portal/issues/519


Please restore logo (URGENT)

Looks like you removed the logo (/admin/lupusec.png ) from the repository.
This causes automatic adapter checks to fail and might impact repostory updates:
Icon of adapter lupusec: https://raw.githubusercontent.com/schmupu/ioBroker.lupusec/master/admin/lupusec.png not gettable
Please restore the file as soon as possible or let us know the new filename if you decided to change it. (In this case and update of io-package.json is required)
If the icon keeps to be unavailable I will remove the adapter from the repository until fixed

fixed

Thanks very much for the ultrafast response


'[HY004] [Microsoft][ODBC Driver 17 for SQL Server]Invalid SQL data type (0) (SQLBindParameter)' When first row of TVP contains None as a cell value

Please first make sure you have looked at:

Documentation: https://github.com/mkleehammer/pyodbc/wiki
Other issues

Environment
To diagnose, we usually need to know the following, including version numbers.  On Windows, be
sure to specify 32-bit Python or 64-bit:

Python: 3.8
pyodbc: 4.0.30
OS: Windows 10 64-bit
DB: SQL Server 2017
driver: ODBC Driver 17 for SQL Server
ODBC Log: SQL.LOG

Issue
The below code fails because the 1st row of param_array contains None. If you swap the rows, the error goes away.. So it seems that TVP will only work when the 1st row contains all non-None values... All other rows can contain None.
It's similar to https://github.com/mkleehammer/pyodbc/issues/520
param_array = [
    [None, 0],
    [1, 1],
    ]
conn_str = "DRIVER={ODBC Driver 17 for SQL Server};SERVER=localhost;UID=sa;DATABASE=hasani"

pyodbc.connect(conn_str).cursor().execute("EXEC TestSelectTVP ?", [param_array]).fetchall()

CREATE TYPE [dbo].[TVPType] AS TABLE(
	[c01] INT NULL,
	[c02] INT NULL
)
GO

CREATE PROCEDURE dbo.TestSelectTVP
(
	@TVP dbo.TVPType READONLY
)
AS
BEGIN
SELECT * FROM @TVP;
END

Often it is easiest to describe your issue as "expected behavior" and "observed behavior".

In general, Nones are a bit problematic since they don't provide any hints of the type that the parameter should be sent as; when the parameter is not a TVP, then the SQLDescribeParam function is used to determine the type, but when it is a TVP, SQLDescribeParam only says that the parameter is a TVP, and does not provide information about its individual columns.
If the parameter is not None, then it can be used to determine the type, which is why it works when the first row doesn't have Nones --- pyODBC uses the contents of that row to determine what types it should bind each of the sub-parameters in the TVP  as.

Hi Chojas,
Thanks for the prompt reply. Is it possible pyodbc can internally retrieve types when the 1st row contains None, or let the client specify the caller of the background.  Also, it may be unreliable to use the value to determine the type because it's not guaranteed to be the same across rows. (E.x.: The first row can contain all strings and the subsequent row can contain an mix of strings and floats strings for a TVP that has all FLOAT columns and this is because SQL server is doing an implicit conversion of the data.
I did try to take a stab at fixing it yesterday. I was able to compile the code and create a python test to replicate it but the ODBC protocol was too much to grok for me... It was my first time dabbling in the ODBC API.

For the 2nd idea, maybe pyodbc can provide a class we can optionally use to send data to pyodbc..
E.x.:
from dataclasses import dataclass

@dataclass
class TableValueParameter:
    column_types: list[type] # the python type that to map the the TVP column
    rows: # the sequence that we normally send to pyodbc for a TVP

If having pyodbc auto-detect the TVP types is not feasible (as suggested by my research into a related issue suggesting that all pyodbc can get from SQL Server itself is that the ParameterType is SQL_SS_TABLE (-153), and the ValueType is SQL_C_BINARY) then perhaps the "let the client/caller specify or register the types" could be accomplished by an extension to .setinputsizes where one of the tuple elements in the outer list of tuples (for each parameter) could itself be a list of tuples (for each column in the TVP).

I give up... I tried using TVP in AWS Lambda and pyodbc caused a core dump. I'll just upload JSON strings instead of TVP.

Closing because

this is a limitation of parameter type discovery for TVPs, and
the workaround is to use OPENJSON as below.

This fails
param_array = [
    [None, 0],
    [1, 1],
]

cnxn = pyodbc.connect(conn_str)
crsr = cnxn.cursor()

crsr.execute("EXEC TestSelectTVP ?", [param_array]).fetchall()

while this works
param_json = [dict(c01=x, c02=y) for x, y in param_array]
print(param_json)
# [{'c01': None, 'c02': 0}, {'c01': 1, 'c02': 1}]

sql = """\
SET NOCOUNT ON;
DECLARE @tvp [dbo].[TVPType];
INSERT INTO @tvp
SELECT c01, c02
FROM OPENJSON(?)
WITH (
    c01 int '$.c01',
    c02 int '$.c02'
);
EXEC [dbo].[TestSelectTVP] @tvp;
"""
crsr.execute(sql, json.dumps(param_json, default=str))
print(crsr.fetchall())
# [(None, 0), (1, 1)]


Add back more properties fields for optinout

What are the schemas that are affected by the issue
optinout
What are examples of products that are impacted by the issue
platform xdm definition

Made obsolete by #139


Smooth equivalent of ConstantInterpolation in terms of integral

Is your feature request related to a problem? Please describe.
A bit of context: In my ODE problem for hydrological modelling, I want to model certain forcing events like precipitation. The precipitation is given as a time series, where constant interpolation ('forward fill') is assumed so that the exact total volume is known. Constant interpolation however does not play nice with ODE solvers because of its lack of smoothness. Therefore I want to have an interpolation type which:

Is at least $C^1$ smooth;
Has for each interval between data points the same integral as forward fill would have.

An additional requirement I have is that it is easy to extend this interpolation with new incoming data.
Describe the solution you’d like
The naive solution of assuming constant interpolation on the first interval and then quadratic on the others to satisfy the above conditions is not stable:
using DataInterpolations
using CairoMakie
using Random
using FindFirstFunctions: searchsortedfirst
using QuadGK

fig = Figure()

ax_itp = Axis(fig[1,1]; title = "Interpolation")
ax_int = Axis(fig[2,1]; title = "Integral")

N = 6
Random.seed!(5)
u = rand(N)
t = cumsum(rand(N))
t_eval = range(first(t), last(t), length = 250)

A = ConstantInterpolation(u, t; cache_parameters = true)
lines!(ax_itp, t_eval, A.(t_eval); label = "ConstantInterpolation")
lines!(ax_int, t_eval, DataInterpolations.integral.(Ref(A), t_eval); label = "ConstantInterpolation")

a = zeros(N - 1)
b = zeros(N - 1)
c = zeros(N - 1)

# Assume constant interpolation on first interval
c[1] = u[1]

for i in 1:(N-2)
    Δt = t[i+1] - t[i]

    # value at t[i+1]
    u_ = @evalpoly Δt c[i] b[i] a[i]

    # derivative at t[i+1]
    du_ = @evalpoly Δt b[i] 2a[i]

    # integral over (t[i+1], t[i+2])
    Δt = t[i+2] - t[i+1]
    I = Δt * u[i+1]

    # Next coefficients
    c[i+1] = u_
    b[i+1] = du_
    a[i+1] = 3 * (u[i+1] - u_ - (du_ * Δt) / 2) / Δt^2
end

function B(t_)
    i = clamp(searchsortedfirst(t, t_)-1, 1, N-1)
    @evalpoly (t_ - t[i]) c[i] b[i] a[i]
end

lines!(ax_itp, t_eval, B.(t_eval); label = "Naive smooth interpolation")
lines!(ax_int, t_eval, [quadgk(B, first(t), t_)[1] for t_ in t_eval]; label = "Naive smooth interpolation")

axislegend(ax_itp)
axislegend(ax_int)

fig

Oh and of course having periods of 'negative precipitation' to compensate does not make sense. It looks like what I want is that the integral of my interpolation is a PCHIPInterpolation of the integrated values.

For now I'm letting go of the condition of being able to easily add new data to the interpolation, as that poses quite some problems. My new idea is to replace each block of precipitation by a hill that has support over the neighboring intervals, using degree 2 B-spline basis functions:
using DataInterpolations
using CairoMakie
using Random
using QuadGK

N = 25
Random.seed!(8)
u = rand(N)
t = collect(1:N)
t_eval = range(first(t), last(t), length = 250)
A1 = ConstantInterpolation(u, t; cache_parameters = true)

k = t[3:end]
pushfirst!(k, t[1])
pushfirst!(k, first(t))
pushfirst!(k, first(t))
push!(k, last(t))
push!(k, last(t))
sc = zeros(N)
out = zeros(length(sc), length(t_eval))

for (i, t_) in enumerate(t_eval)
    sc .= 0
    DataInterpolations.spline_coefficients!(sc, 2, k, t_)
    out[:, i] .= sc
end

I = sum(out, dims=2) * (t_eval[2] - t_eval[1])
c = zeros(N)
c[1:(N-1)] .= diff(t) .* u[1:(N-1)] ./ I[1:(N-1)]

f = Figure()
ax_itp = Axis(f[1,1]; title = "Interpolation")
ax_int = Axis(f[2,1]; title = "Integral")

lines!(ax_itp, t_eval, A1.(t_eval); label = "ConstantInterpolation")
lines!(ax_int, t_eval, DataInterpolations.integral.(Ref(A1), t_eval); label = "ConstantInterpolation")

function A2(t_)
    DataInterpolations.spline_coefficients!(sc, 2, k, t_)
    return sum(sc .* c)
end

lines!(ax_itp, t_eval, A2.(t_eval); label = "QuadraticSpline")
lines!(ax_int, t_eval, [quadgk(A2, first(t), t_)[1] for t_ in t_eval]; label = "QuadraticSpline")

axislegend(ax_itp)
axislegend(ax_int)

f

It must be noted that this only seems to work well with equally spaced data.

Better fit and handling of boundaries:
using DataInterpolations
using CairoMakie
using Random
using QuadGK

N = 25
Random.seed!(8)
u = rand(N)
t = collect(1:N)

function refine(u,t)
    N = 2*length(t)-1
    t_new = zeros(N)
    u_new = zeros(N)
    Δt = diff(t)
    t_new[1:2:end] .= t
    t_new[2:2:end] .= t[1:end-1] + Δt/2
    u_new[1:2:end] .= u
    u_new[2:2:end] .= u[1:end-1]
    N =
    u_new, t_new, N
end

u, t, N = refine(u, t)
t_eval = range(first(t), last(t), length = 500)
A1 = ConstantInterpolation(u, t; cache_parameters = true)

k = copy(t)
pushfirst!(k, first(t))
pushfirst!(k, first(t))
push!(k, last(t))
push!(k, last(t))
sc = zeros(N+1)
out = zeros(N-1, length(t_eval))

for (i, t_) in enumerate(t_eval)
    sc .= 0
    DataInterpolations.spline_coefficients!(sc, 2, k, t_)
    out[1, i] += sc[1]
    out[1, i] += sc[2]
    out[2:(N-2), i] .= sc[3:(N-1)]
    out[N-1, i] += sc[N]
    out[N-1, i] += sc[N+1]
end

Δt = t_eval[2] - t_eval[1]
I = sum(out, dims=2) * Δt
I = I[:,1]
c = diff(t) .* u[1:(N-1)] ./ I
c_extended = copy(c)
pushfirst!(c_extended, first(c))
push!(c_extended, last(c))

f = Figure()
ax_itp = Axis(f[1,1]; title = "Interpolation")
ax_int = Axis(f[2,1]; title = "Integral")
ax_b = Axis(f[3,1]; title = "Basis functions")
scatter!(ax_b, t, zero(t))

for i in 1:(N-1)
    lines!(ax_b, t_eval, out[i, :])
end

lines!(ax_itp, t_eval, A1.(t_eval); label = "ConstantInterpolation")
lines!(ax_int, t_eval, DataInterpolations.integral.(Ref(A1), t_eval); label = "ConstantInterpolation")

function A2(t_)
    DataInterpolations.spline_coefficients!(sc, 2, k, t_)
    return sum(sc .* c_extended)
end

lines!(ax_itp, t_eval, A2.(t_eval); label = "QuadraticSpline")
lines!(ax_int, t_eval, [quadgk(A2, first(t), t_)[1] for t_ in t_eval]; label = "QuadraticSpline")

f

The sigmoid section can be used with constant sections in between to generalize to non-equally spaced data:

🙌
using DataInterpolations
using FindFirstFunctions: searchsortedfirstcorrelated
using CairoMakie
using Random
using QuadGK

N = 25
Random.seed!(8)
u = rand(N)
t = cumsum(rand(N))

t_eval = range(first(t), last(t), length = 500)
A1 = ConstantInterpolation(u, t; cache_parameters = true)

f = Figure()
ax_itp = Axis(f[1,1]; title = "Interpolation")
ax_int = Axis(f[2,1]; title = "Integral")

lines!(ax_itp, t_eval, A1.(t_eval); label = "ConstantInterpolation")
lines!(ax_int, t_eval, DataInterpolations.integral.(Ref(A1), t_eval); label = "ConstantInterpolation")

Δt = diff(t)
d = [min(Δt[i], Δt[i+1])/2 for i in 1:(N-2)]
Δu = diff(u)

function A2(t_)
    idx = clamp(searchsortedfirstcorrelated(t, t_, 1) - 1, 1, length(t) - 1)
    d_lower = (idx == 1) ? 0 : d[idx-1]
    d_upper = (idx == N-1) ? 0 : d[idx]
    return if (t_ - t[idx]) < d_lower
        u[idx] - 0.5 * Δu[idx-1] * (((t_ - t[idx])/d_lower - 1))^2
    elseif (t[idx+1] - t_) < d_upper
        u[idx] + 0.5 * Δu[idx] * ((1 - (t[idx+1] - t_)/d_upper))^2
    else
        u[idx]
    end
end

lines!(ax_itp, t_eval, A2.(t_eval))
lines!(ax_int, t_eval, [quadgk(A2, first(t), t_)[1] for t_ in t_eval])

f

Setting a global maximum on $d$:
using DataInterpolations
using FindFirstFunctions: searchsortedfirstcorrelated
using CairoMakie
using Random
using QuadGK

N = 10
Random.seed!(8)
u = rand(N)
t = cumsum(rand(N))

f = Figure()
ax_itp = Axis(f[1,1]; title = "Interpolation")
ax_int = Axis(f[2,1]; title = "Integral")

Δt = diff(t)
Δu = diff(u)

for d_max in 0.00:0.05:0.3
    d = [min(Δt[i], Δt[i+1], 2d_max)/2 for i in 1:(N-2)]

    function A2(t_)
        idx = clamp(searchsortedfirstcorrelated(t, t_, 1) - 1, 1, length(t) - 1)
        d_lower = (idx == 1) ? 0 : d[idx-1]
        d_upper = (idx == N-1) ? 0 : d[idx]
        return if (t_ - t[idx]) < d_lower
            u[idx] - 0.5 * Δu[idx-1] * (((t_ - t[idx])/d_lower - 1))^2
        elseif (t[idx+1] - t_) < d_upper
            u[idx] + 0.5 * Δu[idx] * ((1 - (t[idx+1] - t_)/d_upper))^2
        else
            u[idx]
        end
    end

    label = "d_max = $d_max"

    lines!(ax_itp, t_eval, A2.(t_eval); label)
    lines!(ax_int, t_eval, [quadgk(A2, first(t), t_)[1] for t_ in t_eval]; label)
end

axislegend(ax_itp)
axislegend(ax_int)

f


Crash with Serene Seasons

Version :
DynamicSurroundings-1.12.2-<IP_ADDRESS>BETA.jar
SereneSeasons-1.12.2-1.2.9-universal.jar
crash-2018-11-28_07.54.10-client.txt
https://github.com/Glitchfiend/SereneSeasons/issues/101

Waiting on updated Serene Seasons JAR.

Both v<IP_ADDRESS> and v<IP_ADDRESS> are both on CurseForge.  Yay.


Install should check for Docker dependency

I guess we should either prevent or warn the use when installing Fuge without having Docker setup.

Not really - fuge will work on just processes alone - it doesn't require Docker to work


Using git bash in windows terminal can't cd into a git project folder!!!

Windows Terminal version
1.13.10983.0(windows terminal preview)
Windows build number
21H2(19044.1645)
Other Software

git version 2.35.1.windows.2，git bash
zsh 5.8 (x86_64-pc-msys)

Steps to reproduce

git clone some project
window termial use git bash
cd the project you just cloned

Expected Behavior
cd into the project folder correctly
Actual Behavior

As you can see, bizplat-ui is a project using git version control,but I can't cd into the project folder,and terminal has no responding.

But when I just use git bash, I can cd into project folder correctly.

my windows terminal git bash config

Does the cd work in an unelevated (not admin) window?
Can you cd into that directory in other shells (cmd, powershell, etc)/?

@zadjii-msft

I can't cd whether is admit or not
I can  cd into the directory by powershell

2. I can  cd into the directory by powershell

Well that's bizarre. That sure doesn't make sense. I wonder if there's some difference between just running git bash directly, and the commandline you have set for that profile. Does the same thing occur if you run c:\Program Files\Git\bin\bash.exe -l -i from the Run Dialog (win+r)/?

@zadjii-msft I've tried running c:\Program Files\Git\bin\bash.exe -l -i from the Run Dialog (win+r).It turns out that I can open it successfully. But still, I can't cd into bizplat-ui directory, and terminal has no responding.

This is probably due to Git Prompt Integration. It is trying to read the git repository, and it cannot do that. So the prompt hangs.
This issue is outside of Terminal or Console's control. You may need to debug this with the Git For Windows project.


Discussion: Populate the device/stream stack if it is empty?

Follow-up of #5705.
Starting CuPy v10 the method Device.use() and the CUDA runtime API runtime.setDevice() are not recorded in the per-thread device stack (for a good reason):
https://github.com/cupy/cupy/blob/fc04c9445f488272c9a1b78999e4e0df15a721b5/cupy/cuda/device.pyx#L180-L191
I am opening this issue mainly to discuss a potentially minor breaking change. My suggestion is when calling __enter__ we first check if the device stack is empty or not. If so, we record the current device in the stack, and proceed as usual, otherwise do nothing. Roughly:
 def __enter__(self):
     stack = _ThreadLocalStack.get()
     curr_dev = runtime.getDevice()
     if len(stack) == 0:
         stack.push_device(curr_dev)
     if self.id != curr_dev:
         runtime.setDevice(self.id)
     stack.push_device(self.id)
     return self

But I would like to know if I miss any subtlety here (like I did earlier today when "re-discovering" the v10 behavior...) 😅
This is useful when the device context is set globally (via either .use() or runtime API) when initializing multi processes. For example, this would not lead to surprises:
import cupy as cp

cp.cuda.runtime.setDevice(1)
with cp.cuda.Device(0):
    pass

cp.cuda.runtime.getDevice()  # --> would become 1 after implementing this proposal

This discussion applies to the stream stack, since the device and stream should go together.

Consequence of this proposal:
import cupy as cp

cp.cuda.runtime.setDevice(1)
with cp.cuda.Device(0):
    pass

cp.cuda.runtime.getDevice()  # --> would become 1

cp.cuda.runtime.setDevice(0)
with cp.cuda.Device(1):
    pass

cp.cuda.runtime.getDevice()  # --> would become 1

Let me close this issue as I prepare for an alternative proposal. I'll link it here once submitted.

Let me close this issue as I prepare for an alternative proposal. I'll link it here once submitted.

#6965.


Editor freezes more than usual when hitting play.

Please fill in the following fields:
Unity editor version: 2019.3.7f1
Firebase Unity SDK version:  6.13.0
Source you installed the SDK (.unitypackage or Unity Package Manager): .unitypackage
Firebase plugins in use (Auth, Database, etc.): FirebaseMessaging.unitypackage
Additional SDKs you are using (Facebook, AdMob, etc.): None
Platform you are using the Unity editor on (Mac, Windows, or Linux): Windows 10
Platform you are targeting (iOS, Android, and/or desktop): Android
Scripting Runtime (Mono, and/or IL2CPP): Tried both
Please describe the issue here:
(Please list the full steps to reproduce the issue. Include device logs, Unity logs, and stack traces if available.)
Everytime i hit play (even on a empty project), Unity takes too long to build, compared to the usual time, this is not practical for me.
Also, when the game starts (the play button becomes blue), the editor completely freezes for a few seconds as if it was compiling the game once again.
Am i doing something wrong ?
To easily reproduce this issue:

Create a empty 3D project
Switch to Android Platform in the build settings
Import FirebaseMessaging.unitypackage from Assets -> Import Package
After import, a popup shows up to enable Android Auto-Resolution, click "Enable"
The Unity Packager Manager Resolver window opens, click "Add selected registries"
Wait until all dependencies update.
Don't add anything else, and hit "Play"
You will notice the editor freezing and taking way too long to build.

This problem happens even if you don't have any base code running.
Also, tried adding the google-services.json, but it doesn't do anything in this case.
Please answer the following, if applicable:
Have you been able to reproduce this issue with just the Firebase Unity quickstarts (this GitHub project)?
Yes, the exact same thing happens here, and it feels worse.
What's the issue repro rate? (eg 100%, 1/5 etc)
100%

Hi @axelbau24 ,
Thanks for your report!
You've done everything right so far, but I've been unable to replicate the issue on my end. Can you tell me if you see any windows, dialogs, &c pop up after the first time you hit play (screenshot them if you like/can)? Do you see anything interesting in your Console window/would you mind sharing your Editor log?

(note that if you don't feel comfortable sharing your Editor log publicly, you can file a bug here and link this issue/say you're trying to add information for this bug).
Is there anything else of interest that you can think of in your setup? Are you using meta files? A version control system other than Git? Is part of your unity project readonly or stored on a synced volume (ex: Dropbox, iCloud, &c)?
I'm checking in with the rest of the team to see if there's anything else we need to verify.
Thanks,
--Patrick

I have similar experiences with Firebase Core and Analytics. If I remove them from the project then everything goes back to normal. I doubt that this would make any difference, but I used External Dependency Manager for Unity (EDM4U) and Unity Package Manager to install the packages.
Nothing is synced in the background for my project, it is a fairly straightforward setup. Code generation is set to .NET 4.x and IL2CPP, but changing the setting to Mono makes no difference.
A similar delay also happens when I switch from my code IDE (JetBrains Rider) to Unity editor, if there was any change on the source code and it is recompiled in the background Unity editor stops responding for a good 15-20 seconds.
I have tried profiling the editor to figure out what is going on. I am no expert on Unity profiling, but as it seems to me when this delay happens then profiler stops capturing anything until the editor starts working again.
Please let me know if there was any other information I could provide.

I have attached the editor log from the moment I press the play button:
editor_log.txt
This line looks rather suspicious to me:
Unloading 534 unused Assets to reduce memory usage. Loaded Objects now: 5522.
Total: 15.606300 ms (FindLiveObjects: 0.836300 ms CreateObjectMapping: 0.170500 ms MarkObjects: 13.140400 ms  DeleteObjects: 1.457900 ms)

I seem to have the same issue. Looking at the editor profiler, there is at least one entry that seems suspicious to me:

Note that I'm running on Windows 10. I can't say for sure if that XCodeProjectPatcher is related to firebase, but looking through some results on google it seems to be the case. But this seems to be a specific thing for IOS, so why is it running on my windows environment? It seems to be responsible for at least 4s of startup time.
I'm using Firebase 6.13, auth, database and analytics. And google admob.

Exact same issue mentioned by MateusMP. It takes almost 4-5 second to enter play mode in the empty project with just Firebase core insalled from package manager.

I am running Unity on Windows too, maybe this is an important piece of information.

Thank you for your answer @patm1987
Here's my Editor Log Editor.log
Also, noticed some iOS stuff happening, even though i'm just compiling for Android, just like @MateusMP did, In my case i don't see anything related to XCode, just the iOS resolver (which i don't think it should be there and also taking a long time to compile.

Is there any way to disable this iOS resolver?

I'm having a similar symptom (8 second slowdown right after entering playmode). Platform is set to Android. Same Unity and SDK version as original poster. I have also tried disabling all auto-resolve functions in the package, but this seems to get called no matter what. This is a critical issue.

I have the same issue.
Freezes at 10-15 seconds on press play button.

Unity 2019.2.21, Firebase 6.12.0 and 6.13.0

The majority of this slowdown appears to be due to the Google Play Resolver / External Dependency Manager. When removed, the slowdown goes away. Of course, Firebase won't load so you'll need to add it in again before you build - and this option only works when using the manual install because if you're using Firebase unity packages it will not let you uninstall the EDM because the firebase libraries rely on them.

@GorillaOne How do I remove that? Thanks!

I have the same issue.
Freezes at 10-15 seconds on press play button.

I’m having the same issue, but on top of this the editor freezing for 20seconds after each code save. It’s getting unbearable. What would normally take me 2 mins to do, takes 3-4x as long

I believe this is the same issue as in #644

I believe this is the same issue as in #644

I think you are right - I was looking at #644 earlier and sounds very similar.

The issue is still present in 6.14.0, EDM version 1.2.153.
As a workaround for those using the Package Manager versions, you can simply delete the Editor folder and Editor.meta file within<EMAIL_ADDRESS>You may have to do it every time you start Unity.
You'll sometimes get an error message about failing to load Firebase.Editor.dll, but it seems you can safely ignore it.

The issue is still present in 6.14.0, EDM version 1.2.153.
As a workaround for those using the Package Manager versions, you can simply delete the Editor folder and Editor.meta file within<EMAIL_ADDRESS>You may have to do it every time you start Unity.
You'll sometimes get an error message about failing to load Firebase.Editor.dll, but it seems you can safely ignore it.

The issue is still present in 6.14.0, EDM version 1.2.153.
As a workaround for those using the Package Manager versions, you can simply delete the Editor folder and Editor.meta file within<EMAIL_ADDRESS>You may have to do it every time you start Unity.
You'll sometimes get an error message about failing to load Firebase.Editor.dll, but it seems you can safely ignore it

What would be the downside of this? Would it be that you can’t  add any new firebase packages or does the dependency manager get re-added (and that’s why we need to remove each time?)

What would be the downside of this? Would it be that you can’t add any new firebase packages or does the dependency manager get re-added (and that’s why we need to remove each time?)

You lose all firebase editor-specific functionality, but you can restore the folder when you actually need the functionality, and keep it removed when you don't. I assume you need it for building, resolving dependencies, probably for installing or removing packages too.
Obviously not ideal but I think it's a serviceable workaround for now, until this issue is fixed.

Same issue here!
When I play a scene, music starts to play but all graphics and the unity editor are frozen for a couple of seconds.
I can confirm this happened only after adding Firebase Analytics (Android + iOS) to my project.

Unity 2019.3.10f1, Firebase 6.14.0

Deleting this folder<EMAIL_ADDRESS>does NOT solve the issue. It is a problem with Firebase and not EDM.
The only workaround I was able to figure out is to remove Firebase from the project while I edit it and put it back when I create a new build.

@almosr The workaround I posted definitely solves it for me. Maybe it's two different issues, or our environments are different somehow.
A reminder just in case, Unity restores the folder when you open the project, so you have to delete it when the project is already open.

@ideka When I deleted that aforementioned folder while Unity is open then the issue disappears, because Firebase fails to initialize. Unity keeps throwing various errors, I would rather recommend to remove Firebase from the dependencies, that is a cleaner way to workaround the issue.

@ideka Thanks! I can confirm that this works for me (and yes, it shows the Editor error), but everything else seems to work fine (fireauth, firestore, cloud functions). I can live with this hack for now :D
@almosr I could, but I'm already fairly far along with firebase

@almosr Removing EDM while working in the project works for me as well. As mentioned above, Unity regenerates the EDM when you open the project and when you open the Package Manager.
If you don't need firebase, then removing firebase is an excellent option. However, I'm assuming that the folks here, like myself, who have this problem need firebase. Removing the EDM when you open the application prevents the slowdown because it prevents the Firebase editor scripts from running, which is where I believe the issue is located. It's not great, but its just a workaround until they fix it.

Not sure if it's already been made explicit or not, but I've been working on the same project in the Windows 10, OS X, and Linux versions of Unity, several different versions, and this only seems to happen on Windows (all 2019 versions I've tried), so you'll have trouble reproing on GLinux or Mac.
My workaround is to locally delete Firebase.Editor.dll and its meta file, and the issue goes away (but I don't commit or it breaks our game in CI builds). This has been happening since at least January, and in particular before the migration to the Unity package manager install, so it's not a particularly new change that's caused it.

For me it's also crippling slow with Unity 2019 and Windows10.
However it seems like deleting the editor.dll doesn't bring any noteworthy improvement.
Startup times are around 20 seconds and also freezes if I change anything in vs code and come back to unity.
Basically impossible to work like this.

Seems #667 is duplicate of this one. The problem seems very straight forward:

GenerateXmlFromGoogleServicesJson class has a [InitializeOnLoad] attribute so Unity will load it and execute its static constructor after each recompile and when entering play mode.
Static constructor will schedule CheckConfiguration() to be called on first next Update,
which will then call UpdateConfigFileDirectory()
which will for every "google-services.json" and "GoogleService-Info.plist" file in project call ReadBundleIds(file)
which will run "generate_xml_from_google_services_json.exe" on Windows and "python generate_xml_from_google_services_json.py" on other platforms and each invocation of that takes more than 5 seconds, keeping the Editor frozen.

Surely Firebase doesn't need to do all this on each load so please just cache it somehow or something so we can continue making games :)

You can speedup recompile time by removing 'generate_xml_from_google_services_json.py'.
After that you will see error message on each recompile. But that's ok.
Also you have to restore this file before 'platform build'.

You can speedup recompile time by removing 'generate_xml_from_google_services_json.py'.
I've tried that but Unity will restore it pretty quickly in my case.

I think it restores 'generate_xml_from_google_services_json.py' on Unity restart.
Can be solved with simple Unity script which only responsibility is to delete it.

I'll copypaste my comment from another closed thread on this topic. I strongly believe that optimizing the performance/caching/removing files when not building are just half measures. The real question is should this code run all the time at all?
I think the best solution is to make an option of not running this code at all. In the end, when no plugins have been changed, it does nothing and takes 10-20 seconds. So my suggestion is to make two options, like:

Run resolvers/version handlers only at build time
Run resolvers/version handlers only manually

I don't mean that there should be no optimization, only that providing these options imo is faster and more reliable way to deal with the issue.

That's 100% true, we need solution from Google, but also we have to make games now)
That's a dirty hack to save some time and continue to use Firebase.

Note in the stack trace that I described that this is done from a method called CheckConfiguration. Its purpose seems to be checking how your project is configured and if it finds something off it presents a dialog to the user telling him about it and offering him a way to fix it. That is probably a reason it can't just run on build since if build is running within some auto build system, there is no user that can be asked to fix the issue.
Still there must be a way to optimize it or have it run only in some special circumstances and not always.

I'm experiencing this issue in Unity 2019 on Windows 10 as well. Unity is effectively impossible to use with Firebase in this state.
I've not been able to find any of the files in the posted workarounds so this issue is a hard blocker for me. Getting a fix for this issue prioritized would be great as Firebase is otherwise quite nice and I'd like to continue using it.

Happening here as well. Workarounds posted above did nothing here. Have an Android project on Windows 10, yet the callstack shows me an Xcode project check.

Can we not run Xcode stuff on windows?
Can we not run this at every enter play mode?
Can we make this a build script?

Running the game takes 13 seconds on the python script alone (on a fast SSD and a 3950X). This prompted an internall talk of looking for alternatives to Firebase.

Not sure if it's already been made explicit or not, but I've been working on the same project in the Windows 10, OS X, and Linux versions of Unity, several different versions, and this only seems to happen on Windows (all 2019 versions I've tried), so you'll have trouble reproing on GLinux or Mac.
My workaround is to locally delete Firebase.Editor.dll and its meta file, and the issue goes away (but I don't commit or it breaks our game in CI builds), but if that dll is anywhere in even an otherwise empty project, things get slow. This has been happening since at least January, and in particular before the migration to the Unity package manager install, so it's not a particularly new change that's caused it.

It's worth noting that after we updated to the newer package manager install, this doesn't work anymore.

The issue is still present in 6.14.0, EDM version 1.2.153.
As a workaround for those using the Package Manager versions, you can simply delete the Editor folder and Editor.meta file within<EMAIL_ADDRESS>You may have to do it every time you start Unity.
You'll sometimes get an error message about failing to load Firebase.Editor.dll, but it seems you can safely ignore it.

This has been working for me. Quick and easy until it's fixed.

This has actually been an issue for a long time (freeze upon hitting Play causes by some part of Firebase), but this latest version (not sure which I had before) is about 2x as slow as it was before which is unbearable.
Please fix this asap with a manual toggle for now. :/

Same freeze issue on macOS, Unity 2019.3.5f1, SDK 6.14 installed from UPM (migrated from earlier .unitypackage installs). In my case, the issue is caused by EDM, not directly by the python generator script (though see @igor84 note in issue #667). The workaround with deleting the EDM Editor folder works, along with the noted side-effects.
I found a more reliable workaround using the new "Enter Play Mode Settings" option introduced in Unity 2019.3. https://blogs.unity3d.com/2019/11/05/enter-play-mode-faster-in-unity-2019-3/
In the Edit menu, Player Settings, Editor section, I enabled "Enter Play Mode Options (Experimental)". For my project, I disabled the "Reload Domain" option, and enabled the "Reload Scene" option.

Play mode starts within a second or so for me now. Note that this feature could introduce other bugs, depending on your project. I seem to have good luck so far.

Thanks for the other workaround Mike!
I actually tried that a month ago and it is indeed quick! But because of some static refs throughout the project it's very troublesome to manage on our end. So a caution for people trying it out, read Unitys warning about it before you start relying on it. :)

We are also having a really hard time with this behavior.
We've tried going over to a manual install to avoid the Dependency Manager entirely, but that has caused difficulties in the build process.
This is extremely frustrating to work with as it will freeze for 30 seconds every time

You launch unity
Recompile from a code change
Press play in the editor

This crushes developer productivity.
I've written a small script that will execute @Whyser 's workaround automatically - deleting the Dependency Manager editor scripts. The initial load of Unity is still slow, but after that the script runs and recompiles/playing should be back to "normal".
Looking forward to a proper resolution!
#if UNITY_EDITOR
using System.Collections;
using System.Collections.Generic;
using System.IO;
using UnityEditor;
using UnityEngine;

[InitializeOnLoad]
public class SuppressDependencyManager : MonoBehaviour
{
    static SuppressDependencyManager()
    {
        Debug.Log("Deleting dependency manager files in attempt to work around https://github.com/firebase/quickstart-unity/issues/639");

        var editorFolderPath = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "Library", "PackageCache"<EMAIL_ADDRESS>"ExternalDependencyManager", "Editor");
        var editorMetaPath = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "Library", "PackageCache"<EMAIL_ADDRESS>"ExternalDependencyManager", "Editor.meta");
        if (Directory.Exists(editorFolderPath))
        {
            Directory.Delete(editorFolderPath, true);
        }
        if (File.Exists(editorMetaPath))
        {
            File.Delete(editorMetaPath);
        }
    }
}
#endif

I was able to patch the Firebase.Editor.dll using dnSpy to avoid the freeze each time you press play.
Everything works and the freeze is gone !
I did the following:

download dnSpy
open Firebase.Editor.dll with dnSpy
open dlls referenced by Firebase.Editor.dll (Unity's dll and Google.JarResolver, Google.IOSResolver, Google.VersionHandlerImpl)
navigate to the method GenerateXmlFromGoogleServicesJson.cctor()
in the code inspector, right-click, Edit Method (C#)
replace the following:

RunOnMainThread.Run(delegate
{
	GenerateXmlFromGoogleServicesJson.CheckConfiguration();
}, false);

with:
if (!EditorApplication.isPlayingOrWillChangePlaymode)
{
	RunOnMainThread.Run(delegate
	{
		GenerateXmlFromGoogleServicesJson.CheckConfiguration();
	}, false);
}

Press "Compile" in the edit method window
Fix compilation issues (there is synthax problems with CompilerGenerated fields. (I renamed those fields like this: <>f__mg$cache0 => f__m0 )
After the compilation success: File -> SaveModule

You should now have a patched version of Firebase.Editor.dll which doesn't call the python script each time you press Play.
This is only a workaround and it fix only the issue when pressing PLAY.
The freeze still occurs after each compilation.
But once you start modifiying code, you can do pretty much anything. So I guess you could add an EditorPrefs per exemple to enable/disable the problematic method.

I was able to patch the Firebase.Editor.dll using dnSpy to avoid the freeze each time you press play.
Everything works and the freeze is gone !
I did the following:

download dnSpy
open Firebase.Editor.dll with dnSpy
make sure dnSpy reference only one version of mscorlib
open dlls referenced by Firebase.Editor.dll (Unity's dll and Google.JarResolver, Google.IOSResolver, Google.VersionHandlerImpl)
navigate to the method GenerateXmlFromGoogleServicesJson.cctor()
in the code inspector, right-click, Edit Method (C#)
replace the following:

RunOnMainThread.Run(delegate
{
	GenerateXmlFromGoogleServicesJson.CheckConfiguration();
}, false);

with:
if (!EditorApplication.isPlayingOrWillChangePlaymode)
{
	RunOnMainThread.Run(delegate
	{
		GenerateXmlFromGoogleServicesJson.CheckConfiguration();
	}, false);
}

Press "Compile" in the edit method window
Fix compilation issues (there is synthax problems with CompilerGenerated fields. (I renamed those fields like this: <>f__mg$cache0 => f__m0 )
After the compilation success: File -> SaveModule

You should now have a patched version of Firebase.Editor.dll which doesn't call the python script each time you press Play.
This is only a workaround and it fix only the issue when pressing PLAY.
The freeze still occurs after each compilation.
But once you start modifiying code, you can do pretty much anything. So I guess you could add an EditorPrefs per exemple to enable/disable the problematic method.
You can find a patched version I made for firebase 6.13.0 here: https://github.com/bhallionOhbibi/FirebaseEditorDll

I'm gonna ping @DellaBitta (sorry, if you're the wrong person!) but maybe you can move/merge/assign this issue so we dont have to download a potentially malicious DLL to fix this issue (no offense bhallionOhbibi, really appreciate your effort!)?

Hi @Whyser,
I'll send it to the team for review.  Thanks!

are there any news about this issue? It is there for soo long :(

The best workaround I came up with is to use dnSpy to change and recompile Firebase.Editor.dll as @bhallionOhbibi described but I changed it a bit differently. I removed the call to GenerateXmlFromGoogleServicesJson.CheckConfiguration() from static constructor but I made that method public and added [MenuItem("Windows/Firebase/Check Configuration") above it. Then in my project I call it from my implementation of IPreprocessBuildWithReport.OnPreprocessBuild described here: https://docs.unity3d.com/ScriptReference/Build.IPreprocessBuildWithReport.html
That way it is automatically called before the build and I can also manually call it anytime from the menu, but it is not invoked on each recompile and when entering play mode.

Hi all, a status update:  the proposed change from @bhallionOhbibi will be part of our official 6.15.0 release -- currently in progress.

Della please don't let it do this process during compilation of scripts. I would say Igor's solution is better to only do during build (or manually).

@Whyser
Your suggestion make sense.  That process creates two files out of google-services.json/GoogleServices.plist

google-services.xml for Android build
google-services-desktop.json for the editor

We might be able to postpone 1 until build time.  It will take some time and we need to investigate the right timing to generate it for different Android build configuration.
There are more improvement can be done for the freeze.  As for next release, we only integrated  @bhallionOhbibi suggestion for temporary relief.  More improvement will be coming from either Firebase or EDM4U.
Thanks for the update!

Still seeing the play mode delay in 6.15.0 on macOS. Anyone else, or am I possibly seeing another issue?

@igor84 I like your approach. I implemented it myself, but I wanted to confirm if I did the correct way. This is a first for me to even dive into this area, let alone dnSpy. Can you confirm or correct what I did to make sure it is correct?

Unity Package Manager. Installed Firebase 6.15.0. (which already implements @bhallionOhbibi solution. Great find btw.
Downloaded dnSpy v6.1.4. Removed all default assemblies. Opened Firebase.Editor.dll and all dependent references.
Edited the GenerateXmlFromGoogleServicesJson class in Firebase.Editor.dll:
a) Removed the following from it's static constructor:

if (!EditorApplication.isPlayingOrWillChangePlaymode)
{
	RunOnMainThread.Run(delegate
	{
		GenerateXmlFromGoogleServicesJson.CheckConfiguration();
	}, false);
}

b) Made the CheckConfiguration static method public and added `[MenuItem("Window/Firebase/Check Configuration")]
c) Fixed all compiler errors (cache references) and saved module.

Copied the modified Firebase.Editor.dll to another folder in the [Project] directory.
Created an EditorStartup.cs script:

using UnityEditor;

[InitializeOnLoad]
public class EditorStartup
{
    static EditorStartup()
    {
        string fromPath = "Support/Firebase.Editor.dll";
        string toPath =<EMAIL_ADDRESS>        FileUtil.ReplaceFile(fromPath, toPath);
    }
}

This was because I noticed it would revert the Firebase.Editor.dll file on each Startup. So this way it overwrites it each time it loads.

Created a CustomBuildProcessor.cs script:

using UnityEditor.Build;
using UnityEditor.Build.Reporting;
using UnityEditor;

public class CustomBuildProcessor : IPreprocessBuildWithReport
{
    public int callbackOrder { get { return 0; } }

    public void OnPreprocessBuild(BuildReport report)
    {
        EditorApplication.ExecuteMenuItem("Window/Firebase/Check Configuration");
    }
}

This is where I realized why you made that MenuItem as I didn't find any other way to call CheckConfiguration. And that was pretty clever of you. Well done.
So would this setup run the Check Configuration on build properly? Or did I miss or misundertand something?

Hi @jasonlawless. Your solution should work but it is different from mine a bit and has some drawbacks:
I did't install Firebase through Unity package manager. I downloaded the unitypackage files of External Dependency Manager and Firebase and then installed them manually. That way their dll files are directly in the project and Unity doesn't overwrite the Firebase one once I change it. That makes a drawback in your solution. Since you had to write that script to constantly overwrite the DLL that might also affect your compile time since Unity might need to again recompile the project when it detects that dll has changed. Although Unity might not do that since it is outside its Assets folder, I am not sure.
Also in step 6 since you made the CheckConfiguration method public you should be able to call it like this:
GenerateXmlFromGoogleServicesJson.CheckConfiguration();

That is what I did.
At first I didn't even implement step 6 I only made a MenuItem because my understanding was that this CheckConfiguration thing is something that needs to be executed only after changing google services json or plist file that is downloaded from Firebase console and maybe when the Firebase plugin is updated and in both of those cases I can now just run it manually from the MenuItem. But since I don't have a complete understanding of what it does I decided, just in case, to also call it just before the build.

@igor84 Thanks for your quick and detailed reply.
Considering your advice regarding Firebase installation via EDM or PM and the possibility of longer compilation times: I have found out that [InitializeOnLoad] indeed is ran on Editor load and each script change, as well as domain reload. It seems I misunderstood this and assumed this only ran on Unity Editor load and that's it. Nonetheless, I decided to test out to see if Firebase.Editor.dll gets "refreshed" on each compilation and if that does increase my compile time before I decided to switch away from PM and go to EDM solution.
I modified the EditorStartup.cs script like so:
using System;
using System.IO;
using UnityEngine;
using UnityEditor;

[InitializeOnLoad]
public class EditorStartup
{
    private static DateTime matchDate;

    static EditorStartup()
    {
        string fromPath = "Support/Firebase.Editor.dll";
        string toPath =<EMAIL_ADDRESS>        if (matchDate == DateTime.MinValue)
            matchDate = File.GetLastWriteTime(fromPath);
        DateTime modifiedAt = File.GetLastWriteTime(toPath);
        Debug.Log($"Custom file mod date: {matchDate} => Firebase.Editor.dll mod date => {modifiedAt}");
        if (modifiedAt != matchDate)
        {
            FileUtil.ReplaceFile(fromPath, toPath);
            Debug.Log("Firebase.Editor.dll was patched.");
        }
    }
}

And tested it by first opening Unity Editor. Then making a random script change. [See Console log screenshot below]. Turns out Firebase.Editor.dll ONLY get's "refreshed" on Editor startup ONLY. It does not go through that process on each compilation. And seeing as how my compilation time has already dropped back to its initial state prior to Firebase installation, I think I'll just stick with this solution for now. Again, @igor84 thank you for pointing this out because I was re-writing the Firebase dll on each script compile. Now that is not the case.

Now. As for GenerateXmlFromGoogleServicesJson.CheckConfiguration() I too though that I would be able to call it like you suggested at first. But for some reason I couldn't figure out which namespace to using and Firebase.Editor either doesn't exist in my project or is not accessable. Anyways, I feel I am in deep waters with only one floaty right now. That is why I decided to call it via the Menu. Can you help me figure out what I need to reference so that I can call it directly via the class?
Thanks

You need to using Firebase.Editor; and the script from which you call it must be under Editor folder anywhere under Assets or if you are using asmdefs then it needs to be a part of Editor asmdef.

Boom! It worked. Thanks @igor84 you're the man!
Also regarding my workaround. If you close your project whilst you have some script (compile) errors, and then next time you load Unity, because of the compiler errors, the EditorStartup script will not execute, therefore the Firebase.Editor.dll file will be restored by Google's SDK.
Minor hiccup, but it's there. Anyways, that's my two cents. I'm happy about getting this resolved.

Hi guys, can you please do a resume with the solution?, i'm a little confuse and is very frustating do a game with this problem every time i change code or just hit play.
Tanks

Hey @jasonlawless. Sorry for the late reply. If you still need some help we can try chatting on https://tlk.io/unityfirebase.
@andruDigital jason made a step by step guide here: https://github.com/firebase/quickstart-unity/issues/639#issuecomment-640161250, and my comment was just not to use unity package manager but download and manually install unitypackages of EDP and Firebase, to avoid some hurdles. I don't see how could we explain it more clearer.

FYI, for anyone wishing to use my guide. This is to add to the "hurdle" that @igor84 mentioned in the previous comment (don't know how to ref link the comments here yet). Anyways, IF you close Unity (or it crashes) whilst there are ANY compilation errors outstanding, next time loading Unity will NOT run the Firebase.Editor.dll script override. You will need to copy the file manually. I been to busy with other things to look for a way to circumvent this while holding on to PM Firebase installation. If I get around to it, I will post an update.
Otherwise, if you would like to avoid this all together, consider using @igor84 advice on installing Firebase using External Dependency Manager instead.

hi @jasonlawless, i don't know how to fix the compile errors in dnSpy, the ones with the cache references... can you pleas guide me on how to fix it. Tanks.

The compiler doesn't like the <>f__mg$cache0 syntax. They are just references or pointers. It's like declaring a variable that also conforms to a certain syntax. So in this case you can change it to something like f__mg0 or iHateFixingCompilerErrors it doesn't matter. Just make sure you follow through with the indexing. Try to keep it simple. So if it is <>f__mg$cache4 change it to f__mg4 or iForgotToDoMyLaundry as long at you rename the the same references with the same name as <>f__mg$cache0 is referenced more than once, if you rename it to iLoveWonderWoman, make sure you rename the next <>f__mg$cache0 to the same variable name. Does that make sense?

Heads up, not quite sure if this has something to do with this patch that we implemented or not, but I thought I'd bring it up here and see if anyone else is or has experienced something like this:
I used Auth and Firestore SDK, no issues. Works good. But it seems that there is an "invisible" exception that gets thrown while in Editor mode. It remained undetected to me until I built the app. When I load app, it throws a funky exception on load:

And this exception keeps coming up with every action you take in the app, which prevents the app from running correctly. So I attached VS debugger to it and did some digging. Wasn't simple to detect, but found out that it all starts from this:
FirebaseAuth auth = FirebaseAuth.DefaultInstance;

Which leads to this exception:
System.IO.FileNotFoundException: Could not load file or assembly 'Firebase.Crashlytics' or one of its dependencies
File name: 'Firebase.Crashlytics'
  at System.AppDomain.Load (System.String assemblyString, System.Security.Policy.Evidence assemblySecurity, System.Boolean refonly) [0x00031] in <437ba245d8404784b9fbab9b439ac908>:0
  at System.AppDomain.Load (System.String assemblyString) [0x00000] in <437ba245d8404784b9fbab9b439ac908>:0
  at (wrapper remoting-invoke-with-check) System.AppDomain.Load(string)
  at System.Reflection.Assembly.Load (System.String assemblyString) [0x00005] in <437ba245d8404784b9fbab9b439ac908>:0
  at Firebase.FirebaseApp.InitializeCrashlyticsIfPresent () [0x00000] in Z:\tmp\tmp.txs8ldQ514\firebase\app\client\unity\proxy\FirebaseApp.cs:487

Now, I thought that this would be a simple fix. I never setup Crashlytics in my app, so I installed the SDK and referenced the Firebase.Crashlytics.dll assembly and built again. But to my disappointment, same error.
So I started to see if there were any "invisible" exceptions during build, and sure enough I found that my CustomBuildProcessor.cs script mentioned earlier in this thread where I had this line:
GenerateXmlFromGoogleServicesJson.CheckConfiguration();

was throwing this error:
"System.IO.FileNotFoundException:
File name<EMAIL_ADDRESS>  at (wrapper managed-to-native) System.Reflection.Assembly.LoadFrom(string,bool)
  at System.Reflection.Assembly.LoadFrom (System.String assemblyFile) [0x00000] in <437ba245d8404784b9fbab9b439ac908>:0
  at System.Reflection.Assembly.InternalGetSatelliteAssembly (System.String name, System.Globalization.CultureInfo culture, System.Version version, System.Boolean throwOnFileNotFound, System.Threading.StackCrawlMark& stackMark) [0x000ad] in <437ba245d8404784b9fbab9b439ac908>:0 "

And when I looked at the referenced path, the folder en-US does not even exist, and neither does that file.
So I commented out that entire script, and built again.
But the DefaultInstance STILL fires that same exception. If someone is experiencing or has experienced this issue, please reach out to me if you have a solution. @igor84 maybe you have some insight? Or we can put our heads together to figure this out? I scoured the internet and found nothing about this anywhere.

I just rechecked our build logs but I couldn't find that FileNotFoundException with en_US folder. Maybe again it has something to do with us not using the Package Manager.
On the other hand I think we did encounter that Could not load file or assembly 'Firebase.Crashlytics' issue and I think we solved it by putting a link.xml file in the Assets folder with this content:
<linker>
	<assembly fullname="Firebase.Crashlytics" ignoreIfMissing="1">
		<namespace fullname="Firebase.Crashlytics" preserve="all"/>
	</assembly>
	<assembly fullname="Firebase.App" ignoreIfMissing="1">
		<namespace fullname="Firebase.App" preserve="all"/>
	</assembly>
</linker>

I made a video on installing and modifying Firebase: https://youtu.be/LVsTVU8cqpk

The compiler doesn't like the <>f__mg$cache0 syntax. They are just references or pointers. It's like declaring a variable that also conforms to a certain syntax. So in this case you can change it to something like f__mg0 or iHateFixingCompilerErrors it doesn't matter. Just make sure you follow through with the indexing. Try to keep it simple. So if it is <>f__mg$cache4 change it to f__mg4 or iForgotToDoMyLaundry as long as you rename the the same references with the same name as <>f__mg$cache0 is referenced more than once, if you rename it to iLoveWonderWoman, make sure you rename the next <>f__mg$cache0 to the same variable name. Does that make sense?

That made the trick, thank you... btw my Firebase.editor.dll is not in Library/PackageCache folder but in Assets/Firebase folder, and the edited dll dont get replaced every run of Unity even if i didn't made the EditorStart.cs

..or Firebase team can prioritize this? It's impossible to work on project which has Firebase in it without doing any of the hax-fixing above. I couldn't care less about added time during build, but when everything freezes for 20sec+ with a small script change...it's unbearable.
Any update on this @chkuang-g ?

Any news ? it should be N1 priority to fix this kind of bug !

Hey Alex,
Wow, thank you for following up on this. I didn’t expect any support to be this thorough, bravo. Unfortunately, not only has so much time passed since then, and due to other issues/limitations that I had to deal with when composing our project, I concluded the head ache that it caused out-weighted the time it would take me to write and deploy my own webserver. And I ended up doing just that, I setup a webserver and now each client connects to it via socket, and it’s working great.
That being said, I am still impressed with what you guys are doing and have done. I have used Firebase before on another platform, and that is precisely why I tried to implement it on the Unity platform, but it seems as though it could use a little more work. You guys are doing a great job with it, keep it up, I’m sure you’ll hit the bulls eye. As for this project, I will end up staying with my new solution as tacking on more layers is now going to be easier than restructuring it back to Firebase solution.
As far as what the issue was that broke the camel’s back. I believe it had to do with the fact that I couldn’t run the same client concurrently and simultaneously on the same machine. When I ran the first session, Firebase would create a temp file and ‘lock’ it, and when the second session loads, it attempts to access that locked file, but it can’t, and so the Unity client (whether it’s editor or runtime, doesn’t matter), just hard crashes. At least that was with the version I used, which if my memory serves me right was 6.15.0? If knowing the version is extremely important to you guys, let me know, I can review my git logs, as I have logged that somewhere in there.
Good luck with everything else,
Jason Lawless™
From: Alex
Sent: Sunday, July 26, 2020 7:55 AM
To: firebase/quickstart-unity
Cc: Jason Lawless™; Mention
Subject: Re: [firebase/quickstart-unity] Editor freezes more than usual whenhitting play. (#639)
Any news ? it should be N1 priority to fix this kind of bug !
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Why is this bug still there? It creates an extra 20 second delay on every run that makes it unusable. The profiler shows that most of this time is spent in XcodeProjectPatcher..cctor() > iOSResolver..cctor() > Mono.JIT which is particularly weird considering that I'm building for Android on a Windows computer.
I knew I was going to regret it when I finally decided to add Firebase to Unity. For now, my workaround is to not use Firebase. I'll check again next year if things have changed (the average delay for a critical bug fix in most Google libraries).

Hey folks,
Sorry for the late response.  The team is currently investigating the issue.  iOSResolver seems to contribute to this behavior the most as well as other factors.
We will compose a patch for this ASAP.
Shawn

I had to switch to OSX to work on project :D

Hi folks,
We just updated EDM4U to 1.2.157.
The most relevant changes to this thread is that now it will only initialize iOSResolver/AndroidResolver only if the current active build target is iOS/Android and not in play mode.  That is, if your current build target is not iOS, iOSResolver initialization will never happen when you click on the Play button.  If your current build target is iOS, iOSResolver initialization will happen after you exit the play mode.
At least, it will not try to do the slow searching for UnityEditor.iOS.Extensions.Xcode.dll unless you switch to iOS build target.  While this is a workaround for some bug in some versions of Unity, if you can switch to iOS build target, there is very high chance that this function will not even be called.
You should be able to get the latest .unitypackage here.  UPM version should be available in 20ish min.
However, I can still see a couple of performance improvement for both EDM4U and Firebase SDK.

[Firebase] Search for google-services.json and/or GoogleServices.plist
[EDM4U] Search for Dependencies.xml

These searches use standard AssetDatabase.FindAsset().  Since we do not require the user to label them or put them in a fixed folder, the code basically need to use a generic search across the whole Asset Database.  And our editor plugins require at least load them once to handle any upcoming events, such as post process event and build request.
The tricky thing is AppDomain reloads whenever play mode starts (there is a flag introduced in later version of Unity which prevent this.  I feel very skeptical about it.)
Still looking into some solutions.  And feel free to make some suggestoins.
I'll keep this thread open until we have a better solution for the searching.
Shawn

Did anyone profile AssetDatabase.FindAsset()? I don't think it's an issue performance-wise, but if it is you can probably cache its path in EditorPrefs after the first FindAsset?

I think it will be slower when the Unity project grows.
Caching is tricky since those files can be added anytime with or without Unity is opened.
Well, only if there is something that can be called only ONCE after the Unity is launched.
The closest thing I get so far is using [InitializeOnLoad], static constructor and EditorApplication.isPlayingOrWillChangePlaymode, which might be good enough. :/

BTW, some improvement from the Firebase SDK will be included in the next release.

Hello!
Thank you for working on improving the performance of the plugin! In the meantime, I think you can allow disabling the automatic checking (but have it enabled by default as to not alter the current behaviour) for those users who don’t need it all the time. Additionally, you can expose the method through the menu to allow manual checking.
Here's a working sample that creates a menu entry that can be toggled on/off and saves its state to the EditorPrefs:
    [InitializeOnLoad]
    public static class AutomaticCheckingSetting
    {
        private const string MENU_NAME = "Window/Firebase/Check Configuration Automatically";
        private const string PREF_NAME = "firebase_check_config_automatically";

        public static bool Enabled
        {
            get { return EditorPrefs.GetBool(PREF_NAME, true); } // Return true if the pref is not set
            set
            {
                EditorPrefs.SetBool(PREF_NAME, value);  // Saving editor state
            }
        }

        // Called on load thanks to the InitializeOnLoad attribute
        static AutomaticCheckingSetting() {

            // Delaying until first editor tick so that the menu
            // will be populated before setting check state, and
            // re-apply correct action

            Enabled = EditorPrefs.GetBool(PREF_NAME, false);
            EditorApplication.delayCall += () => Menu.SetChecked(MENU_NAME, Enabled); // Set checkmark on menu item
        }

        [MenuItem(MENU_NAME)]
        private static void ToggleSetting()
        {
            Enabled = !Enabled;
            Menu.SetChecked(MENU_NAME, Enabled);
        }
    }

And here’s how the current code would use it:
[InitializeOnLoad]
public class GenerateXmlFromGoogleServicesJson
{
    static GenerateXmlFromGoogleServicesJson()
    {
        if (!EditorApplication.isPlayingOrWillChangePlaymode && AutomaticCheckingSetting.Enabled)
        {
            RunOnMainThread.Run(delegate
            {
                GenerateXmlFromGoogleServicesJson.CheckConfiguration();
            }, false);
        }
    }

    [MenuItem("Window/Firebase/Check Configuration Now")]
    public static void CheckConfiguration()
    {
       // Check configuration logic
    }
}

☝ Good proposal @JoNax97 ☝
That would be very useful!

Indeed, checking the configuration is very useful, but It's not necessary on every recompile. Having a toogle to opt-out and a manual check option sounds awesome. @JoNax97's proposal looks great!

@JoNax97  excellent proposal, we are having the same problem.

This issue manifested in another way for me. It completely killed my ability to build the game. Due to a separate issue, our build generates a bunch of assets, and it does so by writing them to a folder that is inside the Assets directory. This was causing there to be a refresh in Unity after every file. Thousands of files were being written, so we were incurring this added delay thousands of times. Builds would go for 5 hours before being killed off, and probably would take days if left running. We sort of fixed the original problem by pausing refreshes with StartAssetEditing() and StopAssetEditing() calls before/after the asset generation, and that has alleviated the problem, but there are other aspects of our build that are outside of that, so still incurring overhead from the Firebase issue.

I have similar kind of issue on Unity 2019.4.16f1, Editor permanently freezes after 2nd/3rd Play or 2nd/3rd code compiling in Visual Studio, so I have to restart Unity every time. I have Firebase Database, Storage and Remote Config. I didn't have this issue before, even though I had freezes from time to time, but now I have this problem and don't quite understand where it came from.

@chkuang-g
It's been years. Is there any news of this issue? GenerateXmlFromGoogleServicesJson not only slow down editor performance, it also produce lots spam in Unity editor logs. This causes so much troubles when debugging CI pipeline.

Guys, come on! Is there any update yet on the issue? I am having serious troubles for some times now, can't work for more then 5 minutes in same project without needing to force kill unity!


sched_getaffinity fails on systems with more than 1024 cores

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.5.0
Python version: 3.9

Describe the current behavior
On doing import tensorflow a warning is shown 2 times:
sched_getaffinity: Invalid argument
can't determine number of CPU cores: assuming 4

Describe the expected behavior
TF works correctly on large systems
Other info / logs
The reason is, that the default struct for the CPU set contains only space for 1024 cores, so it will fail with EINVAL on systems with more cores, even when they are inactive.
The solution is more or less simple, by using the macros of glibc to allocate a larger struct. See e.g. the CPython os Module implementation.

@Flamefire,
Can you please elaborate about your issue so that everyone could understand what the issue exactly is? Thanks!

Sure:
The call at https://github.com/tensorflow/tensorflow/blob/5dcfc51118817f27fad5246812d83e5dccdc5f72/tensorflow/core/platform/default/port.cc#L75 only works for systems with at most 1024 cores
This is due to the usage of the default cpu_set_t at https://github.com/tensorflow/tensorflow/blob/5dcfc51118817f27fad5246812d83e5dccdc5f72/tensorflow/core/platform/default/port.cc#L74 which has only enough space for 1024 cores.
Hence: When using TF on a system with more than 1024 cores the call to sched_getaffinity will fail and TF will fallback to using only 4 cores as indicated by the warning.
The solution is to use the macros CPU_ALLOC_SIZE and CPU_ALLOC provided by glibc to dynamically increase the size of the struct passed to sched_getaffinity until there is enough space for all CPUs. See e.g. https://github.com/python/cpython/blob/0fa282c55f1a45765340cb24ed65c90ffe2aa405/Modules/posixmodule.c#L7129

This issue is not stale!

This issue is not stale!

This issue is not stale!

@Flamefire,
Sorry for the inconvenience caused by the Stale Bot. It doesn't bother you anymore. Thanks!

@Saduf2019 as the problematic code linked at https://github.com/tensorflow/tensorflow/issues/49833#issuecomment-850355351 is still current, yes the issue still persists

I opened a PR with an implementation based on CPythons os.sched_getaffinity and used as a patch for building TF with EasyBuild for a while now: #56633


[REG2.068a] Issue 14746 - Behavior change with struct destructor and alias this

https://issues.dlang.org/show_bug.cgi?id=14746
On stable branch instead of master (#1310)

thx


RuntimeError: Parent directory ~/orthogonal_basis does not exist.

I wanted to try if the Flux training works locally, I think I got quite far with the installation and maybe got the configuration correct to some extent. I'm using Windows. When I'm trying to run the notebook train-flux-concept-sliders, I get a few warnings which look semi-harmless, but then an error:
RuntimeError                              Traceback (most recent call last)
Cell In[8], line 7

(Removed most of the error print)

RuntimeError: Parent directory ~/orthogonal_basis does not exist.

I tried to look for similar issues, and only found this, "Question about training Flux".
https://github.com/rohitgandikota/sliders/issues/109
This is the code, if I'm not mistaken (sliders/flux-sliders/utils/lora.py):
def load_ortho_dict(n):
    path = f'~/orthogonal_basis/{n:09}.ckpt'
    if os.path.isfile(path):
        return torch.load(path)

So, where are the files, and more importantly, what are those files, exactly, and where can one get those, or are they something that
should be downloaded automatically? In any case, the path does not seem to exist, or perhaps it is not created.
Now the code is looking for a files in a path like this?
~/orthogonal_basis/000000042.ckpt

Any help would be appreciated, I would have wanted to try this out since I already tried a few other setups which I did not successfully get to run.

Ok, I see. I got around this by making a local directory:
def load_ortho_dict(n):
    #path = f'~/orthogonal_basis/{n:09}.ckpt'
    path = f'./orthogonal_basis/{n:09}.ckpt'
    if os.path.isfile(path):
        return torch.load(path)

And this folder would be in the flux-sliders directory:
orthogonal_basis
At least now the training seems to have started!


Initial implementation

Initial implementation of the xterm-ligature-support addon. This addresses part 2 of the solution for https://github.com/xtermjs/xterm.js/issues/958 (along with https://github.com/xtermjs/xterm.js/pull/1460). It handles font loading and resolution, wrapping the font in a ligature resolver, which is used for each render call after the font is loaded.
cc: @Tyriar

Yeah I have an Electron app on my box with some other stuff where I've been throwing packed versions of both packages. I'll see if I can whip up a more barebones one so you can play around (and as a possible example for the repo)

😮

When I run vim in the sample it breaks, @princjef did you test an app that enter application mode? (It just could be me?)

I'm getting this error when attempting to pull in the package via git to vscode:
❯ yarn add https://github.com/xtermjs/xterm-ligature-support.git#initial
yarn add v1.3.2
$ node build/npm/preinstall.js
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/opentype.js/-/opentype.js-0.8.0.tgz: ENOENT: no such file or directory, utime'/Users/daimms/Library/Caches/Yarn/v1/npm-opentype.js-0.8.0-acabcfa1642fbe894a3e4d759e43ba694e02bd35/dist/opentype.js.map'".
info If you think this is a bug, please open a bug report with the information provided in "/Users/daimms/dev/Microsoft/vscode/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

This seemed to work though 😕
yarn add file:///Users/daimms/playground/xterm-electron-sample/local_node_modules/xterm-ligature-support-1.0.0.tgz

When trying to include in VS Code I get issues where font-finder successfully getting the list of font files, but parsing the fonts fails such that xterm-ligature-support gets an empty list of fonts back. Still investigating.

This is the exception that's causing the issues:
TypeError: pStream.destroy is not a function
    at parse.ts:109
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:109)

https://github.com/princjef/font-finder/blob/78e28c0f2176fa2621d7768a1b0e8172ee2e5ca5/src/parse.ts#L101
<EMAIL_ADDRESS>is in node_modules

what version of node are you running on? stream destroy was added in node 8

Then I hit "Could not find font" due to fontLigatures.loadFile failing, the trace:
TypeError: util.promisify is not a function
    at Object.loadFile (index.ts:276)
    at Object.load [as default] (font.ts:23)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:109)

These setup problems go away when I run VS Code under Electron 2 (which I just merged into master).

yeah there's a requirement on node 8 right now, which came in sometime in the electron 1.8.x line. it's theoretically possible to get it down to 6, but the requirement comes from the promise-stream lib all the way at the bottom of the stack

Hmm, it looks like they're being drawn together but the ligatures aren't working 😕

i use neovim frequently and it's working (though i'm noticing some clipping of italicized characters). i'll take a look with vim in the sample:

Odd that it's not rendering together. My only guess for that is that the canvas isn't detecting Fira Code. the r of fira code is pretty distinctive so that's how i usually visually confirm that the right font is being rendered

Got it working! Needed to remove the various things turning ligatures off from the past setting:
font-variant-ligatures: none;
font-feature-settings: "liga" 0;

I'm a little surprised that CSS ligature settings impact text in a canvas, but whatever 🎉

i guess it saves them having to define a canvas API instead. definitely good to be aware of

Tried out vim on mac with the sample app and seems to be working alright:

@princjef vim appears to work fine when I got it working in VS Code

Updated with more useful error handling. I'll start looking at the dependencies next

Dependencies updated. Latest npm ls --production:
<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├──<EMAIL_ADDRESS>│ └──<EMAIL_ADDRESS>└─┬<EMAIL_ADDRESS>  ├──<EMAIL_ADDRESS>deduped
  ├─┬<EMAIL_ADDRESS>  │ ├──<EMAIL_ADDRESS>  │ └──<EMAIL_ADDRESS>  └─┬<EMAIL_ADDRESS>    └──<EMAIL_ADDRESS>
I also double checked and all remaining direct/indirect standard dependencies are MIT/ISC licensed so should be good there.


[Build Tools] Fixes docs build error from Node's fs

Building docs is broken. This fixes it.

Cloned a fresh copy of SUI
Ran npm i
Ran gulp build-docs and received error.

fs from Node now requires a callback for the third parameter, which was missing from the docs Gulp build task file, causing the gulp build-docs process to break with the following error:
[15:59:18] Starting 'build-docs'...
Building Metadata
Copying examples
Copying LESS source
Building Semantic for docs
(node:72733) [DEP0013] DeprecationWarning: Calling an asynchronous function without callback is deprecated.
fs.js:113
      throw err;  // Forgot a callback but don't know where? Use NODE_DEBUG=fs
      ^

Error: ENOENT: no such file or directory, open '/Users/username/Development/HTML/docs/out/metadata.json'
ITs-MacBook-Pro-6:ksc-ui photoroom$ ./node_modules/gulp/bin/gulp.js build-docs

Adding a simple error callback (with optional console log for error reporting) fixes this issue. Running the docs build process now works.
Not sure if this has already been hashed out for the next release cycle.

Are there any plans to merge this? We're trying to decide the best way to handle the current Semantic UI build issues (#4727). If the repository isn't being maintained any more at this point, it would be good to know so we can vendor the dependency and start applying our own patches.


flake: use nixpkgs.lib for lib output

Based on my understanding of NixOS/nix#2920, this would improve things. Since devos and anyone else using devlib usually only make use of the lib output, this would decrease download costs. Only the other outputs actually make use of nixpkgs, so nixpkgs shouldn't be pulled as a dependency from this library if only devlib.lib is used.
On a side note: I think lazy-fetching of inputs is a pretty good solution to the "development dependencies" issue.

Maybe we can set a precedent for the ecosystem and name dev-only dependencies with an underscore?
inputs._nixpkgs + a comment.

I would like that, but I ran into a couple issues. First, I don't think inputs can start with an '_', I get this error: error: flake reference '__nixpkgs' is not an absolute path. Might be an upstream bug.
Also deploy-rs makes use of follows = "nixpkgs", so we need to have an input called "nixpkgs" if we use deploy-rs.


don't allocate a wire.Header when parsing short header packets

This PR is part of the ongoing effort to reduce allocations in the fast path (after the handshake, when application data is exchanged). While we already use a sync.Pool for STREAM frames, and #2885 adds one for ACK frames, we still allocate a wire.ExtendedHeader for every incoming packet:
https://github.com/lucas-clemente/quic-go/blob/325bc1694dd84af5d96a1d4d9db7e011ec6f3664/internal/wire/extended_header.go#L20-L31
which contains an embedded wire.Header
https://github.com/lucas-clemente/quic-go/blob/325bc1694dd84af5d96a1d4d9db7e011ec6f3664/internal/wire/header.go#L48-L62
That's a pretty large data structure, especially considering that a QUIC short header packet only ever needs:

the key phase bit
the Destination Connection ID
the packet number

To avoid allocations, we can just define a function that returns (protocol.KeyPhaseBit, protocol.ConnectionID, protocol.PacketNumber) instead of a full wire.ExtendedHeader).
Unfortunately, making this change ended up in a major refactoring, as we have many functions that use a wire.ExtendedHeader as a function argument and then distinguish between long and short header packets in the function body. These functions had to be split up into one version for long and one for short header packets.
It will probably make sense to perform a similar refactoring for the code paths that send out packets.

It's hard to judge whether the tradeoff between code complexity and performance is worth it here. Do you have some numbers on how much inefficiency the GC currently adds, and whether this improves throughput, latency, etc?

We've recently resolved this as part of #3663.


NO-JIRA Update manager-run.sh to parametrize the MC control plane URL

No Jira
PR owner checklist
Please make sure that your PR meets the following requirements:

[ ] Your code is properly formatted according to this configuration.
Run mvn clean verify -DskipTests so that imports are correctly set.
[ ] Your commit messages are clear and reference the JIRA issue e.g: "[MGDOBR-1] - $clear_explanation_of_what_you_did"
[ ] All new functionality is tested
[ ] Pull Request title is properly formatted: MGDOBR-XYZ Subject
[ ] Pull Request contains link to the JIRA issue
[ ] Pull Request contains link to any dependent or related Pull Request
[ ] Pull Request contains description of the issue
[ ] Pull Request does not include fixes for issues other than the main ticket

PR reviewer(s) checklist

[ ] Check if code and Github action workflows have been modified and if the modification is safe
[ ] If the modification is safe, add the safe to test label

How to trigger pipelines and use the bots:

Run the end to end pipeline
Annotate the pull request with the label: safe to test. If you want to run the pipeline again, remove and add it again.

Rebase the pull request
Comment with: /rebase.

@lucamolteni thank you for PR, please have a look on my comment https://github.com/5733d9e2be6485d52ffa08870cabdee0/sandbox/pull/548/files#r838480659 and I think I can approve then.

Thank you please reveiw again