Support backprop through lfilter parameters

This merge solve issue #704.
It moves the original python implementation of lfilter into c++ backend, and register a custom autograd kernel to support torchscript as @vincentqb mentioned in #704 .
A simple test case is added to test whether the gradient is valid or not.
Notes
Some differences to the old lfilter:

The old implementation use direct-form I; the new one use direct-form II.
A mix of indexing and matmul operation at
https://github.com/pytorch/audio/blob/e83d557a0d48249b20eba42952a2ed61a3d9644b/torchaudio/functional/filtering.py#L881

is replaced by a single conv1d function call.
https://github.com/yoyololicon/audio/blob/4e2ff32b50d56ce168fcee872c95ffc6cde82eaa/torchaudio/csrc/lfilter.cpp#L123

Hi @yoyololicon
Thanks for the contribution. This looks a great improvement.
Questions:

Does the new implementation yield the same result as the previous one?
What do you think of the coverage of the input value of the autograd check test? Is it enough or should we testing more in different locations?
Would it be possible to provide a Python version (ideally as a separate PR)? The thing is that currently we do not build C++ extension on Windows so this approach will break the Windows package. We are planning to resolve this in a coming weeks (say, two weeks), until then we cannot merge this PR. Meanwhile, if you can make the original Python version differentiable, then we can merge it until the Windows support situation is resolved. After that we can discuss turning the rest of lfilter implementation into C++.

Hi @yoyololicon
Thanks for the contribution. This looks a great improvement.
Questions:

Does the new implementation yield the same result as the previous one?

It passed the tests listed in test/torchaudio_unittest/sox_compatibility_test.py so I think the output should be the same.

What do you think of the coverage of the input value of the autograd check test? Is it enough or should we testing more in different locations?

I think currently a gradient test on second order filter might be enough, cuz it's very common and is also the basis of biquad filters.

Would it be possible to provide a Python version (ideally as a separate PR)? The thing is that currently we do not build C++ extension on Windows so this approach will break the Windows package. We are planning to resolve this in a coming weeks (say, two weeks), until then we cannot merge this PR. Meanwhile, if you can make the original Python version differentiable, then we can merge it until the Windows support situation is resolved. After that we can discuss turning the rest of lfilter implementation into C++. (if you are fine being blocked on the Windows support, that's okay too)

A custom autograd function in Python frontend would break the torchscript support ability.
I suggest we can fallback to use the original version (which not fully support autograd) when c++ extension is not available on Windows. If this is not the case, I'm fine with waiting for a few weeks.

What is the performance imprecation of conv1d replacement?

My initial idea is to make the source code much more readable, but seems like it can also bring some performance improvement.
Below is the profiling results on this specific part of lfilter:
indexing + matmul
----------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                  Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls
----------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
           aten::copy_        39.23%     236.839ms        39.23%     236.839ms       1.184ms           0 b           0 b           0 b           0 b           200
            aten::take        29.64%     178.937ms        29.64%     178.937ms       1.789ms     807.46 Mb     807.46 Mb           0 b           0 b           100
          aten::matmul         0.39%       2.377ms        28.79%     173.818ms       1.738ms     269.15 Mb    -807.46 Mb           0 b           0 b           100
      aten::contiguous         0.08%     481.909us        20.59%     124.307ms     621.537us     807.46 Mb           0 b           0 b           0 b           200
          aten::repeat         0.20%       1.214ms        19.47%     117.533ms       1.175ms       1.58 Gb           0 b           0 b           0 b           100
            aten::add_        19.26%     116.257ms        19.26%     116.257ms       1.163ms           0 b           0 b           0 b           0 b           100
              aten::mm         7.26%      43.848ms         7.30%      44.086ms     440.863us     269.15 Mb           0 b           0 b           0 b           100
          aten::arange         1.27%       7.642ms         2.62%      15.809ms      26.349us      67.32 Mb           0 b           0 b           0 b           600
             aten::add         0.97%       5.873ms         1.00%       6.010ms      60.097us     100.93 Mb           0 b           0 b           0 b           100
           aten::empty         0.32%       1.934ms         0.32%       1.934ms       2.417us       2.73 Gb       2.73 Gb           0 b           0 b           800
       aten::unsqueeze         0.24%       1.450ms         0.28%       1.698ms       4.246us           0 b           0 b           0 b           0 b           400
          aten::unfold         0.16%     948.707us         0.25%       1.481ms       4.937us           0 b           0 b           0 b           0 b           300
            aten::view         0.18%       1.087ms         0.18%       1.087ms       2.718us           0 b           0 b           0 b           0 b           400
       aten::transpose         0.12%     738.172us         0.17%       1.007ms       5.036us           0 b           0 b           0 b           0 b           200
         aten::reshape         0.06%     373.146us         0.16%     979.680us       4.898us           0 b           0 b           0 b           0 b           200
      aten::empty_like         0.06%     343.304us         0.16%     961.195us       9.612us     807.46 Mb           0 b           0 b           0 b           100
      aten::as_strided         0.15%     894.105us         0.15%     894.105us       0.813us           0 b           0 b           0 b           0 b          1100
             aten::mul         0.10%     615.297us         0.12%     701.129us       7.011us      12.50 Kb           0 b           0 b           0 b           100
         aten::resize_         0.08%     458.102us         0.08%     458.102us       1.527us      33.66 Mb      33.66 Mb           0 b           0 b           300
    aten::_unsafe_view         0.05%     298.902us         0.07%     422.488us       4.225us           0 b           0 b           0 b           0 b           100
               aten::t         0.05%     281.180us         0.06%     384.573us       3.846us           0 b           0 b           0 b           0 b           100
          aten::expand         0.04%     237.944us         0.06%     354.969us       1.775us           0 b           0 b           0 b           0 b           200
          aten::stride         0.05%     272.478us         0.05%     272.478us       0.182us           0 b           0 b           0 b           0 b          1500
       aten::expand_as         0.01%      84.950us         0.04%     230.088us       2.301us           0 b           0 b           0 b           0 b           100
              aten::to         0.02%     132.702us         0.02%     132.702us       1.327us           0 b           0 b           0 b           0 b           100
           aten::alias         0.02%      92.570us         0.02%      92.570us       0.926us           0 b           0 b           0 b           0 b           100
----------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 603.711ms

conv1d
----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                        Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls
----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                aten::conv1d         0.06%     257.202us        99.72%     441.788ms       4.418ms     269.15 Mb           0 b           0 b           0 b           100
           aten::convolution         0.07%     312.007us        99.66%     441.530ms       4.415ms     269.15 Mb           0 b           0 b           0 b           100
          aten::_convolution         0.33%       1.447ms        99.59%     441.218ms       4.412ms     269.15 Mb           0 b           0 b           0 b           100
    aten::mkldnn_convolution        98.70%     437.288ms        98.87%     438.045ms       4.380ms     269.15 Mb           0 b           0 b           0 b           100
               aten::squeeze         0.24%       1.063ms         0.28%       1.237ms       6.186us           0 b           0 b           0 b           0 b           200
             aten::unsqueeze         0.23%       1.006ms         0.26%       1.147ms       3.824us           0 b           0 b           0 b           0 b           300
                 aten::empty         0.12%     551.754us         0.12%     551.754us       5.518us     269.15 Mb     269.15 Mb           0 b           0 b           100
                  aten::view         0.11%     475.994us         0.11%     475.994us       4.760us           0 b           0 b           0 b           0 b           100
            aten::as_strided         0.07%     315.852us         0.07%     315.852us       0.632us           0 b           0 b           0 b           0 b           500
           aten::as_strided_         0.05%     204.563us         0.05%     204.563us       2.046us           0 b           0 b           0 b           0 b           100
            aten::contiguous         0.03%     117.147us         0.03%     117.147us       0.390us           0 b           0 b           0 b           0 b           300
----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 443.038ms

The new implementation reduce number of function calls, make the majority computation rely on a single mkldnn_convolution call; it also greatly reduce memory usage.
The profile script:
import torch
import torch.nn.functional as F
import torch.autograd.profiler as profiler
import torch.utils.benchmark as benchmark

def index_matmul(x, b):
    n_channel, n_sample = x.shape
    dtype = x.dtype
    device = x.device
    n_order = b.shape[0]

    window_idxs = torch.arange(n_sample - n_order + 1, device=device).unsqueeze(
        0) + torch.arange(n_order, device=device).unsqueeze(1)
    window_idxs = window_idxs.repeat(n_channel, 1, 1)
    window_idxs += (
        torch.arange(
            n_channel, device=device).unsqueeze(-1).unsqueeze(-1) * n_sample
    )
    window_idxs = window_idxs.long()
    input_signal_windows = torch.matmul(
        b, torch.take(x, window_idxs)
    )
    return input_signal_windows

def conv1d(x, b):
    n_channel, n_sample = x.shape
    dtype = x.dtype
    device = x.device
    n_order = b.shape[0]
    return F.conv1d(x.unsqueeze(1), b.view(1, 1, n_order)).squeeze(1)

if __name__ == '__main__':
    torch.random.manual_seed(2434)
    b = torch.rand(3)
    x = torch.randn(16, 44100)
    x /= x.abs().max()

    with profiler.profile(profile_memory=True) as prof:
        for _ in range(100):
            index_matmul(x, b)

    print("indexing + matmul")
    print(prof.key_averages().table(sort_by="cpu_time_total"))

    with profiler.profile(profile_memory=True) as prof:
        for _ in range(100):
            conv1d(x, b)

    print("conv1d")
    print(prof.key_averages().table(sort_by="cpu_time_total"))

    y1 = index_matmul(x, b)
    y2 = conv1d(x, b)
    assert torch.allclose(y1, y2, atol=1e-7)

Thank you for this great contribution @yoyololicon! I'm taking this on to take some workload off of Moto :)
There are so many good things in here, I'd suggest we split this into three pieces so we can land more quickly
a) Landing the conv1d optimization within Python only
b) Landing a full C++ forward implementation
c) Landing the C++ backward pass which also adds autograd support
a) and b) are fairly straightforward and should be quick to land, c) will need a bit more discussion due to numerical stability details etc.
While you're doing this we might already have solved Windows support for C++ extension, but your work is not blocked on it.
Thanks!

Christian

a) and b) are fairly straightforward and should be quick to land, c) will need a bit more discussion due to numerical stability details etc.

Should we create seperate request for each one?

a) and b) are fairly straightforward and should be quick to land, c) will need a bit more discussion due to numerical stability details etc.

Should we create seperate request for each one?

I recommend opening separate PRs for a) and b). Then after a) and b) are done, you can either close this one or update it to c). For now this one has a good discussion and benchmark numbers, so let's leave it open.

Profiling results running on a P620 gpu with same parameters:
indexing + matmul
----------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                  Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls
----------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
          aten::matmul        10.66%       2.024ms        39.36%       7.470ms      74.696us           0 b           0 b     149.41 Mb    -403.76 Mb           100
          aten::arange        10.72%       2.034ms        24.99%       4.743ms       7.906us           0 b           0 b      67.58 Mb           0 b           600
          aten::repeat         3.82%     725.022us        18.14%       3.443ms      34.431us           0 b           0 b     807.47 Mb           0 b           100
           aten::copy_         8.69%       1.650ms         8.69%       1.650ms       8.248us           0 b           0 b           0 b           0 b           200
      aten::contiguous         1.34%     254.661us         8.33%       1.581ms       7.903us           0 b           0 b     403.76 Mb           0 b           200
              aten::mm         6.44%       1.223ms         7.65%       1.451ms      14.512us           0 b           0 b     149.41 Mb           0 b           100
          aten::unfold         4.67%     886.657us         7.16%       1.359ms       4.528us           0 b           0 b           0 b           0 b           300
            aten::view         7.09%       1.346ms         7.09%       1.346ms       3.365us           0 b           0 b           0 b           0 b           400
       aten::unsqueeze         5.72%       1.086ms         6.77%       1.285ms       3.212us           0 b           0 b           0 b           0 b           400
           aten::empty         6.39%       1.213ms         6.39%       1.213ms       1.516us           0 b           0 b       1.43 Gb       1.43 Gb           800
             aten::add         4.93%     935.917us         5.88%       1.116ms      11.162us           0 b           0 b     100.98 Mb           0 b           100
            aten::take         5.57%       1.058ms         5.57%       1.058ms      10.578us           0 b           0 b     403.76 Mb     403.76 Mb           100
             aten::mul         4.59%     870.274us         5.55%       1.054ms      10.539us           0 b           0 b      50.00 Kb           0 b           100
    aten::_unsafe_view         0.98%     185.492us         4.24%     805.323us       8.053us           0 b           0 b           0 b           0 b           100
         aten::reshape         1.01%     191.859us         3.65%     692.589us       3.463us           0 b           0 b           0 b           0 b           200
            aten::add_         3.61%     685.445us         3.61%     685.445us       6.854us           0 b           0 b           0 b           0 b           100
      aten::as_strided         3.05%     578.604us         3.05%     578.604us       0.526us           0 b           0 b           0 b           0 b          1100
         aten::resize_         2.73%     518.079us         2.73%     518.079us       1.727us           0 b           0 b      33.79 Mb      33.79 Mb           300
       aten::transpose         1.85%     350.566us         2.47%     467.854us       2.339us           0 b           0 b           0 b           0 b           200
      aten::empty_like         0.88%     167.664us         2.38%     452.493us       4.525us           0 b           0 b     403.76 Mb           0 b           100
          aten::stride         1.96%     371.699us         1.96%     371.699us       0.206us           0 b           0 b           0 b           0 b          1800
               aten::t         1.17%     222.400us         1.72%     326.600us       3.266us           0 b           0 b           0 b           0 b           100
          aten::expand         1.01%     191.119us         1.53%     289.546us       1.448us           0 b           0 b           0 b           0 b           200
       aten::expand_as         0.35%      67.201us         1.02%     193.950us       1.939us           0 b           0 b           0 b           0 b           100
           aten::alias         0.45%      84.716us         0.45%      84.716us       0.847us           0 b           0 b           0 b           0 b           100
              aten::to         0.30%      57.575us         0.30%      57.575us       0.576us           0 b           0 b           0 b           0 b           100
----------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 18.980ms

conv1d
---------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                       Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls
---------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
               aten::conv1d         2.09%     115.218us        87.63%       4.837ms      48.367us           0 b           0 b     134.62 Mb           0 b           100
          aten::convolution         2.07%     114.072us        85.55%       4.721ms      47.214us           0 b           0 b     134.62 Mb           0 b           100
         aten::_convolution        11.17%     616.444us        83.48%       4.607ms      46.074us           0 b           0 b     134.62 Mb           0 b           100
    aten::cudnn_convolution        47.62%       2.628ms        56.20%       3.102ms      31.017us           0 b           0 b     134.62 Mb           0 b           100
            aten::unsqueeze        10.80%     596.204us        13.80%     761.660us       2.539us           0 b           0 b           0 b           0 b           300
              aten::squeeze         8.38%     462.517us         9.91%     546.892us       2.734us           0 b           0 b           0 b           0 b           200
                aten::empty         5.03%     277.499us         5.03%     277.499us       1.387us           0 b           0 b     134.62 Mb     134.62 Mb           200
           aten::as_strided         4.53%     249.831us         4.53%     249.831us       0.500us           0 b           0 b           0 b           0 b           500
                 aten::view         3.45%     190.142us         3.45%     190.142us       1.901us           0 b           0 b           0 b           0 b           100
           aten::contiguous         2.38%     131.520us         2.38%     131.520us       0.329us           0 b           0 b           0 b           0 b           400
               aten::stride         1.32%      72.752us         1.32%      72.752us       0.182us           0 b           0 b           0 b           0 b           400
              aten::resize_         1.18%      64.994us         1.18%      64.994us       0.325us           0 b           0 b           0 b           0 b           200
---------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 5.519ms

The speed improvements is a lot more obvious.
P

@yoyololicon sorry for the confusion.
Now that #1318 merged we landed part a) of this work, which is using conv1d as a performance optimization. Thank you for that!
Next we can work on and merge the C++ version of the forward (b) and then update this PR to only contain the backward function and corresponding harness (c).
Thank you for your contribution and patience.
If you can't add me as a reviewer via the github UI (under section "Reviewers") just cc me on the PR in the description for a faster turn-around.
Thanks,
Christian

@cpuhrsch No problem, will start working on (b).
BTW, I saw you are already listed as reviewers, in my side of view.

Now that the other two PRs were merged, we can rebase and re-review. I expect this PR will become much shorter :) Thank you for working this @yoyololicon!

The custom backward function is added, based on original direct form I.
Seems like skipIfNoExtension was removed, is there any way to skip autograd check on windows?

@yoyololicon
For the time being please please use
import platform

@unittest.skipIf(platform.system() == 'Windows'm "not supported on windows):

Windows support is coming very soon

a) and b) are fairly straightforward and should be quick to land, c) will need a bit more discussion due to numerical stability details etc.

Currently, the custom backward function is actually just filter the gradients in reversed direction with the same parameters. If there is numerical problems in forward pass, it's very likely to happen in backward.
The instability of lfilter in torchaudio can be a result of:

Using float32 single precision (#676). This is unavoidable in PyTorch.
Using Direct Form instead of Transposed Form II. Direct Form is simple to implement (just a convolution and a IIR filter), but is well known to have stability issues; Transposed Form II is good for real-time application, but implement it efficiently in offline application is non-trivial.
Very high order filter (with order > 2). A solution is factoring the filter into cascaded second-order filters, and is also why SciPy recommend using scipy.signal.sosfilt instead of scipy.signal.lfilter in their documentation.

In future I would recommend implementing a custom second-order filter using Transposed Form II, and factorize the filter coefficients using root finding algorithm when higher order filter is needed.

Ok that makes sense. As long as these particularities are documented and the user is aware of it based on the docstring, I think we'll be ok. Could you update the docstring for lfilter to explain potential sources for numerical instability and we can mark your suggestions as a follow-up issue?

@mthrok  Seems custom ops didn't get build on windows, could you help checking it?

@mthrok Seems custom ops didn't get build on windows, could you help checking it?

@yoyololicon Looks like the commits 9395ad6 is missing from your branch. Can you try merge-commit or rebase on the master branch?

@mthrok Seems custom ops didn't get build on windows, could you help checking it?

@yoyololicon Looks like the commits 9395ad6 is missing from your branch. Can you try merge-commit or rebase on the master branch?

commit merged, doesn't work.

Looking at the installation log from https://app.circleci.com/pipelines/github/pytorch/audio/5389/workflows/cb9960d0-3599-4f7a-9a9f-a597b49cba73/jobs/179910/steps
lfilter.cpp is compiled and I see that lfilter.cpp.obj is linked against _torchaudio.dll.
[3/6] C:\PROGRA~2\MICROS~2\2019\COMMUN~1\VC\Tools\MSVC\1428~1.293\bin\Hostx64\x64\cl.exe   /TP -DTORCH_API_INCLUDE_EXTENSION_H -D_torchaudio_EXPORTS -I..\..\..\ -I..\..\..\env\include -I..\..\..\env\Lib\site-packages\torch\include -I..\..\..\env\Lib\site-packages\torch\include\torch\csrc\api\include /DWIN32 /D_WINDOWS /GR /EHsc -Wall  /MD /O2 /Ob2 /DNDEBUG /EHsc /DNOMINMAX /wd4267 /wd4251 /wd4522 /wd4838 /wd4305 /wd4244 /wd4190 /wd4101 /wd4996 /wd4275 /bigobj -std:c++14 /showIncludes /Fotorchaudio\csrc\CMakeFiles\_torchaudio.dir\lfilter.cpp.obj /Fdtorchaudio\csrc\CMakeFiles\_torchaudio.dir\ /FS -c ..\..\..\torchaudio\csrc\lfilter.cpp

[5/6] cmd.exe /C "cd . && C:\Users\circleci\project\env\Lib\site-packages\cmake\data\bin\cmake.exe -E vs_link_dll --intdir=torchaudio\csrc\CMakeFiles\_torchaudio.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~2\2019\COMMUN~1\VC\Tools\MSVC\1428~1.293\bin\Hostx64\x64\link.exe  torchaudio\csrc\CMakeFiles\_torchaudio.dir\pybind.cpp.obj torchaudio\csrc\CMakeFiles\_torchaudio.dir\lfilter.cpp.obj torchaudio\csrc\CMakeFiles\_torchaudio.dir\overdrive.cpp.obj torchaudio\csrc\CMakeFiles\_torchaudio.dir\utils.cpp.obj  /out:torchaudio\csrc\_torchaudio.dll /implib:torchaudio\csrc\_torchaudio.lib /pdb:torchaudio\csrc\_torchaudio.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO  ..\..\..\env\Lib\site-packages\torch\lib\c10.lib  ..\..\..\env\Lib\site-packages\torch\lib\torch_python.lib  ..\..\..\env\libs\python37.lib  ..\..\..\env\Lib\site-packages\torch\lib\torch.lib  ..\..\..\env\Lib\site-packages\torch\lib\torch_cpu.lib  ..\..\..\env\Lib\site-packages\torch\lib\c10.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib  && cd ."

oh the C++ extension is not initialized properly.
..\env\lib\site-packages\torchaudio-0.9.0a0+cd24e10-py3.7-win-amd64.egg\torchaudio\extension\extension.py:14
  c:\users\circleci\project\env\lib\site-packages\torchaudio-0.9.0a0+cd24e10-py3.7-win-amd64.egg\torchaudio\extension\extension.py:14: UserWarning: torchaudio C++ extension is not available.
    warnings.warn('torchaudio C++ extension is not available.')

Looks like we need to fix the C++ extension initialization mechanism. Meanwhile you can skip the Windows test as previously done.

I see. Will wait until the issue been fixed.

I see. Will wait until the issue been fixed.

Hi @yoyololicon
I have fixed the Windows issue in https://github.com/pytorch/audio/pull/1371. Can you merge it to your branch and see if it works?

🎉 it's working on Windows as well.
@cpuhrsch Please have another look. Testing-wise I think it's good to go.

@cpuhrsch A demonstration test is added.
I use a 9th order butterworth hi-pass filter to filter an unit impulse, the output should be the impulse response of filter.
With enough precision, the impulse response should not have absolute value greater than 1.
I adjust the pass band frequency so when using double precision the response is accurate enough, but when using single precision, the precision errors would accumulate enough to make the response have value greater than 1.

Hi @mthrok , the stability test case was updated, could you help me review this?

@yoyololicon Thanks for this great contribution!

@mthrok @cpuhrsch Thank you guys for helping me submit this merge!
I'm really appreciated for your help and take time reviewing it.  It's a wonderful experience.


Add dfs_array

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (676b428) to head (5d1bb11).

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #25   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           14        15    +1
  Lines          235       303   +68
  Branches        24        34   +10
=========================================
+ Hits           235       303   +68

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Feat: Opik Integration

This PR adds an integration to the Opik platform, in includes both a new tracer and some documentation updates.

@jjmachan Opik integration is ready for your review

Thanks for the PR @jverre

Hey @jverre Can you fix the CI issue?

seems like I can't push the changes for some reason - will do that in another PR


Remove references to the unused list of elements-to-bind-to

Pull Request Template
Description
This is more of a note than a real issue. I wasn't sure if you guys realized you no longer use the elements-to-bind-to list, even though it's still imported. If you still want to use it, I'd be more than happy to help implement.
Type of change

[x] Bug fix (kinda)

How Has This Been Tested?
npm run test and npm run test-prod
Checklist:

[x] My code follows the style guidelines of this project. npm run lint passes with no errors.
[x] I have made corresponding changes to the documentation
[ ] ~I have added tests that prove my fix is effective or that my feature works~
[x] New and existing unit tests pass locally with my changes. npm run test passes with no errors.

      this.addAllListeners(elementsToBindTo, events)```

still uses that variable!

Looks like this was added back here: https://github.com/checkly/puppeteer-recorder/commit/9f8f0b1e0eb26345e2ed8cfd8d0d1e5b7c36b199
Was this intentional?


Large select queries running 5-10x slower than with psql

I realize this report is vague, but I'm curious whether this performance is to be expected with LibPQ.jl and whether there's any low hanging fruit that get the performance closer to that of psql.
I'm running a simple select query that returns 10-100 million records with 3 non-null Float64 values.  When I run the query over a not-so-fast wifi connection, psql consistently returns the results 10x faster than with LibPQ.jl (CPU usage is low).  When I run the query over a 10 Gbit LAN connection, psql is consistently 5x faster (CPU usage is maxed out).  binary_format=true and not_null=true didn't make a significant difference.

As a more fair comparison, I ran the same query with python's psycopg2 execute() and fetchall() and the timings are consistent with psql (5-10x faster than LibPQ.jl).

[194296ae] LibPQ v1.14.1 on a recent nightly.
I was measuring both the transit time and time to convert to Tables.rowtable, which I thought would be a lot faster, but if is split to two operations the time breakdown is 70/30 over the  slower connection.  So it looks like both transferring and parsing are each slower with LibPQ.jl than the total time with psycopg2.

Tables.columntable should be significantly faster. Parsing entire columns at once will be a type-stable operation, and we explicitly test for performance on that operation since 1.14.
On the transit time, it would be helpful for replication to have a sense of the schema being pulled. Can this be replicated by pulling 10 million rows of a single double precision column containing 3 unique non-null values?
Are your connections using SSL?

Ah you're right, after making a minimal example locally on my laptop and testing more carefully, nearly all the time is taken up by Tables.rowtable.
But while it's true that Tables.columntable is extremely fast.  It seems like that's because it's a lazy operation.  If I want to actually do anything with the columns, they must first be parsed which allocates for every record.
For example:
postgres=# create table t (x integer, y integer); insert into t (select *,1 from generate_series(1,10000000));
CREATE TABLE
INSERT 0 10000000

julia> c = LibPQ.Connection("postgresql://postgres@localhost:5432/postgres");

julia> @time r = LibPQ.execute(c, "select * from t", binary_format=false, not_null=true);
  1.643387 seconds (85 allocations: 4.484 KiB)

julia> @time t = Tables.columntable(r);
  0.000096 seconds (23 allocations: 1.109 KiB)

julia> @time sum(t.x);
  1.116090 seconds (20.00 M allocations: 686.646 MiB, 8.03% gc time)

julia> typeof(t)
NamedTuple{(:x, :y), Tuple{LibPQ.Column{Int32, 0x00000017, Int32, LibPQ.Result{false}, LibPQ.ParseType{Int32}}, LibPQ.Column{Int32, 0x00000017, Int32, LibPQ.Result{false}, LibPQ.ParseType{Int32}}}}

(everything was already run once to compile)
Also, it seems like Tables.rowtable is much slower than it should be.  In this example, I had to run it on a smaller table because I was impatient, but extrapolating the timing it would take about 500 seconds to complete.

Ah nevermind, I forgot to set binary_format=true in the last set of tests.  When I used that, the results of Tables.columntable don't need additional parsing and are fast.
I also noticed that it's much (orders of magnitude) faster to to Tables.rowtable(Tables.columntable(r)) than just calling Tables.rowtable(r).

I also noticed that it's much (orders of magnitude) faster to to Tables.rowtable(Tables.columntable(r)) than just calling Tables.rowtable(r).

This is a really good catch, we should be able to make a change to get Tables to do that automatically.


HHH-11248 - Test version of persistence.xml not retrieved for JUnit test

https://hibernate.atlassian.net/browse/HHH-11248

Looks good and it restores the old behavior.

Going to close this and implement per comments on the Jira


picasa importer failling since Oauth2 change

Since March 10 when there was a change to the picasa importer from Oauth1 to Oauth2, the importer has been failing.
Symptoms are that it will never finish running (72 hours or more when first noticed).
I've stopped camlistore and restarted and told the importer to "re-login" and received an error that there was something wrong with my credentials added the correct callback URL for my dydns.org registration, then manually ran the picasa importer and it picked up all my new photos (30 or so) of the last ten days, but once again the importer doesn't seem to want to finish.  Pressing the "Pause Import" button has no effect.  The importer status page reports:
Running: true
Started: 11h29m10.651559107s ago
There's nothing appearing in the logs (other than the feed importer, foursquare importer and twitter importer msgs on their schedules).
Stopped & restarted camlistore and again did the re-login, saw the message about updated tokens and then manually started an import (at 08:45).  It once again imported 2 days of new photos and continued re-importing photos for approx 90 minutes (these have all been in my camlistore for months), the last message on the console was of
2016/03/22 09:48:27 Importing media from https://lh3.googleusercontent.com/.../I/20151022_142151.jpg
Once again I can't pause the importer.  System CPU use is up around 90%, all used by camlistored.

I've just run a full import from scratch (on an empty camlistore, with devcam server), and it was successful. But I don't have a very large set - it took ~40 mins - which might be why I'm less likely to be able to reproduce what you're seeing.
I'll try and figure out some way for you to find us more debugging info.
Btw, we didn't Go from OAuth1 to OAuth2, but from a deprecated OAuth2 library to the recommended one.

@ajft you said approx 90 mins, but could you make sure it's not rather starting to fail after ~1 hour? I'm asking because it looks like we're getting tokens that expire after 60 mins, so if token refreshing is what's failing, it would make sense we'd see that after ~1 hour.
Even though my data does not need 1 hour to be all imported, I think I can nevertheless test that theory myself anyway.

@mpl indeed one hour seems to be the key.  I just ran up a copy of devcam, put in my google creds. and watched it for an hour as it sucked down photos, then went silent.  After 21:44 same symptoms as before, the importer is still running but nothing appears on the console and I can't pause it.
2016/03/23 20:43:03 Token = &oauth2.Token{AccessToken:"y..Q", TokenType:"Bearer", RefreshToken:"1/s..6", Expiry:time.Time{sec:63594326583, nsec:167231318, loc:(*time.Location)(0x1c93060)}, raw:map[string]interface {}{"access_token":"y..Q", "token_type":"Bearer", "expires_in":3600, "refresh_token":"1/s//6"}}, error
2016/03/23 20:43:21 Starting picasa importer account, sha1-0..0: userid 1..6 (Adrian Tritschler)
:
2016/03/23 21:43:21 Importing media from https://lh3.googleusercontent.com/-g..c/r..M/I/2..8.jpg

@ajft thanks. I think I've just been able to reproduce this as well. that's good, I know what to debug now.

ok, I found the bug. CL should follow later today.

@ajft Can you tell me if https://camlistore-review.googlesource.com/6156 fixes things for you please?

@mpl -- with that change it seems to have fixed it.  Has been importing
photos for 75 minutes now without stalling at the hour.  Also correctly
responded to the "pause import" button after 75 minutes, then restarted
correctly a few minutes later when I told it to run again.  Not sure how
long I"ll have to watch to see if finish my 3750 photos, I'll leave it
importing overnight and check in the morning.
On Thu, Mar 24, 2016 at 10:40 AM mpl<EMAIL_ADDRESS>wrote:

@ajft https://github.com/ajft Can you tell me if
https://camlistore-review.googlesource.com/6156 fixes things for you
please?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/camlistore/camlistore/issues/698#issuecomment-200584088

@ajft Excellent, thanks. Please note that this probably won't be the actual fix though.

Nice catch! I just wanted to report that it seems to be a token refresh issue: after a while it reproducibly stalls on start, but (after a camlistored restart) after re-login it works.

yeah, I know exactly what the problem is. Just waiting for feedback from Brad on my CL.


Basic push/list Stack example lead to "RAM_EXCEEDED"

I'm submitting a ... (check one with "x")
[x] bug report => search github for a similar issue or PR before submitting
[ ] feature request

Scope (check one with "x")
[x] <root>
[ ] @zetapush/cli
[ ] @zetapush/client
[ ] @zetapush/cometd
[ ] @zetapush/common
[ ] @zetapush/core
[ ] @zetapush/create
[ ] @zetapush/http-server
[ ] @zetapush/platform-legacy
[ ] @zetapush/testing
[ ] @zetapush/troubleshooting
[ ] @zetapush/user-management
[ ] @zetapush/worker

Current behavior

Minimum Viable Code for stack lead to a "RAM_EXCEEDED" critical error

Expected behavior

Calling stack.list after a successful stack.push should work as described.

Minimal reproduction of the problem with instructions

Here MVP worker to try out stack.
import { Stack } from '@zetapush/platform-legacy'
import { Injectable, Context } from '@zetapush/core'

@Injectable()
export default class Api {
  private requestContext: Context
  constructor(private stack: Stack) {}

  async onApplicationBootstrap() {
    const stackId = this.generateId()
    await this.pushStack(stackId)
    await this.readStack(stackId)
  }

  /*
   * Random 4 alpha-numeric id
   */
  generateId() {
    return (((1 + Math.random()) * 0x10000) | 0).toString(16).substring(1)
  }

  /**
   * Push to Stack service
   * @param stackId
   */
  async pushStack(stackId: string) {
    console.log('Stack::push', stackId)
    await this.stack.push({
      stack: stackId,
      data: {
        text: 'toto'
      }
    })
    console.log('Stack::pushed')
  }

  /**
   * Read from Stack service
   * @param stackId
   */
  async readStack(stackId: string) {
    console.log('Stack::read', stackId)
    try {
      const { result } = await this.stack.list({
        stack: stackId
      })
      console.log('Stack::readed', result)
    } catch (e) {
      console.log('Stack::readError', e)
    }
  }
}

And log for this given code:
Stack::push 6a66
Stack::pushed
Stack::read 6a66
Stack::readError { message: 'Ram quota exceeded for this service call : 77',
  code: 'RAM_EXCEEDED' }

Please tell us about your environment:

Operating System: Linux
Node Version:  10.10
Npm Version:  6.4.1
@zetapush/* Version:  3.5.5 & 3.6

Your problem seems to be related to your developer pack.

Issue occured only with Prepackaged Account.
@pablo-abreu


Pre-compiled module is missing: midi and Could not locate the bindings file

I'm trying to use the node-midi package and I'm getting the Pre-compiled module is missing: midi. warning when running t2 run index.js. That doesn't stop the build, but it eventually fails with
Error: Could not locate the bindings file. Tried:
 → /tmp/remote-script/node_modules/midi/build/midi.node
 → /tmp/remote-script/node_modules/midi/build/Debug/midi.node
 → /tmp/remote-script/node_modules/midi/build/Release/midi.node
 → /tmp/remote-script/node_modules/midi/out/Debug/midi.node
 → /tmp/remote-script/node_modules/midi/Debug/midi.node
 → /tmp/remote-script/node_modules/midi/out/Release/midi.node
 → /tmp/remote-script/node_modules/midi/Release/midi.node
 → /tmp/remote-script/node_modules/midi/build/default/midi.node
 → /tmp/remote-script/node_modules/midi/compiled/4.2.1/linux/mipsel/midi.node

There is a midi.node file at node_modules/midi/build/Release/midi.node. I've also tried running it with the --full flag, which fails with
node: '/tmp/remote-script/node_modules/midi/build/Release/midi.node' is not an ELF file

I'm on Node v5.9.1 and node-gyp v3.3.1 on OS X. Same error appears with Node v4.2.1.
Looks like someone had a similar issue with this package on a RaspberryPie.
Any insight into what's causing this? I'm happy to defer this to the node-midi repo, just thought it might be handy to have here too. Thanks!

Thanks! We need to compile midi for T2's architecture (MIPS). @reconbot this is for you.

Great, thanks @johnnyman727! Excited to get hacking on my new Tessel :smile:

Compiled and deployed:

<EMAIL_ADDRESS>
If you need a different version, we can re-open

@rwaldron I'm getting ALSA lib seq_hw.c:457:(snd_seq_hw_open) open /dev/snd/seq failed: No such file or directory when trying to initialize the MIDI library.
After a bit of searching, like a driver/kernel thing. Are there any plans to support this sort of connectivity?

Issue thread for UDOO
Issue thread on an Ubuntu support forum

@albatrocity thanks for researching that. Would you be interested in installing the kernel dependencies on your T2 and reporting back on whether it works?
You would need to:

[ ] Connect your T2 to WIFI
[ ] Run t2 root (must be discoverable via LAN with (t2 list first)
[ ] Once you have shell access, use opkg to install MIDI dependencies. Here is a list with some midi-related deps but I'm not sure which you'll need. You can install them with opkg install PKG_NAME.

@johnnyman727 definitely. Just tried alas-utils which didn't seem to affect much. A bunch of the other packages on that list threw Unknown package errors. I'll check later on for direct downloads to source. Seems like alas-utils-midi and/or kernel-module-snd-usbmidi would be relevant.

yeah anything with midi in the name seems like a good plan :)

Change alas to alsa?
root@bishop:~# opkg install alsa-utils
Package alsa-utils (1.0.28-2) installed in root is up to date.

@rwaldron just a typo in my issue. I've installed the following with opkg:

alas-lib
alas-utils
alsa-utils-seq

I keep getting the ALSA lib seq_hw.c:457:(snd_seq_hw_open) open /dev/snd/seq failed: No such file or directory error, both when attempting to initialize a node-midi connection or running aconnect from the t2 root.
This seemed somewhat relevant. Any idea if this is a kernel thing or something with not actually having a sound card that supports MIDI?

Any idea if this is a kernel thing or something with not actually having a sound card that supports MIDI?

I honestly don't know :\
Can you paste the output of lsmod (ssh to the board to run that)

ac97_bus                 528  1 snd_ac97_codec
arc4                    1312  2
bluetooth             241598 11 rfcomm
bnep                    8365  2
btusb                  17808  0
cdc_acm                13520  0
cfg80211              213345  2 rt2x00lib
ch341                   3680  0
compat                  1332  3 rt2800soc
crc16                   1015  1 bluetooth
crc_ccitt               1019  2 rt2800lib
crc_itu_t               1019  0
crypto_blkcipher       10487  2 ecb
crypto_hash             9698  1 bluetooth
ecb                     1328  0
eeprom_93cx6            1807  0
ehci_hcd               31516  1 ehci_platform
ehci_platform           3744  0
evdev                   8432  0
fat                    46895  1 vfat
ftdi_sio               28624  0
gpio_button_hotplug     6000  0
hci_uart                8799  0
hid                    79970  3 usbhid
hid_generic              592  0
hidp                   10961  0
input_core             24510  4 uvcvideo
ip6_tables              9185  3 ip6table_raw
ip6t_REJECT             1184  2
ip6table_filter          608  1
ip6table_mangle         1040  1
ip6table_raw             576  1
ip_tables               9389  4 iptable_nat
ipt_MASQUERADE           624  0
ipt_REJECT               912  2
iptable_filter           672  1
iptable_mangle           928  1
iptable_nat              768  1
iptable_raw              640  1
ipv6                  256232 32 nf_conntrack_ipv6
leds_gpio               2896  0
ledtrig_usbdev          1936  0
mac80211              378483  3 rt2800lib
nf_conntrack           47691 11 nf_nat_ipv4
nf_conntrack_ftp        5280  1 nf_nat_ftp
nf_conntrack_ipv4       4624  6
nf_conntrack_ipv6       4944  3
nf_conntrack_rtcache    2448  0
nf_defrag_ipv4           806  1 nf_conntrack_ipv4
nf_defrag_ipv6          9031  1 nf_conntrack_ipv6
nf_log_common           2351  2 nf_log_ipv4
nf_log_ipv4             3248  0
nf_log_ipv6             3376  0
nf_nat                  8939  5 nf_nat_ipv4
nf_nat_ftp              1216  0
nf_nat_ipv4             3713  1 iptable_nat
nf_nat_masquerade_ipv4    1372  1 ipt_MASQUERADE
nf_reject_ipv4          1795  1 ipt_REJECT
nf_reject_ipv6          2055  1 ip6t_REJECT
nls_base                5008  6 vfat
nls_cp437               4432  0
nls_iso8859_1           2896  0
nls_utf8                 864  0
ohci_hcd               21951  1 ohci_platform
ohci_platform           3248  0
option                 26976  0
pl2303                  6656  0
ppp_async               6352  0
ppp_generic            20658  3 pppoe
pppoe                   8112  0
pppox                   1354  1 pppoe
rfcomm                 27338  0
rt2800lib              91393  2 rt2800soc
rt2800mmio              6338  1 rt2800soc
rt2800soc               2496  0
rt2x00lib              34239  5 rt2800soc
rt2x00mmio              1904  2 rt2800soc
rt2x00soc               1202  1 rt2800soc
scsi_mod               85591  3 ums_cypress
sd_mod                 26272  0
slhc                    4347  1 ppp_generic
snd                    40656 12 snd_usb_audio
snd_ac97_codec         79790  1 snd_intel8x0
snd_compress            6319  0
snd_hwdep               4558  1 snd_usb_audio
snd_intel8x0           20352  0
snd_mixer_oss          12601  1 snd_pcm_oss
snd_pcm                60391  4 snd_usb_audio
snd_pcm_oss            33873  0
snd_rawmidi            15203  1 snd_usbmidi_lib
snd_seq_device          4381  1 snd_rawmidi
snd_timer              14590  1 snd_pcm
snd_usb_audio          93686  0
snd_usbmidi_lib        15361  1 snd_usb_audio
soundcore               3756  1 snd
ums_alauda              8272  0
ums_cypress             2256  0
ums_datafab             4688  0
ums_freecom             1952  0
ums_isd200              5040  0
ums_jumpshot            3632  0
ums_karma               1536  0
ums_sddr09              8720  0
ums_sddr55              4832  0
ums_usbat               7344  0
usb_common              1740  1 usbcore
usb_storage            37359 10 ums_usbat
usb_wwan                4391  1 option
usbcore               116724 28 option
usbhid                 21557  0
usbserial              18043  5 option
uvcvideo               62151  0
v4l2_common             2966  1 videobuf2_core
vfat                    7840  0
videobuf2_core         28748  1 uvcvideo
videobuf2_memops        1216  1 videobuf2_vmalloc
videobuf2_vmalloc       2145  1 uvcvideo
videodev               97278  3 uvcvideo
x_tables               10741 26 ipt_REJECT
xt_CT                   2224  2
xt_LOG                   768  0
xt_REDIRECT             1056  0
xt_TCPMSS               2656  0
xt_comment               480 18
xt_conntrack            2144  8
xt_id                    480  0
xt_limit                 992  2
xt_mac                   624  0
xt_mark                  672  0
xt_multiport            1200  0
xt_nat                  1072  0
xt_state                 688  0
xt_tcpudp               1728  4
xt_time                 1648  0

Seeing snd_seq_device in there. Now suspecting it's the sound card issue.

My /dev/snd directory only has a timer file, no seq file.
There's a --with-sequencer flag that can be used when compiling alsa. I'm not really sure how it gets compiled in the first place, but maybe it's related?

@albatrocity perhaps alsa should be compiled into the OpenWRT build rather than installed as a package afterwards? I honestly don't know.
I was looking through the build options (you can get this by cloning https://github.com/tessel/openwrt-tessel, cd openwrt, make menuconfig, scroll down to "kernel" and found these options:

Once you select the options you want and save, run make -j50 and it will start compiling. This will take a long time. Once you have your build, you can scp it to the /tmp folder of Tessel and run sysupgrade PATH_TO_NEW_BINARY.
For the sake of convenience, I tried created a build with just the sequencer highlighted but the build failed with:
[mktplinkfw2] *** error: rootfs image is too big

So this fish might not be useful but the fishing pole above might help.

I might also suggest reviewing this guide for ideas.

@johnnyman727 https://github.com/tessel/openwrt-tessel/blob/master/config.mk#L52-L53 ?

Yep, it looks like we already have alsa support, but I'm guessing some kernel modules have to be compiling for sequencing.


Build incorrectly targets old branch serverless

Should be fixed with PR #612 but waiting for build to finish before closing this issue

Closing since build has been proven to work now


nnf.pad error

Hi Junyu,
Thanks for sharing your interesting work! It does help a lot for my project. However, I found some tiny errors related to nnf.pad  function in your code. So I made the below modifications:
x = nnf.pad(x, (0, 0, pad_f, pad_h, pad_t, pad_b, pad_l, pad_r)) # Line 219
# x = nnf.pad(x, (0, 0, pad_l, pad_r, pad_t, pad_b, pad_f, pad_h))

if pad_input: #Line 285
    x = nnf.pad(x, (0, 0, 0, T % 2, 0, W % 2, 0, H % 2))
    # x = nnf.pad(x, (0, 0, 0, W % 2, 0, H % 2, 0, T % 2))

_, _, H, W, T = x.size() # Line 439
if T % self.patch_size[2] != 0:
    x = nnf.pad(x, (0, self.patch_size[2] - T % self.patch_size[2]))
if W % self.patch_size[1] != 0:
    x = nnf.pad(x, (0, 0, 0, self.patch_size[1] - W % self.patch_size[1]))
if H % self.patch_size[0] != 0:
    x = nnf.pad(x, (0, 0, 0, 0, 0, self.patch_size[0] - H % self.patch_size[0]))
# if W % self.patch_size[1] != 0:
#     x = nnf.pad(x, (0, self.patch_size[1] - W % self.patch_size[1]))
# if H % self.patch_size[0] != 0:
#     x = nnf.pad(x, (0, 0, 0, self.patch_size[0] - H % self.patch_size[0]))
# if T % self.patch_size[0] != 0:
#     x = nnf.pad(x, (0, 0, 0, 0, 0, self.patch_size[0] - T % self.patch_size[0]))

Best,
Woaka

Hi @wangkaiwan,
I apologize for the bugs and appreciate your notification.
I modified the code and it seems like the bugs had no effect on the pre-trained models' accuracy.
Thanks,
Junyu

@junyuchen245, yes, the pre-trained models worked well, and I found these tiny bugs when I modified the model structure.
Thanks again for the excellent work!

Thanks!


Media
https://github.com/space-wizards/space-station-14/assets/8107459/3ca9af83-7419-4357-a215-d5719c56a8e7

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog
:cl:

add: Nanotrasen finally decided to give engineering departments something to do, and has finalized the schematics for the thermo-electric generator (TEG).
add: The TEG creates power by exchanging energy between a hot and cold gas on its two sides.
tweak: You can turn on gas heaters/freezers with alt-click now.
tweak: You can open air alarm and pump menus with activate (E) now.

for real tegheads only

conflicts slightly with #18831 probably

fake

funny crash when deleting/deconstructing things linked to a sensor monitor console
wrap the thing in Exists(uid) first
Unhandled exception. System.Collections.Generic.KeyNotFoundException: Entity 46476 does not have a component of type Content.Server.DeviceNetwork.Components.DeviceNetworkComponent
   at Content.Server.SensorMonitoring.SensorMonitoringConsoleSystem.AtmosUpdate(EntityUid uid, SensorMonitoringConsoleComponent comp, AtmosDeviceUpdateEvent args) in Content.Server/SensorMonitoring/SensorMonitoringConsoleSystem.cs:line 293
   at Robust.Shared.GameObjects.EntityEventBus.<>c__DisplayClass47_0`2.<SubscribeLocalEvent>g__EventHandler|0(EntityUid uid, IComponent comp, TEvent& args) in RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 255
   at Robust.Shared.GameObjects.EntityEventBus.<>c__DisplayClass58_0`1.<EntSubscribe>b__0(EntityUid uid, IComponent comp, Unit& ev) in RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 393
   at Robust.Shared.GameObjects.EntityEventBus.EntDispatch(EntityUid euid, Type eventType, Unit& args, Boolean dispatchByReference) in RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 554
   at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEventCore(EntityUid uid, Unit& unitRef, Type type, Boolean broadcast, Boolean byRef) in RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 222
   at Robust.Shared.GameObjects.EntityEventBus.RaiseLocalEvent[TEvent](EntityUid uid, TEvent args, Boolean broadcast) in RobustToolbox/Robust.Shared/GameObjects/EntityEventBus.Directed.cs:line 182
   at Robust.Shared.GameObjects.EntitySystem.RaiseLocalEvent[TEvent](EntityUid uid, TEvent args, Boolean broadcast) in RobustToolbox/Robust.Shared/GameObjects/EntitySystem.cs:line 150
   at Content.Server.Atmos.EntitySystems.AtmosphereSystem.ProcessAtmosDevices(GridAtmosphereComponent atmosphere) in Content.Server/Atmos/EntitySystems/AtmosphereSystem.Processing.cs:line 339
   at Content.Server.Atmos.EntitySystems.AtmosphereSystem.UpdateProcessing(Single frameTime) in Content.Server/Atmos/EntitySystems/AtmosphereSystem.Processing.cs:line 478
   at Content.Server.Atmos.EntitySystems.AtmosphereSystem.Update(Single frameTime) in Content.Server/Atmos/EntitySystems/AtmosphereSystem.cs:line 71
   at Robust.Shared.GameObjects.EntitySystemManager.TickUpdate(Single frameTime, Boolean noPredictions) in RobustToolbox/Robust.Shared/GameObjects/EntitySystemManager.cs:line 303
   at Robust.Shared.GameObjects.EntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 272
   at Robust.Server.GameObjects.ServerEntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in RobustToolbox/Robust.Server/GameObjects/ServerEntityManager.cs:line 153
   at Robust.Server.BaseServer.Update(FrameEventArgs frameEventArgs) in RobustToolbox/Robust.Server/BaseServer.cs:line 718
   at Robust.Server.BaseServer.<SetupMainLoop>b__66_1(Object sender, FrameEventArgs args) in RobustToolbox/Robust.Server/BaseServer.cs:line 535
   at Robust.Shared.Timing.GameLoop.Run() in RobustToolbox/Robust.Shared/Timing/GameLoop.cs:line 228
   at Robust.Server.BaseServer.MainLoop() in RobustToolbox/Robust.Server/BaseServer.cs:line 562
   at Robust.Server.Program.ParsedMain(CommandLineArgs args, Boolean contentStart, ServerOptions options) in RobustToolbox/Robust.Server/Program.cs:line 78
   at Robust.Server.Program.Start(String[] args, ServerOptions options, Boolean contentStart) in RobustToolbox/Robust.Server/Program.cs:line 46
   at Robust.Server.ContentStart.Start(String[] args) in RobustToolbox/Robust.Server/ContentStart.cs:line 10
   at Content.Server.Program.Main(String[] args) in Content.Server/Program.cs:line 9

funny crash when deleting/deconstructing things linked to a sensor monitor console
wrap the thing in Exists(uid) first

That wouldn't be a good fix. Best I can do is a TODO.
That said you reminded me there's similar bugs with the TEG itself. Crap.

FYI @deltanedas the sensor monitoring console would still need a ton of work to make it... actually mappable. There's a reason it's so short-sighted, half-assed and filled with TODOs.

That said you reminded me there's similar bugs with the TEG itself. Crap.

Wait that was also sensor monitoring. My bad.

'testing, do not map' is too vague


Docker improvements

Vastly cleaned up docker directory and files
Removed need to rebuild entire environment just to get dependencies
Normalized all rpm names so they follow the same pattern between OSes
Added 32 bit cross compile support
Move GPG agent from host to docker, to support any boot2docker host now.
Fixed file ownership issue
Enabled integration test on rpm build
Cleaned up README

This is working great for me now.  I definitely want to confirm that gpg signing works before I merge it though.

Thanks for this! I'm excited to try this out for v0.6.0 :)


Form State isDirty not resetting after submit

Hi,
I have a single route that is a simple instance update form. The form submits to itself and the action function then saves the changes to an API. It then returns the latest copy of the instance.
I'm using instance.updated from the API as a key into the form as this changes every time the instance is updated in the backend.
What is the best way to tell remix form that is should reload the default values so that the isDirty field goes back to false and hence the buttons go back to being disabled?
export default function InstanceView() {
    let instance: any = useLoaderData();

    const {
        handleSubmit,
        formState: { errors },
        register,
        control,
        reset,
    } = useRemixForm({
        mode: "onSubmit",
        resolver,
        defaultValues: {
            ...instance,
        },

    });

    const { isDirty} = useFormState({control});

    console.log({updated: instance.updated, isDirty});

    return (

            <Form onSubmit={handleSubmit} key={instance.updated}>
                <CommonFormField
                    name={"name"}
                    label={"Name"}
                    description={"Name for the entry"}
                    register={register}
                    errors={errors}
                />
                <CommonFormField
                    name={"description"}
                    label={"Description"}
                    description={"User-defined descriptive text"}
                    inputType={CommonFormInputType.Textarea}
                    register={register}
                    errors={errors}
                />

                    <Button className="bg-red-500 w-[150px]" disabled={!isDirty} onClick={() => reset()}>
                        Cancel
                    </Button>
                    <Button className="ml-8 bg-green-500 w-[150px]" disabled={!isDirty}>
                        Save
                    </Button>

            </Form>

    )
}

Console output:
index.tsx:133 {updated: '2023-12-09T06:49:35.997Z', isDirty: true}
index.tsx:133 {updated: '2023-12-09T06:49:35.997Z', isDirty: true}
index.tsx:133 {updated: '2023-12-09T06:49:35.997Z', isDirty: true}
index.tsx:133 {updated: '2023-12-09T06:53:31.449Z', isDirty: true}
index.tsx:133 {updated: '2023-12-09T06:53:31.449Z', isDirty: true}

Just needed to reset the form state and pass in the new instance for resetting the default values:
    useEffect(() => {
        if(form) {
            form.reset(instance);
        }
    }, [instance])


False warnings in Error List: "The referenced component 'MyProject' could not be found.", followed by false errors.

After updating to VS 2017 15.6.1, these mysterious warnings have started to appear in the Error List pane from time to time when I do successful builds in my solutions.
"The referenced component 'MyProject' could not be found."
First there's just the warnings. Then when trying to recompile one or more times, the warnings will eventually be accompanied with various errors.

Note that this shows up although the Error List is filtered on "Build Only".
It happens rather frequently, and with solutions with .NET Framework 4.5.2 and 4.6.1. (I don't have anything else.)
These warnings and errors do not show up in the Build Output. The status bar says the build is successful, as it should. Edit: Some errors do show up in the build output, and there is actual compiler errors, but they are false. Restarting VS is the best option to get rid of them quickly.
I am beginning to suspect it only happens when C# projects references F# projects, but haven't done any intensive research on that.
It happens in several solutions, and on several machines.
Repro steps
Rather random behavior. I don't have a repro yet. Repro posted below somewhere.
Known workarounds
None discovered yet.
I think deleting .vs and bin and obj folders will help.
Related information

Operating system - Windows 10, updated.
.NET Runtime - .NET Framework 4.5.2 and 4.6.1
VS 2017 15.6.1
Severity - very annoying. I prefer to roll back if possible. edit1: The found workaround makes this problem considerably less important for me. edit2: The number of actual compiler errors caused by this has increased in frequency, so now it's really annoying.

So I just reported, and then there's a change of course.
The FS0084 that you see in the screenshot above, was the only one of the warnings and errors that also appeared when filtering on "IntelliSense Only".
I restarted VS hoping that would clear the false messages. It didn't. Then I looked in the Build Output, and noticed FS0084 was present there, but none of the other messages.
Then I decided I would run anyway, and it actually worked. The Build Output is flawless. But the Error List is back to displaying false warnings only.

I deleted .vs in the solution root, and all bin and obj folders. This seems to have got rid of the problem, but whether it's permanent I don't know yet.

@BentTranberg Very strange. Do you have a sample solution you could share, or some info about the project types? There's an issue with phantom errror messages in 15.6 for .NET Core SDK-based projects if you have a P2P reference, but they will go away if the file is closed and re-opened (#3952), but I'm not sure if this is what you have.

All the warnings are from C# projects of various types - WPF, WinForms, .NET Framework Library. All the projects that aren't found are F# Framework Library. Solutions are either Framework 4.5.2 or 4.6.1.
I tried only deleting the .vs folder in one case, and that was sufficient to get rid of the problem.

Good to know. We'll keep this open for a bit to see if anyone else comes across. If you don't repro again and nobody else mentions they're seeing it, we'll close the issue.

Just now had a case where deleting the .vs folder earlier did not help permanently. Warnings appeared after some hours work. Nevertheless deleting .vs, bin and obj folders still diminishes the problem greatly. Haven't yet tried to delete only bin and obj, and leave .vs, but will try that next time.
I can't afford to do research on this, but will report findings when stuff happens.

It's getting more annoying, so I thought it worthwhile now to produce a repro. I get warnings and errors immediately when I open and compile this.
TdNet5.bughunt.zip
To produce a repro, I first tried to remove projects that didn't relate to the projects in the warnings. That didn't work - warnings and errors disappeared after some pruning. Then instead I removed pretty much all source from all projects, and references to commercial libraries, but left all references to free NuGet packages. I stopped there and haven't attempted to reduce it further.

Updated to VS 15.6.2, and the problem is still present.

@cartermp, I updated to VS 15.6.3, and the problem seems to be gone. I have worked several hours today and yesterday without sign of it. At 15.6.2 it typically showed up several times in an hour.
Suggest we close this. Also remove the "needs repro", since I've uploaded a zip with repro above.

Agh, sorry @BentTranberg - I've been a bit swamped in preparing for travels in the next two weeks. I'll close this for now, and please do ping to re-open if you see it again.


The location of the toolbar sub-menus show unfriendly on Linux

Storage Explorer Version: 1.20.0-dev
Build Number: 20210518.7
Branch: main
Platform/OS: Linux Ubuntu 16.04/18.04
Architecture: x64
How found: From running test case
Regression From: Previous build (20210511.3)
Steps to Reproduce

Expand one storage account -> Blob Containers.
Open one blob container -> Click 'Upload' -> Observe the sub-menus.
Check whether the location of the sub-menus show friendly.

Expected Experience
The location of the sub-menus show friendly.
Actual Experience
The location of the sub-menus show unfriendly.
On Ubuntu 16.04:

On Ubuntu 18.04:

Additional Context

This issue reproduces for other buttons with sub-menus.
This issue also reproduces for one file share/queue/table.

I downloaded the same build and I was able to reproduce it. However, at some point, it suddenly stops reproducing?! I didn't do anything in between and I have no idea what happened. Could you please check if you can reproduce it consistently in a VM and share it with us?

Hi @JasonYeMSFT I verified on the lasted build main/20210518.9.

On Linux 16.04: This issue reproduces consistently. I share the VM with you on Teams.
On Linux 18.04: This issue reproduces inconsistently.

@v-limu I also can't reproduce consistently on Linux 20.04. It only seems to happen early on first launch, and I can't really get it to happen again afterwards. If you folks can find a reliable way to reproduce, let's revisit.

Hi @craxal  We will pay attention to this issue in testing. if we find a reliable way to reproduce and we will report you.


feat: - Moderation list search & manage

Describe the feature you'd like to request
Moderation list search & Moderation list subscribe manage function
Describe the solution you'd like to see

Moderation list search
Moderation list subscribe manager

Additional information
No response

Not possible at the moment but I will add it once it is added to the API


Ryzen 4000 APU support

If I read correctly ryzen 4000 APUs are not supported. Are there any plans to support them in the future and if not, what opencl ICD should I use? (mesa doesn't work btw)

Thanks @Etaash-mathamsetty for reaching out.
I certainly understood the problem.
I will check with BU team and share some update on this soon.
Thank you.

I switched to arch
and this script kills arch
every single time

OK, I suppose that it is well understood that one should be careful when running a random installation script from the web (more so if it isn't from a well respected source). In fact, I first ran this script (in Gentoo Linux) manually line-by-line to be sure what's going on. So I see that it is better to provide a step-by-step manual installation here instead of a script. In any case I'm not forcing anybody to do it - I'm just giving the procedure which enabled me to get working OpenCL on APU Renoir (4650G, 4750G) and also Cezanne (5700G); this "PAL" OpenCL is also a faster alternative to ROCm for some workloads on R9 Nano (Fiji, gfx803), Vega 56/64 and Radeon VII. I tested it with kernels 5.8 (Renoir) and 5.10 (Cezanne), so I don't know if it works with the newer ones. By the way, with regard to ROCm OpenCL on Renoir - it seems that it works somewhere sometimes (Cezanne is even worse) - it apparently depends also on the motherboard/BIOS - one program may work in one configuration, but not the other, and vice versa (my experience listed above actually involved different motherboards).
So here is the extraction and installation of PAL OpenCL (I removed Vulkan, it proved to be unnecessary):

download AMDGPU-PRO 20.40 and extract it to some temporary directory

wget --referer https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-20-40 https://drivers.amd.com/drivers/linux/amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz
tar xJf amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz

prepare directories /opt/amdgpu and /opt/amdgpu-pro - i.e., if they are present, uninstall corresponding packages or take care not to replace existing files by the ones listed below; if they are not present, create them (including the needed subdirectories). In the steps below, I recommend to create a separate (temporary) directory tree for /opt and /etc, which can be saved and employed for repeated installs of this OpenCL
extract files from *.deb archives by means of

ar x package.deb
tar xJvf data.tar.xz
namely:

extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_amd64.deb, then rename and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/libdrm_amdpro.so.1.0.0
extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_i386.deb, then rename and place it to /opt/amdgpu-pro/lib/i386-linux-gnu/libdrm_amdpro.so.1.0.0
extract amdgpu.ids from libdrm-amdgpu-common_1.0.0-1147286_all.deb, and place it to /opt/amdgpu/share/libdrm/
extract libamd_comgr.so.1.7.0 from opencl-amdgpu-pro-comgr_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl64.so from opencl-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca64.so and libamdocl12cl64.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place them to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca32.so and libamdocl12cl32.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_i386.deb, and place them to /opt/amdgpu-pro/lib/i386-linux-gnu/

create files with simple text contents:

/etc/OpenCL/vendors/amdocl64.icd containing libamdocl64.so (this file can be present from rocm-opencl-runtime, in which case you can simply keep it)
/etc/OpenCL/vendors/amdocl-orca64.icd containing libamdocl-orca64.so
/etc/OpenCL/vendors/amdocl-orca32.icd containing libamdocl-orca32.so
/etc/ld.so.conf.d/zz_amdgpu-pro_x86_64.conf containing

# AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/x86_64-linux-gnu

/etc/ld.so.conf.d/zz_amdgpu-pro_x86.conf containing

# AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/i386-linux-gnu

create symbolic links and modify the binaries to use "amdpro" instead of "amdgpu":

in /opt/amdgpu-pro/lib/i386-linux-gnu/:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca32.so

in /opt/amdgpu-pro/lib/x86_64-linux-gnu:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
ln -s libamd_comgr.so.1.7.0 libamd_comgr.so
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca64.so

now if you didn't copy the files above to the real directories /etc and /opt, do it now, and then run
ldconfig

OK, I suppose that it is well understood that one should be careful when running a random installation script from the web (more so if it isn't from a well respected source). In fact, I first ran this script (in Gentoo Linux) manually line-by-line to be sure what's going on. So I see that it is better to provide a step-by-step manual installation here instead of a script. In any case I'm not forcing anybody to do it - I'm just giving the procedure which enabled me to get working OpenCL on APU Renoir (4650G, 4750G) and also Cezanne (5700G); this "PAL" OpenCL is also a faster alternative to ROCm for some workloads on R9 Nano (Fiji, gfx803), Vega 56/64 and Radeon VII. I tested it with kernels 5.8 (Renoir) and 5.10 (Cezanne), so I don't know if it works with the newer ones. By the way, with regard to ROCm OpenCL on Renoir - it seems that it works somewhere sometimes (Cezanne is even worse) - it apparently depends also on the motherboard/BIOS - one program may work in one configuration, but not the other, and vice versa (my experience listed above actually involved different motherboards).
So here is the extraction and installation of PAL OpenCL (I removed Vulkan, it proved to be unnecessary):

download AMDGPU-PRO 20.40 and extract it to some temporary directory

wget --referer https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-20-40 https://drivers.amd.com/drivers/linux/amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz
tar xJf amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz

prepare directories /opt/amdgpu and /opt/amdgpu-pro - i.e., if they are present, uninstall corresponding packages or take care not to replace existing files by the ones listed below; if they are not present, create them (including the needed subdirectories). In the steps below, I recommend to create a separate (temporary) directory tree for /opt and /etc, which can be saved and employed for repeated installs of this OpenCL
extract files from *.deb archives by means of

ar x package.deb
tar xJvf data.tar.xz
namely:

extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_amd64.deb, then rename and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/libdrm_amdpro.so.1.0.0
extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_i386.deb, then rename and place it to /opt/amdgpu-pro/lib/i386-linux-gnu/libdrm_amdpro.so.1.0.0
extract amdgpu.ids from libdrm-amdgpu-common_1.0.0-1147286_all.deb, and place it to /opt/amdgpu/share/libdrm/
extract libamd_comgr.so.1.7.0 from opencl-amdgpu-pro-comgr_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl64.so from opencl-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca64.so and libamdocl12cl64.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place them to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca32.so and libamdocl12cl32.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_i386.deb, and place them to /opt/amdgpu-pro/lib/i386-linux-gnu/

create files with simple text contents:

/etc/OpenCL/vendors/amdocl64.icd containing libamdocl64.so (this file can be present from rocm-opencl-runtime, in which case you can simply keep it)
/etc/OpenCL/vendors/amdocl-orca64.icd containing libamdocl-orca64.so
/etc/OpenCL/vendors/amdocl-orca32.icd containing libamdocl-orca32.so
/etc/ld.so.conf.d/zz_amdgpu-pro_x86_64.conf containing

# AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/x86_64-linux-gnu

/etc/ld.so.conf.d/zz_amdgpu-pro_x86.conf containing

# AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/i386-linux-gnu

create symbolic links and modify the binaries to use "amdpro" instead of "amdgpu":

in /opt/amdgpu-pro/lib/i386-linux-gnu/:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca32.so

in /opt/amdgpu-pro/lib/x86_64-linux-gnu:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
ln -s libamd_comgr.so.1.7.0 libamd_comgr.so
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca64.so

now if you didn't copy the files above to the real directories /etc and /opt, do it now, and then run
ldconfig

OK, I suppose that it is well understood that one should be careful when running a random installation script from the web (more so if it isn't from a well respected source). In fact, I first ran this script (in Gentoo Linux) manually line-by-line to be sure what's going on. So I see that it is better to provide a step-by-step manual installation here instead of a script. In any case I'm not forcing anybody to do it - I'm just giving the procedure which enabled me to get working OpenCL on APU Renoir (4650G, 4750G) and also Cezanne (5700G); this "PAL" OpenCL is also a faster alternative to ROCm for some workloads on R9 Nano (Fiji, gfx803), Vega 56/64 and Radeon VII. I tested it with kernels 5.8 (Renoir) and 5.10 (Cezanne), so I don't know if it works with the newer ones. By the way, with regard to ROCm OpenCL on Renoir - it seems that it works somewhere sometimes (Cezanne is even worse) - it apparently depends also on the motherboard/BIOS - one program may work in one configuration, but not the other, and vice versa (my experience listed above actually involved different motherboards).
So here is the extraction and installation of PAL OpenCL (I removed Vulkan, it proved to be unnecessary):

download AMDGPU-PRO 20.40 and extract it to some temporary directory

wget --referer https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-20-40 https://drivers.amd.com/drivers/linux/amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz
tar xJf amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz

prepare directories /opt/amdgpu and /opt/amdgpu-pro - i.e., if they are present, uninstall corresponding packages or take care not to replace existing files by the ones listed below; if they are not present, create them (including the needed subdirectories). In the steps below, I recommend to create a separate (temporary) directory tree for /opt and /etc, which can be saved and employed for repeated installs of this OpenCL
extract files from *.deb archives by means of

ar x package.deb
tar xJvf data.tar.xz
namely:

extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_amd64.deb, then rename and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/libdrm_amdpro.so.1.0.0
extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_i386.deb, then rename and place it to /opt/amdgpu-pro/lib/i386-linux-gnu/libdrm_amdpro.so.1.0.0
extract amdgpu.ids from libdrm-amdgpu-common_1.0.0-1147286_all.deb, and place it to /opt/amdgpu/share/libdrm/
extract libamd_comgr.so.1.7.0 from opencl-amdgpu-pro-comgr_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl64.so from opencl-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca64.so and libamdocl12cl64.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place them to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca32.so and libamdocl12cl32.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_i386.deb, and place them to /opt/amdgpu-pro/lib/i386-linux-gnu/

create files with simple text contents:

/etc/OpenCL/vendors/amdocl64.icd containing libamdocl64.so (this file can be present from rocm-opencl-runtime, in which case you can simply keep it)
/etc/OpenCL/vendors/amdocl-orca64.icd containing libamdocl-orca64.so
/etc/OpenCL/vendors/amdocl-orca32.icd containing libamdocl-orca32.so
/etc/ld.so.conf.d/zz_amdgpu-pro_x86_64.conf containing

AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/x86_64-linux-gnu

/etc/ld.so.conf.d/zz_amdgpu-pro_x86.conf containing

AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/i386-linux-gnu

create symbolic links and modify the binaries to use "amdpro" instead of "amdgpu":

in /opt/amdgpu-pro/lib/i386-linux-gnu/:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca32.so

in /opt/amdgpu-pro/lib/x86_64-linux-gnu:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
ln -s libamd_comgr.so.1.7.0 libamd_comgr.so
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca64.so

now if you didn't copy the files above to the real directories /etc and /opt, do it now, and then run
ldconfig

that's exactly what I suspected, the amdgpu-pro vulkan causes arch linux to fail. But even if I read over the script there was no way to realize that.
Anyway, the way I did it was using the 20.40 version from the aur. (thank you arch linux for keeping old versions)

OK, I suppose that it is well understood that one should be careful when running a random installation script from the web (more so if it isn't from a well respected source). In fact, I first ran this script (in Gentoo Linux) manually line-by-line to be sure what's going on. So I see that it is better to provide a step-by-step manual installation here instead of a script. In any case I'm not forcing anybody to do it - I'm just giving the procedure which enabled me to get working OpenCL on APU Renoir (4650G, 4750G) and also Cezanne (5700G); this "PAL" OpenCL is also a faster alternative to ROCm for some workloads on R9 Nano (Fiji, gfx803), Vega 56/64 and Radeon VII. I tested it with kernels 5.8 (Renoir) and 5.10 (Cezanne), so I don't know if it works with the newer ones. By the way, with regard to ROCm OpenCL on Renoir - it seems that it works somewhere sometimes (Cezanne is even worse) - it apparently depends also on the motherboard/BIOS - one program may work in one configuration, but not the other, and vice versa (my experience listed above actually involved different motherboards).
So here is the extraction and installation of PAL OpenCL (I removed Vulkan, it proved to be unnecessary):

download AMDGPU-PRO 20.40 and extract it to some temporary directory

wget --referer https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-20-40 https://drivers.amd.com/drivers/linux/amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz
tar xJf amdgpu-pro-20.40-1147286-ubuntu-20.04.tar.xz

prepare directories /opt/amdgpu and /opt/amdgpu-pro - i.e., if they are present, uninstall corresponding packages or take care not to replace existing files by the ones listed below; if they are not present, create them (including the needed subdirectories). In the steps below, I recommend to create a separate (temporary) directory tree for /opt and /etc, which can be saved and employed for repeated installs of this OpenCL
extract files from *.deb archives by means of

ar x package.deb
tar xJvf data.tar.xz
namely:

extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_amd64.deb, then rename and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/libdrm_amdpro.so.1.0.0
extract libdrm_amdgpu.so.1.0.0 from libdrm-amdgpu-amdgpu1_2.4.100-1147286_i386.deb, then rename and place it to /opt/amdgpu-pro/lib/i386-linux-gnu/libdrm_amdpro.so.1.0.0
extract amdgpu.ids from libdrm-amdgpu-common_1.0.0-1147286_all.deb, and place it to /opt/amdgpu/share/libdrm/
extract libamd_comgr.so.1.7.0 from opencl-amdgpu-pro-comgr_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl64.so from opencl-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place it to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca64.so and libamdocl12cl64.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_amd64.deb, and place them to /opt/amdgpu-pro/lib/x86_64-linux-gnu/
extract libamdocl-orca32.so and libamdocl12cl32.so from opencl-orca-amdgpu-pro-icd_20.40-1147286_i386.deb, and place them to /opt/amdgpu-pro/lib/i386-linux-gnu/

create files with simple text contents:

/etc/OpenCL/vendors/amdocl64.icd containing libamdocl64.so (this file can be present from rocm-opencl-runtime, in which case you can simply keep it)
/etc/OpenCL/vendors/amdocl-orca64.icd containing libamdocl-orca64.so
/etc/OpenCL/vendors/amdocl-orca32.icd containing libamdocl-orca32.so
/etc/ld.so.conf.d/zz_amdgpu-pro_x86_64.conf containing

AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/x86_64-linux-gnu

/etc/ld.so.conf.d/zz_amdgpu-pro_x86.conf containing

AMDGPU-PRO OpenCL support
/opt/amdgpu-pro/lib/i386-linux-gnu

create symbolic links and modify the binaries to use "amdpro" instead of "amdgpu":

in /opt/amdgpu-pro/lib/i386-linux-gnu/:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca32.so

in /opt/amdgpu-pro/lib/x86_64-linux-gnu:

ln -s libdrm_amdpro.so.1.0.0 libdrm_amdpro.so.1
ln -s libamd_comgr.so.1.7.0 libamd_comgr.so
sed -i "s|libdrm_amdgpu|libdrm_amdpro|g" libamdocl-orca64.so

now if you didn't copy the files above to the real directories /etc and /opt, do it now, and then run
ldconfig

that's exactly what I suspected, the amdgpu-pro vulkan causes arch linux to fail. But even if I read over the script there was no way to realize that. Anyway, the way I did it was using the 20.40 version from the aur. (thank you arch linux for keeping old versions)
then I disabled updates for opencl-amd

this now works using opencl-amd 22.10.3 on the AUR, I only have 512 MB of vram though, any for ROCM to use my system ram. (I cannot expand my ram size since my laptop's bios is retarded)

@Etaash-mathamsetty maybe you can upgrade your BIOS? i had to that to increase VRAM on my ThinkCentre system with AMD Ryzen 5 PRO 4650GE with Radeon Graphics. then i selected 2G in BIOS settings for VRAM and i currently see 1983M VRAM in radeontop.

@Etaash-mathamsetty maybe you can upgrade your BIOS? i had to do that to increase VRAM on my ThinkCentre system with AMD Ryzen 5 PRO 4650GE with Radeon Graphics. then i selected 2G in BIOS settings for VRAM and i currently see 1983M VRAM in radeontop.

I am already running the latest bios lol


Login flow when using security key

The login flow in hangar-auth feels really weird. So I click on Security Key, enter my username, and then "Sign in with security key". Then I get the Prepare your WebAuthn device thingy, but below that the "Password" tab is selected and I have to click on "Security Key" again, before I then get a Continue button that feels completely out of place and has the same color as Register/Forgot

Why do I have to manually click on "Continue" after I've already clicked on "Sign in with security key"? Feels like that step is unnecessary I kinda understand it now, but see the rest
On the second page, the "Security Key" tab should ideally be pre-selected, especially since the password-tab is empty there
There should be some sort of visual distinction between the Continue button and Register/Forgot. Could be a blue button as well
Without a text directly above the "Continue" button the spacing looks kinda weird

passwordless signin will most likely be disabled for now

is necessary cause we only load the webauthn js after you pressed the sign in with key button
just removed the tab
just rermoved the buttons
spacing is fixed in browsers that support :has  (https://caniuse.com/css-has)

https://github.com/HangarMC/HangarAuth/commit/9e7f617775280ffc192ba299504ef6bd358a5a02


Hawkular-1295 Migrate Alerts to Prometheus

Update some entities to support alerting.
@lucasponce This PR for the client supports another PR for the provider.  If you prefer not to merge these into your branch it is OK, I can wait and later add PRs against the master branches.  But, this at least allows me to work against your branch for staging, since some of the live-metrics work supports the alerts work.  If you do merge we can consolidate, which may be useful.

Hi Jay,
Sorry, on this repo I used a different branch.
The PR looks ok, but I need to re-arrange mine and consolidate.
I'm going to cherry-pick this commit and prepare a good hawkular-1275 branch.


chore: upgrade node to 18

Node 14 is out of cycle already, and 16 is about to be dropped in couple months, we can move to 18 which is current LTS.
Upgrading jest because our current version doesn't work with node 18 that I'm using locally.

I would recommend to wait for this DigitalOcean dont support v18

Just to confirm this would just force v18 for the test right? Cuz DO still doesn't support v18

Yes, I also tested installing deps with v16 and v14 in case any new packages force specific version, but both work still.


Add choose_action timing configuration option

Adds capability and configuration option to time the choose_action method and create a report summarizing those times.

Tried it out, looks good! Ready to merge


poem.rs down?

I found a link to https://poem.rs here: https://www.reddit.com/r/rust/comments/r4jzo3/poemopenapi_10_released/
That site seems to be down.

Yes, moved the guides all into the source code since I didn't have enough time to maintain it. 🙂

Ahh, got it. Was wondering why it wasn’t linked from anywhere else 🙂


Create SECURITY.md

Create security policy

Thanks! Can you link to this from the README?

Not sure if necessary, I do know that github will automatically show it under the "security" column

but we can certainly also link from readme if we want!

Ah, I had no idea! Great, then we can merge this in as-is.


Refactor DynamoConfig DI to not be awful

What is this PR trying to achieve?
Ensure adding additional config for dynamo is easy and understandable.
Who is this change for?
Developers who wish to stay sane.

Working towards: https://github.com/wellcometrust/platform-api/issues/194


chore: introduce the Operation trait to enforce consistency between operations

I am spinning this pull request out from some of my other in-flight work to extend behaviors with the operations.
The motivation here is to provide a base trait which can act as a standards-bearer for the operations implementations. Right now this is only enforcing IntoFuture but there are a couple of other changes I am actively working on which I hope will dramatically clean up these operations and enable some of the use-cases that @hntd187 and I have been exploring.

To David's point this is why I think a marker trait by itself it maybe not sufficient, we should eventually move towards a trait that implements and expected life cycle of an operation (or multiple smaller ones that implement a part of it) the most ready example I can think of is something similar to Hadoop's commit protocol, but we should do something that makes sense for us, I'm just using as an example of generalization of such a process.

@hntd187 that's certainly the plan here :smile: I just wanted to get something started we could begin hanging functionality off of


buffers.status.actions.open error for empty buffer Vim(print):E749: Empty buffer

Description
Neogit, opening an empty untracked file gives the below error.
This is caused by the 1 command in the function open() in buffers/status/actions.lua. One solution is to make it silent.
Neovim version
NVIM v0.10.2
Build type: Release
LuaJIT 2.1.1713484068
Operating system and version
Linux
Steps to reproduce
:!touch hello.txt
:Neogit

browse to hello.txt, press Enter.
Expected behavior
No response
Actual behavior
Error executing vim.schedule lua callback: vim/_editor.lua:0: nvim_exec2(): Vim(print):E749: Empty buffer
stack traceback:
[C]: in function 'nvim_exec2'
vim/_editor.lua: in function 'cmd'
...e/nvim/lazy/neogit/lua/neogit/buffers/status/actions.lua:83: in function ''
vim/_editor.lua: in function <vim/_editor.lua:0>
Minimal config
-- NOTE: See the end of this file if you are reporting an issue, etc. Ignore all the "scary" functions up top, those are
-- used for setup and other operations.
local M = {}

local base_root_path = vim.fn.fnamemodify(debug.getinfo(1, "S").source:sub(2), ":p:h") .. "/.min"
function M.root(path)
  return base_root_path .. "/" .. (path or "")
end

function M.load_plugin(plugin_name, plugin_url)
  local package_root = M.root("plugins/")
  local install_destination = package_root .. plugin_name
  vim.opt.runtimepath:append(install_destination)

  if not vim.loop.fs_stat(package_root) then
    vim.fn.mkdir(package_root, "p")
  end

  if not vim.loop.fs_stat(install_destination) then
    print(string.format("> Downloading plugin '%s' to '%s'", plugin_name, install_destination))
    vim.fn.system({
      "git",
      "clone",
      "--depth=1",
      plugin_url,
      install_destination,
    })
    if vim.v.shell_error > 0 then
      error(string.format("> Failed to clone plugin: '%s' in '%s'!", plugin_name, install_destination),
        vim.log.levels.ERROR)
    end
  end
end

---@alias PluginName string The plugin name, will be used as part of the git clone destination
---@alias PluginUrl string The git url at which a plugin is located, can be a path. See https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols for details
---@alias MinPlugins table<PluginName, PluginUrl>

---Do the initial setup. Downloads plugins, ensures the minimal init does not pollute the filesystem by keeping
---everything self contained to the CWD of the minimal init file. Run prior to running tests, reproducing issues, etc.
---@param plugins? table<PluginName, PluginUrl>
function M.setup(plugins)
  vim.opt.packpath = {}                      -- Empty the package path so we use only the plugins specified
  vim.opt.runtimepath:append(M.root(".min")) -- Ensure the runtime detects the root min dir

  -- Install required plugins
  if plugins ~= nil then
    for plugin_name, plugin_url in pairs(plugins) do
      M.load_plugin(plugin_name, plugin_url)
    end
  end

  vim.env.XDG_CONFIG_HOME = M.root("xdg/config")
  vim.env.XDG_DATA_HOME = M.root("xdg/data")
  vim.env.XDG_STATE_HOME = M.root("xdg/state")
  vim.env.XDG_CACHE_HOME = M.root("xdg/cache")

  -- NOTE: Cleanup the xdg cache on exit so new runs of the minimal init doesn't share any previous state, e.g. shada
  vim.api.nvim_create_autocmd("VimLeave", {
    callback = function()
      vim.fn.system({
        "rm",
        "-r",
        "-f",
        M.root("xdg")
      })
    end
  })
end

-- NOTE: If you have additional plugins you need to install to reproduce your issue, include them in the plugins
-- table within the setup call below.
M.setup({
  plenary = "https://github.com/nvim-lua/plenary.nvim.git",
  telescope = "https://github.com/nvim-telescope/telescope.nvim",
  diffview = "https://github.com/sindrets/diffview.nvim",
  neogit = "https://github.com/NeogitOrg/neogit"
})
-- WARN: Do all plugin setup, test runs, reproductions, etc. AFTER calling setup with a list of plugins!
-- Basically, do all that stuff AFTER this line.
require("neogit").setup({}) -- For instance, setup Neogit

Actually, after having used neogit some more, this 1 command is quite annoying to me in general, since I've set up neovim to automatically open files at the last known cursor position. But 1 overrides this and forces the cursor to the top line.
Is there a reason for the 1 ? My preferred solution would be to replace it with execute (or some other noop).


i cant find the particle accelerators controller

i have made a small particle accelerator but cant find the controller for it i tried searching it in jei but nothing comes up

@crazyfisheater Which version of the mod is this?

the mod version is 2.18o-1.12.2
уто, 18. феб 2020. у 13:53 Tom Dodd<EMAIL_ADDRESS>је
написао/ла:

@crazyfisheater https://github.com/crazyfisheater Which version of the
mod is this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/turbodiesel4598/NuclearCraft/issues/629?email_source=notifications&email_token=AOS3MXCR3GV4KLKH7FDNN2DRDPK6NA5CNFSM4KXD3HE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMB3MUA#issuecomment-587445840,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOS3MXFNTHNRTAY2FQAWVZDRDPK6NANCNFSM4KXD3HEQ
.

NC for 1.12.2 does not have a particle accelerator - it was not implemented in the end, but may be in the future :)


[Feature Request] Manage permissions

Is your feature request related to a problem? Please describe.
Ability to manage user permissions - view all permissions and update permission.
Describe the solution you'd like
Need a command to view all permission for a user and also to update them.
Additional context
NA
Spec against which we are actively developing the permission management commands:
Permission Management

Scenario
Command
Open Questions

I want to view the permissions I have for a namespace
az devops security permission list  --namespace-id --subject
List all tokens for this user/group

I want to add a permission for a namespace
az devops security permission add --subject --namespace-id --token --allow-bit --deny-bit
Set allow/deny permission bits for a given user/group and token

I want to remove a permission for a namespace
az devops security permission reset --permissions --subject --namespace-id --token

I want to reset all permission for a namespace
`az devops security permission --reset-all --subject --namespace-id --token

Show resolved permissions
az devops security permission show --subject --namespace-id --token

I want to view all namespaces available
az devops security permission namespace list

I want to view all permissions available in a namespace
az devops security permission namespace show --namespace-id

Permission related commands will be available in the next release (i.e 0.9.0.)


updated logic for username / password query to also support nonzenity…

… option.
If only username is set, only password is prompted.
if both are set, nothing will be prompted.
if both are not set, both will be prompted.
tested with my local owncloud, worked fine fore me, but please also test ;-)
Feature addition can probably done a bit nicer, but i'm no code guru...

@schiesbn now that OC 8.1 is out, would you comment? :-) thanks


Bug/issue 13 change dest maps folder

Resolves #12, as well as adds the .min.css extension to minified files when concat is set to false.

Can one of the admins verify this patch?

ok to test


Expose callbacks to manipulate the nodes

Parse the document to generate a tree of nodes that can be manipulated by Elixir functions.
So it can manipulate the output similar to how it's done at https://docs.rs/comrak/latest/comrak/ but on the Elixir side.

This will be very helpful.
And thank you for this Hex package. <3

Done! See examples at https://github.com/leandrocp/mdex/tree/main/examples