Profile caffe opencl version and find bottleneck at init phase of caffe.time() method

For caffe opencl time command in GPU mode
we have 16 sec spent for init  part:
      I0130 10:21:46.310158  3137 caffe.cpp:405] Use GPU with device ID 0
      I0130 10:21:46.310220  3137 caffe.cpp:406] Before Caffe::SetDevice(gpus)
      I0130 10:21:46.313752  3137 device.cpp:62] CL_DEVICE_HOST_UNIFIED_MEMORY: 1
      I0130 10:22:02.722736  3137 caffe.cpp:408] After Caffe::SetDevice(gpus)
      I0130 10:22:02.722839  3137 caffe.cpp:417] Before caffe_net

Need to find exact place (bottleneck) at init phase
and try to optimize (cache some compiled kernels for example)

After additional logging and execution I've found
      I0130 11:02:55.396101 13394 cl_kernels.cpp:5237] Before add_program(kernel_program)
      I0130 11:03:11.751402 13394 cl_kernels.cpp:5240] After add_program(kernel_program)

Place at code
caffe/src/caffe/greentea/cl_kernels.cpp (5238)^

So second usage of same kernel could be loaded from cache. I suppose cache path should be set before first kernel init:
void viennacl::ocl::context::cache_path	(	std::string 	new_path	)

See fore details: http://viennacl.sourceforge.net/doc/classviennacl_1_1ocl_1_1context.html#a6aa5ef5cb563ed047ff72f286eb5718b

Yes, I figured that out too, but it seems that setting VIENNACL_CACHE_PATH to some path, say /data/local/tmp/ocl-cache- forces ViennaCL to start caching and reusing compiled kernels. But it seems that cached kernels are in IR format (not executable) and still compiled each time which is the most time consuming part. Also it seems that ViennaCL calculates hashes of all files and also save cached files extra time to disk - don't know if it can be optimized or not ...
See in viennacl:

include/viennacl/ocl/context.hpp
...
temp = clCreateProgramWithBinary(h_.get(),1,&devid,&len, &bufdata,&status,&err);
vs
temp = clCreateProgramWithSource(h_.get(), 1, (const char **)&source_text, &source_size, &err);
...
err = clBuildProgram(temp, 0, NULL, options, NULL, NULL);
...

Extra ViennaCL debug if needed:
log-viennacl-debug.txt

I've tried
ck run program:caffe-time-opencl --target_os=android21-arm-v7a --env.CAFFE_COMPUTE_DEVICE_ID=0 --env.VIENNACL_CACHE_PATH=/data/local/tmp/tmp
but can't see  ViennaCL has been started caching and reusing compiled kernels ...
will continue ...

How do you check caching?
VIENNACL doesn't show debug info by default.
When recompiling Caffe packges, you need to add
to the beggining of the file "install/include/viennacl/ocl/context.hpp"
in the installed ViennaCL:
#define VIENNACL_DEBUG_ALL 1
#defined VIENNACL_DEBUG_CONTEXT 1
You can also probably put your own prints there
(std::out << ...) to check if cached files were found ...
Thanks!
On 2017-01-30 14:23, Daniil Efremov wrote:

I've tried
ck run program:caffe-time-opencl --target_os=android21-arm-v7a
--env.CAFFE_COMPUTE_DEVICE_ID=0
--env.VIENNACL_CACHE_PATH=/data/local/tmp/tmp
but can't see ViennaCL has been started caching and reusing compiled
kernels ...
will continue ...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dividiti/ck-caffe/issues/44#issuecomment-276061303,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHCu5fHWen4X_nMvw-y0GPdo-_tHeiZPks5rXeRogaJpZM4LxBeV.

http://viennacl.sourceforge.net/doc/faq.html:
I'm using OpenCL and my program seems to hang for about a second when I call ViennaCL functions?

This is due to the just-in-time compilation of OpenCL kernels. The NVIDIA graphics driver caches the compiled kernels, therefore the overhead is only seen at the first run on a particular machine. The OpenCL SDKs of AMD and Intel, however, recompile all kernels with each program launch. Other OpenCL SDKs most likely show similar behavior.

I think OpenCL kernel caching in ViennaCL is a quite recent feature ... But again, as discussed with @psyhtest , we need to check what is cached there - sources, IR or binary (I see ELF files being recorded for kernels on Linux/Windows/Android when using VIENNACL_CACHE_PATH) ...

I've tried for linux
ck run program:caffe-time-opencl --env.VIENNACL_CACHE_PATH=/home/daniil/vienacl_cache
and for android
ck run program:caffe-time-opencl --target_os=android21-arm-v7a env.VIENNACL_CACHE_PATH=/data/local/tmp/tmp
but can't  see any files at provided dirs

I've checked the source. Just as I thought, caching is done before any kernels are compiled. Therefore, all kernels need to be recompiled, once the binary is loaded. I'll experiment with changing the order of compilation to see if that helps.

@DVEfremov: on your machine compilation seems to take 15-16 s. On my machine, it takes about 11-12 s.
I0202 17:53:22.211339  1313 cl_kernels.cpp:5237] just before add_program 1
I0202 17:53:22.222057  1313 cl_kernels.cpp:5240] just after add_program 1

To see this, I modified ~/CK_TOOLS/lib-caffe-bvlc-opencl-viennacl-master-gcc-5.4.1-linux-64/src/caffe/greentea/cl_kernels.cpp as follows:
@@ -5234,8 +5234,10 @@ viennacl::ocl::program & RegisterKernels(viennacl::ocl::context *ctx) {
 #ifdef USE_FFT
   ctx->build_options("-DFFT");
 #endif
+  LOG(INFO) << "just before add_program 1\n";
   viennacl::ocl::program &program = ctx->add_program(kernel_program,
       "kernel_program");
+  LOG(INFO) << "just after add_program 1\n";
   return program;
 }

and then rebuilt and rerun using:
$ cd ~/CK_TOOLS/lib-caffe-bvlc-opencl-viennacl-master-gcc-5.4.1-linux-64/obj
$ LIBRARY_PATH=/home/anton/CK_TOOLS/lib-openblas-0.2.19-gcc-5.4.1-linux-64/install/lib:/home/anton/CK_TOOLS/lib-gflags-2.2.0-gcc-5.4.1-linux-64/install/lib:$LIBRARY_PATH  make && make install
$ ck run program:caffe

(These are notes for myself to try on another machine.)

OK, apparently NVIDIA cache compiled kernels somewhere. This cache interferes with my measurements, but can be disabled as follows:
$ ck run program:caffe --env.CUDA_CACHE_DISABLE=1

resulting in about 2.4 s:
I0202 18:25:51.529376  2609 cl_kernels.cpp:5237] just before add_program 1
I0202 18:25:53.906580  2609 cl_kernels.cpp:5240] just after add_program 1

By the way, you can use the following handy function to compute the elapsed time between two timestaps:
#
# timestamp_delta.py
#
import dateutil.parser

# Return the difference between the end and start timestamps in seconds.
def timestamp_delta_s(ts_end, ts_start):
    delta = dateutil.parser.parse(ts_end) - dateutil.parser.parse(ts_start)
    delta_s = delta.total_seconds()
    return delta_s

For example, if you %load timestamp_delta.py in IPython, you can compute the time difference as follows:
In [5]: timestamp_delta_s("0202 18:25:53.906580","0202 18:25:51.529376")
Out[5]: 2.377204

The following command also runs in about 2.4 s:
$ ck run program:caffe --env.VIENNACL_CACHE_PATH=/home/anton/CK_REPOS/ck-caffe/program/caffe/tmp/viennacl_cache/ --env.CUDA_CACHE_DISABLE=1

and populates the cache with the following 5 files:
$ ls -la /home/anton/CK_REPOS/ck-caffe/program/caffe/tmp/viennacl_cache/
total 1012
drwxrwxr-x 2 anton dvdt   4096 Feb  3 09:42 .
drwxrwxr-x 3 anton dvdt  40960 Feb  3 09:42 ..
-rw-rw-r-- 1 anton dvdt 121937 Feb  3 09:42 039e228d3c5a327d29b7c1ddae5b3a87a2a6b582
-rw-rw-r-- 1 anton dvdt  25029 Feb  3 09:42 1a45ff74948f016e7c31803583a1dcceace1796f
-rw-rw-r-- 1 anton dvdt  68288 Feb  3 09:42 25a82fda93da2cf1f1a3482e3008b3837c371390
-rw-rw-r-- 1 anton dvdt 739419 Feb  3 09:42 2fc1511e148c546bd0343ea3043027942b59b1d8
-rw-rw-r-- 1 anton dvdt  21034 Feb  3 09:42 53cef1874cc4eddbc0c1ad3595c5ac5c2310bf10

Seems to be working on Windows, but I need to check it deeper - just don't have time right now ...

Made a start on the Odroid XU3, but got the following with dvdt-prof:
I0203 16:21:38.867061   978 caffe.cpp:423] Testing for 1 iterations.
*** Aborted at<PHONE_NUMBER> (unix time) try "date -d @1486138900" if you are using GNU date ***
PC: @ 0xb57bb742 mcl_entrypoints_valid_event_list_common
*** SIGSEGV (@0x22ce5164) received by PID 978 (TID 0xafe3b000) from PID 583946596; stack trace: ***

mcl stands for "Mali OpenCL" - it's a function internal to the driver. I will try updating the driver from v10 to v12, before blaming it on dvdt-prof.

Shocking ;) ...
On 2017-02-03 17:26, Anton Lokhmotov wrote:

Made a start on the Odroid XU3, but got the following with |dvdt-prof|
https://github.com/dividiti/dvdt-prof:
|I0203 16:21:38.867061 978 caffe.cpp:423] Testing for 1 iterations. ***
Aborted at<PHONE_NUMBER> (unix time) try "date -d @1486138900" if you are
using GNU date *** PC: @ 0xb57bb742
mcl_entrypoints_valid_event_list_common *** SIGSEGV (@0x22ce5164)
received by PID 978 (TID 0xafe3b000) from PID 583946596; stack trace:
*** |
|mcl| stands for "Mali OpenCL" - it's a function internal to the
driver. I will try updating the driver from v10 to v12, before blaming
it on |dvdt-prof|.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dividiti/ck-caffe/issues/44#issuecomment-277293176,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHCu5eJfry_v3YpjWVdCUTtetKJMHyI1ks5rY1U2gaJpZM4LxBeV.

OK, I'm ready to make some experiments on the Odroid XU3. I only have one Caffe lib, one Caffe model and one dividiti's OpenCL profiler registered on the platform:
$ ck show env --tags=caffemodel
Env UID:         Target OS: Bits: Name:                                         Version: Tags:
e8811419b9c1149c   linux-32    32 Caffe model (net and weights) (bvlc, alexnet) trunk    32bits,alexnet,bvlc,caffe,caffemodel,host-os-linux-32,mirror,net,target-os-linux-32,v0,weights
$ ck show env --tags=lib,caffe
Env UID:         Target OS: Bits: Name:                                  Version:       Tags:
53834f239eff6c18   linux-32    32 BVLC Caffe framework (opencl,viennacl) master-69f35c5 32bits,bvlc,caffe,host-os-linux-32,lib,target-os-linux-32,v0,v0.0,vmaster,vopencl
$ ck show env --tags=dvdt,prof
Env UID:         Target OS: Bits: Name:                                  Version: Tags:
2ea881239f688658   linux-32    32 dividiti's OpenCL API profiler (cjson) trunk    32bits,cjson,dividiti,dvdt,host-os-linux-32,opencl,prof,profiler,target-os-linux-32,tool,v0,vtrunk

This allows to make a quick profiling run of program:caffe e.g. as follows:
$ ck benchmark program:caffe \
    --record --record_repo=local --record_uoa=caffe-explore-build-compile --skip_stat_analysis \
    --cmd_key=time_gpu --cpu_freq=max \
    --dvdt_prof --repetitions=1 \
    --env.CK_CAFFE_BATCH_SIZE=1

$ rm -rf /tmp/viennacl-cache && mkdir /tmp/viennacl-cache
$ ck benchmark program:caffe \
    --record --record_repo=local --record_uoa=caffe-explore-build-compile --skip_stat_analysis \
    --cmd_key=time_gpu --cpu_freq=max --dvdt_prof \
    --repetitions=3 \
    --env.CK_CAFFE_BATCH_SIZE=1 \
    --env.VIENNACL_CACHE_PATH=/tmp/viennacl-cache \
    --tags=time,build,compile,original \
    --subtags=empty-cache

Will next make some modifications to ViennaCL, and reinstall using:
$ ck install package:lib-caffe-bvlc-opencl-viennacl-universal \
  --env.DISABLE_DOUBLE_SUPPORT=ON \
  --env.DISABLE_DEVICE_HOST_UNIFIED_MEMORY=ON \
  --env.CK_HOST_CPU_NUMBER_OF_PROCESSORS=2

If storing to cache after all the kernels are created:
$ ls -la $VIENNACL_CACHE_DIR/
total 616
drwxr-xr-x  2 anton dvdt   4096 Feb 24 00:02 .
drwxrwxrwt 10 root  root  20480 Feb 24 00:01 ..
-rw-r--r--  1 anton dvdt  52603 Feb 24 00:01 0a142aebde6485b18653b5fa0be8ad372350b6a8
-rw-r--r--  1 anton dvdt 334286 Feb 24 00:01 71ded351ed4bf39e36515c4fc0162d80afdf3c7c
-rw-r--r--  1 anton dvdt  62052 Feb 24 00:01 98026695c419af93aabbc354c5a5ee77a4493f67
-rw-r--r--  1 anton dvdt 108399 Feb 24 00:02 ed2e1ee786038a1cd3f842f57cdd6878404345de
-rw-r--r--  1 anton dvdt  36315 Feb 24 00:01 faace8064282b7b739bcffc5515df19ca2de1de5

Submitted a PR to ViennaCL to move binary caching after kernel compilation, which has a dramatic effect on Mali powered platforms on subsequent invocations.

The above mentioned PR has been merged to the ViennaCL master, so all ViennaCL packages (except 1.7.1 of course) should now have the improved caching mechanism.
Don't forget that you need to set the environment variable (e.g. `VIENNACL_CACHE_PATH=/tmp/viennacl-cache/') for caching to work!


Is there a cdn release? Want to try siwe with Flutter Web

Flutter Web can call JS code. I already use ethers.js and Metamask this way and now want to try adding siwe, but can't seem to find how to install it with

Unfortunately there is not.

Going to close this for now but will also note that there is a community member that made a siwe-flutter library!
https://github.com/onemoongate/Flutter-SIWE-Plugin


[GSSoC '23] Contributing a Shopping Cart Project to Code-Canvas

Description
I am making this issue to contribute a shopping cart project build with vanilla JS to Code-Canvas open source project.
Objective
By contributing this project I want to help beginners build and learn more JavaScript Projects and have a good understanding in JavaScript, so this website will be a great project for everyone to have more knowledge about JS.
So Please assign this to me under GSSoC'23 @Dharmesh177

assign to @debsouryadatta while making PR follow contirbuting.md guidelines and complete it in 1 week

Sure ✌

assign to @debsouryadatta while making PR follow contirbuting.md guidelines and complete it in 1 week


Feature request: allow for NLog to render nested innerexceptions to JSON

Feature request for NLog version: 4.2.3
The current JSONLayout renderer only supports the rendering of the first exception in fully structured JSON.  I would like to be able to render out all the exceptions, including the inner exceptions, in structured JSON format.
This would allow for the separate exceptions to be inspected and allow for a deeper understanding of what happened.
Currently you can do something like this:
<layout xsi:type="JsonLayout">
          <attribute name="datetime" layout="${iso8601dateformat}" />
          <attribute name="level" layout="${level:uppercase=true}" />
          <attribute name="logger" layout="${logger}" />
          <attribute name="thread" layout="${threadid}" />
          <attribute name="machine" layout="${machinename}" />
          <attribute name="message" layout="${message}" />
          <attribute name="exceptions" encode="false">
            <layout type="JsonLayout">
              <attribute name="exception" encode="false">
                <layout type="JsonLayout">
                  <attribute name="message" layout="${exception:format=message}" />
                  <attribute name="stacktrace" layout="${exception:format=stacktrace}" />
                  <attribute name="data" layout="${exception:format=data}" />
                  <attribute name="type" layout="${exception:format=type}" />
                  <attribute name="method" layout="${exception:format=method}" />
                </layout>
              </attribute>
            </layout>
          </attribute>
        </layout>

Which will render out something like this:
{
    "datetime": "2016-03-18T10:28:30.810",
    "level": "ERROR",
    "logger": "Fairfax.Classifieds.Ore.Company.Business.Controller.Common.ExceptionHandler",
    "thread": "82",
    "machine": "R5001764W",
    "exceptions": {
        "exception": {
            "message": "Check this out",
            "stacktrace": "   at Company.Default.Page_Load(Object sender, EventArgs e) in C:\\Source\\Company\\src\\services\\Company\\Default.aspx.cs:line 20\r\n   at System.EventHandler.Invoke(Object sender, EventArgs e)\r\n   at System.Web.UI.Control.OnLoad(EventArgs e)\r\n   at System.Web.UI.Control.LoadRecursive()\r\n   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)",
            "type": "System.ApplicationException",
            "method": "Void Page_Load(System.Object, System.EventArgs)"
        }
    },
}

I would like to be able to render out is something like this:
{
    "datetime": "2016-03-18T10:28:30.810",
    "level": "ERROR",
    "logger": "Fairfax.Classifieds.Ore.Company.Business.Controller.Common.ExceptionHandler",
    "thread": "82",
    "machine": "R5001764W",
    "exceptions": {
        "exception": {
            "message": "Check this out",
            "stacktrace": "   at Company.Default.Page_Load(Object sender, EventArgs e) in C:\\Source\\Company\\src\\services\\Company\\Default.aspx.cs:line 20\r\n   at System.EventHandler.Invoke(Object sender, EventArgs e)\r\n   at System.Web.UI.Control.OnLoad(EventArgs e)\r\n   at System.Web.UI.Control.LoadRecursive()\r\n   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)",
            "type": "System.ApplicationException",
            "method": "Void Page_Load(System.Object, System.EventArgs)"
        },
        "exception": {
            "message": "Inner Exception 1",
            "stacktrace": "   at Company.Default.Method1(Object sender, EventArgs e) in C:\\Source\\Company\\src\\services\\Company\\Default.aspx.cs:line 20\r\n   at System.EventHandler.Invoke(Object sender, EventArgs e)\r\n   at System.Web.UI.Control.OnLoad(EventArgs e)\r\n   at System.Web.UI.Control.LoadRecursive()\r\n   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)",
            "type": "System.InvalidOperationException",
            "method": "Void Method1(System.Object, System.EventArgs)"
        }
    },
}

I have a technical background so I'm happy to help implement but would like some guidance on architecturally how this can be best achieved.
Many thanks,
Lach

Cool.
Aby idea how the config could look like? Would a double nested exception (in json) also OK?

Actually Maybe a structure like this would be better?
{
	"exception": {
		"message": "Check this out",
		"stacktrace": "   at Company.Default.Page_Load(Object sender, EventArgs e) in C:\\Source\\Company\\src\\services\\Company\\Default.aspx.cs:line 20\r\n   at System.EventHandler.Invoke(Object sender, EventArgs e)\r\n   at System.Web.UI.Control.OnLoad(EventArgs e)\r\n   at System.Web.UI.Control.LoadRecursive()\r\n   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)",
		"type": "System.ApplicationException",
		"method": "Void Page_Load(System.Object, System.EventArgs)",
		"innerException": {
			"message": "Inner Exception 1",
			"stacktrace": "   at Company.Default.Method1(Object sender, EventArgs e) in C:\\Source\\Company\\src\\services\\Company\\Default.aspx.cs:line 20\r\n   at System.EventHandler.Invoke(Object sender, EventArgs e)\r\n   at System.Web.UI.Control.OnLoad(EventArgs e)\r\n   at System.Web.UI.Control.LoadRecursive()\r\n   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)",
			"type": "System.InvalidOperationException",
			"method": "Void Method1(System.Object, System.EventArgs)",
			"innerException": {
				"message": "Inner Exception 2",
				"stacktrace": "   at Company.Default.Method1(Object sender, EventArgs e) in C:\\Source\\Company\\src\\services\\Company\\Default.aspx.cs:line 20\r\n   at System.EventHandler.Invoke(Object sender, EventArgs e)\r\n   at System.Web.UI.Control.OnLoad(EventArgs e)\r\n   at System.Web.UI.Control.LoadRecursive()\r\n   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)",
				"type": "System.InvalidOperationException",
				"method": "Void Method1(System.Object, System.EventArgs)"
			}
		}
	}
}

and maybe using something like this?

          <attribute name="exception" encode="false">
            <layout type="JsonNestedExceptionLayout" source="${exception}" maxInnerExceptionLevel="5">
                  <attribute name="message" layout="${exception:format=message}" />
                  <attribute name="stacktrace" layout="${exception:format=stacktrace}" />
                  <attribute name="data" layout="${exception:format=data}" />
                  <attribute name="type" layout="${exception:format=type}" />
                  <attribute name="method" layout="${exception:format=method}" />
                  <attribute name="innerException" layout="${exception:innerexception}" />
            </layout>
          </attribute>

It would be nice if we could implement this without a fully specialised layout (JsonNestedExceptionLayout isn't generic)

Sounds good.  I'm only new to using this project so could you give an example of how you think it could be achieved?

Well I don't know a good way to configure this generic.

Maybe for inspiration:
https://libraries.io/github/justeat/NLog.StructuredLogging.Json

Not the most elegant solution, but here goes:
        <layout type='JsonLayout'>
          <attribute name='logger' layout='${logger}' />
          <attribute name='message' layout='${message}' />
          <attribute name='exceptions' encode='false'>
            <layout type='CompoundLayout'>
                <layout type='SimpleLayout' text='{' />
                <layout type='JsonLayout' SuppressCurlyBraces='true'>
                  <attribute name='exception' encode='false'>
                    <layout type='JsonLayout'>
                      <attribute name='type' layout='${exception:format=type}' />
                      <attribute name='message' layout='${exception:format=message}' />
                    </layout>
                  </attribute>
                </layout>
                <layout type='InnerExceptions'>
                    <layout type='SimpleLayout' text=',' />
                    <layout type='JsonLayout' SuppressCurlyBraces='true'>
                      <attribute name='exception' encode='false'>
                        <layout type='JsonLayout'>
                          <attribute name='type' layout='${exception:format=type}' />
                          <attribute name='message' layout='${exception:format=message}' />
                        </layout>
                      </attribute>
                    </layout>
                </layout>
                <layout type='SimpleLayout' text='}' />
            </layout>
          </attribute>
        </layout>

This give the following output:
{ "logger": "A",
 "message": "msg",
 "exceptions": {
"exception": { "type": "System.AggregateException", "message": "One or more errors occurred." },
"exception": { "type": "System.Exception", "message": "Test exception 1" },
"exception": { "type": "System.Exception", "message": "Test Inner 1" },
"exception": { "type": "System.Exception", "message": "Test exception 2" },
"exception": { "type": "System.Exception", "message": "Test Inner 2" }
} }

Have added new property SuppressCurlyBraces to JsonLayout.
Have created new Layout called InnerExceptions (Inherits from CompoundLayout). It creates a artificial LogEventInfo that is a clone of the original LogEventInfo. It then loops through the InnerExceptions and renders the artificial LogEventInfo with each individual exception injected.

The nested inner-exception output is now supported with #2357. It just uses reflection to output all Exception-properties (so no control on the key-names)
One can override the NLog-IJsonConverter and provide a custom formatter for exception serialization.
But one might consider to have an NLog-Json-Option that specifies that all json-keys should be lowercase.

There is one problem with the current reflection-based solution - it doesn't include the name of the exception type. Currently I'm including the name of the top-level type separately, but then inner exception types are not recorded.
A generalisation of this problem would be the ability to easily add a object's type name as one of the keys in the serialised output. I could use a custom IJsonConverter for this, but it looks like the converter - and the IValueFormatter which provides it - is fixed.
What would be the best way to go about this? Either as configuration in the dependent project, or an NLog PR?

@tobz1000 I was planning to go down in the machine room after #2670 has been merged. Allowing the new XML-Layout to also benefit from the reflection logic. In that operation then I will add explicit Type-property for Exception-objects and also allow easier override of object-serialization.

@tobz1000 Created #2989 that includes Type as implicit property for Exception-objects

You can find Prerelease Nuget package here:
https://ci.appveyor.com/project/nlog/nlog/builds/20044070/artifacts
Sent from my Sony Xperia
---- Toby Dimmick wrote ----
Great! I'll give it a go when I can.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/1311#issuecomment-435825705, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnKhQWrIuOKCsbq7QAn0XLUp8Uv6qks5usBIhgaJpZM4HzdUE.

#2989 has been merged, so NLog 4.6


Masking zeros not supported in some layers

Hi,
I am trying to implement a model over zero-padded sequences. The problem is when I use mask_zero=True some layers do not support it. For example, in the following code, the Dense layer throws an error that says it does not support masking:
# Mean over time implementation
def MeanOverTime():
    layer = Lambda(lambda x: K.mean(x, axis=1), output_shape=lambda s: (s[0], s[2]))
    return layer

model.add(Embedding(vocab_size, emb_dim, mask_zero=True))
model.add(LSTM(lstm_dim, return_sequences=True))
model.add(MeanOverTime())
model.add(Dense(10))
model.add(Activation('softmax'))

Is there an easy way to fix this? Thanks.
Kaveh

I changed the MeanOverTime() implementation and it seems to be working:
def MeanOverTime():
	layer = Lambda(lambda x: K.mean(x, axis=1), output_shape=lambda s: (s[0], s[2]))
	layer.supports_masking = True
	def compute_mask(input, mask):
		return None
	layer.compute_mask = compute_mask
	return layer

Can you confirm that this fix is OK?

Do you even need a mask past the LSTM?  You are squashing over time as it is.
Which, by the way, you aren't actually using your mask to do the mean, so the zero-padded items aren't being ignored.
My suggestion is to use a custom layer to implement MeanOverTime. I have this one which may work: https://github.com/braingineer/ikelos/blob/master/ikelos/layers/utility.py#L29
Basically, it lets you pass in a function to modify the mask rather than the layer. It's quick and dirty and a bit hacky, but it works in a pinch.
However, in thinking about your problem, you can't just use this. I would take the LambdaMask layer, modify it so that it in call it outputs
if mask is not None:
    if K.ndim(mask) == K.ndim(x) - 1:
        mask = K.expand_dims(mask)
    x *= mask
return x

this way, your mask is being given a broadcastable dimension so it will be able to multiply across your feature dimension. it is also correctly 0ing out the time values you don't care about.
in the compute mask portion, you will want to just return None.  this gets rid of the mask in the pipeline.  Now, your Dense can properly do its job without having to worry.

Yeah, I don't need the mask after MeanOverTime layer. I just did not know how to remove mask after MeanOverTime. I implemented this really really ugly layer instead (it works though):
def MeanOverTime():
	mean_func = lambda x: K.cast((x.sum(axis=1) / (x.shape[1] - K.equal(x, 0).all(axis=2).sum(axis=1, keepdims=True))), K.floatx())
	layer = Lambda(mean_func, output_shape=lambda s: (s[0], s[2]))
	layer.supports_masking = True
	def compute_mask(input, mask):
		return None
	layer.compute_mask = compute_mask
	return layer

I also did not know how to access the mask inside this function. That's why I am using K.equal(...).all(...) hack. I am gonna fix it now. Thanks!

#1579 issue seems to have already tackled your problem in an elegant way.
You could use the following function for mean over time.
def lambda_mask_average(x,mask=None):
    return K.batch_dot(x,mask,axes=1) / K.sum(mask, axis=-1, keepdims=True)

main_input = Input(shape=(input_length,),dtype='int32')
m = Embedding(vocab_size+1, emb_size, input_length=input_length, mask_zero=True)(main_input)
m = LSTM(lstm_dim, return_sequences=True)(m)
m = MaskEatingLambda(lambda_mask_average, output_shape=(lstm_dim,))(m)
# no more mask layer.
# insert whatever other layers you want here
model = Model(input=main_input, output=m)

Thanks @mpavankumarreddy. The problem is solved. I am closing the issue.

Hey @kavehtp,
I am using the same architecture from @sergeyf for a toy problem. I added a dense layer after the averaging layer as such
def lambda_mask_average(x,mask=None):
    return K.batch_dot(x,mask,axes=1) / K.sum(mask, axis=-1, keepdims=True)

def lambda_mask_sum(x,mask=None):
    return K.batch_dot(x,mask,axes=1)

main_input = Input(shape=(maxlen,), dtype='int32')
x = Embedding(max_features+1, embed_dim, input_length=maxlen, dropout=0.2, mask_zero=True)(main_input)
x = LSTM(lstm_dim, dropout_W=0.2, dropout_U=0.2, return_sequences=True)(x)
x = MaskEatingLambda(lambda_mask_average,output_shape=(lstm_dim,))(x)
pred = Dense(1,activation='sigmoid')(x)

However, Dense layer gives an error because the input type is upcasted by MaskEatingLambda layer to float64 and it expects float32. On the other hand, that problem does not occur if I use lambda_mask_sum function.
Have you come across with a similar problem in your implementation? Can you suggest a fix for it?
Thanks

You'll probably have to use K.cast on the sum over mask.
On Fri, Jun 17, 2016 at 9:05 AM, Caglayan Dicle<EMAIL_ADDRESS>wrote:

Hey @kavehtp https://github.com/kavehtp,
I am using the same architecture from @sergeyf
https://github.com/sergeyf for a toy problem. I added a dense layer
after the averaging layer as such
def lambda_mask_average(x,mask=None):
return K.batch_dot(x,mask,axes=1) / K.sum(mask, axis=-1, keepdims=True)
def lambda_mask_sum(x,mask=None):
return K.batch_dot(x,mask,axes=1)
main_input = Input(shape=(maxlen,), dtype='int32')
x = Embedding(max_features+1, embed_dim, input_length=maxlen, dropout=0.2, mask_zero=True)(main_input)
x = LSTM(lstm_dim, dropout_W=0.2, dropout_U=0.2, return_sequences=True)(x)
x = MaskEatingLambda(lambda_mask_average,output_shape=(lstm_dim,))(x)
pred = Dense(1,activation='sigmoid')(x)
However, Dense layer gives an error because the input type is upcasted by
MaskEatingLambda layer to float64 and it expects float32. On the other
hand, that problem does not occur if I use lambda_mask_sum function.
Have you come across with a similar problem in your implementation? Can
you suggest a fix for it?
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2728#issuecomment-226810579,
or mute the thread
https://github.com/notifications/unsubscribe/ABya7HXMtWMXFcKBjuhzpICz9D6UqU3gks5qMsW6gaJpZM4IfBxy
.

Hey @sergeyf. Thanks for quick reply.
I tried
return K.batch_dot(x,mask,axes=1) / K.cast_to_floatx(K.sum(mask, axis=-1, keepdims=True))

and got the error
ValueError: setting an array element with a sequence.

Now, I could not fix that one, mainly, due to the lack of my python knowledge, but I believe you guys can help me.

I use it like this: K.cast(x,'float32').
On Fri, Jun 17, 2016 at 10:48 AM, Caglayan Dicle<EMAIL_ADDRESS>wrote:

Hey @sergeyf https://github.com/sergeyf. Thanks for quick reply.
I tried
return K.batch_dot(x,mask,axes=1) / K.cast_to_floatx(K.sum(mask, axis=-1, keepdims=True))
and got the error
ValueError: setting an array element with a sequence.
Now, I could not fix that one, mainly, due to the lack of my python
knowledge, but I believe you guys can help me.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2728#issuecomment-226835691,
or mute the thread
https://github.com/notifications/unsubscribe/ABya7HDTBlZP9BeJxW9iKNrYKsGby7Vjks5qMt38gaJpZM4IfBxy
.

That solved the problem. Thanks man!

Has anyone checked if these work with the tensorflow backend? Trying the code by @mpavankumarreddy
I get this error:
/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.pyc in batch_dot(x, y, axes)
    247         adj_x = None
    248         adj_y = None
--> 249     out = tf.batch_matmul(x, y, adj_x=adj_x, adj_y=adj_y)
    250     if ndim(out) == 1:
    251         out = expand_dims(out, 1)

/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.pyc in _batch_mat_mul(x, y, adj_x, adj_y, name)
    387   """
    388   result = _op_def_lib.apply_op("BatchMatMul", x=x, y=y, adj_x=adj_x,
--> 389                                 adj_y=adj_y, name=name)
    390   return result
    391

/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.pyc in apply_op(self, op_type_name, name, **keywords)
    702           op = g.create_op(op_type_name, inputs, output_types, name=scope,
    703                            input_types=input_types, attrs=attr_protos,
--> 704                            op_def=op_def)
    705           outputs = op.outputs
    706           return _Restructure(ops.convert_n_to_tensor(outputs),

/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)
   2260                     original_op=self._default_original_op, op_def=op_def)
   2261     if compute_shapes:
-> 2262       set_shapes_for_outputs(ret)
   2263     self._add_op(ret)
   2264     self._record_op_seen_by_control_dependencies(ret)

/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in set_shapes_for_outputs(op)
   1700       raise RuntimeError("No shape function registered for standard op: %s"
   1701                          % op.type)
-> 1702   shapes = shape_func(op)
   1703   if shapes is None:
   1704     raise RuntimeError(

/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.pyc in _BatchMatMulShape(op)
   1383   if a_shape.dims is None and b_shape.dims is None:
   1384     return [tensor_shape.unknown_shape()]
-> 1385   batch_dims = a_shape[:-2].merge_with(b_shape[:-2])
   1386   output_rows = a_shape[-1] if adj_a else a_shape[-2]
   1387   output_cols = b_shape[-2] if adj_b else b_shape[-1]

/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.pyc in merge_with(self, other)
    568       except ValueError:
    569         raise ValueError("Shapes %s and %s are not compatible" %
--> 570                          (self, other))
    571
    572   def concatenate(self, other):

ValueError: Shapes (?,) and () are not compatible

Hi @braingineer , I'm new to keras and I want to process sentences with different number of words in CNN. I used zero-padding, but layers after Embedding layer doesn't support masking. Is there any way to solve it?
inputs = Input(shape=(1,max_len),dtype='int32')
x= Embedding(vocab_size, dim,weights = GloVe, input_length=max_len)(inputs)
x = Reshape((1,max_len,50))(x)
x = Convolution2D(nb_filter, n_gram, dim,init='glorot_uniform', activation='linear',border_mode='valid', subsample=(1,1))(x)
x = MaxPooling2D(pool_size=(2,1))(x1)
x = Flatten()(x)
out = Dense(10)(x)
Conv_sen= Model(inputs,out)
By the way, does keras support global pooling?
Thanks.

For anyone who stumbles onto this post looking to deal with Embeddings, zeros, and masks, the following works in both Theano and TF.
My solution to this problem is as follows:
(1) Make a custom ZeroMaskedEntries layer that (a) zeros out all of the masked-out embedding rows and (b) swallows the mask so it doesn't pass on.
(2) Use a lambda function called mask_aware_mean that knows to ignore all-zero rows when taking the mean.
This is a little bit silly (inefficient) because first I get rid of the mask, and then I reconstruct, but it gets rid of the whole MaskEatingLambda business. You can also use ZeroMaskedEntries in other places, and easily modify it to pass on the mask if need be.
Here is ZeroMaskedEntries:
import keras.backend as K
from keras.engine.topology import Layer

class ZeroMaskedEntries(Layer):
    """
    This layer is called after an Embedding layer.
    It zeros out all of the masked-out embeddings.
    It also swallows the mask without passing it on.
    You can change this to default pass-on behavior as follows:

    def compute_mask(self, x, mask=None):
        if not self.mask_zero:
            return None
        else:
            return K.not_equal(x, 0)
    """

    def __init__(self, **kwargs):
        self.support_mask = True
        super(ZeroMaskedEntries, self).__init__(**kwargs)

    def build(self, input_shape):
        self.output_dim = input_shape[1]
        self.repeat_dim = input_shape[2]

    def call(self, x, mask=None):
        mask = K.cast(mask, 'float32')
        mask = K.repeat(mask, self.repeat_dim)
        mask = K.permute_dimensions(mask, (0, 2, 1))
        return x * mask

    def compute_mask(self, input_shape, input_mask=None):
        return None

Below is a way to take the mean of what comes out of ZeroMaskedEntries. It does the silly business mentioned above of reconstructing the mask, but the computational hit is minor in my experience.
def mask_aware_mean(x):
    # recreate the masks - all zero rows have been masked
    mask = K.not_equal(K.sum(K.abs(x), axis=2, keepdims=True), 0)

    # number of that rows are not all zeros
    n = K.sum(K.cast(mask, 'float32'), axis=1, keepdims=False)

    # compute mask-aware mean of x
    x_mean = K.sum(x, axis=1, keepdims=False) / n

    return x_mean

And here is a test to make sure it all works:
import numpy as np
from keras.layers import Input, Embedding, Lambda
from keras.models import Model

main_input = Input(shape=(input_length,), dtype='int32')
embed = Embedding(output_dim=2, input_dim=25, input_length=4, mask_zero=True)(main_input)
embed_zeroed = ZeroMaskedEntries()(embed)
lambda_mean = Lambda(mask_aware_mean)(embed_zeroed)

model = Model(input=main_input,output=lambda_mean)
model.compile(optimizer='rmsprop',loss='mse')

# test
test_input = [[0,0,2,0],[0,0,0,1],[0,0,2,1]]
test_output =  model.predict(test_input)
print('Mean is working?', np.all(np.isclose(test_output[0:2,:].mean(0),test_output[2,:])))

@sergeyf Thank you very much for your solution above!
I know my solution below is wrong, but can you explain to me why it is incorrect in achieving the average embedding vector?
def means(x):
    return K.mean(x, axis=1)

model = Sequential()
model.add(Embedding(num_features+2, 128))
model.add(Lambda(means, output_shape=(128,)))
model.add(Masking(mask_value=0))
model.add(Dense(64, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

I'm not getting any errors like the OP but I assume it's still computing the average with the zeros, correct?
Thanks!

I think because Dense doesn't support masking.

By chance I currently have the same problem: I want to average the outputs of a TimeDistributed Dense layer across all non-masked timesteps. Is @kavehtp's MeanOverTime the best way to do it?

@iskandr I haven't tried my approach after a TimeDistributed anything, so not sure how it would work. If it does, please put up an example here for posterity!

Here is another (slightly cleaner?) alternative implementation:
class MeanOverTime(Layer):
	def __init__(self, **kwargs):
		self.supports_masking = True
		super(MeanOverTime, self).__init__(**kwargs)

	def call(self, x, mask=None):
		return K.cast(x.sum(axis=1) / mask.sum(axis=1, keepdims=True), K.floatx())

	def get_output_shape_for(self, input_shape):
		return (input_shape[0], input_shape[2])

	def compute_mask(self, x, mask):
		return None

	def get_config(self):
		config = {}
		base_config = super(MeanOverTime, self).get_config()
		return dict(list(base_config.items()))

@sergeyf Just a follow-up question... I am observing that using your solution, my model always predicts the most frequent class for each and every test example. My goal is to average the embeddings of variable-length vectors (which I padded with zeros) and to predict one of n_classes classes.
Here is my implementation using your solution:
model = Sequential()
model.add(Embedding(num_features+2, 128, mask_zero=True))
model.add(ZeroMaskedEntries())
model.add(Lambda(mask_aware_mean))
model.add(Dense(n_classes, activation='softmax'))

When I disable the mask_zero flag and remove the ZeroMaskedEntries layer, it seems to suddenly work (i.e., doesn't always predict the same class for every example) as follows:
model = Sequential()
model.add(Embedding(num_features+2, 128))
model.add(Lambda(mask_aware_mean))
model.add(Dense(n_classes, activation='softmax'))

Why could this phenomenon be happening and what may I be doing wrong? Thanks!

@Qululu I am not sure. The use case I had was the same as yours. I have a bunch of variable-length text, and I wanted to train a smart average of it, without getting messed up by the zero-index vector that comes out of the Embedding layer. I found a model that works better for my use-case (sorry for the extra classes etc):
from keras.engine import Layer

class NamedLambda(Lambda):
    def __init__(self, name=None):
        Lambda.__init__(self, self.fn, name=name)

    @classmethod
    def invoke(cls, args, **kw):
        return cls(**kw)(args)

    def __repr__(self):
        return '%s(%s)' % (self.__class__.__name__, self.name)

class L2Normalize(NamedLambda):
    def fn(self, x):
        return K.l2_normalize(x, axis=-1)

class Sum(NamedLambda):
    def fn(self, x):
        return K.sum(x, axis=1)

embed_direction = Embedding(output_dim=output_dim,
                                input_dim=input_dim,  mask_zero=True)
mask_direction = ZeroMaskedEntries()
embedding = mask_direction(embed_direction(main_input)
sum = Sum.invoke(embedding, name='the_sum')
l2_normed_sum = L2Normalize.invoke(sum, name='l2_sum')

Try that.  This will confirm that it's not ZeroMaskedEntries that's causing the problem.
(2) Alter ZeroMaskedEntries to just return x. This will confirm that it's not mask_aware_mean that is causing the problem.

@sergeyf Ok, I tried both debug suggestions. While (2) confirmed mask_aware_mean was indeed working, (1) led me to stumble upon an interesting phenomenon...
When I changed the mask_aware_mean implementation to ignore the mask as such:
def mask_aware_mean(x):
    n = K.sum(K.cast(x, 'float32'), axis=1, keepdims=False)
    x_mean = K.sum(x, axis=1, keepdims=False) / n
    return x_mean

then I observed the problem still occurring. However, if I used the following dumb averaging implementation:
def means(x)
    return K.mean(x, axis=1)

then it worked.
So what is the difference between the two implementations? Aren't they both equivalent?
Also, if this implies that ZeroMaskedEntries is causing the problem, then how can I debug it too? Thanks!

@Qululu This looks wrong: n = K.sum(K.cast(x, 'float32'), axis=1, keepdims=False)
n should be the number of rows. Originally it was n = K.sum(K.cast(mask, 'float32'), axis=1, keepdims=False), which makes sense since mask is binary. x is not binary, so you're not going to get the number of rows from your operation.

@sergeyf Hi Sergey, oops you're right. Here, I corrected it to the best of my ability:
def mask_aware_mean(x):
    # All values will meet the criterion >= 0
    mask = K.greater_equal(K.sum(K.abs(x), axis=2, keepdims=True), 0)
    n = K.sum(K.cast(mask, 'float32'), axis=1, keepdims=False)
    x_mean = K.sum(x, axis=1, keepdims=False) / n
    return x_mean

The above implementation computes the dumb average and indeed the phenomenon (model predicting most dominant class for every example) goes away. You suggested in (1) that this could mean ZeroMaskedEntries is causing the problem. Now I'm stuck, any help would be appreciated.
Not sure this is relevant, but I'm trying to compute the average embeddings for variable-length and non-sequential sets of one-hot encoded words. However, I still need to treat these sets as lists and pad them with zeros to be of uniform size for feeding into the Embedding layer, right?
from keras.preprocessing import sequence
embedding_layer_input = sequence.pad_sequences(np.array(word_idxs), maxlen=MAX_WORD_IDX_LEN)

Is this the correct way to handle embeddings of variable-length non-sequential tokens?

@Qululu Yes, your padding looks right assuming that word_idxs is something like [[1,2,3],[4,2,1],...]
I'm not sure what the problem is. I think you need to set up a test bed to carefully go through and figure out where things stop behaving the way you want with your particular dataset. It sounds like you haven't found any particular part of my code to be broken in any obvious way, so I don't know how to help you debug further.
Also, try my K.l2_normalize solution.
Sorry I can't be of more help. Please report anything you find though - I'm sure others will find it useful.

Hi, i have i little problem with masking. There are training examples in my datasets for which no masking is applied because their length is equal (or slightly less) than the input_length in the Embedding Layer. My problem is that using the following Layer:
class MeanOverTime(Layer):
    def __init__(self, **kwargs):
        self.supports_masking = True
        super(MeanOverTime, self).__init__(**kwargs)

    def call(self, x, mask=None):
        if mask is not None:
            return K.cast(x.sum(axis=1) / mask.sum(axis=1, keepdims=True), K.floatx())
        else:
            return K.mean(x, axis=1)

    def get_output_shape_for(self, input_shape):
        return input_shape[0], input_shape[-1]

    def compute_mask(self, input, input_mask=None):
        return None

there are case that this mask.sum(axis=1, keepdims=True) leads to division by zero. When this happens loss becomes nan.
I don't know how to work with tensors  and i need just to add this little check. In order to bypass this i have increased the input_length so it covers all my training examples. Also i tried adding a try/except but this also didn't work.

This is what i did. Hope it helps someone...
class MeanOverTime(Layer):
    def __init__(self, **kwargs):
        self.supports_masking = True
        super(MeanOverTime, self).__init__(**kwargs)

    def call(self, x, mask=None):
        if mask is not None:
            mask = K.cast(mask, 'float32')
            sum = mask.sum(axis=1, keepdims=True)
            if K.equal(sum, K.zeros_like(sum)):
                return K.mean(x, axis=1)
            else:
                return K.cast(x.sum(axis=1) / mask.sum(axis=1, keepdims=True), K.floatx())
        else:
            return K.mean(x, axis=1)

    def get_output_shape_for(self, input_shape):
        return input_shape[0], input_shape[-1]

    def compute_mask(self, input, input_mask=None):
        return None


CTRL D should not automatically query the API

🚀 feature request
Is your feature request related to a problem? Please describe.
The app is making too many API calls, and most are useless.
 There is no reason to have it begin querying every time a single affix is selected or modified, or before any are chosen. There needs to be a manual search button.
Describe alternatives you've considered
Additional context

Exactly, 1 more click won't be that nightmare and maybe will stop trigger ggg.

completely agree, while it sure is QOL, the OG trade macro did not do that as far as I am concerened. you had to set your stats and your ranges, and then you had to click a button to get values. especially when you are still adjusting the ranges for some values, it basically queries every time, I can 100% see this being at least part of the issue that was caused.
I would suggest the following:

if you have anything where there are no variables (currency, fragments, fossils, whatever), you can immediately query.
if there is anything which has variables (basically every equippable item with stats) it first opens, you have to set whatever stats and ranges, and then you have to press a button to do a query once. if you change anything, again it does not auto query, and you have to press a button manually to trigger the query again.

100% agree with the previous comments.

yeah, I also think that's one huge step the tool can make to reduce the amount of while we are not really loosing anything. Actually I think it's very unnecessary to directly sent the tradesearch-request since I always have to change some mods to get the actual search I want...

* if you have anything where there are no variables (currency, fragments, fossils, whatever), you can immediately query.

There is no need to query this, just get the price once an hour/day for all items from poe.ninja. They come in a nice json.  Example! The price of currency and fragments tends to be stable!

Some user like me prefer it this way, if you would like to make an additional command, thats cool.
But not a replacement.

Set the “Debounce Time” in the settings “Evaluate/Query” to infinity (move the slider to the right).

* if you have anything where there are no variables (currency, fragments, fossils, whatever), you can immediately query.

There is no need to query this, just get the price once an hour/day for all items from poe.ninja. They come in a nice json. Example! The price of currency and fragments tends to be stable!

disagree. even though they might not anymore, they can change very quickly and frequently, and especially so on league start, or maybe even during the league when a fancy video guide or clip to anything goes viral. I do not want the tool to show me outdated price. Especially if the currency quering hardly was the problem with poe overlay, it always was the problem that it almost instantly queries if you change a setting on the ranges. let it just work like the OG trade macro, change your settings, press a button manually to query once. and whatever you change, it does not query again unless you click the button again. Problem solved. usability goes down maybe like 1% but probability to get banned because of the tool goes down most likely by 100%.

Set the “Debounce Time” in the settings “Evaluate/Query” to infinity (move the slider to the right).

this should be a) the default setting and b) not changeable. whether or not someone wouldn't like it if it is not changeable doesn't matter, because it is a requirement on the tool so the servers are not flooded with queries.

0.6.26 (2020-04-29)

add initial search toggle - off by default (#672)


Textures used by SPIR-V shader in OpenGL are not recognized

Description
I'm working on porting my graphics engine to Vulkan and as an intermediate step I added SPIR-V shader support to the OpenGL renderer.  While debugging this I noticed that RenderDoc does not show the textures used by the SPIR-V shader in any way.  They're not in the input panel in the texture viewer nor in the fragment shader section of pipeline viewer.  The timeline shows no accesses to the texture.  Replay also does not work; the color output shows up as blank.
If I add a binding layout qualifier matching what the program uses to the sampler uniform in the shader, replay in RenderDoc does work, but the texture is still not recognized as used anywhere.
Steps to reproduce

Create a SPIR-V shader program using glShaderBinary and glSpecializeShader
Bind textures and other necessary things for the shader
Render a mesh

Capture of affected application: opengl_spirv.zip
opengl_spirv.rdc has only location layout qualifier on the sampler uniform (the compiler generates a binding of zero), opengl_spirv_binding.rdc additionally has a manually specified binding.
I can share the application if needed, but it requires some preparation so I'll wait for your word on whether you need it in addition to the capture.
Environment

RenderDoc version: v1.14 (cd5d0ede4)
Operating System: Debian unstable
Graphics API: OpenGL
GPU: GeForce 980Ti
Driver: Nvidia 460.56

FYI I'm not working for the next couple of weeks so I can look into this when I get back. Taking an extremely quick look I'd guess this is because you have two entry points named "main" in the SPIR-V module. I don't remember off hand if that's legal, try giving them unique names or having a SPIR-V module each.

The SPIR-V specification says "A module must not have two OpEntryPoint instructions with the same Execution Model and the same Name string." (in the description of the OpEntryPoint instruction).  To me this means that the combination of execution model and name must be unique, but there may be two identical names if the execution model is different.
I'm not sure how to generate a SPIR-V module consisting of multiple stages with different entry point names using existing tools.  glslangValidator seems very insistent that the entry point must be named "main".  I might try separate modules to see if that's the problem here, though I would like to use linked modules.  I'm also working on my own shader compiler which will give me more freedom about what goes into the SPIR-V.

My own compiler isn't ready to produce SPIR-V yet, so I compiled both stages separately with glslangValidator and then combined them with spirv-link.  But glslangValidator doesn't accept anything other than "main" for the source entry point and seems to ignore the -e option which I had assumed would rename the entry point put into SPIR-V.
I realized that it would be easy enough to tweak the SPIR-V binary with a hex editor, so I did a quick test by changing the entry point names and the associated OpName instructions to "mainv" and "mainf".  This indeed makes RenderDoc recognize that the fragment shader is using a texture in the pipeline state panel, however it's still shown as "no resource" and the inputs panel in the texture viewer shows nothing.  This could be a different bug; I don't know how usual it is to use glUniform1i to change sampler bindings for SPIR-V shaders, and I plan to discontinue doing that as I develop my engine further.
Here's a capture for this: opengl_spirv2.zip
Workaround is how I interpreted your suggestion.  I'll try it if I come across a situation where the need for debugging exceeds the effort of making the necessary code changes.

Ah right I wasn't aware of spirv-link, that makes sense.
I'm not sure it's valid to set textures in SPIR-V shaders with glUniform, I think the expectation is that only new drivers would support SPIR-V so you can specify the binding directly in the shader rather than using legacy methods. From the GL extension spec:

How do we change the location of samplers and images?
RESOLVED. You don't. Well you can using Uniform1i as usual, but you have
no way of knowing which location corresponds to which sampler or image
variable as GetUniformLocation won't work without a named variable. Best
to just treat them as immutable bindings which are specified in the
shader source or binary.

Though you can totally specify a location for a SPIR-V variable even if you can't query it, so that's not entirely accurate. It seems like it wasn't intended to work, whether or not there's a way to do it.

That commit should fix the handling of entry points as (name, stage) pairs instead of just looking them up by name. I also put in a second commit which will ignore the Location decoration if a Binding exists (i.e. for texture/buffer type uniforms), so that Location is only used for bare value uniforms as I believe is the intention of the extension.


Support for MS Edge is broken

MS Edge crashes when clearing cookies:
https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/5751773/
This always happens on browser start in prepare_browser:
https://github.com/pytest-dev/pytest-splinter/blob/c4cf38683ee8eff0a6cecc0fb21c118d5cf1d34a/pytest_splinter/plugin.py#L411
The infinite retry behavior in prepare_browser (see #71, #72, #73) hides this error and restarts the browser infinitely.

From what I understand the cookie cleaning is somewhat broken in another way too:

In session scoped browsers cookies is cleared at browser startup (when the cookies is already clean anyways) but since this happens in prepare_browser, it won't be called between tests.

In function scoped browsers cookie cleanup happens, but it is not very useful since a clean browser is constructed for each test anyways.

My preferred solution would be to just avoid cookie cleaning on function-scoped browsers altogether even if splinter_clean_cookies_urls is configured, fix splinter_clean_cookies_urls to be called before each test with session scoped browsers and change the default of splinter_clean_cookies_urls to ['about:blank'] to preserve the existing behaviour. Users of MS Edge+session scoped browsers just have to set splinter_clean_cookies_urls to a real url or [] until the issue is fixed in edge/selenium.
(FYI I cannot use session scoped browsers at all since we store a bunch of stuff in localstorage which cannot be reliably cleared via webdriver APIs)
@bubenkoff thoughts?

For session scoped browser prepare browser is Always called per test
On Nov 5, 2016 11:50 AM, "Andreas Pelme"<EMAIL_ADDRESS>wrote:

From what I understand the cookie cleaning is somewhat broken in another
way too:

In session scoped browsers cookies is cleared at browser startup (when
the cookies is already clean anyways) but since this happens in
prepare_browser, it won't be called between tests.
In function scoped browsers cookie cleanup happens, but it is not very
useful since a clean browser is constructed for each test anyways.
My preferred solution would be to just avoid cookie cleaning on
function-scoped browsers altogether even if splinter_clean_cookies_urls
is configured, fix splinter_clean_cookies_urls to be called before each
test with session scoped browsers and change the default of
splinter_clean_cookies_urls to ['about:blank'] to preserve the existing
behaviour. Users of MS Edge+session scoped browsers just have to set
splinter_clean_cookies_urls to a real url or [] until the issue is fixed
in edge/selenium.
(FYI I cannot use session scoped browsers at all since we store a bunch of
stuff in localstorage which cannot be reliably cleared via webdriver APIs)
@bubenkoff https://github.com/bubenkoff thoughts?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pytest-dev/pytest-splinter/issues/81#issuecomment-258604130,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAaEgGwjgBZ57U7TYIf5L5-pKRphEJo2ks5q7F9egaJpZM4KmS3Q
.

@pelme, please try related pull request and see if it solves both issues (retry and cookies)


Added providers modal

Added modal to add providers. Although we were not able to use the PF3 wizard from frontigrade, we used the PF4 modal. This will be replaced when the PF4 wizard is available.
The modal is designed in a way to be opened from any page. We're currently only opening this from the dashboard, but believe we may launch from the cost details as well.
When the user logs in, the app fetches a list of providers which is saved via the Redux store. If there are no providers, the dashboard is replaced with a simple empty state asking the user to add an account.
When the user launches the providers modal, they are presented with 3 input fields; provider name, a bucket name, and the provider resource name.
I've added some basic input validation; for example, to check if the bucket name has special characters, spaces, and is less than 255 chars. I also check the provider resource name to ensure  it begins with arn:aws:.
Empty state:

Providers Modal:

Modal with error checking:

Deploy preview for koku ready!
Built with commit 4210786bb0acab751ec609c6c197a1ddcf3686c5
https://deploy-preview-97--koku.netlify.com

Deploy preview for project-koku ready!
Built with commit 4210786bb0acab751ec609c6c197a1ddcf3686c5
https://deploy-preview-97--project-koku.netlify.com

Codecov Report

Merging #97 into master will decrease coverage by 0.71%.
The diff coverage is 79.16%.

@@            Coverage Diff             @@
##           master      #97      +/-   ##
==========================================
- Coverage   84.46%   83.75%   -0.72%
==========================================
  Files         103      110       +7
  Lines         998     1139     +141
  Branches      107      129      +22
==========================================
+ Hits          843      954     +111
- Misses        145      172      +27
- Partials       10       13       +3

Impacted Files
Coverage Δ

src/testIds.ts
100% <ø> (ø)
:arrow_up:

src/store/ui/uiReducer.ts
100% <100%> (ø)
:arrow_up:

src/app.tsx
96.55% <100%> (+0.39%)
:arrow_up:

src/store/ui/uiSelectors.ts
100% <100%> (ø)
:arrow_up:

src/store/rootReducer.ts
100% <100%> (ø)
:arrow_up:

src/pages/providersModal/providersModal.styles.ts
100% <100%> (ø)

src/store/ui/uiActions.ts
100% <100%> (ø)
:arrow_up:

src/store/providers/index.ts
100% <100%> (ø)

src/api/providers.ts
60% <60%> (ø)

src/store/providers/providersSelectors.ts
66.66% <66.66%> (ø)

... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7282e3e...9893d44. Read the comment docs.


Append to single output file (netCDF/zarr)

In a data cube setting we don't need a tile directory, but one single file that holds all data. netCDF and zarr can be subset to retrieve only the data that's needed.
Current situation
A tile directory with many zarrs is output (one per row-column combination)
Expected situation
mapchete appends data to one single file.
Possible difficulties
mapchete must deal with file locks if multiple processed want to write to the file (netCDF). Not an issue with zarr, as it is designed as a data sink: https://zarr.readthedocs.io/en/stable/tutorial.html#parallel-computing-and-synchronization

Thanks @StefanBrand!
What do you mean by "append"? That subsequent processes update the datacube?
The single file output should be generally feasible for zarr. I would bin netCDF (maybe for the whole extension) anyway because I don't really see a future advantage over zarr.

What do you mean by "append"? That subsequent processes update the datacube?

From the docs:

Write to an array concurrently from multiple threads or processes.
[...]
By data sink we mean that multiple concurrent write operations may occur, with each writer updating a different region of the array.
[...]
If each worker in a parallel computation is writing to a separate region of the array, and if region boundaries are perfectly aligned with chunk boundaries, then no synchronization is required. However, if region and chunk boundaries are not perfectly aligned, then synchronization is required to avoid two workers attempting to modify the same chunk at the same time, which could result in data loss.

So chunking should be set to the tile size.

A, I understand. Yes, this is feasible. We can limit the pixelbuffer to 0 in order to assure tile borders are aligned with chunk boundaries.
I just was not sure whether you meant that you want to run multiple execute commands on the same output and mapchete should "append" outputs to the zarr archive. This would have been more complicated.

Follow-up: netCDF is really not the best idea for using it as an object storage...

Zarr allows appending time data or ranges of time data without overwriting the dataset. NetCDF-4 allows this as well but the underlying object store would require overwriting the entire dataset.

Source: Yee, C., Durbin, C., Quinn, P., & Shum, D. (2020). Task 51—Cloud-Optimized Format Study (Technical Paper No. EED2-TP-125, Rev. 01). Raytheon Company. http://wiki.esipfed.org/images/3/38/EED2-TP-125_Rev01_CloudOptimizeStudy.pdf


feat: integrated magic link handling/processing

Issue #: /https://github.com/bcgov/entity/issues/23357

Allowed the handling of magic link

Successful:

Business already been added:

Expired link (clicked on link after 15 minutes):

~Note: I will add unit tests in another PR.~ Added the unit tests here.

/gcbrun

Temporary Url for review: https://business-registry-dev--pr-29-fi1utig1.web.app

/gcbrun

Temporary Url for review: https://business-registry-dev--pr-29-fi1utig1.web.app

If you are talking to Auth a lot, consider just using $authApi It is in the common-layer https://github.com/bcgov/sbc-connect-common/blob/main/nuxt/layers/core/app/plugins/auth-api.ts and as example used here https://github.com/bcgov/sbc-connect-common/blob/main/nuxt/layers/core/app/stores/connect-account.ts#L53 with some error sugar here https://github.com/bcgov/sbc-connect-common/blob/main/nuxt/layers/core/app/utils/logFetchError.ts

Thanks for pointing this out Thor. Yes, I removed my function and now I'm using authApi from the common-layer. Much cleaner now.

I just noticed, unexpectedly, that the new BRD is slower (by several seconds) than the old MBR. Thoughts?
Also, I noticed that the account number is not part of the route. Will it be, eventually?

I don't know if this was intentional, but as a developer, I like seeing the Temp Reg Number.

I just noticed, unexpectedly, that the new BRD is slower (by several seconds) than the old MBR. Thoughts?
Also, I noticed that the account number is not part of the route. Will it be, eventually?

That's very true. Even in https://business-registry-dev.web.app/en-CA and when running locally, it's slower. I've noticed this since I started working in this repo. Not quite sure, this'll probably need a deep dive.
As for the account number, I don't think so. I didn't see any tickets to include that.

/gcbrun

Temporary Url for review: https://business-registry-dev--pr-29-gba5iq6k.web.app


LUCENE-10005: Improve AlreadyClosedException logging

Description
AlreadyClosedException logging is misleading.
Solution
Improve logging.
Tests
Tested by backporting to Lucene 8.4.1
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[x] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[x] I have added tests for my changes.

Lucene doesn't do any logging. If you want to improve the logging, please open an issue in SOLR.
I don't think we should do this. However we format the date, it may not be in the correct format / timezone / etc to be of any use. In fact, it may just cause more confusion to see a GMT-preformatted date when the rest of my dates in logs are in localtime.
Also keep in mind "tragic exception" in lucene is very delicate handling. We shouldn't be allocating memory, formatting dates, stuff like this. It is supposed to be only for handling things like virtual machine error such as OutOfMemoryError!
I don't know why your solr HDFS example becomes "tragic", perhaps Solr is abusing this mechanism? Again, that is a problem to fix over there.

Thanks @asalamon74 -- looks like we shouldn't fix this in Lucene, but instead Solr.


Pin Multibag and node_package

LGTM +1 a00b331

Thanks, @russelldb . But node_package 2.0.2 is now weird, so I removed it.

+1 93c986f
I KNOW!
But this looks OK too

Come on, @borshop

Build #989 successfully finished. Merging.


promise

Need Promise snippet like this:
prom →
return new Promise((resolve, reject) => {
});

Resolved with the latest publish version 1.2.2


Account modification error on ammount

The amount rises a error when trying to remove special characters

cant replicate..

when the amount comes like "3000" looks like it takes it as number not string - which is ok in this case


⚠ GitLab Server has degraded performance

In f55d283, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 6753 ms

Resolved: GitLab Server performance has improved in 17c4d9f.

Resolved: GitLab Server performance has improved in 17c4d9f.


Bulk edit.

Hi this is great app.
Can we Make the app handle bulk editing?
Thx

It is already

It is already

How it can even work? For these purposes you can use crop resize mode, or fit to strip images to given dimensions or aspect ratio


Running BFCL data on Cohere

I am trying to run the BFCL on cohere's command+r model. While running everything in pipeline is a later task. For now, I am trying to search the function - evaluators.
 If the probability of rolling a 6 is 0.6, what is the probability that the player will roll a 6 exactly 5 times?",     "function":     {         "name": "calc_binomial_probability",         "description": "Calculates the probability of getting k successes in n trials.",         "parameters":         {             "type": "dict",             "properties":             {                 "n":                 {                     "type": "integer",                     "description": "The number of trials."                 },                 "k":                 {                     "type": "integer",                     "description": "The number of successes."                 },                 "p":                 {                     "type": "float",                     "description": "The probability of success."                 }             },             "required":             [                 "n",                 "k",                 "p"             ]         }     },     "execution_result":     [         0.0012944935222876579     ],     "execution_result_type": "exact_match" }
Where can I find the executor-function for the above function?.

Hi there,
All the executable functions are in  berkeley-function-call-leaderboard/eval_checker/executable_python_function.py. For example, the one you mentioned above is at https://github.com/ShishirPatil/gorilla/blob/main/berkeley-function-call-leaderboard/eval_checker/executable_python_function.py#L189-L201


ModSecurity (v3) Sending logs to syslog server

I built ModSecurity using the following tutorial:
https://www.netnea.com/cms/nginx-modsecurity-tutorials/
I searched a lot about sending ModSecurity v3 logs to a Syslog server. Using pipe is not supported and I don't know how to send my logs. My Syslog server is not an HTTP server. It just listened on a specific port (514). I don't know how to compile and use mlogc according to the tutorial: https://www.netnea.com/cms/nginx-modsecurity-tutorials/
Please help me.

Tnx a lot @martinhsv for the comment


Add sound file upload

Is your feature request related to a problem? Please describe.
I'd like to add sound files to observations, in the same way that is supported for photos
Describe the solution you'd like
An analogue of add_photo_to_observation

Sure, we can add that... as soon as I can figure out how. Strangely I don't see and endpoint for observation sounds listed anywhere in the API docs, but there must be one somewhere. I'll look into that some more and get back to you.

Okay, I found an undocumented POST /observation_sounds endpoint. I should be able to get that added soon. I can also add a sounds parameter to upload sound files with create_observation() and update_observation().

This is great. I'm working on a tool for automating the uploaded of bat recordings. This could make it easier for bat workers to share recording for use in classifiers.
FYI I couldn't get the local image argument in the createobservation function to work, had to use the function to add the image afterwards.

@JWCook I see your merge. Is this functionality ready to use?

Changes are in the latest pre-release build, which you can get with:
pip install pyinaturalist==0.14.0dev343

Pre-releases are more likely to contain bugs, but please let me know if you run into any issues. Notes on changes are in the changelog. I also just fixed the issue with image uploads (#150). So now you can do:
from pyinaturalist import create_observation

create_observation(
    ...,
    photos='photo.jpg',
    sounds='sound.mp3',
)

Or upload multiple files at once:
create_observation(
    ...,
    photos=['photo1.jpg', `photo2.png`],
    sounds=['sound1.mp3', sound2.wav'],
)

Same goes for update_observation().

Your tool for working with bat recordings sounds interesting. Do you plan to use audio metadata or some other data source to automatically fill out observation details (time, location, species guess, etc.)? Just curious because I've dabbled with something similar for observation photos + XMP metadata.

Boom! That's great. I'll test it out this evening. Your on the money. I use a bat detector which produces wav files packed with metadata (echometer touch). Lots of other people use the same detector. I'm building a pipeline so that I can point to a directory with lots of recordings in it and it can churn away sending them up to iNat. Obviously having the audio files up there is key for verification and for future research use, so thanks for the rapid updates! There is also metadata on acoustic trigger frequency and stuff. In guessing iNat Doesn't have field for that so I'm putting it into the description for now, along with software and hardware data. Does that seem the right way to go?

That sounds like a good reason to use observation fields. In practice, there's no standardization for how those are used, and many of the fields out there are project-specific, but they're still fairly useful. You can define your own, but it would also be worth checking to see if any fields already exist for the metadata you want to add. For example, I found some existing fields for echolocation frequency and bat detector model.
Pyinaturalist has basic support for those in create_observation():
create_observation(
    access_token=token,
    species_guess='Pieris rapae',
    photos='~/observation_photos/2020_09_01_14003156.jpg',
    observation_fields={297: 1},
)

In that example, 297 is the observation field ID for Number of individuals.
And here's a quick example to see what those look like in the response JSON:
>>> obs = get_observation(74566731)
>>> obs_field_summary = [f"{field['name']}: {field['value']}" for field in obs['ofvs']]
>>> print('\n'.join(obs_field_summary))
Habitat (s Afr): Dwellings
SA Veg: Grassveld Types: Northern Free State Shrubland Gm 7
Body mass:
Forearm length: 31.25
Bat detector model:
Field number:
Photographer:
Age: adult
Echolocation call frequency:
Original observer:
Habitat: Indoors
Sampling rate:
Sex: unknown
Recording method:
Reproduction (mammals):
Count: 2
Museum number:

Oh boy this is great. I can certainly use some of those existing fields!

So I'm running the new version using sounds and photos. Its not complaining but the sound files are not ending up on iNat. Here is the response I get (I'm calling python via R because that my home turf!)
[[1]]
[[1]]$id
[1] 81068458

[[1]]$observed_on
[1] "2021-05-26"

[[1]]$description
[1] "Recorded on Echo Meter Touch 2 Pro Android App 2.8.3 \n Recorder settings\n [{\"rate\":\"256000\",\"gain\":\"0.00\",\"trig level\":\"0.00\",\"trig max len\":\"0.00\",\"trig window\":\"0.00\",\"trig min freq\":\"0.00\",\"trig max freq\":\"0.00\",\"prefix\":null}]"

[[1]]$latitude
[1] "51.60157821"

[[1]]$longitude
[1] "-1.11353538"

[[1]]$map_scale
NULL

[[1]]$timeframe
NULL

[[1]]$species_guess
[1] "Pipistrellus pipistrellus"

[[1]]$user_id
[1] 1866606

[[1]]$taxon_id
[1] 40364

[[1]]$created_at
[1] "2021-05-31T18:38:48.970Z"

[[1]]$updated_at
[1] "2021-05-31T18:38:49.023Z"

[[1]]$place_guess
[1] "Oxfordshire, England, GB"

[[1]]$id_please
[1] FALSE

[[1]]$observed_on_string
[1] "2021-05-26"

[[1]]$iconic_taxon_id
[1] 40151

[[1]]$num_identification_agreements
[1] 0

[[1]]$num_identification_disagreements
[1] 0

[[1]]$time_observed_at
NULL

[[1]]$time_zone
[1] "UTC"

[[1]]$location_is_exact
[1] FALSE

[[1]]$delta
[1] FALSE

[[1]]$positional_accuracy
NULL

[[1]]$private_latitude
NULL

[[1]]$private_longitude
NULL

[[1]]$geoprivacy
NULL

[[1]]$quality_grade
[1] "casual"

[[1]]$positioning_method
NULL

[[1]]$positioning_device
NULL

[[1]]$out_of_range
NULL

[[1]]$license
NULL

[[1]]$uri
NULL

[[1]]$observation_photos_count
[1] 0

[[1]]$comments_count
[1] 0

[[1]]$zic_time_zone
[1] "Etc/UTC"

[[1]]$oauth_application_id
[1] 612

[[1]]$observation_sounds_count
[1] 0

[[1]]$identifications_count
[1] 1

[[1]]$captive
[1] FALSE

[[1]]$community_taxon_id
NULL

[[1]]$site_id
[1] 1

[[1]]$old_uuid
NULL

[[1]]$public_positional_accuracy
NULL

[[1]]$mappable
[1] TRUE

[[1]]$cached_votes_total
[1] 0

[[1]]$last_indexed_at
NULL

[[1]]$private_place_guess
NULL

[[1]]$uuid
[1] "693f6a9d-ee30-476b-bdf7-6a23f54e36c4"

[[1]]$taxon_geoprivacy
[1] "open"

[[1]]$user_login
[1] "tomaugust"

[[1]]$iconic_taxon_name
[1] "Mammalia"

[[1]]$project_observations
list()

[[1]]$created_at_utc
[1] "2021-05-31T18:38:48.970Z"

[[1]]$updated_at_utc
[1] "2021-05-31T18:38:49.023Z"

[[1]]$time_observed_at_utc
NULL

[[1]]$faves_count
[1] 0

[[1]]$owners_identification_from_vision
[1] FALSE

The path to the sound file I'm passing is data/PIPPIP_20210526_221147.wav. The file is 3.4MB

Hmm, I had assumed POST /observation_sounds would work the same as POST /observation_photos, but maybe the request body format is different. I'll look into it.

Found it. POST /observation_photos accepts the observation ID in either the request body or request params, while POST /observation_sounds only accepts it in request params. I'll get that fixed shortly.

Great work @JWCook ! Let me know once it is ready to test

Done. Give the latest build a try (0.14.0dev356).

Works a charm

This would be a different issue, but as you are talking about sounds you might already have done things like this.
Figure out you already have sound files uploaded to some other site like https://www.xeno-canto.org/
(you could also have them in some other website, like your own blog, or a Google Drive folder)
What if you want now to submit those observations (and files) to iNaturalist?
Do you know if it's possible to upload files from a remote url, instead doing it from your local drive?
From what I see in the iNaturalist web interface, it looks like some of these options are not working right now.  But I am not meaning linking to soundcloud or google stuff accounts.
I mean using public file urls which are perfectly available without any authentication process.
Perhaps you know if iNaturalist api provides this option somewhere or not (and if it is also available in pyinaturalist)?
Of course a python script would make this migration quite easy, but I would prefer to avoid download/reupload band usage.
Thanks

@abubelinha I just checked, and unfortunately the API doesn't provide a way to upload from a publicly accessible URL. It only accepts multipart POST data, so one way or another it will require using download and upload bandwidth.
You could consider making a feature request either on the Feature Requests forum section, or on the iNaturalistAPI repo.
For pyinaturalist, we could make this a little more convenient by:

Accepting URLs as arguments for pyinaturalist.upload(), the same way you can currently pass file paths
Use streaming requests to download and upload the file in chunks; this would use the same amount of bandwidth, but would be faster than downloading the whole file before uploading.

If the volume of sound files you want to upload is so large that it would be too expensive or slow for you (depending on your ISP's bandwidth limits), you could potentially run this from some free or cheap cloud computing resources like Heroku or AWS Lambda instead of running it on your own computer.
The free tier of Lambda includes up to 1,000,000 requests and 400,000 GB-seconds per month. That's based on memory usage, for example if you used 4GB of RAM you could run it for about 28 hours per month before getting charged.

Thanks a lot @JWCook
In my case it's more like this:

I am testing this in my computer and with my data, but the final idea is using this script in a university server (with limited space, and I guess also limited band usage)
The script would read remote observations data (google sheet) and sound/image urls (mostly google drive files) and try to process them as observations into iNaturalist. Each user of the script uses a different Google account and shares his/her Gsheet with me. Their Gdrive folder should contain only public links.
Still not sure if we will use a shared iNaturalist account for the project, or different ones. But that's not the point. The point is I don't know in advance how much stuff there would be inside each user's Gdrive folder.

I am not that concerned about the speed of the process. I am just worried about hitting the shared server storage/band usage limits.
The "chunks" you mention would be temporary files, or would be them be just in memory until the whole file is uploaded to iNaturalist? (so we can control how much memory we want to use for this process)


[Feature] Routine load process with a exist group id

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
routine load now not support modify columns, so when my table schema changed, we must stop current routine load job and submit a new one. but doris will reset offset to lastest or earilest by property.kafka_default_offsets setting, that may cause data repeat or data lost. the only way we avoid this problem is appoint kafka_partitions and kafka_offsets settings, and this also bring new problems:

the configuration is difficult when we have a large number of partitions;
when we increase partiton in topic, we will loss data in the new partitions.

in summary， we need to consume kafka topic with a exist consumer group an do not reset the offset.
Use case
No response
Related issues
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Can we support this feature in future versions?

Sounds reasonable

Can we support this feature in future versions?

yep

Thanks for reporting this, I'd like to help support this feature.


typo

This seems to add a typo.

This seems to add a typo.

Yea, this PR is invalid. Can be closed.


fix: Added Dark Mode to Contact Form

Related Issue
Closes #157

Description
added dark mode in contact form ui and other minor ui bugs.

Screenshots (if applicable)

Original
Updated

Checklist

[ x] My code adheres to the established style guidelines of the project.
[x ] I have included comments in areas that may be difficult to understand.
[ x] My changes have not introduced any new warnings.
[x ] I have conducted a self-review of my code.

In dark mode the size of form is less than size of phone img.
Make them equivalent like in light mode

@Kritika30032002 actually i forgot to mention it in the PR .
i have done it intentionally, as the UI looks better this way.
If you want me to make it same , do let me know .

https://nicepage.com/html-code/preview/simple-contact-us-form-2882096?device=desktop
design like this


Don't show required error when loading a safe

What it solves
Resolves #2539
How this PR fixes it
Initializes the safeName variable with undefined so the required check is not made
How to test it
-Open Add existing safe
The required error should be ther

Looks good to me

Should it rather check if the field was touched?

@mikheevm I think that with this change we are restoring the previous behavior and it already does what you say. Once the input is touched is set to '' and so if the user leaves it empty the error is shown.


Move DB configure to central palce

Put harbor_db, clair_db configuration together.

Coverage decreased (-0.03%) to 59.759% when pulling beec66028dc9cfab51787b16c3399b5196b9b71f on yixingjia:reorderconfig into 3b36236dc0af9d9839ff2d4cc5f4d61cd85a6bf0 on vmware:master.


Unable to set OSGi configuration that is not hardcoded in swagger.rb, instead yields strange downcase error

https://github.com/shinesolutions/ruby_aem/blob/7e0405baafc889953cb2f182d1f589eba798355c/lib/ruby_aem/swagger.rb#L56
Error: /Stage[main]/Aemprovisioning::Aem::Configuration/Aem_config_property[author: My custom configuration]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: undefined method `downcase' for nil:NilClass (file: /vagrant/aempuppet/modules/aemprovisioning/manifests/aem/configuration.pp, line: 164)

Hi @henrykuijpers  Thanks for the ticket. I also opened ticket shinesolutions/puppet-aem-resources#87 to address this isuse.
PR #33 should fix this issue.

PR #32  adds the support to configure all available OSGI Configuration nodes in AEM by using the new API Client from https://github.com/shinesolutions/swagger-aem-osgi

@henrykuijpers I finally manage to update the ruby_aem code so it supports the configuration of all OSGI configuration nodes at /apps/system/config/{configNodeName}.
Am I right to assume that you are trying to use this function to configure other configuration nodes than the ones which are hardcoded ?
btw. the resource config_property is expecting that the configuration file at /apps/system/config/ already exists. If not than you have to create the osgi configuration node by either using the puppet module aem_node (https://github.com/shinesolutions/puppet-aem-resources/blob/master/manifests/author_publish_enable_ssl.pp#L14-L22) or by using the ruby_aem resource node(https://github.com/shinesolutions/ruby_aem/blob/master/lib/ruby_aem/resources/node.rb#L39).

Yes indeed! There's a lot more that we have to configure for our systems. :)
I.e. the load balancer SSL termination according to the documentation of Adobe.
The Apache Felix SSL filter (as this is also mentioned there and we were running into an issue where the author redirects are pointing towards http instead of https).
The OsgiManager password (for when the Adobe osgi admin code has not taken over yet, to prevent people from logging in with default passwords during that window, part of the security checklist as well).
But also configs of our custom services (as we see the puppet code as infrastructure code and hence some things like hostnames etc. should be part of the infrastructure code).

good news @henrykuijpers  I've tested the code changes successful and they are 100 % backaeds compatible.

@henrykuijpers Changes to support the configuraiton of all configuration node is now included in ruby_aem 3.6.0.
Let me know if you are still experiencing some issues.


[HxGrid] ContentNavigationMode.Custom to support custom Pager

@hakenr Can you clarify, how can we disable built-in pager? I am not able to get rid of it :)

@vaclavek Currently, it is quite hacky 🤞, it should be achievable by setting ContentNavigationMode to LoadMore and empty LoadMoreTemplate. We should probably add ContentNavigationMode.Custom.

Originally posted by @vaclavek in https://github.com/havit/Havit.Blazor/issues/279#issuecomment-1780121732

Also see #279 [HxGrid] Feature Request: set location of Pager


Create akrish4.md

@NishkarshRaj  , Is there anything wrong? , its showing 1 check failed.

@NishkarshRaj , Is there anything wrong? , its showing 1 check failed.

No, the GitHub Actions is not working properly.


Incorrect highlighting for object spread operator

@skyronic Got another one for you!
Babel syntax highlighter - correct

Vue syntax highlighter - incorrect

Hey @Herteby. From what I can make out, this is an issue with sublime's JavaScript package. You should possibly file an issue with sublimehq/Packages repo.

@skyronic Ah, that makes sense. And actually it's still in the proposal stage, which is probably why the default highlighter doesn't support it yet.
I figured out a solution though! Even though I had Babel set as the highlighter for .js files, it was using the default highlighter for js in .vue files. However, if you disable the JavaScript package, that forces it to switch to the Babel highlighter!


Bringing only relative data

I am a novice to Meteor.
I have CATEGORY and PRODUCTS collection.
When user selects a CATEGORY, I would like to bring only valid PRODUCTS list.
How can I do this with meteor-kitchen?
Your hint or any help will be greatly appreciated.
Thank you!

If I can have query of PRODUCTS based on CATEGORY field, it would be simple.
But so far, I had no luck.

I'm recently in trouble with that too. But seems like there is no support for that dependent data in meteor-kitchen yet.

@quangmach Really?
There must be a way. Hum..

@innocube there is two ways (maybe more). Now I'm typing from mobile phone, so I'll tel you a principle without code, later I can write example.

using url params

using session variable

using url params
in your page's route_params define param itemId? (ending with question marke means it's not mandatory). For dataview with master dataset define details_route to be this page route (the same page) and details_route_params this.__id`. So when use clicks on the master data item, will be redirected to thesame url but with item id in the url.
Now, in your details query, filter by this.params.itemId. That's it.
using session variable
master on click event set Session.set("itemId", clicked item id), and in details dataset query filter by Session.get("itemId")

@perak Wow.. So I have expected, there are ways to do although I am not sure if I can just do in meteor-kitchen. If you could show me a little bit of code, it would be much helpful for a novice in meteor like me.
Thank you so much!

Let me explain a little more:
I have CATEGORY selection and PRODUCT selection on one page.
When user select a category, PRODUCT needs to load corresponding choices.

@perak
I am getting CATEGORY ID when a client selects one. However I am still struggling how to retrieve PRODUCT LIST according to this CATEGRORY ID.
Since meteor-kitchen generated code is composed of XXcontroller.js, XX.html, and XX.js, I am not sure how to set to bring relevant data only according to client's previous selection on CATEGORY.
If you could  show me a code example, I would really appreciate it.
Thank you again for your good work!

@innocube This works for me but my case is everytime I change select input, it re-subscribe and publish with id filter on server side as I have lots of upper items to store in client database. This one I write for case of subscribing all and filter on client side upon change on select input.
I use Session as google, haven't test it yet so please try and feedback to me.
change selectinput_name to your one and copy this one to Page->Component-> Events where select input located (For my case is Insert form)
"change select[name='selectinput_name']": function (e,t){
var cat_id = $(e.target).val();
Session.set("cat_id", cat_id);
console.log(Session.get("cat_id"));
},

This one goes to Helpers section. Change helpers_name to your one. (I take cat_id example for your category id)
"helpers_name": function(){
    var cat_id = Session.get("cat_id");
    return Product.find({category_id: cat_id}, {});
},

This one goes to Template rendered code section (Set cat_id = 0 to make select input not display all Product list items)
if(!Session.get("cat_id")) Session.set("cat_id","0");

@quangmach thanks, your solution is OK, more correct one is filtering server side, and filtering should be done in the controller. If data is filtered in a controller then you can access filtered data in all events and helpers:
You need to define your product query like this:
{
  "name": "product",
  "filter": { "categoryId": ":categoryId" }
}

In page where you use "product" query, I believe you need to subscribe both to category and product, so your page or component looks something like this:
{
  "name": "your_page_name",
  "query_name": "your_page_main_query_name",
  "query_params: [],

  "related_queries": [

    {
      "name": "category",
      "params": []
    },

    {
      "name": "product",
      "params": [
        { "name": "categoryId", "value": "Session.get(\"cat_id\")" }
      ]
    }

  ]
}

Now, your form's event should look like @quangmach suggested:
{
  ...
  "events_code": "\"change select[name='selectinput_name']\": function (e,t) { Session.set(\"cat_id\", $(e.target).val()); }"
  ...
}

Resulting code should be:
in controller:
...
  Meteor.Subscribe("product", Session.get("cat_id"));
...
  product = Product.find({ categoryId: Session.get("cat_id") });

in server/publish:
...
  "product": function(categoryId) {
    return Product.find({ categoryId: categoryId });
  }
...

in your_page.js
Template.your_page_name.events({
  ...
  "change select[name='selectinput_name']": function (e, t) {
    Session.set("cat_id", $(e.target).val());
  },
  ...
});

And that should work as expected.

@quangmach
It is working perfectly fine now!
So I just needed to return in helper class.
I got it.
Thank you so much!

@innocube Good to see that works for you. However, like @perak said, mine one just grab all the database and filter on client side, so it is not so good if you have lots of items. It is better to filter on server side then publish -> client subscribed to it.

@perak Thank you for letting me know a proper way to publish & subscribe according to previous selection!
Now, I am wondering is there anyway that I can limit or expand accessibility of data set according to the level of user. For instance, for admin & manager, I would like to display everything, but for user, I would like to limit items only specific to that user. How can I do that?

Just set "roles_allowed_to_read": ["owner", "manager"]  in a collection. So only owner user (user who created document) and user with "manager" role will have access to it.

@perak
My question was a little bit different. What I want is on a page according to user level I would like to limit or allow contents, not limiting the page access.
Thank you!


Implement Command box and help bar UI

[ ] Delete help window related files and documents (HelpWindow.java, HelpWindow.fxml)
[ ] Change DeveloperGuide and UserGuide regrading help function
[ ] Align the command box and help bar according to figma design

Codecov Report
Base: 69.48% // Head: 70.35% // Increases project coverage by +0.86% :tada:

Coverage data is based on head (c631668) compared to base (9247284).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #199      +/-   ##
============================================
+ Coverage     69.48%   70.35%   +0.86%
+ Complexity      591      590       -1
============================================
  Files           102      101       -1
  Lines          1881     1855      -26
  Branches        210      210
============================================
- Hits           1307     1305       -2
+ Misses          500      476      -24
  Partials         74       74

Impacted Files
Coverage Δ

src/main/java/seedu/taassist/ui/MainWindow.java
0.00% <0.00%> (ø)

src/main/java/seedu/taassist/ui/UiManager.java
0.00% <ø> (ø)

...main/java/seedu/taassist/commons/util/AppUtil.java
66.66% <0.00%> (-22.23%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Implemented #83 in this pr as it is part of the main screen ui

Implemented main screen UI in the same branch as they are all part of UI #155


Restore the CRDs from 1.0

This avoids Tiller deleting the CRDs because it believes they are no longer present.
The CRDs are now created with kubectl apply or by using the new istio-init chart, and
new CRDs are no longer under control of helm.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: costinm
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS [costinm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@costinm: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
9cf8b32772c0996e229fb185febe227b28bfe7d8
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@costinm: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
9cf8b32772c0996e229fb185febe227b28bfe7d8
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
9cf8b32772c0996e229fb185febe227b28bfe7d8
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@costinm: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@costinm considering the world hasn't ended even though we didn't merge this, I think this would be better off in master and perhaps 1.2


tolerance for pivot_cache

This PR is intended to address #31 by making the default tolerance depend on the eps() of the data type for the pivot_cache method, instead of being hard-wired to 1e-8.
(This refinement is needed in other places too, eventually.)
There is also some whitespace cleanup to move closer to https://github.com/invenia/BlueStyle

I think this is ready to go but I'd like at least one other set of eyes on it before committing...