Yul optimisation

Solidity does  a pretty good job generally but some things help, for example:
        //uint8(LOOKUP_TABLE_256[(isolateLS1B256(bb) * DEBRUIJN_256) >> 248]);
        bytes memory table = LOOKUP_TABLE_256;
        assembly {
            bb := and(bb, sub(0, bb))
            bb := mul(bb, DEBRUIJN_256)
            bb := shr(248, bb)
            result := mload(add(table, add(bb, 1)))
            result := and(0xff, result)
        }
    }

Whereas this function was the same either way:
    function set(BitMap storage bitmap, uint256 index) internal {
        uint256 bucket = index >> 8;
        uint256 mask = MASK_INDEX_ZERO >> (index & 0xff);
        bitmap._data[bucket] |= mask;
        //assembly {
        //    let bucket := shr(8, index)
        //    let mask := shr(and(index, 0xff), 0x8000000000000000000000000000000000000000000000000000000000000000)
        //    mstore(0, bucket)
        //    mstore(0x20, bitmap.slot)
        //    let slot := keccak256(0, 0x40)
        //    let current := sload(slot)
        //    sstore(slot, or(current, mask))
        //}
    }

However, I've done a test version with Chiru's 4.2 ERC721A (combined with a lengthless array optimisation around storage for this and another struct) and it was easiest to simply port across the two functions used and merge them in rather than me go through and yul/test the solidity-bits lib.
Happy to help if you want the lib improved though.

Another function I converted:
        bytes memory table = LOOKUP_TABLE_256;
        assembly {
             let bucketIndex
             let bb
            //bucket = index >> 8;
            let bucket := shr(8, index)
            // index within the bucket
            //uint256 bucketIndex = (index & 0xff);
            bucketIndex := and(index, 0xff)

            for {
                // load a bitboard from the bitmap.
                //uint256 bb = bitmap._data[bucket];
                mstore(0, bucket)
                mstore(0x20, bitmap.slot)
                bb := sload(keccak256(0, 0x40))
                // offset the bitboard to scan from `bucketIndex`.
                //bb = bb >> (0xff ^ bucketIndex); // bb >> (255 - bucketIndex)
                bb := shr(xor(0xff, bucketIndex), bb)
                if iszero(bb) {
                    bucketIndex := 255
                }
            } iszero(bb) {
                mstore(0, bucket)
                bb := sload(keccak256(0, 0x40))
            } {
                if iszero(bucket) {
                    revert(0, 0)
                }
                bucket := sub(bucket, 1)
            }

            //bb.bitScanForward256()
            bb := and(bb, sub(0, bb))
            bb := mul(bb, DEBRUIJN_256)
            bb := shr(248, bb)
            bb := mload(add(table, add(bb, 1)))
            bb := and(0xff, bb)
            //result = (bucket << 8) | (bucketIndex - bb.bitScanForward256());
            result := or(shl(8, bucket), sub(bucketIndex, bb))
        }
    }

The 721A version and notes can be found in my comment here:
https://github.com/chiru-labs/ERC721A/issues/391#issuecomment-1199728087

@sillytuna This is interesting. Do you have a gas usage comparison (with solidity optimizer enabled)?

I was testing in situ. Marginal gains from the second two funcs but worth it on something like this, esp if could be called a lot.
Should be quick for you to put in your code and run a quick gas comparison tho.

My concern is it may affect the code readability. I also saw you lengthless array PR for ERC721A.
Maybe shifting from mapping to lengthless array for the bitMap datastructure will be a bigger improvement.

My view on that is it depends on use. Code like this can be needed in gas performant functions and will be write once use many. I'd keep the original solidity in hand to be read for sure and yul would need well testing and gas checks.
Your code is already quite hard to understand just because of the algorithm so I'm not sure yul really changes that. Can be well commented and include solidity comments.

        //uint8(LOOKUP_TABLE_256[(isolateLS1B256(bb) * DEBRUIJN_256) >> 248]);
        bytes memory table = LOOKUP_TABLE_256;
        assembly {
            bb := and(bb, sub(0, bb))
            bb := mul(bb, DEBRUIJN_256)
            bb := shr(248, bb)
            result := mload(add(table, add(bb, 1)))
            result := and(0xff, result)
        }
    }

I took a look and it seems the gas saving comes from two factors:

removing the require(bb>0) check.
inline the isolateLS1B

Unless there is a good reason, I don't think require(bb>0) should be removed. As for the merging the function, solidity optimizer now have this optimization stage and should take care that automatically based on the optimization parameter you specify. It's a deployment gas vs run time gas problem.


🚀 Send an error message when VideoTrack is null while trying to render video

🚀 Enhancement

If the app sends a trackId of a video track which is not currently in the Room
Then send an error message to apps

We are already sending this with an error code : 6004


Add Threshold Proxy Re-encryption Scheme

This PR was originally initiated by @Cryptographer63, see https://github.com/secretflow/yacl/pull/50. Now we need to merge this primitive into the main branch.
We sincerely thank him for the contribution.

lgtm


pkg/k8s/watchers follow-up for #14864

Unexport two methods only used inside the package and while at it address https://github.com/cilium/cilium/pull/14864#pullrequestreview-584603874

GH actions and Travis CI passed, full CI run not necessary. Marking as ready to merge.


Exported Models Show No Materials on Facebook or Oculus

I downloaded this exporter so that I could create custom models and scenes in Blender and then add them to my Oculus Home environment. However, I have discovered that no materials, textures, or even colors are being exported despite me having checked all material and PBR-related options. I am using Blender 2.79, and have tested the exported models in both Oculus Home and Facebook's 3D model embedding system. I have attached both the .blend file for the test model and the exported .glb file.
Test GLB Export.zip

Hi @AwesomeAnimationGamer — this exporter does not yet support Principled BSDF materials, per https://github.com/KhronosGroup/glTF-Blender-Exporter/issues/101. In the meantime we provide custom PBR nodes, described in the user documentation.

Oh, I missed that part. Thanks!

I'm still having issues. I found the file with the nodes and was able to transfer my current model over just fine. However, the exported model still shows no colors. Metallic roughness is preserved, but color data seems to be getting lost.

Yeah sorry, so a confusing bit on these nodes is that BaseColor needs to be an Image Texture. BaseColorFactor accepts simple color information like you're doing here. Hoping we can support Principled BSDF soon so that the normal workflows just work. 😅

And something is still going wrong. The file exports as 1 KB and Facebook rejects it, saying it is unsupported. These are the options I have ticked in the export:

And here is the bugged exported file:
Hyperwarp Engine.zip
All I changed between the versions was to migrate the color data to BaseColorFactor.

Hm well the file certainly looks empty. Are you able to share the .blend file?  Maybe something to do with that "selected only" checkbox?

You may be able to get a little more info about the file from http://github.khronos.org/glTF-Validator/, which I believe is a large subset of the validation that Facebook uses during upload.

I can confirm it was caused by the "Selected Only" checkbox. I guess it wasn't registering that I had the object selected already. It's working now, it seems the only thing that's still having a problem is the Emissive texture. It stays gray in Facebook's 3D view regardless of the color set in Blender. (Emissive component on PBR node has no differing effect to the basic emissive shader)

Hm, I don't know what to make of the emissive texture issue. Does the emissive color appear on http://sandbox.babylonjs.com/ or https://gltf-viewer.donmccurdy.com/? You also can't use an Emissive node, the texture must be wired into that same glTF PBR node's Emissive slot.
Note that neither of those viewers (nor Facebook) apply bloom post-processing, so you won't get that nice emissive glow, but just a solid color unaffected by lighting.

It does indeed show up that way. Hopefully it affects lighting in Oculus Home, but if not then this will be just fine too. Thank you so much for your help! Hopefully I'm closing this for good now.

Sure thing! I think Sketchfab's viewer should show emissive nicely if you want to upload the GLB there.  We also have a glTF gitter chat (https://gitter.im/KhronosGroup/glTF) if you want to ask one-off questions. :)

I just tested. I have to tweak the model itself, but it works perfectly in Oculus Home! The Emissive part doesn't necessarily cast light but it has a nice little glow effect around it.


Line Breaks not escaped

Hi
I am currently using the class in a vbs and have noticed that line breaks in values are not escaped. I work around this by doing the replacement before I pass the values so this is not really a problem for me.
This might however be an issue only concerning the scripting host, I wanted to inform anyway.
Regards

I will sure take a look at this.
Thank you!

This should be fixed now in 3.3.0.


grid proxy in manual

how to use, review the api specs
put link to swagger
some examples...

@sameh-farouk please make sure your grid proxy docs to exist in this repo

wiki editor gitpod doesn't start.
publishing tools won't compile and seems broken atm.
possible related to https://github.com/freeflowuniverse/crystallib/issues/72 ?
logs from the gitpod
/home/gitpod/.vmodules/despiegk/crystallib/process/ps_tool.v:46:20: error: `despiegk.crystallib.process.processmap` is immutable, cannot have a mutable reference to an immutable object
   44 | //then you can get it everywhere
   45 | pub fn processmap_get() ?&ProcessMap {
   46 |     mut pm := process.processmap
      |                       ~~~~~~~~~~
   47 |     pm.scan()?
   48 |     return pm

~/code/github/freeflowuniverse/crystaltools/crystaltools /workspace/web_wiki_editor
 - COMPILE
publishtools.v:8:8: warning: module 'gittools (despiegk.crystallib.gittools)' is imported but never used
    6 | import despiegk.crystallib.publisher_core
    7 | import despiegk.crystallib.publisher_config
    8 | import despiegk.crystallib.gittools
      |        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    9 | import readline
   10 |
/home/gitpod/.vmodules/despiegk/crystallib/coinmarketcap/coinmarketcap.v:185:16: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  183 |         req.header = h.header() ?
  184 |         res := req.do() ?
  185 |         result = res.text
      |                      ~~~~
  186 |     }
  187 |     // means empty result from cache
/home/gitpod/.vmodules/despiegk/crystallib/coinmarketcap/coinmarketcap.v:219:16: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  217 |         req.header = h.header() ?
  218 |         res := req.do() ?
  219 |         result = res.text
      |                      ~~~~
  220 |     }
  221 |     // means empty result from cache
/home/gitpod/.vmodules/despiegk/crystallib/liquid/liquid.v:162:21: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  160 |         println(req)
  161 |         response := req.do() ?
  162 |         result = response.text
      |                           ~~~~
  163 |     }
  164 |     // Post without auth header
/home/gitpod/.vmodules/despiegk/crystallib/liquid/liquid.v:167:21: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  165 |     else {
  166 |         response := http.post_json('$h.url/$prefix', postdata) ?
  167 |         result = response.text
      |                           ~~~~
  168 |     }
  169 |     h.cache_set(prefix, postdata, result, cache) ?
/home/gitpod/.vmodules/despiegk/crystallib/liquid/liquid.v:194:21: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  192 |         println(req)
  193 |         response := req.do() ?
  194 |         result = response.text
      |                           ~~~~
  195 |     }
  196 |     // Post without auth header
/home/gitpod/.vmodules/despiegk/crystallib/liquid/liquid.v:199:21: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  197 |     else {
  198 |         response := http.post_json('$h.url/$prefix', postdata) ?
  199 |         result = response.text
      |                           ~~~~
  200 |     }
  201 |     h.cache_set(prefix, postdata, result, cache) ?
/home/gitpod/.vmodules/despiegk/crystallib/liquid/liquid.v:222:16: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  220 |         req.header = h.header()
  221 |         res := req.do() ?
  222 |         result = res.text
      |                      ~~~~
  223 |     }
  224 |     // means empty result from cache
/home/gitpod/.vmodules/despiegk/crystallib/liquid/liquid.v:251:16: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  249 |         req.header = h.header()
  250 |         res := req.do() ?
  251 |         result = res.text
      |                      ~~~~
  252 |     }
  253 |     // means empty result from cache
/home/gitpod/.vmodules/despiegk/crystallib/liquid/liquid.v:276:16: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
  274 |     req.header = h.header()
  275 |     res := req.do() ?
  276 |     result := res.text
      |                   ~~~~
  277 |     h.cache_set(prefix, data, result, cache) ?
  278 |     data_raw := json2.raw_decode(result) ?
/home/gitpod/.vmodules/despiegk/crystallib/httpcache/httpcache.v:31:13: notice: field `text` will be deprecated after 2022-10-03, and will become an error after 2023-04-01; use Response.body instead
   29 |
   30 |         r := http.get(url) ?
   31 |         data := r.text
      |                   ~~~~
   32 |
   33 |         status := http.status_from_int(r.status_code)
/home/gitpod/.vmodules/despiegk/crystallib/process/ps_tool.v:46:20: error: `despiegk.crystallib.process.processmap` is immutable, cannot have a mutable reference to an immutable object
   44 | //then you can get it everywhere
   45 | pub fn processmap_get() ?&ProcessMap {
   46 |     mut pm := process.processmap
      |                       ~~~~~~~~~~
   47 |     pm.scan()?
   48 |     return pm
/home/gitpod/.vmodules/despiegk/crystallib/publisher_config/model_site.v:12:20: error: struct `despiegk.crystallib.gittools.GitRepo` was declared as private to module `despiegk.crystallib.gittools`, so it can not be used inside module `despiegk.crystallib.publisher_config`
   10 | pub struct SiteConfig {
   11 | mut:
   12 |     repo        gittools.GitRepo
      |                          ~~~~~~~
   13 | pub mut:
   14 |     name       string
/home/gitpod/.vmodules/despiegk/crystallib/publisher_config/model_site.v:125:37: error: struct `despiegk.crystallib.gittools.GitRepo` was declared as private to module `despiegk.crystallib.gittools`, so it can not be used inside module `despiegk.crystallib.publisher_config`
  123 | }
  124 |
  125 | pub fn (site SiteConfig) repo_get() gittools.GitRepo {
      |                                     ~~~~~~~~~~~~~~~~
  126 |     if site.repo.addr.name == ""{
  127 |         panic("Cannot return repo, because was not initialized, name empty")
/home/gitpod/.vmodules/despiegk/crystallib/builder/node.v:36:20: error: struct `despiegk.crystallib.rediscache.RedisCache` was declared as private to module `despiegk.crystallib.rediscache`, so it can not be used inside module `despiegk.crystallib.builder`
   34 |     db &DB [skip]
   35 |     done map[string]string
   36 |     cache  rediscache.RedisCache [skip]
      |                       ~~~~~~~~~~
   37 |     environment map[string]string
   38 | }
/home/gitpod/.vmodules/despiegk/crystallib/publisher_core/publisher_model.v:26:17: error: struct `despiegk.crystallib.texttools.ReplaceInstructions` was declared as private to module `despiegk.crystallib.texttools`, so it can not be used inside module `despiegk.crystallib.publisher_core`
   24 | struct ReplacerInstructions {
   25 | pub mut:
   26 |     site texttools.ReplaceInstructions
      |                    ~~~~~~~~~~~~~~~~~~~
   27 |     file texttools.ReplaceInstructions
   28 |     word texttools.ReplaceInstructions
/home/gitpod/.vmodules/despiegk/crystallib/publisher_core/publisher_model.v:27:17: error: struct `despiegk.crystallib.texttools.ReplaceInstructions` was declared as private to module `despiegk.crystallib.texttools`, so it can not be used inside module `despiegk.crystallib.publisher_core`
   25 | pub mut:
   26 |     site texttools.ReplaceInstructions
   27 |     file texttools.ReplaceInstructions
      |                    ~~~~~~~~~~~~~~~~~~~
   28 |     word texttools.ReplaceInstructions
   29 |     defs texttools.ReplaceInstructions
/home/gitpod/.vmodules/despiegk/crystallib/publisher_core/publisher_model.v:28:17: error: struct `despiegk.crystallib.texttools.ReplaceInstructions` was declared as private to module `despiegk.crystallib.texttools`, so it can not be used inside module `despiegk.crystallib.publisher_core`
   26 |     site texttools.ReplaceInstructions
   27 |     file texttools.ReplaceInstructions
   28 |     word texttools.ReplaceInstructions
      |                    ~~~~~~~~~~~~~~~~~~~
   29 |     defs texttools.ReplaceInstructions
   30 | }
/home/gitpod/.vmodules/despiegk/crystallib/publisher_core/publisher_model.v:29:17: error: struct `despiegk.crystallib.texttools.ReplaceInstructions` was declared as private to module `despiegk.crystallib.texttools`, so it can not be used inside module `despiegk.crystallib.publisher_core`
   27 |     file texttools.ReplaceInstructions
   28 |     word texttools.ReplaceInstructions
   29 |     defs texttools.ReplaceInstructions
      |                    ~~~~~~~~~~~~~~~~~~~
   30 | }
   31 |
publishtools.v:116:9: error: struct `despiegk.crystallib.publisher_core.Publisher` was declared as private to module `despiegk.crystallib.publisher_core`, so it can not be used inside module `main`
  114 |                 exit(1)
  115 |             }
  116 |             publ.develop = true
      |                  ~~~~~~~
  117 |             publisher_core.webserver_run(mut &publ) or {
  118 |                 println('Could not run webserver for wiki.\nError:\n$err')

@sameh-farouk please do it manually, clone the repo and create grid_proxy directory under wiki, and your content there

Update:
PR ready for review:
https://github.com/threefoldfoundation/info_manual3/pull/80


Only half of the test images can be run via 'python -m runx.runx scripts/dump_folder.yml'.

Thank you for your sharing.
But Only half of the test images can be run via 'python -m runx.runx scripts/dump_folder.yml'. In addition, when it run the cityscapes testing images, it only got mIoU about 84.6, not the 85.4 like your submitting on the cityscapes benchmark.
Could please tell me what the problem is? thank you very much.

it only got mIoU about 84.6 -- Is 84.6 your validation score or are you saying that you submitted to Cityscapes server and got this test score?
I don't know what the half test images problem is, are you running dump_folder.yml on your own images?

Hi @ajtao ,
I am trying to run inference on my own images using this command python -m runx.runx scripts/dump_folder.yml -i
Below is the output:
(nvidia_hrnet) C:\Workspace\msjmf59\VirtualEnvironments\nvidia_hrnet\semantic-segmentation>python -m runx.runx scripts/dump_folder.yml -i -n
cd logs\dump_folder\piquant-gazelle_2021.02.26_17.42/code; PYTHONPATH=logs\dump_folder\piquant-gazelle_2021.02.26_17.42/code exec python -m torch.distributed.launch --nproc_per_node=1 train.py --dataset cityscapes --cv 0 --syncbn --apex --fp16 --bs_val 1 --eval folder --eval_folder ./imgs/test_imgs --dump_assets --dump_all_images --n_scales 0.5,1.0,2.0 --snapshot ASSETS_PATH/seg_weights/cityscapes_ocrnet.HRNet_Mscale_outstanding-turtle.pth --arch ocrnet.HRNet_Mscale --result_dir logs\dump_folder\piquant-gazelle_2021.02.26_17.42

(nvidia_hrnet) C:\Workspace\msjmf59\VirtualEnvironments\nvidia_hrnet\semantic-segmentation>python -m runx.runx scripts/dump_folder.yml -i
Copying codebase to logs\dump_folder\neat-skink_2021.02.26_17.42 ...
Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Workspace\msjmf59\VirtualEnvironments\nvidia_hrnet\lib\site-packages\runx\runx.py", line 394, in <module>
    main()
  File "C:\Workspace\msjmf59\VirtualEnvironments\nvidia_hrnet\lib\site-packages\runx\runx.py", line 387, in main
    run_experiment(args.exp_yml)
  File "C:\Workspace\msjmf59\VirtualEnvironments\nvidia_hrnet\lib\site-packages\runx\runx.py", line 380, in run_experiment
    run_yaml(experiment_copy, runroot)
  File "C:\Workspace\msjmf59\VirtualEnvironments\nvidia_hrnet\lib\site-packages\runx\runx.py", line 346, in run_yaml
    subprocess.call(['chmod', '-R', 'a+rw', expdir])
  File "C:\Python37\lib\subprocess.py", line 339, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Python37\lib\subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "C:\Python37\lib\subprocess.py", line 1207, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

Have updated the weights folder in config.py as well.
Not sure what am I missing here.
Thanks in advance!

It's telling you that it can't create the experiment directory. The experiment directory is created under LOGROOT, defined in your .runx file. You should have created a .runx file.  Most likely either your LOGROOT isn't defined or it's pointing to a location that doesn't exist.


Interactive Intake Worksheet Design

**Objective: ** The UNTD team would like to add an interactive worksheet to be associated with the "Use Cases" Page.
Context: This interactive worksheet will support students who are connected with a nonprofit to redesign programs that encourage economic mobility. The worksheet will help students evaluate how aligned a nonprofit is with existing research on socioeconomic mobility. In the future, responses will produce a result, but for now we will be designing an interactive questionnaire that records participant responses. More information here
Scope: Hyperobjekt has agreed to present preliminary mockups for this feature to UNTD for feedback.

I think designing this for "students" will turn out to have been short-sighted, despite the fact that that is the narrative Owen has provided at this point. The initial discussion suggested that this tool would ultimately be used for a variety of people in different roles, including people who run nonprofits and people who write grants for nonprofits. There is no need to phrase this tool as being "for students," and if it is designed for this one narrative only it may limit its applicability for other kinds of users.

Excellent point, thanks for raising this Lou! I've tweaked the description to reflect that in the future, more than students might be using this tool (expanded target user to be any local decision-maker).


moved setup go envirnoment before installation

Change order in documentation to first create a go env' and then install ent.
This is meant to avoid having ent on your global pkgs.

Thanks @yonidavidson 😃


Gangsters: Organized Crime not starting

Hi!
Using the latest version of the fix on Windows 11. Game does not start only black screen then exits. Log attached.
dxwrapper-gangsters.log

I believe that is an old build with an issue on Windows 11.  Try with this update.  Just unzip this file into the game folder overwriting the older file:  dxwrapper.zip


webfist.org should be deployed on HTTPS

Currently webfist.org is HTTP only.
This fallback should be deployed behind SSL and using HTTPS should be the recommended way to access it.

+1
https://github.com/silverbucket/webfinger.js/issues/12


configure dependabot for github composite actions

🚀 Pull Request
Description

This pull-request updates the dependabot configuration to include coverage of our composite GHAs.
Note that, dependabot has not updated the actions/cache GHA version used by the composite actions, which has been at v4.0.2 since 19 Mar 2024.
As a result, the composite actions have been using a deprecated version of node.js and forced to run on node20.

Consult Iris pull request check list

Add any of the below labels to trigger actions on this PR:

https://github.com/SciTools/iris/labels/benchmark_this

Let's not manually update the actions/cache version in the composite actions, and leave it to dependabot, which is a good test of this PR change.


fix: add collected data types to privacy manifests

Issue #

Description

This PR adds the data collected section in the privacy manifests. In each target are the following reasons

Amplify: no corresponding AWS service, no data is collected
AWSAPIPlugin - adds Amplify's user-agent to AppSync / APIGateway requests
AWSPinpointAnalyticsPlugin - adds Amplify's user-agent to Amazon Pinpoint requests
AWSCognitoAuthPlugin - adds Amplify's user-agent to Cognito requests
AWSPluginsCore - no corresponding AWS service, no data is collected
AWSDataStorePlugin - Makes calls through AWSAPIPlugin, thus it's privacy manifest is reflected by AWSAPIPlugin
AWSLocationGeoPlugin - adds Amplify's user-agent to AWS Location requests
InternalAWSPinpoint - adds Amplify's user-agent to Amazon Pinpoint requests
AWSCloudWatchLoggingPlugin - adds Amplify's user-agent to AWS Cloudwatch requests
AWSPinpointPushNotification - adds Amplify's user-agent to Amazon Pinpoint requests
AWSPredictionsPlugin - adds Amplify's user-agent to various AWS services, depending on which is utilized.
CoreMLPredictionsPlugin - no corresponding AWS service, no data is collected
AWSS3StoragePlugin - adds Amplify's user-agent to AWS S3 requests

The user agent information contains the Amplify version, OS name and version which corresponds to Other Data types (https://docs.amplify.aws/swift/build-a-backend/auth/data-usage-policy/).
In the current version of the privacy manifest format, there isn't a way to specify in more detail the type of "other data type", which is why I have opened a question in the Apple Developer Forums to ask about this: https://developer.apple.com/forums/thread/743559
Tracking is set to false, despite the current version of our docs shows tracking is true for certain data types, however I have a separate proposal based on my understanding of AppTrackingTransparency Framework that defines what tracking is. We may wait for that proposal to finalize on the Tracking value for the data types listed here (either set NSPrivacyCollectedDataTypeTracking to true or leave it as false).
This PR also fixes a problem with generating the privacy manifest report, without this PR, the privacy report shows errors expecting the CollectedDataType key to exist:

To generate a privacy report, Archive the app which consumes the library, and then in Xcode -> Window -> Organizer -> Under Archives, right click on the archive -> Generate Privacy Report. The privacy report only shows tracking related information, thus with this change the privacy report will be an empty file.
General Checklist

[ ] Added new tests to cover change, if needed
[ ] Build succeeds with all target using Swift Package Manager
[ ] All unit tests pass
[ ] All integration tests pass
[ ] Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)
[ ] Documentation update for the change if required
[ ] PR title conforms to conventional commit style
[ ] New or updated tests include Given When Then inline code documentation and are named accordingly testThing_condition_expectation()
[ ] If breaking change, documentation/changelog update with migration instructions

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 68.07%. Comparing base (f1d354c) to head (ac03722).
Report is 10 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3531      +/-   ##
==========================================
- Coverage   68.07%   68.07%   -0.01%
==========================================
  Files        1080     1080
  Lines       36363    36363
==========================================
- Hits        24754    24753       -1
- Misses      11609    11610       +1

Flag
Coverage Δ

API_plugin_unit_test
67.50% <ø> (ø)

AWSPluginsCore
64.14% <ø> (ø)

Amplify
48.10% <ø> (ø)

Analytics_plugin_unit_test
81.16% <ø> (ø)

Auth_plugin_unit_test
79.02% <ø> (+0.01%)
:arrow_up:

CoreMLPredictions_plugin_unit_test
59.44% <ø> (ø)

DataStore_plugin_unit_test
81.28% <ø> (-0.05%)
:arrow_down:

Geo_plugin_unit_test
70.75% <ø> (ø)

Logging_plugin_unit_test
62.87% <ø> (ø)

Predictions_plugin_unit_test
37.16% <ø> (ø)

PushNotifications_plugin_unit_test
87.13% <ø> (ø)

Storage_plugin_unit_test
77.51% <ø> (ø)

unit_tests
68.07% <ø> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add missing initialization to basic example

@samjbmason ran into an issue in the #micro channel 😉

Thanks!


https://github.com/StackStorm/orquesta/issues/119

Failure to publish True to a variable using shorthand format
The root cause:
For this issue is that when parse_inline_params is called, it checks regular expression against 'true' (REGEX_TRUE = 'true').  Since input value is True. it returns empty list v=[]. This is why publish value always equals to input value which is false.
If modify expression to REGEX_TRUE = '(?i)true' to ignore case, it passes regular expression check and return v=['True'], when later v = json.loads(v)is called, it treatsTrueas string and publish output will be"True"instead oftrue`.

Codecov Report

Merging #129 into master will increase coverage by 0.14%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #129      +/-   ##
==========================================
+ Coverage   94.03%   94.18%   +0.14%
==========================================
  Files          38       38
  Lines        2448     2545      +97
  Branches      493      539      +46
==========================================
+ Hits         2302     2397      +95
+ Misses         85       84       -1
- Partials       61       64       +3

Impacted Files
Coverage Δ

orquesta/utils/parameters.py
100% <100%> (ø)
:arrow_up:

orquesta/composers/native.py
93.83% <0%> (+0.28%)
:arrow_up:

orquesta/graphing.py
96.15% <0%> (+1.86%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 774123b...22699f7. Read the comment docs.

Can you change the title of this commit?  It's not standard practice to use the link of the issue.  You can include the link in the description but the title should be more descriptive.  Thanks.

Can you include some unit tests?

Done!

@jinpingh What happens if we pass the value with quotes or double quotes (i.e. x="true" or x='true')?  Is the value evaluated to boolean or kept as string?  I'm expecting it to be string.

@jinpingh What happens if we pass the value with quotes or double quotes (i.e. x="true" or x='true')? Is the value evaluated to boolean or kept as string? I'm expecting it to be string.

It  will evaluated as boolean since json.loads(v) convert string "true" to boolean true.

@jinpingh I added another issue to address the issue where x="true" is evaluated to boolean.  If the user puts the value in quotes, I think the value should be treated as string. https://github.com/StackStorm/orquesta/issues/130


Improve binning of histograms

Consider re-implementing Vega's binning strategy: https://github.com/vega/vega/blob/72b9b3bbf912212e7879b6acaccc84aff969ef1c/packages/vega-statistics/src/bin.js#L23

Is there a writeup of that binning style?  I found this.  https://vega.github.io/vega-lite/docs/bin.html
When I implemented binning for my project, I made an extra bin for the 1st and 99th percentile, capturing most outliers and making a fatter higher resolution middle.  Ideally I would include that first and last percentile in the first and last regular bins and communicate the change via mousover for Bin sizes.  the 1/99 will only ever have 2% of total values and will never have a high bar.

So I dont know the name of the algorithm they use in vega, right now in AutoProfiler we use equal width bins. This issue was about doing something smarter to pick the number of bins. Right now we do bins = min(unique values, 20) iirc which is very simple and vega seemingly has a better approach depending on cardinality or range of data


Partnership for Energy Progress

Partnership for Energy Progress table also lists countries and not just the states with ad dollars

fixed!


Support parsing translations

It is missing translation information :(
>>> word = parser.fetch('smartphone')
>>> print(word)
[{'etymology': 'smart +\u200e phone', 'definitions': [{'partOfSpeech': 'noun', 'text': 'smartphone \u200e(plural smartphones)\nA mobile phone with more advanced features and greater computing capacity than a featurephone.\n', 'relatedWords': [], 'exampleUses': []}], 'audioLinks': [], 'pronunciations': ['(UK) IPA: /ˈsmɑːtfəʊn/', '(US) IPA: /ˈsmɑɹtfoʊn/']}]
>>>

But thank you for this interesting and nice project!

Hey, I've added translations now but they're not sorted by languages, they're just a list of words in the relatedWords field.  Thanks!


HTTP Deadlines

Updated HTTP call to use a deadline set to the distant future to use the HTTP configuration's timeouts directly. This matches what APNSwift is doing, and observing the code in AsyncHTTPClient, seems to be what the NIO-based execution methods do as well when it is set to nil (though in that case, they skip adding a deadline task to cancel the request).

This check is wrong:
        } else if expiration >= Expiration.recommendedMaximum {
            logger.warning("The message expiration should be less than \(Expiration.recommendedMaximum) seconds.", metadata: ["expiration": "\(expiration)"])

it should be >, a maximum should be allowed. (it triggers the log if Expiration.recommendedMaximum is used as the maximum value)

This check is wrong:
        } else if expiration >= Expiration.recommendedMaximum {
            logger.warning("The message expiration should be less than \(Expiration.recommendedMaximum) seconds.", metadata: ["expiration": "\(expiration)"])

it should be >, a maximum should be allowed. (it triggers the log if Expiration.recommendedMaximum is used as the maximum value)

Oops, let me fix and make a new tag.

@helje5 Fixed in 0.2.1, sorry about that!


shm memory file mutex should not be unlinked on subscriber destruction

shm memory file mutex should not be unlinked if memory file is not owned (created) by the caller (fixing issue #854)
Pull request type
Please check the type of change your PR introduces:

[X] Bugfix

What is the current behavior?
See issue description.
Fixes #854
Does this introduce a breaking change?

[X] No

I would propose to fix this issue on my shm-monitoring branch, since i removed the current memfile mutex api and replaced them with a RAII style NamedMutex class.


OSS-Fuzz: Use-of-uninitialized-value in spvtools::CFAspvtools::val::BasicBlock::CalculateDominators

https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=48566
@alan-baker for info; wouldn't be surprised if this is related to #4862

Fixed by #4971


Why is there a bbox-head in the segmentation model

Because in MaskRCNN during inference the mask head gets masks from the results of the bbox head
image from https://www.kaggle.com/c/data-science-bowl-2018/discussion/49692


data_layer.py "while len(sample_person) > 0:" ?

Hi, hizhangp,
In this while loop, len(sample_person) does not decrease, right? How would it become 0?

@youhangtian Yes, the keys in sample_person need to be deleted when its length == 0, I've added the missing expressions.

@hizhangp 这个工程里面如果训练样本里有人只有一张图片的话，是不是跑不了？

你程序里好像是分割了训练样本，分成训练和测试


implemented local Outlier Probabilites (LoOP)

See  https://dl.acm.org/citation.cfm?id=1646195

This is implemented here -https://github.com/vc1492a/PyNomaly

Exactly. I am aware of the great library implemented by Valentino. However, I think it could be better to include LoOP as part of Pyod, which implies to rewrite the model under the current framework shown in pyod/models/base.py.
You are more than welcome to see the examples of implemented models and submit a PR for LoOP:

https://github.com/yzhao062/Pyod/blob/master/pyod/models/abod.py
https://github.com/yzhao062/Pyod/blob/master/pyod/models/pca.py
https://github.com/yzhao062/Pyod/blob/master/pyod/models/cblof.py

The key is to inherit from base.py and implement fit() and decision_function().
Cheers.

Just to add to the discussion, I've been working on adding LoOP
functionality to scikit-learn that can be viewed in this pull request:
https://github.com/scikit-learn/scikit-learn/pull/10121.
The scikit-learn implementation is faster than that of PyNomaly, as it's
able to take advantage of scikit-learn's rapid nereast neighbor algorithms
and other nice features. I'm waiting on core scikit-learn contributors to
provide some feedback and comments before continuing with a lot of further
work, but maybe this would be the best way to integrate LoOP into Pyod
going forward if it is accepted into scikit-learn (more maintainable than
PyNonaly as there are more contributors, it's faster, etc.).
Happy to help with this issue when I'm back from vacation in early July.
Thought the above may add some helpful information.
On Thu, Jun 21, 2018, 20:42 Yue Zhao<EMAIL_ADDRESS>wrote:

Exactly. I am aware of the great library implemented by Valentino.
However, I think it could be better to include LoOP as part of Pyod, which
implies to rewrite the model under the current framework shown in
pyod/models/base.py.
You are more than welcome to see the examples of implemented models and
submit a PR for LoOP:

https://github.com/yzhao062/Pyod/blob/master/pyod/models/abod.py
https://github.com/yzhao062/Pyod/blob/master/pyod/models/pca.py
https://github.com/yzhao062/Pyod/blob/master/pyod/models/cblof.py

The key is to inherit from base.py and implement fit() and
decision_function().
Cheers.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/yzhao062/Pyod/issues/8#issuecomment-399186706, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB9mqPhjiwnaU9x3OiitubrnT-mOq27lks5t-9sSgaJpZM4UoBkb
.

I have opened a pull request scikit-learn/scikit-learn#27337 that adds knn, knn weight, and loop outlier detectors to sklearn.
If you are interested in seeing this included in core sklearn, you could express this by a thumbs-up on the pull request to finally have this included (note that this issue has been open on PyOD since 2018!)


Tune local kafka cluster config to be more stable

Make the timeout longer so that the test is more stable in a resource limited environment.
Default for the following timeout are:
replica.socket.timeout.ms: 30s
controller.socket.timeout.ms: 30s
controlled.shutdown.retry.backoff.ms: 5s
Reference: https://kafka.apache.org/documentation/#brokerconfigs
One example failure test run: https://github.com/apache/pinot/runs/4388914598

Codecov Report

Merging #7854 (c494ecf) into master (b9da141) will decrease coverage by 50.71%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #7854       +/-   ##
=============================================
- Coverage     65.28%   14.57%   -50.72%
+ Complexity     4087       80     -4007
=============================================
  Files          1535     1535
  Lines         79224    79224
  Branches      11865    11865
=============================================
- Hits          51722    11547    -40175
- Misses        23823    66819    +42996
+ Partials       3679      858     -2821

Flag
Coverage Δ

unittests1
?

unittests2
14.57% <ø> (-0.03%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9da141...c494ecf. Read the comment docs.


move.py - add email notification when transfer is complete

Yup, awesome idea. Btw I made a different branch with some new change. I'll merge with the master branch once I'm sure it works. https://github.com/kieranjol/IFIscripts/tree/kieranjol-file-count


Refinement stage query

Hi,
Awesome work! Thanks for sharing it.
I had a couple of questions regarding the refinement stage:

In the paper you mention that you use the output from the 1st fconvnet and feed it (after some adjustments) into a 2nd fconvnet. Do I understand this correctly? So if we were to use the refined network it would in fact be 2 fconvnets joined end-to-end?
When training the refined network, does the model_best.pth obtained contain BOTH these fconvnets? or does it only contain the 2nd fconvnet that uses the results of the 1st one?
In the case of training the model for car, pedestrian and cyclists, the number of classes for fconvnet would still remain 2 right (object, background)? Is the classification then taken from the 2D object detection model?

Cheers!

All you said is right.

Yes. We use prediction of the 1st f-convnet as the input of our 2nd f-convnet during evaluation. The  command of python kitti/prepare_data_refine.py --car_only --gen_val_rgb_detection save the prediction of the first stage. The whole pipeline includes two f-convnet networks.
Only contain parameters of 2nd f-convnet.
Yes. We encode class prediction from 2D detection as one-hot label  and use it as the intermediate features in our network. So, we only need to distinguish foreground and background. Actually, we found in our experiments the RGB features are more distinctive than point clouds.

Thanks for the quick reply back 👍

Hi!
Thanks for sharing this information. @sharma-n @zhixinwang
I have some further questions about this issue.
If the classification directly taken from 2D detector, doesn't this mean that frustum-convnet does not train category classification at all? So this architecture only trains localization?
For training, do you use the ground truth 2D bounding boxes and ground truth classification to get the frustums as input?

From what i understand, FConvNet is responsible for the 3D box regression. The two classification classes of background/foreground allows us to get some kind of "confidence score" from FConvNet.
You could also potentially train 3 separate FConvNets for each class (slightly higher accuracy?). This would require you to pipe them accordingly during inference time based on the 2d detection. But you'll be using 3 times the GPU memory to keep the 3 networks loaded.
And yes, to train the FConvNet you'll need 2D detection boxes and classifications. Once you have that, the code provided by @zhixinwang automatically calculates the frustums. Also, @zhixinwang also already provides those rgb detections for the KITTI dataset (thanks!), stored in frustum-convnet/kitti/rgb_detections/rgb_detections_train.txt

@sharma-n  Thanks for your quick response and answer!!


add private drishti_gltest library for portable windowless opengl context

A number of changes related to gpu testing, related code and cmake cleanup, the ogles_gpgpu dependency, and silent GTest logs

Add drishti_gltest for portable windowless GLContext

GLFWContext : public GLContext (NOT (IOS OR ANDROID))
QGLContext : public GLContext

global query replace in filenames, C++, and CMakeLists.txt: ’s/qtplus/gltest/g’
disable qt + mobile (IOS OR ANDROID) GPU gtests for drishti_{hci,acf,drishti}
control clang formatting w/ on/off switches
disable logging during gtests

add —silent option to turn loggin goff
add option to name logger for FaceTracker
remove unused threshold shader param for Nms2

add simple test-ogles_gpgpu for ogles_gpgpu sanity test (link, etc)
ogles_gpgpu package: add find_dependency(OpenGL)
prefer lower case for local (i.e. not option) vars in some cases
rework some preprocessor logo: prefer #if defined(SOME_DEFINE) to #if SOME_DEFINE (is 0 or NOT)
indentation fixes and formatting guards // clang-format {off,on}

Clean PR + fixes origin/context == #446

Status: OpenGL linking + testing works locally, but it seems the context creation is failing for CI tests. I'll improve testing in the ogles_gpgpu layer, and will investigate GPU allocation/handling on Travis + Appveyor.

GPU testing isn't currently supported on travis:
https://github.com/travis-ci/travis-ci/issues/5911
https://github.com/travis-ci/travis-ci/issues/5911#issuecomment-293615127

GLFW should provide a CPU rendering context using mesa, as of this commit:
https://github.com/glfw/glfw/issues/281#issuecomment-283162003
We can do:
glfwWindowHint(GLFW_CONTEXT_CREATION_API, GLFW_OSMESA_CONTEXT_API)

It will search for libMesa.{dylib,so}.  Will this be compatible with glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE);?
Note:

OSMesa: As its name implies, an OpenGL context created with OSMesa does not update the window contents when its buffers are swapped. Use OpenGL functions or the OSMesa native access functions glfwGetOSMesaColorBuffer and glfwGetOSMesaDepthBuffer to retrieve the framebuffer contents

Installing libMesa from source (autoconf) on Mac OS Sierra is not trivial.
https://mesa.freedesktop.org/archive/mesa-17.0.0-rc1.tar.gz

Installing libMesa from source (autoconf) on Mac OS Sierra is not trivial

Same for Linux, this is the reason why we have to run apt-get install libgl1-mesa-dev. I can take a look if we need this in Hunter though.

Will this be compatible with glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE);

It may not be possible.  Even on ubuntu where mesa can be installed easily, glfwInit() fails. This post suggests a window basedssh -X connection is required.
https://github.com/glfw/glfw/issues/696#issuecomment-176353504
I'll do some more digging.

http://stackoverflow.com/q/35092531

This recent Mesa extension - MESA_platform_surfaceless - provides a native display independent of any native window system. Given this solution already works and is testable via CI, we can pursuit this as a follow-up work.

https://github.com/mapbox/mapbox-gl-native/issues/7020#issue-188742720

Portable SW only OpenGL + CI testing (without GPU) will require a bit more work.  There seem to be at least two different approaches for this, both of which rely on Mesa3D.  I've experimented with each, starting on an Ubuntu Trusty server, and haven't yet managed to get either one working.  These are roughly outlined below:
Approach 1: GLFW + MESA

glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE) + glfwWindowHint(GLFW_CONTEXT_CREATION_API, GLFW_OSMESA_CONTEXT_API) + Xvfb (local x SW X server)
seems to be the approach used by mapbox folks for OpenGL CI testing
Tested on Ubuntu Trusty but seemed to hit an Xvfb error

Approach 2: MESA w/  EGL_MESA_platform_surfaceless

http://www.phoronix.com/scan.php?page=news_item&ppx=mesa-surfaceless-spec
https://github.com/mapbox/mapbox-gl-native/issues/7020

The core issue of CI + OpenGL testing is somewhat orthogonal to the changes in this PR.  I'll disable OpenGL tests for CI builds (relying on local OpenGL testing for now) so this PR can be merged.  I'll create a separate issue to track the CI + OpenGL requirements above.  Ideally, this will be a separate library that can be shared by drishti and ogles_gpgpu.  It may be some combination of:

https://github.com/Uroc327Mirrors/mesa3d : EGL_MESA_platform_surfaceless ...
https://github.com/glfw/glfw: (OS X, Linux, MSVC)
https://github.com/brackeen/glfm (iOS, Android)


Curved Linechart displays values below the x-axis when there's no negative value

Description
When I pass in relatively small values, everything seems okay, but when the values are bigger and far apart, the curved version adds a dip before moving up
Also, datapoints are far out of the screen and I don't know how to make sure there's some top padding.
Without the curved prop

With the curved prop

PS: I intentionally added a noOfSectionsBelowwXAxis so this can be observed properly. Without it, you can still see that the line drops below the visible view.
Looking forward to any help. Thank you.
Steps to reproduce

Use this line chart:

              <LineChart
                curved
                isAnimated
                data={incomingPaymentTrends}
                data2={outgoingPaymentTrends}
                maxValue={Math.max(
    ...[...outgoingPaymentTrends, ...incomingPaymentTrends].map((i) => i.value)
  )}
                // mostNegativeValue={0}
                noOfSectionsBelowXAxis={1}
                xAxisLabelTexts={ ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]}}
                formatYLabel={formatToAbbreviation} // this can be removed but I'm leaving it so you can see how i use it
                height={400}
                thickness={5}
                spacing={60}
                initialSpacing={20}
                endSpacing={0}
                color1="#4880FF"
                color2="#FF9000"
                textColor1={"white"}
                textColor2={ "white"}
                dataPointsWidth={6}
                focusEnabled
                showTextOnFocus
                showDataPointOnFocus
                dataPointsColor="#fff"
                textShiftY={-2}
                textShiftX={-10}
                textFontSize={13}
                xAxisColor={"white"}
                yAxisColor={"white"}
                xAxisLabelTextStyle={{color: "white"}}
                yAxisTextStyle={{color: "white"}}
              />

Plug in these values:

{"incomingPaymentTrends": [{"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "7.1K", "value": 7058}, {"dataPointText": "38.9K", "value": 38891}, {"dataPointText": "140", "value": 140}, {"dataPointText": "94.3K", "value": 94287}]}
{"outgoingPaymentTrends": [{"dataPointText": "5K", "value": 5020}, {"dataPointText": "48K", "value": 48000}, {"dataPointText": "12M", "value": 12008500}, {"dataPointText": "", "value": 0}, {"dataPointText": "6.8K", "value": 6800}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "", "value": 0}, {"dataPointText": "5K", "value": 5000}, {"dataPointText": "44K", "value": 44000}, {"dataPointText": "14.1K", "value": 14100}]}

Snack or a link to a repository
No response
version of react-native-gifted-charts
1.4.7
React Native version
0.76.3
Platforms
Android, iOS
Workflow
Expo Go

I see that my scenario is the same as this issue and it fixed there by adding
curveType={1}
i.e. (CurveType.QUADRATIC which I can't seem to import). Thank you for the great library @Abhinandan-Kushwaha


remove https from sandbox config

Sreekumar, I saw this request just after I had made the changes. Github unfortunately won't let me apply the pull request. But thanks again for your time in providing us feedback and helping us.
Thanks,
Abhishek


Expand DockWidget API and add hookspec?

🚀 Feature
In thinking about trying to integrate well-established community projects, like ilastik or deep lab cut, (or at least small modular parts of them)... we are, of course, a long way away from a full functional hook-based plugin architecture that would support all of their UX needs.
I'm wondering if we couldn't expand the dock widget API as a "loose" type of plugin that would immediately meet some needs, while we consider development of stricter hook-based plugins in parallel.  Specifically, when someone runs viewer.window.add_dock_widget, we pass back the QtViewerDockWidget object... and at the moment, it is a very thin wrapper a plain QDockWidget.  However, it would be pretty easy to establish an API and write up some documentation that says "here's how you access the viewer that your widget is docked in"... and here's how you can use it, and perhaps inject some useful properties into their QWidget class.
We could define a new plugin hookspec that allowed plugins to declare these widgets, along with some metadata that would let us put a new item in the Plugins Menu, and let the end user select it and get that new dockwidget.
It's certainly not as "locked" down, and it does require more from developers (they'd need to pass a QWidget... or a function that we run magicgui on?), but I think we'd quickly get some very cool components (based on the many cool community dock widget examples we've seen on twitter and image.sc)
Motivation
Letting advanced developers extend the napari gui with components that are well-beyond the current plugin design.
Pitch
# the spec

@napari_hook_specification
def napari_provide_dock_widget(): -> dict:
    """Return a dict that can be passed as kwargs to viewer.add_dock_widget"""

# the plugin

class MyWidget(QWidget):
    ...

    # these things would be true once added as a dock widget
    # we could maybe "inject" these properties for plugin developers too
    @property
    def dock_widget(self):
        return self.parent()

    @property
    def viewer(self):
        return self.get_dock_widget().parent()

@napari_hook_implementation
def napari_provide_dock_widget():
    return {
        'widget': MyWidget,
        'name': 'Cool Tracking Plugin',
        'area': 'right',
        'allowed_areas': ['right', 'left'],
        'shortcut': Alt+Shift-T,
    }

This is exactly what I want to do for some projects of mine, so I'm happy to see things moving in this direction! Can I try working out an initial implementation?

@tlambert03 Love it! One modification I would suggest is:
@napari_hook_specification
def napari_provide_dock_widget(): -> Tuple[QWidget, dict]:
    """Return a dict that can be passed as kwargs to viewer.add_dock_widget"""

To ensure that a widget is always provided.

This is exactly what I want to do for some projects of mine, so I'm happy to see things moving in this direction! Can I try working out an initial implementation?

Sure that would be great!
The only non-obvious thing I might suggest is to implement it as a “historic” spec (and call it with call historic)... similar to how I wrote this one: https://github.com/napari/napari/issues/1272#issuecomment-633064212

I have fears here about opening up the plugin API too quickly if we start accepting Qt widgets that can directly access the viewer is the way you describe - see comments from @royerloic here https://github.com/napari/napari/issues/1272#issuecomment-633697572 about insulation.
It basically means we have no control over backwards compatibility beyond the full API of napari (as opposed to the reader/ writers which are insulated).
On the other hand I see the immediate use cases, arguments that GUI's a different, and I can see an argument that a more flexible/ open plugin interface at this time is better than no plugin interface at this time, and then we can always add in more insulated ones
My big concern is that plugin developers will start to hate us and we'll get a reputation for instability if we keep breaking our APIs that affect plugins, and there are so many major refactors in the words right now (event handler, layer groups, world-coordinates, multi-canvas to name a few) that it seems hard to avoid.
I guess right now there just has to be some shared understanding that if you're adding a Qt element to a plugin you're a little more advanced, and we should make a space for the more advanced users.
Maybe even just making this napari_provide_qt_dock_widget to make it clear that this isn't going the magic gui route would make me feel better about this.
Final question - if a plugin wants to provide multiples doc widgets can this return a list?

I'm still not 100% sure I agree with the comments about insulation made in #1272.  I feel like there is a little bit of an illusion going on about exactly what we can and can't prevent users from doing in python.  Moreover, this proposal does almost nothing that isn't there already.  By making something a dock widget, it already can find the viewer (by just looking at the widget's parent).  So pretty much the only thing we're doing here is saying "hey, got a widget?  use this hook and if a user installs your plugin, will put it in the plugin menu for you".
it's the same as someone doing
pip install napari some-plugin

import napari, some_plugin

with napari.gui_qt():
    v = napari.Viewer()
    v.window.add_dock_widget(some_plugin.Widget, name="Plugin Widget")

I am sensitive to the longevity argument for sure... but it's hard to even conceive a way in which we could eventually enable complicated GUI extensions in an insulated way such that developers aren't still just passing us a QWidget... so we might be waiting an extremely long time for that (without a professional team of full-time coders).  And the flipside of the stability argument is the idea that people will figure out their own ways to do this, and there will start to be as many different ways to do it as there are people trying to do it.   (remember, we can't control the use of the word plugin... and there are already github packages that call themselves "napari plugins" that don't use hooks... and each of them is going to have a different "install" procedure in their readme).   And then it might not be the plugin developers that hate us, but it could be the community that gets frustrated by heterogeneity...

(remember, we can't control the use of the word plugin... and there are already github packages that call themselves "napari plugins" that don't use hooks... and each of them is going to have a different "install" procedure in their readme)

We can control though what get's installed into the bundled app through our "add plugin" dialog, and thinking 2 years from now that's how I see most biologists using napari + plugins, and it's that user experience that I think needs to be really robust and not a wild west (like so much of the tools we use is)

I feel like there is a little bit of an illusion going on about exactly what we can and can't prevent users from doing in python. Moreover, this proposal does almost nothing that isn't there already.

The distinction is really about what can a plugin that ultimately gets installed via our "add plugin" dialog can do, not about what is possible more broadly. For people that build their own "plugins" that use napari as a component of half and half etc that people install themselves we have no control over, and so in many ways for them we don't really need a plugin interface/ hook specs, they can just call add_dock_widget themselves.
Now I do appreciate the complexity of GUI widgets, so having a hook spec with qt in the name seems the right middle ground for where we are now, and if we ever migrate away from that, deprecate that it will be very clear why - because we moved to an API that didn't allow passage of a qt object

yeah, I don't want wild west either.  So definitely happy to keep this conversation going and get more input from all @napari/core-devs
but while we're talking about what a plugin can and can't do that get's discovered via some hookspec, don't forget that we are importing their module to find their hook implementations.  And once we run importlib.import_module("some_plugin"), i hate to say it, but they can pretty much do or monkeypatch whatever they want!  Again though, I do appreciate @royerloic's point that "at least they know they're being naughty" ... and that's super important.  But there's a big spectrum of just how naughty they feel like they're being.

My big concern is that plugin developers will start to hate us and we'll get a reputation for instability if we keep breaking our APIs that affect plugins, and there are so many major refactors in the words right now (event handler, layer groups, world-coordinates, multi-canvas to name a few) that it seems hard to avoid.

Totally.
In this type of conversation I always return to the @torvalds commitment for Linux, “don’t break user space.” It’s especially wise for scientific software where code should be able to run, without fuss, decades from now. Furthermore, it has the benefit of simplify decision making by weighing the benefits of a proposed feature against the possible complications of maintaining the “don’t break user space” commitment, e.g., in this spirit, does this PR make it harder to transition to a new version of Qt? A different widget framework? A different execution environment?

Those are all good points @sofroniewn
Can you clarify what you mean by "going the magicgui route"?

Can you clarify what you mean by "going the magicgui route"?

This would mean that we accept python functions with typed signatures and then internally we use magicgui (or some similar process) to generate the qt elements.
Note here that plugin developers are not using magicgui and do not depend on it, napari is using magicgui, and so when the qt5 -> qt6 release happens (which will be in 2020!!!) we just have to update magicgui and all our plugins keep on working. Similarly in 2 years if we swap to a web-based UI that calls python code we just make a new version of magicgui that makes web-based UI elements and everything keeps on working.

Can you clarify what you mean by "going the magicgui route"?

This would mean that we accept python functions with typed signatures and then internally we use magicgui (or some similar process) to generate the qt elements.
Note here that plugin developers are not using magicgui and do not depend on it, napari is using magicgui, and so when the qt5 -> qt6 release happens (which will be in 2020!!!) we just have to update magicgui and all our plugins keep on working. Similarly in 2 years if we swap to a web-based UI that calls python code we just make a new version of magicgui that makes web-based UI elements and everything keeps on working.

So you think the magicgui route is good or bad?
(Your first comment made it sound like you thought that would be bad, but this comment sounds like you think it's a good idea)

So you think the magicgui route is good or bad?

All of this is more nuanced than good or bad for me, I really can see pros/ cons to all the approaches, but I probably prefer a magicgui-like approach where Python functions/ objects are passed instead of full Qt objects, but I'd want to discuss more the current method of passing layers directly to magicgui before solidifying that in the plugin architecture.

yeah, I see these as two independent things, which both have their merits and which both may be implemented at some point.  constructing guis for people (using, e.g. magicgui) definitely has the advantage that we can swap out the backend anytime, and I do see that as a longer-term goal.  For the moment, actually connecting signals/interactivity would be rather hard with magicgui if we do the wrapping for them... (all of the magicgui examples connect events after the widget is created).  If we accepted a class, then they could declare those connections.  So maybe someday a magicgui-like solution will be great here.
this is all why I stated in my first post:

expand the dock widget API as a "loose" type of plugin that would immediately meet some needs, while we consider development of stricter hook-based plugins in parallel.

If we want to abstract all of the interactivity that someone could achieve by passing us a QWidget (instead of us building it for them), then we're basically rewriting Qt ourselves.  It's absolutely a "desirable" goal... I just don't see it as achievable.  And maybe that's fine, maybe we just don't really support rich extension of the gui like that.
If it would help, i would (literally) not be opposed to this hookspec being called:
@napari_hook_specification
def napari_provide_qt_widget_that_is_not_guaranteed_to_be_supported_in_the_future():

@tlambert03 A common approach in cross-platform applications that need to stay agnostic about their underlying widget toolkit is to shim (and use) just HTML widgets, e.g. input and select. It provides a simple common denominator that is more than likely going to match semantics with whatever widget toolkit you’d consider using.

So I think what I'm most concerned about is what you might call the lowest common denominator use case for plugins. That is something that can:

Install into the bundled napari app
Add itself to the plugin menu list, so users can click to start the plugin
When the plugin is launched it should add a little GUI, so the user can select the input layer(s) plus any other needed parameters
Allow the user to be able to click a button to run it, and spit back (or save) a layer with results

I'd be pretty happy if all of this was possible in a very structured/limited way using hookspecs.
But so far I don't think we have a hookspec to take in layers, compute something, and return a result as a layer; and we definitely don't have a way for users to select parameters from a plugin GUI.
We do have writing our own scripts with magicgui that then call napari, but the conda setup is still tricky for users. But fairly frequently I have conversations with my colleagues where someone is stuck because they're not super familiar with conda and can't quite get the setup working. The two biggest things that would improve this are (1) simplifying installation with app bundles, and (2) giving them a plugin GUI they can click.
Do we have a way to move forward on this type of use case, or is everything on hold until we work out the overall architecture?

Do we have a way to move forward on this type of use case, or is everything on hold until we work out the overall architecture?

I would hold on this for a little, it will certainly need some dedicated discussion.

A few thoughts from someone who's trying to build an extension for napari.  One of the things that has been so useful with napari, is that we can easily customize it directly for different tasks by bolting on extra pieces of code that augment the functionality. I really believe that's one of the true strengths here - so features like the add_dock_widget have been invaluable.
However, i'm totally convinced by the 'dependency hell' argument against such extensions being installed directly through napari - it seems that is going to be the biggest challenge of all. Some insulation/containerization could help, but my experience is that those come with performance overheads, especially when dealing with large datasets.
Perhaps you could have something akin to a napari extension zoo, where extensions (not plugins) live and are discoverable.  These extensions would make use of an API similar to the one Talley describes, but with the caveat of needing certain environment to work (not dissimilar to the kernels idea in jupyter notebooks).
BTW - I removed the napari-plugin tag for arboretum in light of these discussions, since we're not using any of the existing hookspecs.

Perhaps you could have something akin to a napari extension zoo, where extensions (not plugins) live and are discoverable.

I think this makes a lot of sense to me. You see similar concepts in a lot of different platforms that support 3rd-party tooling... On Github, for example, you have Apps (which have a very wide surface, but are siloed from one another) vs Actions (which are more restrictive, but offer compatibility and composability). Lots of platforms separate Themes from Extensions from Recipes.
So yeah, having a distinct concept like an "Extension" which is more restrictive in what the extension developer can do, but offers a stronger guarantee of compatibility with other Extensions could be highly valuable for plugin/extension developers and end users.
In napari-land, I could imagine things like Tasks (take in data & parameters, and return data) or Components (specialized viz or UI) or Apps (a bundle of Tasks & Components which work together) or Workflows (a DAG of Tasks). They would have different areas on the Zoo or App Store where users find them. On the implementation-side, they could all use the "plugin" API, but would be defined in part by which part of the plugin API they are restricted to touching (perhaps a restricted set of allowed dependencies, too).
And at the end of the day, an enterprising developer could still build a monolithic plugin, incompatible with the rest of the ecosystem by ignoring those conventions and working directly with the plugin API and telling their users to install via pip.

Oh wait, I think I inverted @quantumjot 's Extension/Plugin relationship. Extension = less restrictive API surface, more restrictive allowed dependencies. Still, the broader concepts of different flavors of "plugins" with different rules could be super super valuable.

Ok, sounds like wording is very important here, and everyone is a bit more comfortable opening up various APIs as long as we protect the word "plugin".  I can certainly get behind that.  I do think we need some sort of avenue to promote the amazing work that early adopters are doing, and I genuinely do think we will kill some of the buzz/excitement if we make everyone wait until we come up with some abstraction between the viewer API and some imagined extension API (that may or may not ever come). so how about we "formalize" the concept of a napari extension:
Proposal for napari "extensions":

extensions are marked with a @napari_extension decorator imported from napari-plugin-engine (behind the scenes they will be registered using napari-plugin-engine PluginManager ... but that needn't have any implications beyond the fact that it would be silly to replicate all that machinery)
(they could be labeled with napari-extension in github)
Extension hooks WILL be passed napari objects, like Viewer and Layer, but will NOT be hooked into of the public napari "API" (in other words, an extension would never modify the behavior of a napari function like Viewer.open, whereas a plugin might).
We are unlikely to "use" data returned from an extension, instead, they are expected to call the native APIs of the napari objects they are given.
we make it clear that it is up to the extension author to maintain compatibility with the napari API.  We make no guarantees that changes to napari will not break extensions.
we can provide at least some way for an extension to declare what version of napari it was designed for... but no runtime checking will be performed.

I think the hookspec proposed in my first post here would be a reasonable place to start with extensions.  They provide a widget, and some metadata, and we will make it visible in some "extensions" menu (we can create a seperate menu for extensions or a sub-menu of plugins, doesn't matter to me) that a user can find.  this would already be great for @quantumjot's stuff, the napari-omero, etc...
thoughts?

Hi @tlambert03 thanks for keeping this thread going. Following on, here are a couple additional thoughts on how people could maybe build on top of napari, thinking more from the user/ builder perspective rather than implementation perspective. Note that what follows are just some proposals and ideas, not anything written in stone:
Extensions
I see extensions as interactivity focused. They allow people to customize the interactivity napari provides, provide their own GUI elements. You wouldn't use an extension "headless" because there is no need for custom plots, mouse clicks, or key presses "headless". Extensions are additive, and you're generally only using one or two at a time and explicitly turning them on or off when you want to use them.
I see extensions as being able to do four main things:

provide custom dock widgets (with say custom plots or data displays)
provide custom mouse functions
provide custom keybindings
connect into our event system to provide callbacks

You might have 50 extensions installed, but only 2 enabled A and B. Extension A might provide 3 dock widgets and 2 mouse functions, and 3 callbacks. Extension B might provide 2 dock widgets, 1 mouse function, and 3 keybindings, and no callbacks. These would be combined additively so that you'd have access to 5 dock widgets, 3 mouse functions, 3 keybindings, and 3 callbacks.
Examples of extensions could now be

napari-console if we pull the console out of the main repo
napari-movie see https://github.com/guiwitz/naparimovie and by #780 @guiwitz
https://github.com/hci-unihd/covid-if-annotations by @constantinpape
https://github.com/quantumjot/arboretum by @quantumjot
https://github.com/SainsburyWellcomeCentre/cellfinder by @adamltyson
https://github.com/tlambert03/napari-omero

Note here would these extensions be able to provide an actual Qt widget - what if it is PySide2 vs PyQt5, what happens during the Qt6 transition in December? Are all these extensions provided the full viewer object? Will that lead to extensions breaking when they are used together as they will be trying to do incompatible things. Can we pass only the ViewerModel to our extensions, not the full Viewer (but that might not allow functionality like screenshot). If we simplify our models like in #1353, might be only provide subsets of the models to the extensions?
Ideally extensions would be able to customize stylings and theme, but that conflicts somewhat with the "additive" nature of extensions, now we'd need a priority status.
Plugins
I see plugins as IO/ analysis focused. These should be pure python and highly independent of any GUI elements. They could often be described as python libraries. They probably need to understand napari-types like images, labels, shapes etc. but not napari specific objects. They should be modular, composable, and easily runable "headless" as they are pure python.
I see the two main types of plugins as doing

IO, taking a path and providing a list of layer type data or vice versa, taking a list of layer type data and saving it to a path
Analysis, taking a list of layer type data operating on it and returning a list of layer type data.

You might have 50 plugins installed each of which may provide many of those IO or analysis functions that are callable from the GUI (either from drop downs, or very simple autogenerate GUI elements, which allow additional parameter values to be passed). Note here all drop downs are autogenerated, all GUI elements are autogenerated and focused around parameter input (and not say custom plotting, which would appear in an extension).
Examples of plugins could now be

scikit-image
https://github.com/AllenCellModeling/napari-aicsimageio by @JacksonMaxfield
https://github.com/manzt/napari-lazy-openslide by @manzt
https://github.com/tlambert03/napari-dv
https://github.com/transformify-plugins/segmentify

Note here these plugins are functional and stateless so no napari objects like layers or the viewer would pass through them, only the associated data.
Workflows
I see workflows as simple python scripts that describe the chaining together of IO and analysis calls (which use plugins as needed). They should be pure python, runnable headless, and the CLI tool to run them should just be python. They should be recordable from the GUI (like a "macro recorder", like ImageJ) or maybe one day they can be made via a graphical programming language (like "protocols" in Icy), but they should always be valid runnable python code. These would allow a user to a workflow composed of many plugins, with say registration, segmentation, and tracking steps.

How does it all fit together?
I can see some problems where someone might be building a tool that wants to provide both extension-like and plugin-like functionality, and maybe also some workflows. Should that be allowed in the same repo. Probably from a convenience standpoint, yes, but I'd prefer a world where not all our plugins lived in repos that also had GUI code/ pyqt dependencies etc as the goal is to make install of that super robust everywhere.
I'm still a little nervous about letting an extension return a Qt widget that has the full viewer on it, as I worry that in a couple years we'll have lots of extensions that no longer work as Qt has changed or our viewer has changed, but I do see the value in opening up this approach now as extensions will be installable into the bundled app and I think for that reason are preferable to people using napari as a component in their own tool which then must be installed and run completely independently.

I'm also interested in add plugin system for widget. Maybe good idea to be use Protocols and TypedDict? So:
From my experience I prefer to use Protocols with classmethod than description in additional structure. runtime_checkable decorator allow to check which protocol is implemented.
Additionally napari does not use all popular widgets, and unused one are styled symilary to plain QWidget. There should be some instruction how to fix it and how prepare style matching others and some hook for "Toggle Theme"
And maybe some instruction to create additional icons which will match napari style (you spend big effort to get clean and nice view of napari window).
And build more complex plugins need better API stability.


can not sync for timestamp for mysql with default value

This is the mysql table
desc test;
+-----------+-----------+------+-----+---------------------+-------+
| Field     | Type      | Null | Key | Default             | Extra |
+-----------+-----------+------+-----+---------------------+-------+
| update_at | timestamp | NO   |     | 0000-00-00 00:00:00 |       |
+-----------+-----------+------+-----+---------------------+-------+

this is the hcl file:
table "test" {
        schema  = schema.test
        column "update_at" {
                type = timestamp
                default = "0000-00-00 00:00:00"
                null = false
        }
}
schema "test" {
        charset = "utf8mb4"
        collate = "utf8mb4_unicode_ci"
}

with latest version atlas, alway ask to change table:
atlas schema apply -u<EMAIL_ADDRESS>-f test.hcl -s test
-- Planned Changes:
-- Modify "test" table
ALTER TABLE `test`.`test` MODIFY COLUMN `update_at` timestamp NOT NULL DEFAULT "0000-00-00 00:00:00";
Use the arrow keys to navigate: ↓ ↑ → ←
? Are you sure?:
  ▸ Apply
    Abort

Hey @calvin2021y, and thanks for reporting this issue 🙏
https://github.com/ariga/atlas/pull/1437 should fix this, but note that you can also use the --dev-url option to normalize your schema before diffing and applying it on the database to prevent cases like this. You can read more about it here: https://atlasgo.io/concepts/dev-database


Fix order_independent_hash on 32-bit platforms

Closes #1913

Sorry, I missed this until the merged notification.  Good that it works, but it might be a good idea to still use 64-bit hashes on 32-bit platforms.


Add 1.3.2 version to main causing gradle check failures

Signed-off-by: Suraj Singh<EMAIL_ADDRESS>Description
Add 1.3.2 in main post 1.3 version bump to 1.3.2 here
Issues Resolved
Existing gradle check failures on main
* What went wrong:
Execution failed for task ':distribution:bwc:bugfix:buildBwcLinuxTar'.
> Building 1.3.1 didn't generate expected file /var/CITOOL/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/search/distribution/bwc/bugfix/build/bwc/checkout-1.3/distribution/archives/linux-tar/build/distributions/opensearch-min-1.3.1-SNAPSHOT-linux-x64.tar.gz

Sample impacted PR: https://github.com/opensearch-project/OpenSearch/pull/2601
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dreamer-89 Could you please backport these to 2.x and 2.0? Thank you!


CLI should require username and password for authentication

This would be useful both for ensuring that the requests are bing made by a valid user and to associate any triggered events with the authenticated user.

Implemented in #32


Add "callee" scope type

This scope type would select the function being called in a function call, so eg in foo(bar), it would select foo.  We'd also use this one for the tag function in Javscript tagged templates, as proposed in #641.  So saying "take callee" within either of the following lines should take the foo on the given line:
foo(bar);
foo`hello ${bar}`;

I'm interested in attempting this. I assume for a new scope I'd need to add to the talon capture as well as each language file. Anywhere that existing ScopeType's are used will be somewhere to check as well as of course the ScopeType definition itself. Is there anything non-obvious I should know to update or check?
Also, I'm concerned that "callee" might conflict with existing scope "call" for talon recognition, I'll try to think of an alternative default if that's alright?
Another question, are you sure this shouldn't be an extension of the functionName scope? i.e. "take funk name bat" to take the foo in foo(bar)

I'm interested in attempting this. I assume for a new scope I'd need to add to the talon capture as well as each language file. Anywhere that existing ScopeType's are used will be somewhere to check as well as of course the ScopeType definition itself. Is there anything non-obvious I should know to update or check?

Nope that all sounds good to me

Also, I'm concerned that "callee" might conflict with existing scope "call" for talon recognition, I'll try to think of an alternative default if that's alright?

Fine by me

Another question, are you sure this shouldn't be an extension of the functionName scope? i.e. "take funk name bat" to take the foo in foo(bar)

We pretty much always use "name" to refer to the place that the given name is defined, rather than a reference to it. Also, it's possible that the callee is not a name. Ie it could be the return value of a function, or it could be a chain, etc
That being said, I don't feel super strongly, and these distinctions can sometimes be too subtle to register when coding fast cc/ @AndreasArvidsson @wenkokke @Will-Sommers

Could you give me an example of what you mean by it could be the return value of a function?

Could you give me an example of what you mean by it could be the return value of a function?

Sure: if you have the expression foo()(), then there are two nested function calls. The outer one is calling the value returned by foo() as the function. So saying "callee" from within the second pair of parens should select foo(). Fwiw, the expression is equivalent to
const bar = foo()
bar()


GET, POST, DELETE, PUT and PATCH now check error status

This is to address issue #28.

I'm currently knee deep in some other projects. This is why work on Traverson has come to a halt for the last two month. I'll start working on Traverson again soon (~2 weeks) and will have a look at it then. Sorry for the delay.

Thanks for the PR! <3
This one had tabs instead of spaces - please try to keep PRs aligned with the existing code style :-)  I added an .editorconfig that might make this easier, if your editor of choice supports it.

this has been published with 1.0.0


How to wire-up JSON-RPC manually to a "source/conduit"

Not really an issue, but don't know where else to ask. Any help with http://stackoverflow.com/questions/35011946/how-to-wire-up-json-rpc-manually-to-a-source-conduit will be really appreciated.

Look at the examples. They should clarify how to do this. Read this manual if you still do not understand.


Use react-native.config.js

I have used this plugin. but I have an issue.
When I purchase successful but It throws to catch the exception. I don't know why. I think this is library problems.
Any idea helps me. Thanks a lot.

Thanks for the PR @superandrew213
Have you tested this locally and see that it works?

@cbrevik yes. I used this to convert from rnpm to react-native.config.js.
https://github.com/react-native-community/cli/blob/master/docs/configuration.md#libraries
I still left the rnpm settings.

Thanks, looks good to me 👍


Add register-js! for registering js libraries

Docstring as example:
Given some string libname and a JS namespace object (and an optional symbolic alias), makes the JS namespace available to SCI.
For example, given a namespace with (:require [\"react\" :as re]):
(register-js! \"react\" re)

Makes it possible to call
(clerk/eval-cljs '(require '[\"react\" :as react]))

in a notebook.
Calling
(register-js! \"react\" re 'react)

makes the react alias available to every consumer of the SCI environment.

Good catch that we didn't expose react. We already had a different way to expose existing javascript classes to sci so I've used that in the attached commit. You can also require react now but we don't need users to make the register call before hand. Let me know if this solves your original problem!

Maybe I shouldn't have picked this up before my first coffee. Looking at your commit again, you're going through libname->class so your problem is more about write access to this mapping to expose libs other than react? In that case I think I'm in favor of turning libname->class into an atom (plus maybe find a better name).

The goal of this PR was to let people add JS libraries to the SCI config. It was not about react specifically. Instead of the eval-form the helper function can also call load-fn directly.

@borkdude should turning libname->class into an atom as suggested above also enable letting people add JS libraries to SCI? There no sci concept for this we could use here, correct?

The SCI concept is already used in the load-fn.

If we gloss over what is currently in the :load-fn:
(defn load-fn [{:keys [libname ctx opts ns]}]
  (when (contains? libname->class libname)
    (let [{:keys [as refer]} opts
          munged-libname (symbol (munge libname))
          klass (libname->class libname)]
      (sci/add-class! ctx munged-libname klass)
      (when as
        (sci/add-import! ctx ns munged-libname as))
      (doseq [r refer]
        (when-some [prop (j/get klass (name r))]
          (let [sub-libname (str munged-libname "$" r)]
            (sci/add-class! ctx sub-libname prop)
            (sci/add-import! ctx ns sub-libname r))))
      {:handled true})))

To add a class by name into the SCI context you can use:
(sci/add-class! ctx munged-libname klass)

To add an import (here used for alias) to a namespace in the context you can use:
(sci/add-import! ctx ns sub-libname r)

I was more asking if we can unify how folks extend the sci env for regular namespaces vs js classes. For regular js namespaces you go through the global sci context. It might be nice to also go through this route for regular js classes to have this state in one place?
One idea might be to store the (mutable) libname->class map in the sci context?

I'm afraid I don't understand what you mean. Unify where, in clerk? What is "regular js namespaces" vs "regular js classes"? What do you mean with "go through the global sci context". I don't see why you need libname->class at all if you want to extend it yourself, but the above post by Sam was just a quick hack based on what was already present. With sci/add-class! and sci/add-import you can do it yourself as well.

I mean (sci) namespaces like reagent.core and js classes like "@codemirror/api" (how I would require them.
We currently provide sci namespaces when setting up the context, and js classes in load-fn. There's no place (afaik) currently to store a mapping of (unmunged) string names to js classes in the sci context.
Or should I just add the munged symbol to :classes and things will work without the libname->class map in Clerk?
Not sure if this clarifies things. Also happy to talk about this on a call next week.

JS libs (called classes in SCI for unifying the concept between JVM and JS) and clojure namespaces are not unified since they are two different concepts. SCI leans more towards the JVM side here where vars are mutable objects with metadata, etc. things whereas classes are host objects that are not aware of Clojure. SCI has an API to add classes to the context and to define imports for a namespace. This is unified between JVM and JS but I don't have plants to unify "classes" with "namespaces" as classes have different semantics.

clojure namespaces are not unified since they are two different concepts.

That's clear. I meant supporting passing string keys in :classes when initializing the sci context. See 7aac5bda214fa9b1481fb8bd12229bdd36493e00 in which I dropped the libname->class map and let load-fn use the already existent class. I think it might be nice if

sci supported string keys in :classes doing the (symbol (munge libname)) conversion in that case
sci would support requiring string classes out of the box so we could pass these classes without needing a :load-fn in that case

Can do a PR if you think this is reasonable.

Right, experimental PR welcome!


Setup basic python packaging

I am aware the tests are failing. Merging in so the requirements get updated

I am aware the tests are failing. Merging in so the requirements get updated


implement cacheing of the liturgical calendar by year

The liturgical calendar is calculated by year. The General Roman Calendar data has now been defined, and will not change until a new decree of the Congregation for Divine Worship will come out. Any decree that will come out will apply to future years, not past years. The same for national calendars: so far we have defined data for Italy and the United States. This data might need some small update for future years, but not for past years.
Since this data doesn't change (unless a typo is found), there really isn't any need to recalculate it each time the API receives a request. So responses will be much faster if the data is cached in a static manner by writing for example the JSON or XML files. Then the JSON result would be served immediately instead of calculating the data each time. Perhaps these JSON files could be recalculated when a change is made to the data (for example a typo is fixed), and this could be handles by versioning.
TODO: implement cacheing of calculated results in a static JSON file, which should be versioned based on the API version.

done with commit c9c0eded31f540d60b7cd245768067346d194e13 (and a36346874f85b5adf5b4819b77f6b2ba7cfc5783 - 0bb0c8b914185fed734331c35cd7cd9e835ef1fa for the calculation of the date of Easter: so much faster now!)


Is it really necessary to rely on a login shell?

The installer script suggests using the file ~/.bash_profile for bash environment settings, which is only read by login shells:
https://github.com/pyenv/pyenv-installer/blob/92c4cd2812d12c8170f4970f8d4d71e18d093bd7/bin/pyenv-installer#L53-L56
Is it not possible to use either ~/.profile or ~/.bashrc instead? They are sourced by interactive shells of all types, which seems to make things easier for the user.

I was also wondering whether to use .profile vs .bash_profile.
Any insights anybody?
(I am using Ubuntu Bionic)

Works for me flawlessly with ~/.bashrc on Ubuntu 16.04

~/.bashrc is also sourced in the default ~/.bash_profile on CentOS and also RedHat. Also on the CI front, ~/.bashrc is all that's sourced automatically on steps within CircleCI, I'm guessing it similar on Jekins and Travis, but I currently have no confirmation of that.

It looks like we can merge #74 to suggest that .bashrc as the right place in most cases then ...


The column Properties must be an Xml datatype when autoCreateSqlTable is set to true

I work on a console developed in .net core 2.0.
In the appsettings.json file, i have this declaration:
"Serilog": {
    "Using": [ "Serilog.Sinks.MSSqlServer" ],
    "MinimumLevel": "Information",
    "WriteTo": [
      {
        "Name": "MSSqlServer",
        "Args": {
          "connectionString": "Server=.;Database=Serilog;Trusted_Connection=True;MultipleActiveResultSets=true;",
          "tableName": "log",
          "columnOptionsSection": {
            "customColumns": [
              {
                "ColumnName": "EventType",
                "DataType": "int",
                "AllowNull": false
              },
              {
                "ColumnName": "Release",
                "DataType": "varchar",
                "DataLength": 32
              }
            ]
          },
          "autoCreateSqlTable": "true"
        }
      }
    ]
  }

The log table have a properties column's datatype set to nvarchar(max), not xml.

A workaround is to exexute this command :
ALTER TABLE [dbo].[log] ALTER COLUMN Properties XML

I'd strongly disagree with making this a default. Although SQL Server stores the XML data type much more efficiently than the same XML stored as text, it requires extra processing which incurs additional CPU overhead, which potentially reduces write performance -- something you definitely want to avoid when logging. The storage footprint savings are probably irrelevant given that you shouldn't be logging huge data structures. Additionally all of my deployments use JSON for structured data and I'd argue that XML is rapidly falling out of favor.

I'll tag this as an enhancement request since it comes up periodically. Once I grind through the open bugs, I'm going to start looking at new feature requests.
I think it should be implemented as an option with nvarchar remaining the default.
I'd like to start building up a Wiki for this repo since the README is getting unwieldy, and there I'd caution against the XML type in high-frequency logging scenarios.

Currently overhauling configuration and better control over column data types is part of it, including XML column support. Docs will recommend against this for write-performance reasons, most likely.


Improve cover

Coverage increased (+2.0%) to 82.581% when pulling 358693f95a62b5832c7e86c6718f30b6f096ee87 on cover into 90140edf39c1be2d533f6cbe42190b69ad01daa5 on master.

Coverage increased (+2.0%) to 82.581% when pulling ff2c6308ee4154d0a3ca89600a48a1c6973f2897 on cover into 90140edf39c1be2d533f6cbe42190b69ad01daa5 on master.


Appstudio update test-component-pac-lcul

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-lcul-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
16 seconds
appstudio-init

✅ Succeeded
14 seconds
clone-repository

✅ Succeeded
14 seconds
appstudio-configure-build

✅ Succeeded
2 minutes
build-container

✅ Succeeded
18 seconds
show-summary

Pipelines as Code CI/test-component-pac-lcul-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
appstudio-init

✅ Succeeded
14 seconds
clone-repository

✅ Succeeded
7 seconds
appstudio-configure-build

✅ Succeeded
2 minutes
build-container

✅ Succeeded
9 seconds
show-summary


fix(useDetails): update ref type

Closes #2620
Closes #2146
Changelog
New
Changed

Update type of ref from useDetails to HTMLDetailsElement

Removed
Testing & Reviewing
I couldn't figure out a good test strategy locally for this, let me know if you have any tips! I ended up having to create a distinct folder and use the published package (or link locally). Something like:
npx create-react-app details-example --template typescript
cd details-example
npm i @primer/react styled-components
npm run build

Initially, this will fail with the error from the parent issues. When updating @primer/react in package.json, this error will no longer show up when building.

@siddharthkp hope the changes make sense, let me know if I'm missing anything!


cpu support mmdet3d==0.17.1?

I install mmdet3d==0.17.1 using pip and using setup.py after git checkout this version but it is checking for cuda, this version only support cuda, it is not support support?

Some operations do not support cpu version, such as spconv etc.

with cpu i'm not able to install it, it is getting failed while installing it. Look like with gpu I can' even use?