Convert demo to standard GeoTrellis deployment framework

Overview
This PR adds a pipeline to build/deploy API Server and Nginx Containers to AWS. Also, add Terraform resources for an AWS ECS Service, using CloudFront as a CDN.
Notable changes

Update Docker/Compose
Remove azavea.* Ansible scripts from the repository (they're downloaded with ansible-galaxy at the beginning of every provisioning run)
Add a dockerized deployment environment
Add containers to serve static files and run the API server
Replace webpack-dev-server with nginx as the default way to serve static files in development
Add Terraform configuration for an AWS ECS service using our terraform-aws-ecs-web-service module
Add STRTA to necessary for build/deployment pipeline
Configure TravisCI for deployment

Notes

The JavaScript for main site is broken, and won't pass tests. Scala also throws compilation errors when compiling the ingest JAR, but not the server one. As a workaround, I've skipped JS tests, and only build the server JAR (which I think is the only one we need anyway).

I ran into some Scala dependency resolution issues, and added a Sonatype repo to get everything to work. I'm not sure if this is WAI, or if the LocationTech repo needs to be updated.

Although I wanted to keep using webpack-dev-server running in development, making the frontend work with both dev and production URLs proved to be tricky. I opted towards just using Nginx instead.

The backend service displays a small weighted Overlay, but I'm unsure if it should be larger than it is. @lossyrob can you confirm that the backend is functioning?

Testing

see https://pointcloud.geotrellis.io
See Travis CI output
in development, provision a VM, and run scripts/server. Nginx should be available at http://localhost:8000.

I also hit http://localhost:8000/ping to be sure that nginx is rewriting URLs properly.

Fixes #11

Having trouble running this. I ran scripts/setup.sh, and then inside the vm scripts/cibuild.sh and scripts/server.sh. Navigating to localhost:8000 I'm getting:

Am I skipping a step?

I don't think so; I just rebuilt my VM from scratch and was able to run scripts/setup.sh and scripts/server.sh, then hit localhost:8000. scripts/setup.sh runs scripts/update.sh; did any part of that script fail? It seems like the JavaScript bundle didn't build properly.

@lossyrob Just wanted to circle back and see if you were able to fix your Dev environment issues. Did you have any success after running scripts/update again?

Thanks for looping back - I didn't have a watch on on this repo until today, and so didn't see your comment. The Azavea GeoTrellis team is putting this on their next sprint - and if someone else doesn't pick it up I'll be able to take a look when I get from State of the Map travels next week.

Encountered the same error as @lossyrob but after vagrant destroy vagrant up ./scripts/setup.sh and ./scripts/update.sh inside the container I was able to see the index.html but the page is unresponsive and the service is is throwing 500 which I have not investigated yet.
I had to bump the dependencies for geotrellis-pointcloud to newly published branch. I assume you guys must have been doing publish-local from feature branch in order to have 1.1.0-PC-SNAPSHOT artifact available? I also assume the demo was working fine with that so I'm possibly looking at some kind of regression with the update to published geotrellis-pointcloud.

I was able to see the index.html but the page is unresponsive and the service is is throwing 500 which I have not investigated yet.

I ran into this locally, and it was because the API server was silently failing to start with OOM errors. I had some success by setting JAVA_OPTS="-Xmx512M". Doing
$ docker-compose run --rm pc-api-server

Will start the API server only, so you can see if that is indeed the issue without seeing any other output from Nginx.

II had to bump the dependencies for geotrellis-pointcloud to newly published branch. I assume you guys must have been doing publish-local from feature branch in order to have 1.1.0-PC-SNAPSHOT artifact available?

Do you mean did I update the geotrellis-pointcloud version? I didn't, this demo is still using 0.1.0-PC-SNAPSHOT. I just corrected a typo in the JAR name.

It is a part of https://github.com/geotrellis/geotrellis-pointcloud-demo/pull/16 now


Apple keyboard 'fn' key not being handled correctly

The keyboard on my macbook contains a 'fn' key that emits a separate keycode (0x1d0/464) when pressed. I would like to use this key as a modifier, but it doesn't seem to be handled correctly at the moment. In fact, in v2.3.0-rc it seems to trigger the wrong key name.
Output of keyd -m when I press the key:
v2.2.7-beta
Out of bounds evdev keycode: 464
v2.3.0-rc
Apple Inc. Apple Internal Keyboard / Trackpad	05ac:025a	fastforward down
Apple Inc. Apple Internal Keyboard / Trackpad	05ac:025a	fastforward up

I believe this should be fixed in the latest commit.
On Mon, Mar 21, 2022 at 7:48 AM Tjisse van der Wal @.***>
wrote:

The keyboard on my macbook contains a 'fn' key that emits a separate
keycode (0x1d0/464) when pressed. I would like to use key this as a
modifier, but it doesn't seem to be handled correctly at the moment. In
fact, in v2.3.0-rc it seems to trigger the wrong key name.
Output of keyd -m when I press the key:
v2.2.7-beta
Out of bounds evdev keycode: 464
v2.3.0-rc
Apple Inc. Apple Internal Keyboard / Trackpad 05ac:025a fastforward down
Apple Inc. Apple Internal Keyboard / Trackpad 05ac:025a fastforward up
—
Reply to this email directly, view it on GitHub
https://github.com/rvaiya/keyd/issues/159, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOP7GKHV5B3WQGND6NDSB3VBBOXHANCNFSM5RHNWUNQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Yes it works, thanks!


fix: iterate over newSettings to avoid setting unchanged settings

description
By going with this:
// Create complete updated settings object
      const updatedSettings = { ...settings, ...newSettings };

// Save each setting individually to the store
      for (const [key, value] of Object.entries(updatedSettings)) {
        await store!.set(key, value);
      }

store.set is called even for settings that remain unchanged.
Also noticed issue with my code was that I did this:
// Save each setting individually to the store
      for (const key of newSettings) {
        await store!.set(key, value);
      }

instead of:
// Save each setting individually to the store
      for (const [key, value] of Object.entries(newSettings)) {
        await store!.set(key, value);
      }

Apologies for that.
Rest of code that was removed in commit #648c89abeb770ab5a2630fa90d9fcd80e1bbc813, I did consider removing but didnt want to cause any side effects as I didnt really understand how the settings feature was implemented. So I tried to make my solution as non invasive as possible
EDIT:
Not quite sure if issue came from my code, as key of newSettings is working properly in my local instance

While going over the code I realized theres two copies of settings, which could cause issues in the future:
https://github.com/mediar-ai/screenpipe/blob/main/screenpipe-app-tauri/lib/hooks/use-settings.tsx#L151
export function useSettings() {
  const [settings, setSettings] = useState<Settings>(DEFAULT_SETTINGS);
  const [localSettings, setLocalSettings] = useState<Settings>(settings);


Revert "replace "first ROM bank" with "ROM bank 0""

Reverts gbdev/pandocs#186

The bank at 0000-3FFF is able to be swapped around by some MBCs (most notably the MBC1). It should rather be called either "ROM0", "ROM0 space", or "$0000-3FFF".
Thanks @ISSOtm , @CasualPokePlayer


Add support for custom filters to attempt_predicate_pushdown

Follow-up to https://github.com/dask-contrib/dask-sql/pull/1160
Makes it possible to combine (1) pre-existing, (2) extracted, and (3) manually-specified filters within attempt_predicate_pushdown. Here, we assume that these three kinds of filters must always be combined as an and conjunction. However, any one of these three kinds of filters may correspond to a general Or(And(...), ...) DNF expression.

@ayushdg - This kind of functionality should make it a lot easier to manually "add on" filters that cannot be extracted from the graph.

Codecov Report

Merging #1173 (87a7f5c) into main (51cd0c8) will decrease coverage by 0.10%.
The diff coverage is 82.55%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main    #1173      +/-   ##
==========================================
- Coverage   81.42%   81.32%   -0.10%
==========================================
  Files          78       78
  Lines        4474     4509      +35
  Branches      817      828      +11
==========================================
+ Hits         3643     3667      +24
- Misses        650      654       +4
- Partials      181      188       +7

Impacted Files
Coverage Δ

dask_sql/physical/utils/filter.py
81.92% <82.55%> (+1.03%)
:arrow_up:

... and 2 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


CRA message should be deprecated

This is what Create React App suggest after running npm run build:
The project was built assuming it is hosted at the server root.
You can control this with the homepage field in your package.json.
For example, add this to build it for GitHub Pages:

  "homepage" : "http://myname.github.io/myapp",

The build folder is ready to be deployed.
You may serve it with a static server:

  npm install -g serve
  serve -s build

Find out more about deployment here:

http://bit.ly/2vY88Kr

This is deprecated since V7, as the new api is not compatible at all with previous versions.
I will cross post to CRA github, but I think it's also related to this package somehow. maybe you should consider tagging the V7 @next instead of @latest...

v7 is the latest stable release. Everything looks fine from Serve's end, we should update CRA's docs and CLI.

https://github.com/facebook/create-react-app/pull/4532

Ok, thanks for the feedback

This is very frustrating on our side.
We didn’t add Serve instructions to CRA ourselves. We took a PR from Zeit team. Of course at the time we assumed we can rely on the arguments not changing.
Now all CRA apps have non-working hardcoded instructions.

My sentiment exactly, as that's what happened to me on a production environment and it took me at least 30 minutes to realize and fix the problem

@sulliwane This is now solved, as per this comment. Sorry!


[Bug] 0922 Local variable '${value}' is overwritten before usage (variable-overwritten-before-usage) warning when using TRY/EXCEPT

What happened?
There seems to be an unnecessary "0922 Local variable '${value}' is overwritten before usage (variable-overwritten-before-usage)" warning when using TRY/EXCEPT.
What command/code did you try to run?
robocop Example.robot
Example.robot:
*** Settings ***
Documentation     Incorrect 0922 warning example

*** Test Cases ***
Example Test
    [Documentation]    Incorrect 0922 warning example test
    TRY
        ${value}=    Randomly Failing Keyword    # In real use case something inside the TRY block may sometimes fail
    EXCEPT
        ${value}=    Set Variable    Keyword failed
    END
    Log To Console    ${value}

*** Keywords ***
Randomly Failing Keyword
    [Documentation]     This keyword randomly fails
    IF    random.randint(0, 1) == 1    Fail    ELSE    RETURN    Keyword succeeded

What is the full error message?
Example.robot:9:9 [W] 0922 Local variable '${value}' is overwritten before usage (variable-overwritten-before-usage)
What did you expect to happen instead?
I expected no 0922 warning because the variable ${value} is set either in the TRY block or in the EXCEPT block before use.
Operating System
Windows
Robocop version
4.0.1

Thanks for the report. It should indeed be considered as separate namespace but currently all variables TRY EXCEPT (with all branches) are loaded and checked together. We will fix it


Unsynced Music

Hello! I wanted this bot on my server, so I downloaded source code, still alive mp3 and put token. But there is a problem: music is unsynced. Down here a stillalive.mp3 file(rename .zip to .mp3). Hope you can help me(or upload your version of stillalive.mp3, if problem with music)
stillalive.zip

Hi there! this version of the .mp3 should work :) Let me know if you're still having trouble and ill mess around some more when I've got time!
stillalivemp3.zip

Just one thing I would like to highlight, but the timings in the bot are largely dependent on your internet connection. For example, faster internet connections need more delay than slower connections.
I mainly noticed this because when I was adding the code for the bot to support "Want You Gone" too, my internet was being very dodgy and I noticed that the exact same code was getting inconsistent results. This may just be something that if you want it done right, you have to go and edit the timings yourself.

One other thing I would like to mention on the timing, I noticed that the code was mostly right for me if I added .5 seconds onto each time. Maybe just change the code so the wait time is something like ORIGINALTIME - difference, with an example being 5 - x. Then you can change the value of x to modify all the values by either adding or subtracting a value. If you then change x by adding or subtracting .1 each time while you are getting closer to having the song and text synchronized, you can get fairly close.
I don't think there really is anything we can do to help fix the sync issues for everyone, maybe someone could write an algorithm that modifies the time based off ping or something. Anyway, what I listed above is a bit of a fix for anyone that REALLY wants to use the bot


Make navigation menu collapsible/expandable

The current change will have all menu levels collapsed as the default behavior. What's the preferred default behavior? @agracey @jimmykarily?

In the current version I see no way to collapse a menu. So after a menu is expanded, it can't be collapsed again? Also I'm not sure what the default behavior should be. I'm not sure our top level menus are 100% indicative of what sub-menus lie inside.

In the current version I see no way to collapse a menu. So after a menu is expanded, it can't be collapsed again?

With output.html.fold enabled a chevron/triangle thingy is added to the menu to allow it to collapse and expand. The title was not clear and should've included expandable too.

Also I'm not sure what the default behavior should be. I'm not sure our top level menus are 100% indicative of what sub-menus lie inside.

Some of the submenus go 5/6 levels deep which reduce visibility of other menu items. Maybe I could set level to 1 instead so that the first level of submenus start expanded and any deeper submenus start collapsed.

level 1 sound like a good solution. Let's do that


Use updated checkboxes component in option select

Checkbox component has been retired and merged into checkboxes component. This PR updates finder-frontend to use the checkboxes component in the option select component.
This PR will not pass tests until a version of the publishing components gem has been released that contains this PR and should not be merged until the publishing components gem version has been bumped to include this change and added here. <- this has now been done, we're all good 👍
Examples of checkbox usage:

https://finder-frontend-pr-745.herokuapp.com/news-and-communications
https://finder-frontend-pr-745.herokuapp.com/prepare-business-uk-leaving-eu
https://finder-frontend-pr-745.herokuapp.com/cma-cases
https://finder-frontend-pr-745.herokuapp.com/administrative-appeals-tribunal-decisions

@vanitabarrett the components gem has now been upgraded in this PR to include the options for visually hiding the checkboxes legend, which I think addresses your concern. Do you want to take a look?

@vanitabarrett are you happy to approve this PR? The comment from Bill is still outstanding, trying to get some advice on whether it needs changing.

@vanitabarrett thanks, think I'll wait til after Christmas before merging, have made a note on the card.


Why do you use argmax() to find the eos token

eos_idx = tokenized[idx].argmax()
this gives me index number 3 on my custom words while using context length 16.

eos_idx = int(self.token_ids[0].argmax())
soft_embeddings = self.attr_dropout(self.soft_embeddings)
token_tensor[:, eos_idx - 2, :] = soft_embeddings[
    attr_idx
].type(self.clip_model.dtype)
token_tensor[:, eos_idx - 1, :] = soft_embeddings[
    obj_idx + self.offset
].type(self.clip_model.dtype)

How can you say that eos_idx-1 would be attribute and eos_idx-2 will be the place for object embedding in context length 8? What if i want another word in in between attribute and object? i.e 'a photo of car with wet look'. How would i manage to do replacement then?

You bring up a good point. We assume the class names will be of the form <attribute> <object> in the compositional zero-shot learning task. But, our framework can be easily adapted to a more generalized setting as well.
Tokenization in CLIP. We use the clip tokenizer to build token ids for our prompt, i.e. the tokenizer converts tokens to ids. For example, if you tokenize the prompt a photo of car with wet look, you will get the following output:
In [3]: clip.tokenize("a photo of car with wet look", context_length=10)
Out[3]: tensor([[49406,   320,  1125,   539,  1615,   593,  6682,  1012, 49407,     0]])

where 49406 corresponds to the start token, 49407 corresponds to the end token, and 0 corresponds to the padding. You can play around with the context_length to add more padding. A smaller context_length creates a smaller attention mask in the clip architecture, which saves memory during training and testing. But, if the context_length is too small then it might throw an error that the input prompt is too long for the context length.
Extention. To extend our framework to your setting, you could have a map for the token id 1615 which corresponds to car to a custom token or learned representation and replace them in your token_tensor.
Tip: Keep in mind that sometimes words split into subwords that might not correspond to a 1:1 mapping. For example, the word compositional corresponds to the token ids 7783,   760,   918,. You might need a heuristic to deal with these corner cases.
I hope this helps! Good luck!

thanks for your answer. I can see you call the 'mean' function to cover up the corner cases where words don't correspond to one token. However, I have used your clip module and csp.py code files, and the soft embedding gradients are not being calculated. Optimizer is constructed the same way by passing only the soft_embeddings variable. However, after loss.backward, I get soft_embeddings.grad as none. Due to this reason, there is no update on weights either. I am not sure where I am going wrong. Even if my soft_embeddings are incorrectly calculated, there should be grad calculation.


[Question] GCP alignment

Hello :hand:
I have a small dataset captured from a drone. Each image has GPS lat/lon/alt information included.
For additional accuracy I've tried to include a GCP file. The resulting point cloud however, exported with opensfm export_geocoords --dense does not align well with the GCPs (it's offset by ~1 meter in the X and ~3 meters in the Y axix).
Point cloud coordinates (Xg and Yg) of GCP point (WGS84 UTM 16N):

Entry from GCP file:
609865.7259757777 4950688.244601672 170.675678 2704 2559 DJI_0027.JPG

GCP point (top left) vs actual position.

OpenSfM config file:
use_exif_size: no
optimize_camera_parameters: yes
use_altitude_tag: True
align_method: naive
bundle_use_gcp: yes
bundle_use_gps: no

I'm wondering if this is expected, if I need to toggle a setting, or if there's something I should check (or did I find a bug)?
I should mention there are 5 GCPs in total and none of them align correctly.

Hi,
I'm not sure what went wrong, but the way we do align is by realigning the reconstruction on GPS+GCP after any whole bundle adjustment. It might not be the optimal way of doing it, and we're currently working on it.
So at the very end of the process, after last bundle, the reconstruction is aligned on GPS+GCP. It's already misleading since you might want not to use GPS (as GPS can be off by several meters compared to GCPs).
I made a tentative-untested branch feat-gcp-align that does not align on GPS AT ALL is bundle_use_gps is no, to see if it solves your case. let me know how it goes.
Yann

Hey @YanNoun, thanks for the reply! :hand:
Like you say, it looks like both GPS+GCP observations are used even though budle_use_gcp is set to no.
I will try the changes in your branch and report back my findings.

Looking good!!

This is with:
bundle_use_gps: no
bundle_use_gcp: yes

And the changes from the feat-gcp-align branch.

Running with defaults (bundle_use_gps: yes, bundle_use_gcp: no) however yields:
During ./opensfm reconstruct:
File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/reconstruct.py", line 23, in run
    incremental_reconstruction(data, graph)
  File "/code/SuperBuild/src/opensfm/opensfm/reconstruction.py", line 1323, in incremental_reconstruction
    data, graph, graph_inliers, reconstruction, remaining_images, gcp)
  File "/code/SuperBuild/src/opensfm/opensfm/reconstruction.py", line 1184, in grow_reconstruction
    align_reconstruction(reconstruction, gcp, config)
  File "/code/SuperBuild/src/opensfm/opensfm/align.py", line 16, in align_reconstruction
    res = align_reconstruction_similarity(reconstruction, gcp, config)
  File "/code/SuperBuild/src/opensfm/opensfm/align.py", line 58, in align_reconstruction_similarity
    return align_reconstruction_naive_similarity(config, reconstruction, gcp)
  File "/code/SuperBuild/src/opensfm/opensfm/align.py", line 87, in align_reconstruction_naive_similarity
    s = np.linalg.norm(Xp[0] - Xp[1])/np.linalg.norm(X[0] - X[1])
TypeError: unsupported operand type(s) for -: 'list' and 'list'

Hi,
I just updated the branch. Let me know how it goes.
However, it seems that in your tests, you have only two images, right ? In that very specific case of two images (or any sets of images aligned on a line), as since we can't align a line in 3D, the result won't be good.
For such case (which is not ideal capture practice), it is better to use 'align_methodwithorientation_prioralong withalign_orientation_priorset tovertical`.
Maybe an additional smart setting would be useful by mixing naive and orientation_prior would be better : use naive when sufficiently constrained (non-aligned point), otherwise, switch to orientation_prior.
Yann

Hey @YanNoun,
I've further modified line 87 to wrap the other list into a numpy array (the same error had appeared without this change):
s = np.linalg.norm(np.array(Xp[0]) - np.array(Xp[1]))/np.linalg.norm(np.array(X[0]) - np.array(X[1]))
The dataset has 18 images in total (https://github.com/pierotofy/drone_dataset_brighton_beach) and I'm executing it with the following:
use_exif_size: no
feature_process_size: 2048
feature_min_frames: 8000
processes: 4
matching_gps_neighbors: 8
matching_gps_distance: 0
depthmap_method: PATCH_MATCH
depthmap_resolution: 640
depthmap_min_patch_sd: 1
depthmap_min_consistent_views: 3
optimize_camera_parameters: yes
undistorted_image_format: png
use_altitude_tag: yes

So that defaults to align_method: naive. I could certainly switch it to align_method: prior and align_orientation_prior: vertical. A bit of background, this is related to some work we're doing with OpenDroneMap (https://github.com/OpenDroneMap/ODM). The reason I've left it at naive is because while most datasets are of aerial images, we occasionally have users that use the program to reconstruct non-aerial datasets (perhaps of architectural buildings) in which the orientation is not vertical, but horizontal. Would that create problems if we imposed the orientation prior to vertical (would the program adjust for non-vertical inputs)? Sometimes there's also mixes of vertical images and horizontal images (mapping an area with vertical shots, followed by a "point of interest" set of horizontal images).

Result looks pretty good! Will run some further tests.

Branch with fix in https://github.com/mapillary/OpenSfM/compare/master...pierotofy:feat-gcp-align-fix

Ok, then you're right, in your case, it is better to let it to naive. The orientation prior is nice for strips of images, but the drawback is someone needs to decide for the scene orientation wrt. images. At Mapillary, our street-view datasets often consist in long strips of images, here's why we have this kind of option.
Regarding your dataset, does it result in a single reconstruction or several partial ones ? You must have some chunk of two cameras somewhere since it entered the buggy code somehow.
BTW, we did a bit of improvment on the SfM these past weeks :

Robustified the way we do incremental SfM : https://github.com/mapillary/OpenSfM/pull/427 . Improves a lot of some tricky scenes. Some options can greatly improve accuracy as well (see bundle_outlier_filtering_type in the PR)
Added image-based (BoW) matching. Maybe not of major interested for drone mapping, but allows matching of scene without GPS.

It results into a single reconstruction (which seems odd, given your explanation).
I've seen BoW matching and tried to use it a few days ago, albeit unsuccessfully for a few datasets (like the one I've linked above), when I switch the matcher_type to WORDS I always get:
opensfm reconstruct /datasets/brighton2/opensfm
2019-06-20 13:19:06,488 INFO: Starting incremental reconstruction
2019-06-20 13:19:06,491 INFO: 0 partial reconstructions in total.

Works fine with FLANN though.
I hadn't seen #427, I will for sure check it out!! Great to see all of these improvements in OpenSfM!

https://github.com/mapillary/OpenSfM/pull/429 add matching_bow_neighbors for using images themselves to decide which pairs to match (might not be relevant since you'll use GPS anyway).
It is different from bow_matcher_type which is the matching method for matching the pair. Anyway, we ran benchmarks recently and FLANN is still the best option.

Mm, still no luck (full output is available here: https://pastebin.com/hrTQsyTk)
Perhaps the datasets that I'm trying are not ideal for bow?

With the following config.yaml :
processes: 12
use_exif_size: no
feature_process_size: 2048
feature_min_frames: 8000
depthmap_method: PATCH_MATCH
depthmap_resolution: 640
depthmap_min_patch_sd: 1
depthmap_min_consistent_views: 3
optimize_camera_parameters: yes
undistorted_image_format: png
use_altitude_tag: yes
matching_bow_neighbors: 8
matching_gps_neighbors: 0
matching_gps_distance: 0

Using bow_matcher_type: BOW I have after create_tracks : Good tracks: 10077
Using bow_matcher_type: FLANN instead (FLANN being default) I have : Good tracks: 10081`
My camera_models_overrides.json is (I tried with a Brown model to see if it removes the slight bowl effect, but didn't)
    "v2 dji fc300s 4000 2250 perspective 0.5555": {
        "focal_x_prior": 0.5555555555555556,
        "focal_y_prior": 0.5555555555555556,
        "width": 4000,
        "k1": 0.0,
        "k2": 0.0,
        "k1_prior": 0.0,
        "k2_prior": 0.0,
        "projection_type": "brown",
        "focal_x": 0.5555555555555556,
        "focal_y": 0.5555555555555556,
        "height": 2250
    }

Oh, I think I understand now. I just need to set matching_bow_neighbors to a value > 0 to enable BoW. I kept setting matcher_type to WORDS, but that needs to remain FLANN? (Can you confirm I'm doing this right?)
p.s. you can use this camera model which has been computed using a better dataset.
{
    "v2 dji fc300s 4000 2250 perspective 0.5555": {
        "focal_prior": 0.5555555555555556,
        "width": 4000,
        "k1": -0.009399196808062386,
        "k2": 0.015120083762071154,
        "k1_prior": 0.0,
        "k2_prior": 0.0,
        "projection_type": "perspective",
        "focal": 0.5852784573769383,
        "height": 2250
    }
}```

The two are independant :

matching_bow_neighbors > 0 will use BoW for selecting which pairs of images to match.
matcher_type can be set either BOW or FLANN and used when matching pairs, FLANN being better than WORDS.

Thanks for the explanation! I think there might be some bug (you mentioned you were able to process it OK, but still having issues, I swear I'm not crazy :) )
With:
processes: 4
use_exif_size: no
feature_process_size: 2048
feature_min_frames: 8000
depthmap_method: PATCH_MATCH
depthmap_resolution: 640
depthmap_min_patch_sd: 1
depthmap_min_consistent_views: 3
optimize_camera_parameters: yes
undistorted_image_format: png
use_altitude_tag: yes
matching_bow_neighbors: 8
matching_gps_neighbors: 0
matching_gps_distance: 0
bow_matcher_type: BOW
matcher_type: WORDS

root@07628e9f82a2:/source/OpenSfM# bin/opensfm create_tracks /datasets/brighton
2019-06-21 15:13:12,033 INFO: reading features
2019-06-21 15:13:12,215 DEBUG: Merging features onto tracks
2019-06-21 15:13:12,215 DEBUG: Good tracks: 0

Changing matcher_type to FLANN works.
This is the master branch of OpenSfM, built with docker this morning from a fresh clone.

OK, my bad, I was misleading and I forgot about the bow_matcher_type and matcher_type.
So correct BoW setting are :

bow_matcher_type: FLANN (default). Using BOW makes the command detect_features actually crash on my machine, as it is not a valid option. Then BoW assignment are not generated, and then further BoW use doesn't work (actually the command match_features should crash.
matcher_type: FLANN (default, best trade-off) or matcher_type: WORDS (faster, worse)

Additionally, the new matching_bow_neighbors > 0 setting allow matching dataset without GPS and/or bad GPS (indoor).
Overall the process is the following :

Extract features
Assign each feature to some words (hence Bag-of-Words, BoW). It's where bow_matcher_type is used.
Construct a list of pairs to match. This where matching_{gps, bow, order, time}_neighbors comes into action.
For each of these pairs, match their features using the matcher of matcher_type.

Hope that helps !

It does, thanks for taking the time to explain in more details!
I'm still unable to get this dataset to process, but I'll keep trying. I must be doing something wrong.
I will close this issue for now since the original discussion was about GCP alignment. I'll re-open a new issue to follow up with the BOW feature.
Thanks again! :+1:


Add BSD LICENSE file

Similar to https://github.com/ros2/urdf/pull/32 , for compliance with ROS 2 developer guide https://docs.ros.org/en/humble/The-ROS2-Project/Contributing/Developer-Guide.html#documentation :

Each package must have a LICENSE file

Yep, this looks good.  Thanks!

Thanks @clalancette !


KBFS won't open

Windows 10
Keybase GUI Version: 1.0.18-20170120130100+e2dc6c4
2017-01-26T07:59:20.422270 - [DEBU kbfs tlf.go:70] 3f59 Loading root directory for folder ghallford (public: false, filter error: true) for FindFiles [tags:DID=ficBfnzFNDwfzecvZjMmPA]
2017-01-26T07:59:20.422270 - [DEBU kbfs kbfs_ops.go:400] 3f5a getMaybeCreateRootNode(/keybase/private/ghallford, , false) [tags:DID=ficBfnzFNDwfzecvZjMmPA]
2017-01-26T07:59:20.422770 - [DEBU kbfs connection.go:486] 3f5b (CONN MDServerRemote d6dad5f8) Connection: getReconnectChan
2017-01-26T07:59:20.422770 - [DEBU kbfs connection.go:453] 3f5c (CONN MDServerRemote d6dad5f8) Connection: waitForConnection; status: 1
2017-01-26T07:59:20.422770 - [DEBU kbfs kbfs_ops.go:408] 3f5d Done: context.deadlineExceededError{} [tags:DID=ficBfnzFNDwfzecvZjMmPA]
2017-01-26T07:59:20.422770 - [DEBU kbfs dir.go:134] 3f5e context deadline exceeded [tags:DID=ficBfnzFNDwfzecvZjMmPA]
keybase.kbfs.txt
keybase.service.txt

Hi @ghallford, we'll check this out. For what it's worth, I don't recommend sharing keybase.kbfs.txt publicly (as you have), while we try not to log any secrets, the log does include names of files you've accessed etc.
Better to run "keybase log send", which sends us your logs securely.

@ghallford: this seems to be the issue:
2017-01-26T07:59:20.401756 - [WARN kbfs connection.go:370] 3f44 (CONN MDServerRemote d6dad5f8) Connection: error dialing transport: dial tcp: lookup mdserver.kbfs.keybase.io: getaddrinfow: No such host is known.
2017-01-26T07:59:20.402256 - [WARN kbfs connection.go:535] 3f45 MDServerRemote: connection error: "dial tcp: lookup mdserver.kbfs.keybase.io: getaddrinfow: No such host is known."; retrying in 2s
2017-01-26T07:59:20.403756 - [WARN kbfs(BSR) connection.go:370] 3f46 (CONN BlockServerRemoteGet 759e55fc) Connection: error dialing transport: dial tcp: lookup bserver.kbfs.keybase.io: getaddrinfow: No such host is known.

That seems to indicate that you can't connect to our servers over the network.  Are you behind some kind of firewall that needs exceptions for kbfs.keybase.io subdomains?

Hi, the actual resolution was a reboot. Network for all other apps was working fine.

Geoff Hallford
On Jan 26, 2017, 1:47 PM, at 1:47 PM, Jeremy Stribling<EMAIL_ADDRESS>wrote:

@ghallford: this seems to be the issue:
2017-01-26T07:59:20.401756 - [WARN kbfs connection.go:370] 3f44 (CONN
MDServerRemote d6dad5f8) Connection: error dialing transport: dial tcp:
lookup mdserver.kbfs.keybase.io: getaddrinfow: No such host is known.
2017-01-26T07:59:20.402256 - [WARN kbfs connection.go:535] 3f45
MDServerRemote: connection error: "dial tcp: lookup
mdserver.kbfs.keybase.io: getaddrinfow: No such host is known.";
retrying in 2s
2017-01-26T07:59:20.403756 - [WARN kbfs(BSR) connection.go:370] 3f46
(CONN BlockServerRemoteGet 759e55fc) Connection: error dialing
transport: dial tcp: lookup bserver.kbfs.keybase.io: getaddrinfow: No
such host is known.

That seems to indicate that you can't connect to our servers over the
network.  Are you behind some kind of firewall that needs exceptions
for kbfs.keybase.io subdomains?
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/keybase/client/issues/5564#issuecomment-275475059


CORE: Ensure if the user changes the weight on the new recipe page right from the page that the weight variables on page are updated dynamically.

OMG. That was hard


Inventor Names are not working correctly

The way that Inventor names are added to the query is not working correctly and has a tendency to be over inclusive.  Need to be reworked.
Maybe add_inventor(first_name, last_name)?

Will be fixed with roll out of v3 of the API


[Feature Request] Golang to Serve Frontend Assets (like React) instead of Node.Js

Tell us about your feature request
I work on projects that require pretty high security standards, and I have yet to get Node.Js pushed through (maybe Deno in the future??). Has anyone tried deploying go-blueprint with React and without Node - letting Golang serve the frontend assets instead?
This would be similar to the official React + ASP.NET or Vue + ASP.NET templates. Vite becomes a proxy server during development for DX reasons (HMR, tailwind, etc.). When it's time to deploy, Vite builds the frontend assets into a public folder, which Golang picks up and serves. All API endpoints will get served through 'https://example.com/api/' (it would feel kinda like Next.Js).
It could simplify the deployment story because you wouldn't necessarily need docker-compose just to serve HTML/CSS/JS assets. One Golang server would do all the work, and latency could be reduced (depending on how the project is deployed).
Deploying without Node means you can't deploy any server-side JS packages. But go-blueprint is a Golang backend project - if you're deploying backend JS, you're probably doing it wrong.
If this seems interesting, I am happy to tinker and create a PR, but I don't want to expand this project's scope because of a specific deployment opinion or security need.
Any thoughts?
Huge fan!! Go Blueprint is incredible!!
Disclaimer

[X] I agree

@EarliestFall988
I agree with you, the current implementation is not ideal. The use of serve for serving static content can be removed. Instead, we can adopt a more efficient approach. Nginx, for instance, is highly efficient, fast, secure, and production-ready.
Bundling the frontend and backend into one monolith is not something I favor. We should keep these two components separate. This separation aligns with cloud-native patterns.
If you are deploying into k8s, as most production apps are, you gain granular control over resources, autoscaling, rolling updates, and fault isolation. It also simplifies scaling specific components and removes the need to redeploy the backend if only the frontend changes, and vice versa.
The only remaining question is whether to serve the frontend using a Go  or Nginx.


Allow non-core OrientDB modules to register streamable types with OStreamableHelper

At the moment OStreamableHelper relies on Class.forName to lookup OStreamable types. While that works for types that are on the same classpath as the core module, it doesn't work well in OSGi or other modular systems, where types might be isolated to their defining module.
The simplest solution I found was to let non-core modules register their OStreamable types, using weak references so those types can still be unloaded when their defining module is unloaded. (To keep things simple dangling keys aren't cleaned up, but this shouldn't matter because there will only be a few entries in the cache and the keys are simple strings.)
We're using this change to deploy OrientDB distributed on top of Karaf and would also like it backported to 2.2.x. (Note the 2.1.x branch doesn't have this issue because it used a different approach for the distributed protocol.)

Found an edge case which isn't covered by this change, so will rework it and re-push


sklearn.utils.compute_class_weight incorrectly called by dtreeviz

When trying to plot my tree, I got the following error:
/opt/anaconda/envs/swb/lib/python3.7/site-packages/dtreeviz/models/sklearn_decision_trees.py in get_class_weights(self)
     28         if self.is_classifier():
     29             unique_target_values = np.unique(self.y_data)
---> 30             return compute_class_weight(self.tree_model.class_weight, unique_target_values, self.y_data)
     31
     32     def get_thresholds(self):

TypeError: compute_class_weight() takes 1 positional argument but 3 were given

Looking at sklearn's documentation, dtreeviz should have called the function as such:
compute_class_weight(self.tree_model.class_weight, classes=unique_target_values, y=self.y_data)

I've implemented a monkey patch for myself, which resolves the error.

Hi @thomsentner, I don't see a difference in proposed method call, it only uses named parameters, which is a default option in python.
Can you double check it please? Thanks

Sklearn includes a star in its function definition:
sklearn.utils.class_weight.compute_class_weight(class_weight, *, classes, y)

As per PEP 3102, this means that any arguments right of the star are keyword-only arguments.
Some example code:
def my_func(x, *, a):
    print(a)

# this works
my_func(1, a="a")

# this doesn't work
my_func(1, "a")

Output:
>>> my_func(1, a="a")
a
>>> my_func(1, "a")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: my_func() takes 1 positional argument but 2 were given

Thanks @thomsentner, didn't know this feature. I'm in vacation right know. Will try to solve it soon


Data is not in JSON format.

Invalid response from the backend. The data is not in JSON format.
Thanks

Do you have an example? Which URL does not return JSON? What is returned instead?

URL ->  http://localhost:8080/elfinder/connection?instanceName=pratica-1&cmd=open&target=&init=1&tree=1&_=1627567646415

The URL seems to be wrong. With the solution I coded, elfinder is a submodule of pratica, so the URL should look like this:
ttp://localhost:8080/pratica/elfinder/connection?instanceName=pratica-1&cmd=open&target=&init=1&tree=1&_=1627567646415

Also happens if I use 'default' connectionSet as in documentation, here the url:
http://localhost:8080/elfinder/connection?instanceName=default&cmd=open&target=&init=1&tree=1&_=1627570414917

sorry my mistake, I had added a VarDumper :: dump in the @vendor code, I close the issue.


Adding functionality for the edit command to remove a DateOfBirth field from an existing Person

Closing as our requirements have changed and DateOfBirth fields are now compulsory


New package: EasyQP v0.1.0

Registering package: EasyQP
Repository: https://github.com/PharosAbad/EasyQP.jl
Created by: @PharosAbad
Version: v0.1.0
Commit: b2ccd90eeea1529c0f8cfea7f862d685caa9d01a
Reviewed by: @PharosAbad
Reference: https://github.com/PharosAbad/EasyQP.jl/commit/b2ccd90eeea1529c0f8cfea7f862d685caa9d01a#commitcomment-97385575

give up


Use vendor-id when discovering main classes (#839)

Fixes #839

Thanks @2m. The extra tests are cool!


[BUG] CMS_TOOLBAR_HIDE = True breaks ToolbarMiddleware in v3.11.x

Description
We upgraded to v3.11.x and were getting hit by middleware issues. Eventually tracked it down to the ToolbarMiddlware.
There are likely other ways that this bug could get hit but this is specifically how it hit us.
Steps to reproduce

Enable CMS_TOOLBAR_HIDE = True
Visit any page.
See clickjacking exception thrown because that's the next middleware that needs a response.

Expected behaviour
To enable CMS_TOOLBAR_HIDE without causing problems with other middlewares.
Actual behaviour
We need to disable CMS_TOOLBAR_HIDE for other middlewares to work properly.
Additional information (CMS/Python/Django versions)
Django CMS v3.11.0
Do you want to help fix this issue?

[x] Yes, I want to help fix this issue and I will join #workgroup-pr-review on Slack to confirm with the community that a PR is welcome.
[ ] No, I only want to report the issue.

Pull Request Created: https://github.com/django-cms/django-cms/pull/7856

My PR was merged to resolve the issue so closing bug


Make lulls message in Python SDK to be consistent with the implementation in Java SDK

Resolves #26278
Implementation in Java: [link](https://github.com/bzablocki/beam/blob/ff270573316b29c5a1d303ed53335ef6f7916635/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ExecutionStateSampler.java
https://github.com/apache/beam/blob/ff270573316b29c5a1d303ed53335ef6f7916635/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ExecutionStateSampler.java)
The message in Java is in the following format:
String.format(
    "Operation ongoing in bundle %s for PTransform{id=%s, name=%s, state=%s} for at least %s without outputting or completing:%n  at %s",
    processBundleId.get(),
    currentExecutionState.ptransformId,
    currentExecutionState.ptransformUniqueName,
    currentExecutionState.stateName,
    DURATION_FORMATTER.print(Duration.millis(lullTimeMs).toPeriod()),
    Joiner.on("\n  at ").join(thread.getStackTrace())));

The implementation in this PR will bring Python's lulls message closer to Java's format.
step_name_log = (
    ' for PTransform{name=%s, state=%s}' % (step_name, state_name))

stack_trace = self._get_stack_trace(sampler_info)

_LOGGER.warning(
    (
        'Operation ongoing in bundle %s%s for at least %.2f seconds'
        ' without outputting or completing.\n'
        'Current Traceback:\n%s'),
    instruction,
    step_name_log,
    lull_seconds,
    stack_trace,
)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

remind me after tests pass

waiting on author

Python tests / Python Unit Tests (windows-latest, 3.7, py37) (pull_request)

Run Python tests / Python Unit Tests (windows-latest, 3.7, py37)

Thanks for reviewing @riteshghorse !


ENH: Support Series in read_hdf

Closes https://github.com/dask/dask/issues/1564
Modifies the reader to check for name in the case of Series,
and columns in the case of DataFrame.

This looks great to me.

Thanks! Merging.


MK upload custom json using ui and support for debugging a service running in Docker

This PR features was handled within "IBM/sarama Go client library for Apache Kafka support #68"

IBM/sarama Go client library for Apache Kafka support #68


Added conditional include of the OpenMP header.

Since the OMP pragmas are used, it seems appropriate to include the openMP header (if the -fopenmp flag is specified).

This makes sense to me... actually wait why does OpenMP work on my Linux system without this import?

It currently works because only #pragma is used.
I tried to use the omp_get_num_devices() and noticed the header was missing.

It currently works because only #pragma is used. I tried to use the omp_get_num_devices() and noticed the header was missing. So, it's not strictly needed but you never know if some compiler uses it or not (I use gcc 11 but someone might use gcc 12 or clang).

I use clang, gcc and mingw to side-by-side perf them.
so again. confirmed that someone does that (and why I already roll like that)


Feature/normalize function

We decided to split normalize and average functions into two branches and also two pull request.
In this one the normalize function got revised, new tests are implemented and also the new decibel parameter "prefix_return"
was used in the normalize function.

What are meaningful combinations of the normalize parameter with mode='power'?

Another idea following the comments above on phrasing and the meaning of the different parameters:
What do you think of removing the 'log_magnitude' option and instead introducing a dB parameter. Then the parameters might even get more clear:

select the data the normalization value is computed from ('time', 'freq', 'power') (according to the code, all the operations act on the absolute values - also for 'time' - so 'magnitude' might be replaced with 'freq' and taking the absolute value is part of the operation)
select the method to operate ('max', 'mean', 'rms') (all on absolute values)
set wether the operation should operate on the logarithmized data (dB=True) (in accordance with the signal properties)
if dB=True, both the value to normalize to and returned normalization value are in dB (in accordance with the signal properties)

What should be made clear is how 'power' and signals with fft_norm='power' in combination with dB=True are handled.

Still to do:

Docstring
New tests for revised function (also adjust old tests)
Comment function


Removed service provider disposing and added missing dependencies for mining feature

And also removed extra ConnectionManager.Disposed() call.
Fix #799

Nice @noescape00 this will up that pesky #799 issue nicely. I'm happy with it if @dangershony s happy?


Ability to cache older-but-frequently-accessed data

Setting value_retention_factor = 1536 to conf file has no impact. Only responses that have less than default 1024 sample are fully cached.
Config file is correctly used because I can switch to filesystem caching.
This is issue when Grafana Prometheus datasource use Resolution 1/1. It result small possible range_query "step", sample per pixel. For example querying 24h data the step is set to 60s --> 1441 sample returned and only 1024 are cached resulting always kmiss and phit.
Workaround:
When Grafana resolution is set to 1/2 --> step size increase to 120s --> 721 sample returned and Trickster has 100% cache hit rate.

@jtorkkel thanks for the report - would you mind copy/pasting the relevant config snippet? That would help me to reproduce, thanks again!

I renamed https://github.com/Comcast/trickster/blob/next/cmd/trickster/conf/example.conf to trickster.conf and pass the file to docker as shown in your example.
I have modified three lines:
[caches]
[caches.default]
type = 'filesystem' # change from memory
[origins]
host = '10.x.y.z:19090'.   # point to host where Prometheus is running
value_retention_factor = 1536 # Remove comment and change default 1024

Two first configuration change are used and works, because metrics and logs report filesystem and because real Prometheus can be accessed.
Thus I suspect that third value is not read correctly.

Thanks! I will take a look at this more today with those instructions to repro. I'm having a 'works on my machine' situation but will get to the bottom of it.

If you test with Prometheus graph directly you need to set step manually to 60 for 1d/86400s. Prometheus Graph probably divide screen width with 5 or six to calculate how many sample to show, and then divide range with that number resulting step between 300-400 for 86400s (response has ~500 sample).
Grafana use configured min steps if resolution is 1/1 and unfortunately can not adjust step intelligently (I thought it could but maybe not working anymore in 6.x)

I tested this more and cannot find any error from my configuration.
I also tried larger 3072 value because actually Prometheus is returning 2881 sample when Grafana query Graph which is screen width. Still only 1024 datapoints are cached. I also took logs which also indicate that only 1024 out of 1441 are cached.
Seems that Grafana use step size which result samples ScreenResolution < samples < 2x ScreenSize.
Resolution needs to be adjusted to 1/3 to workaround this and datasource scrape interval needs to put to real scrape interval/3, otherwise last 5 min query would use 3x larger step than scraping value.
 Default is 1024
    value_retention_factor = 1536

# default is info
log_level = 'debug'

Log when resolution is 1/2
after changing the Grafana Resolution to 1/2 --> step 120 for range 86400s, reply size=24334
Reply logging contain size: userAgent=Grafana/6.1.2 durationMS=5809 originName=default originType=prometheus handlerName=QueryRangeHandler method=GET code=200 size=24334
time=2019-06-20T07:44:28.784279352Z app=trickster caller=cache/filesystem/filesystem.go:108 level=debug event="filesystem cache miss" key=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz dataFile=/tmp/trickster/<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz.data
time=2019-06-20T07:44:34.596933471Z app=trickster caller=proxy/engines/access_logs.go:23 level=debug event="upstream request" uri="http://<IP_ADDRESS>:19090/api/v1/query_range?end=1560978000&query=sum(rate(http_server_requests_seconds_count{job%3D~".*"%2C+host%3D~".*"%2C+status%3D~"2.."%2C+app%3D~".*"%2C+exception%3D"None"}[5m]))&start=1560891600&step=120" userAgent=Grafana/6.1.2 durationMS=5809 originName=default originType=prometheus handlerName=QueryRangeHandler method=GET code=200 size=24334
time=2019-06-20T07:44:34.598662822Z app=trickster caller=proxy/engines/deltaproxycache.go:293 level=debug event="delta routine completed" cacheKey=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30 cacheStatus=kmiss reqStart=2019-06-18T21:00:00Z reqEnd=2019-06-19T21:00:00Z
time=2019-06-20T07:44:34.60083272Z app=trickster caller=proxy/engines/cache.go:71 level=debug event="compressing cached data" cacheKey=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz
time=2019-06-20T07:44:34.601228198Z app=trickster caller=cache/filesystem/filesystem.go:86 level=debug event="filesystem cache store" key=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz dataFile=/tmp/trickster/<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz.data indexed=true
time=2019-06-20T07:44:35.294144663Z app=trickster caller=cache/filesystem/filesystem.go:86 level=debug event="filesystem cache store" key=cache.index dataFile=/tmp/trickster/cache.index.data indexed=false
time=2019-06-20T07:44:41.551302785Z app=trickster caller=cache/filesystem/filesystem.go:118 level=debug event="filesystem cache retrieve" key=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz dataFile=/tmp/trickster/<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz.data
time=2019-06-20T07:44:41.551390211Z app=trickster caller=proxy/engines/cache.go:50 level=debug event="decompressing cached data" cacheKey=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz
time=2019-06-20T07:44:41.5563185Z app=trickster caller=proxy/engines/deltaproxycache.go:293 level=debug event="delta routine completed" cacheStatus=hit reqStart=2019-06-18T21:00:00Z reqEnd=2019-06-19T21:00:00Z cacheKey=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30
Now hit
time=2019-06-20T07:44:43.863177433Z app=trickster caller=cache/filesystem/filesystem.go:118 level=debug event="filesystem cache retrieve" key=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz dataFile=/tmp/trickster/<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz.data
time=2019-06-20T07:44:43.863302715Z app=trickster caller=proxy/engines/cache.go:50 level=debug event="decompressing cached data" cacheKey=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30.sz
time=2019-06-20T07:44:43.867255435Z app=trickster caller=proxy/engines/deltaproxycache.go:293 level=debug event="delta routine completed" cacheKey=<IP_ADDRESS>:19090.463d281e25a33a9d7a667439336c3d30 cacheStatus=hit reqStart=2019-06-18T21:00:00Z reqEnd=2019-06-19T21:00:00Z
-bash-4.2$

@jtorkkel can you compare the clock time of your Prometheus and Trickster instances to make sure they are closely synced? You should be able to run something like curl -s --head http://PROMETHEUS:9090/api/v1/query?query=up | grep 'Date:' && curl -s --head http://TRICKSTER:9090/v1/query?query=up | grep 'Date:'

Will check on Monday. Trickster, Prometheus and Grafana are on same Linux host so time should match. Trickster is docker, Prometheus and Grafana not.  Browser using Grafana is different timezone, 1h difference as seen when converting the time. Trickster tz in utc+1, mine utc+2.
I checked the query start and end time and range was always 86400s.
In phit partial fetch was 8h44min. This hints that 15h16min was cached. With step=60s and 1024 sample trickster should be able to store 17h04min but based on cache only 15h16min was cached which equal 916 sample.
I was wondering why initially query logging did not record response size at all. Sounds that cache can store only 916 sample even configuration is larger. This hints that maybe 1h is lost due time difference but does not explain why trickster is making 8h44min query. Even 2h or 3h difference in timezone does not explain why 8h44min extended query was needed.
On the other hand if we assume that cache size is indeed 1536 but from some reason every second  entry is utilized —> cache size 768 equals 12h48min but that does not explain 8h44min extended query.
I already tested 3072 cache size and did not see any behavior change, which hints that new value in not used.
I also tested that step=120 meaning 721 sample were cached correctly.
I can recheck by using range 15h, 15h15min, 15h20min, 16h, 17h, 22h and 23h to see if hit or phit —> see how many entries can be actually stored.
Might make sense to expose /status or /config endpoint exposing actual configuration or log all config changes to log, currently only some config changes are logged.

I run the curl command and output was:
Date: Sun, 23 Jun 2019 15:45:22 GMT
Date: Sun, 23 Jun 2019 15:45:22 GMT
I made testing with "memory" cache and did not find any changes in behavior compared to filesystem.
I also tested different queries using setting value_retention_factor=3072 and made strange findings which relates to query range and/or system time and Trickster capability to update the cache.
I made two queries using same 48h range but different start time (two hour difference, using memory caching this time). Tests were done after restart

One query get correctly cached and new query read 100% from cache
Another query only partially cached, new query result phit every time

This indicates that Trickster has problems storing all result to cache for specific start and end times but also indicates that value_retention_factor configuration "works" . I could not find any sequence what works and what not. I got feeling that time might also impact if caching succeed or not.  I noticed for example the following behavior:

after phit the new results are not stored at all resulting same phit with same extentsFetched for every new query
after phit part of the result are stored and next query results slightly different extentsFetched range, and maybe after few query all results are stored resulting hit

I do not know how cachekey works but I assume that cachekey is calculated from query excluding start and end time, and then cachekey has index containing separate ranges for each start/end ranges. I assumed that extentsFetched should update the cache and maybe ranges can be combined if overlapping results exists.
48h range_query, 2881 sample returned, 21.6 17.30 - 23.6 17.30 resulting phit
url:"api/datasources/proxy/12/api/v1/query_range?query=sum(rate(http_server_requests_seconds_count%7Bjob%3D~%22.%22%2C%20host%3D~%22.%22%2C%20status%3D~%222..%22%2C%20app%3D~%22.*%22%2C%20exception%3D%22None%22%7D%5B10m%5D))&start=1561127400&end=1561300200&step=60"
time=2019-06-23T19:11:47.382741509Z app=trickster caller=proxy/engines/access_logs.go:23 level=debug event="upstream request" originType=prometheus method=GET code=200 size=96170 durationMS=20027 originName=default handlerName=QueryRangeHandler uri="http://<IP_ADDRESS>:19090/api/v1/query_range?end=1561300200&query=sum(rate(http_server_requests_seconds_count{job%3D~".*"%2C+host%3D~".*"%2C+status%3D~"2.."%2C+app%3D~".*"%2C+exception%3D"None"}[10m]))&start=1561127400&step=60" userAgent=Grafana/6.1.2
time=2019-06-23T19:11:47.402693291Z app=trickster caller=proxy/engines/deltaproxycache.go:293 level=debug event="delta routine completed" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd cacheStatus=kmiss reqStart=2019-06-21T14:30:00Z reqEnd=2019-06-23T14:30:00Z
time=2019-06-23T19:11:47.413750478Z app=trickster caller=proxy/engines/cache.go:71 level=debug event="compressing cached data" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:11:47.414279047Z app=trickster caller=cache/memory/memory.go:50 level=debug event="memorycache cache store" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz length=57890 ttl=6h0m0s
time=2019-06-23T19:12:01.375960723Z app=trickster caller=cache/memory/memory.go:63 level=debug event="memorycache cache retrieve" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:12:01.376058001Z app=trickster caller=proxy/engines/cache.go:50 level=debug event="decompressing cached data" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:12:01.975050692Z app=trickster caller=proxy/engines/access_logs.go:23 level=debug event="upstream request" handlerName=QueryRangeHandler uri="http://<IP_ADDRESS>:19090/api/v1/query_range?end=1561132680&query=sum(rate(http_server_requests_seconds_count{job%3D~".*"%2C+host%3D~".*"%2C+status%3D~"2.."%2C+app%3D~".*"%2C+exception%3D"None"}[10m]))&start=1561127400&step=60" size=3018 originName=default originType=prometheus method=GET userAgent=Grafana/6.1.2 code=200 durationMS=592
time=2019-06-23T19:12:01.986591092Z app=trickster caller=proxy/engines/deltaproxycache.go:293 level=debug event="delta routine completed" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd cacheStatus=phit reqStart=2019-06-21T14:30:00Z reqEnd=2019-06-23T14:30:00Z extentsFetched=1561127400-1561132680
time=2019-06-23T19:12:01.994917552Z app=trickster caller=proxy/engines/cache.go:71 level=debug event="compressing cached data" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:12:01.995479093Z app=trickster caller=cache/memory/memory.go:50 level=debug event="memorycache cache store" length=57867 ttl=6h0m0s cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
48h range_query, 2881 sample returned, 21.6 19.30 - 23.6 19.30 resulting hit
url:"api/datasources/proxy/12/api/v1/query_range?query=sum(rate(http_server_requests_seconds_count%7Bjob%3D~%22.%22%2C%20host%3D~%22.%22%2C%20status%3D~%222..%22%2C%20app%3D~%22.*%22%2C%20exception%3D%22None%22%7D%5B10m%5D))&start=1561134600&end=1561307400&step=60"
2878:Array[1561307280,1276.5750793299735]
2879:Array[1561307340,1279.0319298692425]
2880:Array[1561307400,1286.8592604758749]
time=2019-06-23T19:15:18.128265716Z app=trickster caller=proxy/engines/access_logs.go:23 level=debug event="upstream request" originName=default size=96270 method=GET uri="http://<IP_ADDRESS>:19090/api/v1/query_range?end=1561307400&query=sum(rate(http_server_requests_seconds_count{job%3D~".*"%2C+host%3D~".*"%2C+status%3D~"2.."%2C+app%3D~".*"%2C+exception%3D"None"}[10m]))&start=1561134600&step=60" userAgent=Grafana/6.1.2 code=200 durationMS=17630 originType=prometheus handlerName=QueryRangeHandler
time=2019-06-23T19:15:18.149565888Z app=trickster caller=proxy/engines/deltaproxycache.go:293 level=debug event="delta routine completed" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd cacheStatus=kmiss reqStart=2019-06-21T16:30:00Z reqEnd=2019-06-23T16:30:00Z
time=2019-06-23T19:15:18.161051275Z app=trickster caller=proxy/engines/cache.go:71 level=debug event="compressing cached data" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:15:18.161534142Z app=trickster caller=cache/memory/memory.go:50 level=debug event="memorycache cache store" length=59647 ttl=6h0m0s cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:15:23.82039096Z app=trickster caller=cache/memory/memory.go:63 level=debug event="memorycache cache retrieve" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:15:23.820560766Z app=trickster caller=proxy/engines/cache.go:50 level=debug event="decompressing cached data" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd.sz
time=2019-06-23T19:15:23.837470201Z app=trickster caller=proxy/engines/deltaproxycache.go:293 level=debug event="delta routine completed" cacheKey=<IP_ADDRESS>:19090.acc693992ec9c75ccd2f1abe664529fd cacheStatus=hit reqStart=2019-06-21T16:30:00Z reqEnd=2019-06-23T16:30:00Z

I made more testing by disabling compression and disabling fast_forward_disable but no help. I was suspecting that system cache data correctly but incorrectly made unnecessary fast forward but that was not the case.
I changed back to value_retention_factor = 1024 and checked the the content of the non compressed /tmp/trickster/keyfile.
I noticed that only cache contained only 126 entries out of 2881 from query and cache file ended to
extents":[{"start":"2019-06-23T14:24:00Z","end":"2019-06-23T16:30:00Z"}],"step":60000000000}
After 10s  the query resulted phit and the size just decrease which indicate that cache size is somehow bound to time and reduce until revert back to larger (closer to configured 1024 size).
I finally set value_retention_factor to 4096 and now cache works for also for 2881 entries.
I have changed the Grafana Resolution to 1/3 and Grafana scrape interval 10s (We use 30s prometheus scrape interval). This result that no query has more than 960 sample, but query over short period use 30s step --> value_retention_factor to 2048 should be enough to store 1000 sample (assuming 1000 is always available). Value_retention_factor = 4096 would allow Resolution 1/1 usage --> step=10s for short range like 1h while 48h range would use step=60s --> 2881 sample.
I made more testing and found that 48h query from 20.6 to 22.6 but it did not get fully cached (2881 sample, and only 1200 got cached when value_retention_factor =4096) but 72h query from 20.6 -  23.6 got cached (step was 120 --> 2160 sample).
Sounds that only max half of value_retention_factor entries are available. I finally set value_retention_factor = 16384.
Would be good to find out why all "value_retention_factor" entries are not available for cache.

@jtorkkel I think I understand your situation now, and believe Trickster is working as (currently) designed, but I propose a feature update below to make it work according to your expectations.
It is true that the Value Retention Factor helps determine what data is stored in cached for a given query. However the VRF does not guarantee that you will always have that number of timestamps in cache, and does not mean that if data set has fewer timestamps in a response than the Value Retention Factor that all of them are cacheable. It also does not guarantee that if you have more timestamps in your result than the VRF, that the cached number will then be precisely equal to the VRF.
Value Retention Factor works like this: When you make a request to Trickster, it will look at the step parameter in your query, and multiply it by the Value Retention Factor to determine a cacheable window of time for that query. For example, if your VRF is 1024 and your Step is 15s, then the window duration calculates to 15,360 (4h16m0s). We then then subtract that cacheable window duration from time.Now() to determine an oldest cacheable absolute timestamp for the query. Any timestamp in the data set that is predates the oldest eligible timestamp (time.Now() - 4h16m, in my example) is not cached, no matter the VRF.
The primary intent of Trickster is to cache recent (relative to time.Now()) data points. So if you are querying a data set from a few days ago (as it appears you are doing based on the comments above), many of the data points are likely no longer eligible for caching, even if the final count of timestamps in the cache is less than the Value Retention Factor. And if you repeatedly query the same time range, then the number of cached elements will begin to drop over time, as you have noted, because over time more of them will predate the Oldest Eligible Timestamp.
I believe the behavior you are expecting (and I would definitely like to implement for our users) is to cache any timestamps for a query, irrespective of age, so long as the VRF is not exceeded; and if the VRF is exceeded, crop the oldest timestamps until we reach the VRF.
If you agree with this assessment and the proposed change in behavior, give me a thumbs up here, and we will retitle and label this ticket as an enhancement to implement that behavior.
Thank you for taking so much time to get to the bottom of this one!

@jtorkkel I have evaluated the option of allowing the storage of up to ValueRetentionFactor timestamps in a query's cache object, regardless of age, and here is what i have found.
There would be a bit of a performance hit to operate in this mode, because it would mean that every time series request, regardless of age, would then require polling the cache to determine how many free timestamps are available, just to know if the request should just be punted to the basic proxy or ran through the delta proxy cache routine. The beauty of the Oldest Eligible Timestamp is that we can check this information, and if eligible, punt to the proxy, all before ever even trying to obtain the lock for the cache record, with the downside being that you are not able to cache older timestamps at a high resolution without a high ValueRetentionFactor.
So what I propose is creating a new configuration called ValueRetentionPolicyName that can define either the current date-based methodology (which will remain the default for peformance), and the size-based methodology that is slightly less performant, and users would be able to choose which works best for their needs. Let me know if that sounds OK and I will get it ready for the next beta.

I did not know how cache was expected to work and I did not realize it was optimize to work with recent data only. I use 1d time range because then you see more clearly the cache benefit. Now when I tested more I noticed that even with settings of 16384 caching did not work for "this day last week". When you explained how it works it is not expected to work either.
I agree that main benefit of the cache is to cache recent data when multiple dashboards query and/or auto-refresh their content, and not cache the queries to the past to keep cache data relative small.
Need for more generic caching comes when you deep dive the metrics you typically start with larger range and then deep dive to into problem and then again go back to larger range. These dashboards /queries are often very heavy and caching 100% for "relative short time" would be valuable. Normally only few user or just one is making such queries.
Would this new called ValueRetentionPolicyName be able to mix both modes, I mean use current mode for last xxh ranges and new mode for queries outside the eligible cache range?
At lest making queries back in time should not flush last xxh cache. Setting two separate instance would be alternative but then you need to select datasource in advance which is not optimal. Or maybe allocate half of cache to last xxh and rest outside xxh.

@jtorkkel the size-based retention policy will use a Least-Recently-Used methodology for evicting time ranges from a query's cache object to remain at or under the Value Retention Factor. So you would be able to approximate the behavior of having both cache types by using the size-based policy and doubling the Value Retention Factor. The older values that are frequently accessed, as well as any realtime dashboard views would remain in cache, while somewhat recent data that is no longer rendering on users' dashboards, as well as older data that was once requested for single-use, will be evicted first.

Make sense. We have duplicate Prometheus / Grafana setup so we can have current policy in one and new policy in another to measure  the difference.

@jtorkkel Beta 9 includes this feature request, see the release notes for details. Please let us know how this works out for you. Thanks!

I did quick testing in new “lru” mode and cache correctly had 100% hit in second query in multiple range_queries (24h range) whereas default mode did not cache it.
Could you clarify which cache parameter impact “lru” mode.

timeseries_ttl_secs
object_ttl_secs
timeseries_retention_factor
I have increased two first one but not sure how they should impact.

I think in our config timeseries_retention_factor was 16384 or 4096 but new endpoint
/trickster/config
reported 1024.
Could not validated real input value.

/trickster/config report timeseries_retention_factor correctly.

closing due to inactivity, please reopen if this is still an issue


Adding toggle dark switch mode

I want to add the toggle dark switch mode in your website .
please assign me this issue @Shubham-Zone so that I can work on it and make the necessary changes.
please mark it as gssoc'24 ext and hactoberfest .

@Shubham-Zone Please assign it to me . Thank You!


When you turn on Glow ESP, the Game crashes

When you turn on Glow ESP, the game simply crashes.
Windows version: 10 Pro
Injector: Public Injector
Version Indigo: 4.4.3
Video: https://youtu.be/PoX_1RYKwaA
P.S Other details are shown in the video

// Glow ESP is in BETA

@smefcc But it still does not work, I do not think it should be added to the menu if it even clumsily does not work, I think it is in the Alpha version.

We just added it jessu christ. If you do not want it on the menu remove it yourself.
Its open source not everything will 100% work

@smefcc Okay, I understand, please go to the source breathless, little pulls created


Pocketsphinx and UPD Streaming

I was plaing with wakewords when I noticed that pocketsphinx (the only wakeword I got to understand me to date) does not work with UDP streaming enabled. Other wakewords have an input field for the UDP port, pocketshpinx doesn't.
I could get rhasspy to wake when I removed the UDP port, but with that included in my Audio Recording setup I can not get any reaction. I tried adding "udp_audio": "12111" to the pocketsphinx config but even then it didn't work.
Is this supposed to not work with pocketsphinx or is this a bug?

Fixed in 2.5.6


Unable to compile android project

/.gradle/caches/transforms-1/files-1.1/appcompat-v7-27.1.1.aar/a586b0a3d421c5bb7413374bc558cd65/res/values/values.xml:255:5-69: AAPT: error: resource android:at

/Users/account/.gradle/caches/transforms-1/files-1.1/appcompat-v7-27.1.1.aar/a586b0a3d421c5bb7413374bc558cd65/res/values/values.xml:255:5-69: AAPT: error: resource android:attr/fontVariationSettings not found.
/Users/account/.gradle/caches/transforms-1/files-1.1/appcompat-v7-27.1.1.aar/a586b0a3d421c5bb7413374bc558cd65/res/values/values.xml:255:5-69: AAPT: error: resource android:attr/ttcIndex not found.
error: failed linking references.

this happening because of using com.android.support:support-core-utils:28.0.0 manually change it to 27

this happening because of using com.android.support:support-core-utils:28.0.0 manually change it to 27

Thanks @ramzi27 , it worked for me...

Thank you @ramzi27! Such an awesome solution that gave me additional hours to sleep!

@ramzitannous i did it but i have this problem yet
in react-native-fcm/android/build.gradle i have this "com.android.support:support-core-utils:27.1.1"
can anyone help me


Update GRPC to fix GCC 9.2.1

Update GRPC to support GCC 9.2.1 with https://github.com/tikv/grpc-rs/pull/386 which contains https://github.com/pingcap/grpc/pull/33 which is a backport of https://github.com/grpc/grpc/pull/18950 .
What have you changed?
Users with Ubuntu 19.10 or similar distros can now build TiKV easily!
What is the type of the changes?
Engineering
How is the PR tested?
Manually and by the CI
Does this PR affect documentation (docs) or should it be mentioned in the release notes?
No
Does this PR affect tidb-ansible?
No
Refer to a related PR or issue link (optional)
https://github.com/tikv/grpc-rs/pull/386 which contains https://github.com/pingcap/grpc/pull/33 which is a backport of https://github.com/grpc/grpc/pull/18950 .
Benchmark result if necessary (optional)
Any examples? (optional)

LOL 🤣


Updates

/botio build

From: MushinBot (botio-academy.horizen.global)

Received
Command cmd_build from @tuanhvdev received. Current queue size: 0
Live output at: https://botio-academy.horizen.global/905409a9e1b8c3e/output.txt

From: MushinBot (botio-academy.horizen.global)

Success
Full output at https://botio-academy.horizen.global/905409a9e1b8c3e/output.txt
Total script time: 1.83 mins
You can download the generated site at: https://botio-academy.horizen.global/latest.tar.gz
Or view a preview at https://devweb-academy.horizen.global


"This is the solution of the issue"

          > "This is the solution of the issue"

this is a quote
Originally posted by @jhon-elen in https://github.com/jhon-elen/Prueba3/issues/7#issuecomment-2267641691

Anothe comment

@jhonelentest

Header
Header

Cell
Cell

Cell
Cell

import os

os.chdir()

Details

This is a detail


PWS should report cache headers to clients

If a client wants to keep a local cache, the cache headers from the website would make it easier to do that properly.

Interesting, but this is complex.  Typically cache headers are associated
with an http GET, which we are not.  For the final page content, which is
cacheable, the browser already implements this, not us.
We could create a custom protocol and add to our REST api, but the scan is
very likely for a short-url with no cache lifetime, and so you shouldn't
circumvent the lookup.  (Though this isn't always the case, some urls are
short).  Other times, the metadata associated with the request affects the
reply (like distance), which means clients would have to implement all our
filtering logic.
So, I guess we should brainstorm this further.  While I'm not convinced
this will help very often, some people have asked for an offline solution
and I think this is as elegant as I've seen proposed.
On Tue, Jun 16, 2015 at 5:56 PM Conley Owens<EMAIL_ADDRESS>wrote:

If a client wants to keep a local cache, the cache headers from the
website would make it easier to do that properly.
—
Reply to this email directly or view it on GitHub
https://github.com/google/physical-web/issues/415.

Well, this isn't so much for an offline solution as it is for the situation where I'm sitting at my desk and every five minutes check the news on my phone.  While the scan that I trigger doesn't take up too much battery, I'm going to be seeing the same beacons each time and shouldn't be hitting the radio.

Typically cache headers are associated with an http GET, which we are not.

Could you explain more?  PWS does a GET request to the url and receives cache headers.  I'm not seeing the issue with forwarding that along in the metadata object.

I mean that the App uses a POST to resolve-scan.  We cannot just include the cache-control header as part of the response in the typical way which lets the infrastructure leverage it.  We would have to add it to our api and have clients actually implement caching.
At that point, there is no specific reason to use the http cache header format (except that it may be good to borrow concepts).

Oh, right, I wasn't suggesting we keep the format or put it in the http response.  I was just saying that the metadata should include a value that lets the client know how long the metadata is going to be good for.
It doesn't make sense to put it in resolve-scan at the http level because you can call resolve-scan on multiple urls at once.

Sounds good.  Prerequisite is actually parsing cache-control headers (I thought we had a feature files for this, but cannot find it), and also making we understand all the situations where the cache can get invalidated.

#416


Fix comment on default for serverModuleFormat

Since the default for serverModuleFormat is now esm, the comments in the v2 guide for the different configurations of serverBuildTarget were incorrect.

[x] Docs

These comments are tricky since they are intended (originally) for v1 users preparing for v2.  So CJS is still the default on v1.  Now that v2 is released, ESM is the default but there are still 2 types of users reading these docs:

Users upgrading to 1.19.x and adopting all future flags to smooth the v2 upgrade

Current comments are accurate for these folks

Users going straight to v2 without the in-between step

Your proposed new comments are accurate for these folks

I think it's probably best to either remove these specific comments or change them to default value in 1.x/default value in 2.x for clarity?

These comments are tricky since they are intended (originally) for v1 users preparing for v2. So CJS is still the default on v1. Now that v2 is released, ESM is the default but there are still 2 types of users reading these docs:

Users upgrading to 1.19.x and adopting all future flags to smooth the v2 upgrade

Current comments are accurate for these folks

Yes, I considered that, but the document is, at some level, upgrading a user to v2, not to v1.9 with the v2 flags. For users that are upgrading to v1.9 with all flags enabled, would be best served to use the v1.9 documentation.

but the document is, at some level, upgrading a user to v2

If the document were only intended to be read by users upgrading straight to v2, then it wouldn't include the steps of telling them to set future.v2_* flags to true since those flags don't exist in v2.  Successfully or not, at the moment it is intentionally trying to be something consumable by both sets of users I mentioned above.

would be best served to use the v1.19 documentation

In an ideal world, yes.  But in practice these users are going to land on the main docs and the vast majority of them won't know/remember to switch over to the v1.x docs, so we have to be pragmatic.

I updated the comments to the following:
serverModuleFormat: "cjs", // default value in 1.x, add before upgrading
...
serverModuleFormat: "esm", // default value in 2.x, can be removed once upgraded


Store motion in EventMouse

This pull request changes EventMouse to also store whether or not the event was sent as a motion event. In the current version of tcell, the motion bit is just thrown away, but this can be useful information on systems which are more limited in mouse reporting and don't always report correctly.
For example, on Elementary OS, the mouse button is always sent as 32 when there is motion, even when mouse button 1 is not pressed. With this PR I can tell whether the key was actually pushed down or not by storing when the key was first pressed without motion and then registering the motion events.

I’d really like to see how you use this change in application code?  It
seems like carrying the motion bit for these brain-damaged OS’ may wind up
pushing more work into the application, which is an antipattern; is there
something more intelligent we can do here in tcell to make life better for
applications?
Part of what would be helpful, I think, would be a sample stream of events
from both busted and non-busted OS’, so we can see whether there’s an
opportunity to do better post processing of these events.
On Mon, Apr 25, 2016 at 11:52 AM, Zachary Yedidia<EMAIL_ADDRESS>wrote:

This pull request changes EventMouse to also store whether or not the
event was sent as a motion event. In the current version of tcell, the
motion bit is just thrown away, but this can be useful information on
systems which are more limited in mouse reporting and don't always report
correctly.
For example, on Elementary OS, the mouse button is always sent as 32 when
there is motion, even when mouse button 1 is not pressed. With this PR I
can tell whether the key was actually pushed down or not by storing when
the key was first pressed without motion and then registering the motion
events.
You can view, comment on, or merge this pull request online at:
https://github.com/gdamore/tcell/pull/97
Commit Summary

Store motion in EventMouse

File Changes

M console_win.go
https://github.com/gdamore/tcell/pull/97/files#diff-0 (2)
M mouse.go https://github.com/gdamore/tcell/pull/97/files#diff-1
(20)
M simulation.go
https://github.com/gdamore/tcell/pull/97/files#diff-2 (2)
M tscreen.go
https://github.com/gdamore/tcell/pull/97/files#diff-3 (12)

Patch Links:

https://github.com/gdamore/tcell/pull/97.patch
https://github.com/gdamore/tcell/pull/97.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/gdamore/tcell/pull/97

I'm using it in my code like this:
case tcell.Button1:
    if mouseReleased && !e.HasMotion() {
        // Mouse click with no motion
        // ...
        mouseReleased = false
    } else if !mouseReleased {
        // Mouse click with motion (mouse drag)
        // ...
    }

    // Mouse motion with no click
    // which has been incorrectly reported by the terminal as button 1

case tcell.ButtonNone:
    // Mouse event with no click
    if !mouseReleased {
        // Mouse was just released
        mouseReleased = true
        // ...
    }

I think we can do better here. Stay tuned.

So I'm confused here... can you show me the string of event data that this terminal reports?   I think we should do a little more post-processing of these events to prevent you from having to go to such extremes in your application, so if we can make tcell smarter here I would prefer that to just exposing the motion event directly.

The terminal sends the string \x1b[<32;x;yM for every mouse motion event, even when no mouse buttons are pressed. However, tcell (rightly so) thinks that there is a left click because of the 32.

Please look at PR #99  -- I think this represents a better fix to the problem, by teaching tcell about this broken behavior and correcting its reporting, rather than forcing applications to cope with this brain damage.
(The bug in ElementaryOS is that it should be using 35 in these reports instead of 32.)
If you can verify that PR #99 resolves your problem, without an application workaround, then I'll integrate that instead.

@zyedidia Any chance you can test out my PR #99 ?


[ycsb] Upgrade YCSB to 0.9.0 and support latest logging format.

YCSB 0.9.0 output results and log to stdout and stderr.
Previously we only parse stdout log to get the info we care about, but with YCSB 0.9.0, that is not the case anymore. We need to combine both log files to get all the info.

Which results are we missing?

@hildrum The client version and command line info are now output to stderr. Previously, we only get info from stdout. So if we use any YCSB version > 0.7.0, we cannot get these 2 lines info.

LGTM

Ready to merge


add support for type meta to service method arguments

This PR expands on the type.go.tmpl renaming it to field.go.tmpl so it can account for the type meta annotations. A field can be used in many places outside of the struct, so moved some of the logic around, and added some flags if to include json / struct tags or not, etc.

Thanks for reviewing


Settings Page CSS

This is the updated CSS to allow for the new styling of the settings and partially admin pages of zulip.
This includes new checkboxes, new input grids, and a new avatar upload module look.

Automated message from Dropbox CLA bot
@brockwhittaker, it looks like you've already signed the Dropbox CLA. Thanks!

Closing this in favor of #1999.


added files for docker, moved cypress to dev dependencies

Closes #374
Need docker and docker-compose installed
"sudo docker-compose up" should build and deploy the containers
"sudo docker-compose down" should shut them all down
once they're running you'll need the IP of the container to get to the UI, or to attempt an API call with postman
sudo docker inspect beima_<container>_1 | grep "IPAddress"
<container> being: web, api, mongo, minio
Moved Cypress to dev dependencies so that it's not pulled/built with the web container

Me neither for sure. at this point it's just the Draft that Ashlyn asked for. it's the proof of concept that "it builds" tomorrow I'll be looking to see how they communicate with each other and getting it to DO something.
Right now the main test is "can someone pull the branch, install docker and docker compose, and actually build it"
It should work on any OS since it's independent, but if there's some other config that's needed that I missed, or something I'd rather know now as she suggested than get 4 more levels deep and figure out there's something that breaks the downstream that I've put up.
Storage mapping, port mapping, etc is a little up in the air at the moment

I was able to get this to work using docker desktop. I did notice the api is using the azure backend app, which I thought was odd since I have my local.settings.json to point to my local stuff. But good work so far!

When running all the docker stuff, I sometimes don't get port numbers for the web client or the db. And when I do, they are not our usual port numbers. Thought I'd bring it up

When running all the docker stuff, I sometimes don't get port numbers for the web client or the db. And when I do, they are not our usual port numbers. Thought I'd bring it up

We'll probably want to talk with Professor Henderson to see if we can get a range of ports that only our team can use on the sdp server.

Backend docker builds the API, I was able to get results inside the actual container via curl on localhost, accessing with postman on the external docker IP (the 172...* address)  also yielded results. The console reports it's listening on localhost, but it appears to be listening on all interfaces

ports from localhost are mapped to the containers, if they need to be changed, or the custom network needs to be changed it can all be done from the docker-compose file
until a "deploy" environment for CurrentEnv is available
Local.settings.json changes
MinioBucket should be "files"
LocalMongoConnection in the local.settings.json needs to be changed to "mongodb://<IP_ADDRESS>:27017"
modify BEIMA.Backend\StorageService\MinioStorageProvider.cs with:
client = new MinioClient().WithCredentials(accessKey, secretKey).WithEndpoint("<IP_ADDRESS>",9000);

approvals can wait on this one until it's converted to a real PR from Draft. It's not ready to merge until then, and I don't know if it will dismiss approvals when I convert it either


3x Vis Related Improvements

3 distinct improvements. (Haven't actually done a test build to check they're valid syntax though)
Closes #138
Closes https://github.com/FLAMEGPU/FLAMEGPU2/issues/1047

Have checked the artefact built by CI.


[ACR] az acr run --cmd: disable working directory override

Description of PR (Mandatory)
az acr run -r myregistry --cmd mcr.microsoft.com/dotnet/core/samples:dotnetapp /dev/null
The dotnetapp image sets the workdir to /app folder. Its entrypoint and cmd settings assume the current working directory is /app folder. However ACR Tasks by default launches the container using /workspace as the workdir so the above cli command will fail.
Since the main usage of --cmd is to launch the container image as it is, it's better to minimize the override behavior, especially avoid overriding the workdir folder.
So when generating the task step in the request, the cli command should pass disableWorkingDirectoryOverride: true
https://github.com/Azure/acr/issues/346

This checklist is used to make sure that common guidelines for a pull request are followed.

[X] The PR title and description has followed the guideline in Submitting Pull Requests.

[X] I adhere to the Command Guidelines.

acr

@yungezz for customization, the users can define the steps in a yaml file https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tasks-reference-yaml.
--cmd is targeting single step case.

@yungezz for customization, the users can define the steps in a yaml file https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tasks-reference-yaml.
--cmd is targeting single step case.

thanks for clarification. is this PR ready to mrege? Will merge it after ack

@yungezz the pr is ready to merge. thanks.


SmartThings could not validate the endpoint configured in ‘base_url’

I’ve been migrating Hass.io to a RasPi4 and am having an issue with the SmartThings Integration. I get the error: “SmartThings could not validate the endpoint configured in ‘base_url’. Please review the component requirements”. I don’t know how to resolve this.
I set up DDNS and SSL certificate using the DuckDNS and Let’s Encrypt add-on prior to my attempt.
My http configuration is formatted correctly (I am not using port 8123…or 12345):
base_url: https://*********.duckdns.org:12345
ssl_certificate: /ssl/fullchain.pem
ssl_key: /ssl/privkey.pem
server_port: 12345

External access works fine. However, when I input my URL into https://www.digicert.com/help/, I get the error: “Error: *********.duckdns.org is not a fully qualified public domain name or public IP address.”.

Thanks for reporting this issue.  Based on the error you reported from Digicert, this tells me that you likely have a misconfiguration somewhere and your HA isn't actually accessible from the internet.  This isn't an HA issue or ST issue.  Are you positive you entered the full URL into Digicert (including the port) correctly?  Unfortunately, I don't use DuckDNS, so I'm going to be of limited help.

@andrewsayre, thanks for your response.  I have tried entering my full URL into Digicert with and without the port.  I am able to access my HA page remotely from my phone (iOS app) and laptop via https, so it is accessible from the internet with an SSL certificate.

I think you'll need to figure out why Digicert think it's invalid/unreachable...  If they can't access it, it's a safe bet ST won't either.

#andrewsayre, one day later and Digicert now gives me the green light - all verified.  However, I still receive the same error with the ST Integration.

Someone on Reddit seems to be having the same issue this week.

@andrewsayre I got it resolved by removing the port # from my base_url, which I thought was required.  I'm glad it was a simple solution (that should've been more obvious!).

I had this same issue.  had to wrap the base_url value in quotes
  base_url: "https://homeassistant.mydomain.com"


Unable to use Infinite Tracing with a forking webserver (puma)

Description
We have a Rails application that is deployed with a forking webserver (puma) that we have recently tried to migrate onto Infinite tracing. While it works locally when puma is set to a single work mode we have found that it doesn't work when the tracer is set to clustered(forking) mode where the rails app is preloaded (for performance reasons) before forking .
We've found that after a fork, while other parts of the agent are able to restart successfully, the infinite tracer is not restarted successfully.
If we try to manually restart the infinite tracer's connection then a GRPC error is raised: RuntimeError: grpc cannot be used before and after forking. This issue discusses the issue and suggests that no fix is forthcoming soon: https://github.com/grpc/grpc/issues/8798.
Do you have any recommendations for fixes/optimal configuration for puma? Is it possible to defer the infinite tracer to boot after a fork?
Expected Behavior
Tracing continues to work after a fork
Steps to Reproduce

Create a rails 7 app
Add infinite tracing
Run the rails app with puma in clustered mode with a preloaded app

Your Environment
Our application is a Rail 7 application running on Ruby 3.1 and we are currently running puma 5.6.
We have the following puma settings:
workers 2
preload_app!

Hi @chrisholmes! Thank you for bringing attention to this and sharing the GRPC issue! We will take a look at workarounds and follow up with updates.

Hi @hannahramadan, thanks for quick reply.
After raising this, I realised I could do a workaround by not requiring the infinite tracing source until after work book. The workaround looks like:
#in Gemfile
gem 'newrelic-infinite_tracing', require: false

#in config/puma.rb
on_worker_boot do
  require 'newrelic/infinite_tracing'
  NewRelic::Agent.manual_start
end

@chrisholmes thanks for sharing your solution and it's great to see you've found a workaround. We are still interested in addressing this, so I'll leave the issue open while we continue to take a look :)

Hi @chrisholmes! Thanks again for reporting this and sharing your workaround. I'm changing the "bug" label just for some internal bookkeeping that differentiates between agent code bugs and agent compatibility issues, but we're going to keep this issue open for others to find your workaround and for us to continue to brainstorm compatibility solutions with forking. Cheers!

Oh, I've been experimenting with newrelic, puma and sinatra as of late and have some understanding of how they work internally. But I've never used infinite tracing, so your steps to reproduce are kind of lacking for me. Still, you might want to try the fork_worker setting. It seems to be generally a better option than preload_app. Additionally, the NewRelic agent should start in the workers.
On a side note, from the description of newrelic-infinite_tracing:

If you want distributed tracing to use tail-based sampling (Infinite Tracing), you need to add both newrelic_rpm and newrelic-infinite_tracing to your application's Gemfile.

This seems outdated. newrelic-infinite_tracing-8.14.0 depends on newrelic_rpm and newrelic/infinite_tracing requires newrelic_rpm. And therefore should start the agent.

Hello @chrisholmes
I've spent some time looking into this issue but I haven't been able to reproduce it.
I was using the latest version of the agent (tried both 9.0 and dev branch), rails 7, grpc 1.53.0 and puma 5.6.5
Originally I was seeing issues with infinite tracing when forking and thought I had reproduced the issue, but after looking into it more to try to solve the issue, it turns out that I was encountering a different issue that seems to only affect grpc forking on macs. I wasn't able to get it working using the workaround you provided and then saw that it was not the same RuntimeError: grpc cannot be used before and after forking you reported. In that situation, my puma workers kept crashing and giving the error
objc[42550]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

whenever grpc would try to connect. This happened even if the agent was not started or loaded until after forking. However, this didn't happen if it never forked, or if grpc was turned off. Since this seemed to be specific to macs, I used docker after this.
Once I switched to using the ruby docker image, that error was no longer present and I was able to focus on infinite tracing and puma, but I was not able to reproduce the original infinite tracing issue you reported. I tried changing a couple settings in puma, including number of workers, and using/not using preload_app!, but didn't see any change or see the grpc forking error.
The agent has built in logic to defer startup until after forking when a forking dispatcher is detected. Since infinite tracing doesn't startup until after the agent has connected, this also forces grpc to wait until after forking as well. One thing you could check what dispatcher the agent is detecting by looking at agent logs. You should be able to find a line that includes INFO : Dispatcher: puma. If the agent isn't properly detecting the dispatcher, it would instead say INFO : No known dispatcher detected.. If that's the case, you could try forcing the dispatcher to puma by adding dispatcher: :puma to your newrelic.yml and see if that changes the behavior you're seeing at all.
For me though, even when no known dispatcher was detected I was still unable to reproduce the issue, but my test app is a very simple rails 7 app. It's possible my simple rails 7 test app is missing something that is affecting the issue.
What agent version are you using when you see the error? Any other details that you think might be helpful in reproducing the issue would also be appreciated.
Thank you!

Closing this, since we have been unable to reproduce this behavior and have not heard back. Please reopen this if setting the dispatcher doesn't make any change or if there is some additional info you can provide to help with a reproduction.